Minimalism vs. Memes on the Command Line

[@alexstone691]

'grep is good for grepping' who would've thought..

[@xthebumpx]

But it's far from the best. At least if best means fastest.

[@KingJellyfishII]

@@bashisobsolete.pythonismyn6321 Nice uname

[@wulymammoth]

ah, tautologies

[@serrag97]

I cat into grep, because I like to write cat 🐈 on the terminal. 2 seconds longer, but 2 seconds of happiness

[@jezzamobile]

with you on this one 🐈

[@zerquex4520]

🐈

[@VvC_Lad]

alias \ 🐈="cat" \

[@baaz5642]

😸

[@KyzenHD72]

🐈

[@giobot]

> doesn't approve of command line memes > 4.20 seconds

[@josephgoebbels1605]

what are you comparing there with those [more than symbol]s sir?

[@thelastmagethelastmage2084]

@@josephgoebbels1605 greentext sir

[@bbseal6174]

meme arrows

[@genericgamer1319]

people die when they''re killed

[@jarro2778]

@@genericgamer1319 why

[@bruderdasisteinschwerermangel]

for maximum efficiency, consider the following: cat config | grep i3 | sed 11q

[@eirikol1]

Nikola Tesla cat config | awk /i3/ | sed 11q

[@MisterTommyD]

@@eirikol1 cat config > tmp_file.txt; cat tmp_file.txt | awk /i3/ > tmp_file2.txt; gzip -c tmp_file2.txt | zcat | sed 11q; rm -f tmp_file*.txt

[@brxne]

@@MisterTommyD no

[@rufine4114]

I don't think that yours is efficient as mine: while read line; do if [ ! -z "$(echo $line | grep i3)" ]; then echo $line; fi ; done

[@williambennett4360]

grep and awk are bloated; print out the file and highlight the occurences

[@JINORU_]

Too much paper and time bloat

[@pokefreak2112]

All these new video's from luke... can't stop consooming...

[@pokefreak2112]

Edit: thanks for the upcummy

[@jan_harald]

can confirm, it's midnight, still watching Luke's stuff ;P

[@jimbarino2]

Don't ask questions... Just watch Luke videos... and wait for new Luke Videos...

[@ChrisTitusTech]

Man, this is so good, I can't make any videos, but just watch you and DistroTube go back and forth... LOL

[@avishekpdsyouthoob9083]

Heyyo! xD

[@rodrigosouto9502]

Lol

[@bendover4728]

Grep is good for grepping, awk is good for awking.. --Luke, 2019

[@Nicolas-qc3jf]

dude just ctrl + f

[@littleblack2309]

Distotube: replace everything with awk. Luck: No, grep is the most efficient way for greping, unless you already using awk for something. Distotube: replace everything with awk.

[@KingZero69]

luke: i don't endorse memes also luke: *his whole life is a meme*

[@ariathyf144]

They should teach programmers to build the habit of bench marking the commands like you did with the timer. The source code of proprietary programs run on windows must be a nightmare in that regard.

[@luke_filewalker]

"if we do something silly like awk it.." - shots fired! :D

[@Chimperly]

lol I caught that too and I was like - what a fricken world we live in. or rather dont live in.... haha

[@ethanp5215]

DistroTube vs Luke Smith is, Pewd's vs T-series: Virgin edition

[@epenguin3509]

Lmaoooooooooo

[@paulk314]

Something else I learned recently: grep has a -F option for searching for fixed strings (i.e. no regex of any kind). In many cases, this can speed up the search if you don't need regex. I routinely deal with very large text files at my work, sometimes multiple gigiabyes with 10s of millions of lines and I often want to quickly extract out specific information from them, parse it and display it in a convenient form. I used to slurp the files in perl and iterate over the lines applying pattern matching. It actually wasn't terribly slow, but I was amazed how fast it become when I instead had the script run grep first and then slurp the output of grep instead.

[@jan_harald]

reading this I immediately got a basic idea, if done in a compiled language like C(++) or assembly, might even perform well... iterate over the string byte-for-byte non-matching bytes read again, matching ones build into a buffer until they form the searched phrase (or more likely, a static buffer with the inputted string to compare to, then a changing int or something storing how much is left to match) , then print it out, or if they don't match then flush blank at next read...

[@allegoricalstatue]

Came here to comment this too. Has tons of other nice features too and they're just easy to use built-in flags, like grep -rlF "test" --exclude-dir=vendor --include=*.php --include=*.html -C 5 .

[@joinedupjon]

calling fgrep instead of grep -F saves typing and should be exactly equivalent... I'd say using grep (or fgrep | egrep) for grepping also makes it instantly comprehensible to the next person who has to read and maintain your script... especially good if that next person is likely to be you.

[@ilyasazonov3901]

Th guy from Distro Tube could just have said - "Why use cat or grep when you can just use Java!" . And you definitely can, BTW

[@zZGzHD]

Let's rewrite plan9 core utils in Java. DURGA TIME.

[@con-f-use]

Right ultimately it's "why have specialized programs at all, if you can solve everything computable with a humble Turing machine?"

[@KingJellyfishII]

As much as java gets hate, I like it. Now hear me out - 1) it's faster than python, and actually (ignoring memory) it's not _that_ slow. 2) it's easier than C. By an order of magnitude. Developer time is often more expensive than runtime, especially if that runtime is negligible. I can't express how much easier it is to _actually have strings_, rather than having to do some cryptic magic with char pointers or whatever CPP does. And it gives proper errors not just "segmentation fault". 3)OOP is easy for the programs that benefit from it, although it is a bit forced 4) yes it's overly verbose but it's not _obstructively_ verbose. 5) ok I agree there are some bad things but overall its pretty good imo

[@axelforsman1642]

1024 Oh come on. The JVM is one of the fastest interpreters out there. Java dances around Python. Don't spread misconceptions

[@ilyasazonov3901]

@@KingJellyfishII Ok, I have two questions for you )) . 1. How long does it take for grep to start? 2. How long does it take for java program to start?

[@CvnDqnrU]

For maximum efficiency, call it "mememalism" instead or "meme minimalism".

[@hmagellanlinux307]

You may not endorse memes, but they have endorsed you

[@peterjansen4826]

This is getting fun. :) The battle of the command-line users, round 3. In the blue corner...

[@an2qzavok]

The main bottleneck is not in computer power but in operator's brain. This is why suckless.org stresses counting sloc over performance. Less code = less code to understand and keep in your head.

[@scoreunder]

A couple of things to correct: 1. Total time is more important than user time. If the process takes a lot of time stat-ing files, forking, or something like that, then it's going to have a lot of cpu time outside of user mode. 2. One of the reasons cat|grep is so slow is not because it's another process, but because it must read the file and send it over a pipe, and then grep must read from that pipe. This results in a lot of copying of data that grep would normally not do.

[@johnc3403]

It's great to learn the correct way to do things but I hope videos like this don't create a rift between your channels. I have a gorgeous Archlinux install thanks to both you gentlemen. Your channels complement each other wonderfully, are educational, entertaining and make linux accessable to us newbies. Keep up the great work Luke. ...and you too DistroTube

[@alcesmir]

You can also do `grep pattern < infile` which turns out to be about as fast as supplying the file name to grep. So if you ever find yourself piping cat, then a file indirection will be a drop in replacement. Ideally you would do the right thing and just supply the argument, but just getting rid of cat goes a long way.

[@mina86]

Also, little known fact, you can put redirections at the start of the command so ‘cat file | blah’ can pretty much always be converted to ‘

[@alcesmir]

@@mina86 That's actually not true in a lot of cases. Redirection from file is actually smarter than that. Try generating a big file and doing tail with the file as an argument vs redirection. You shouldn't see much difference (however cat | tail should be slow). From the program side you can essentially read the file that's redirected as if it was just a file rather than a stream.

[@mina86]

@@alcesmir, oh, yeah. I was actually expecting tail to do something smarter (i.e. what you’ve described) but after doing quick tests got results to the contrary. Running the same test a few more times evens results for both just as expected.

[@user-ic7cj8lq9z]

Use sed instead of cat, ok got it.

[@jan_harald]

instructions unclear, got cat stuck in pipe

[@stumbling]

@@jan_harald For real, piping cats is dangerous: kzbin.info/www/bejne/bZLYiYecnLSBncU (poor kitteh :3)

[@ChungusAm0ngus]

sed $(wc -l file | awk '{print $1}')q file

[@IWantSomethingNew]

@@stumbling Piping cats is not just dangerous, it is also distasteful!

[@Yaxqb]

luke sees inefficiency at work, this can't wait until getting home, gotta get da truth out there!!!!

[@fox0ps22]

Don't use awk, use mawk - it beats grep by 30% and awk by 100%.

[@TheAnigai]

i always pipe cat into grep because I like to run the command multiple times to search for different things and having the regex at the end of my line instead of having to move past the filename in grep back to my search pattern is just easier for me. Obviously not in scripts though.

[@andrej4342]

Wtf Luke? Catting into grep is my will and you question it?

[@jordanc.m.6735]

which chipmunk do you think gives the best sed 11q?

[@TimeTravelingFetus]

Nice meme

[@biehdc]

3:34 you sure made it in a way that its going to be 4.20 sec

[@thsdsyt]

Glad to see luke back again. Watching his videos has made me migrate to Linux a while ago.

[@zZGzHD]

- Sent from my Wagie Cagie

[@hielke2332]

There is a blazingly fast implementation of awk called mawk, which is (on my machine) even faster than using grep. I also don't believe it, but check it on yours and see if it is true for you as well. One plausible explanation I have for it, is that grep is older and nobody has actually decided to really optimize grep for speed with a total rewrite or something. So if you really care about performance you should check out mawk. Here is an article comparing multiple different programs for a data munging problem: brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-and-most-elegant-big-data-munging-language/

[@hielke2332]

Scrolling to the other comments I found another interesting detail about this. Grep interprets your string as regex. If you disable that with -F it will be faster, and will in that case be faster than mawk. However, a regex search will be faster with mawk than with grep. I don't think there is a way to do a fixed string search in awk, so we can't make that fair comparison.

[@joinedupjon]

@@hielke2332 fgrep should be exactly equivalent to grep -F and saves keystrokes ;)

[@saeedbaig4249]

@@joinedupjon From the man page for grep: "In addition, the variant programs egrep, fgrep and rgrep are the same as grep -E, grep -F, and grep -r, respectively. These variants are *deprecated*, but are provided for backward compatibility."

[@joinedupjon]

@@saeedbaig4249 Hi Saeed - OK go with the manual... fun fact; when implemented, fgrep, egrep and grep are usually the exact same binary

[@RushilKasetty]

If you really want speed then use ripgrep

[@jalfredprufrock1915]

sed 11q outputs 11 lines not 10 try it yourself diff

[@LukeSmithxyz]

Yeah, it's a meme: it's supposed to be with capital Q.

[@jalfredprufrock1915]

i have learned something today (and it wasnt chemistry thank God)

[@Jango1989]

Great video! I must admit that I have been actively trying to break my 'cat a | grep b' habit ever since I saw your first video. I have to use some hideous pipes of cat | grep | cut | sed | grep etc for work sometimes and I think these videos just confirm that I should suck it up and gt gd with regex.

[@jarrod752]

1992 called, they want their system resourc.... SEGFAULT ERROR: BUFFER OVERFLOW

[@queenannsrevenge100]

Loving this channel after having just discovered it. Looking forward to more!

[@Andrath]

or use perl, you can replace all those things with perl. ;)

[@borisdappen5295]

Because that's actually the origin story and first use case of Perl

[@Andrath]

@@borisdappen5295 Ah, you got the joke. :)

[@hielke2332]

Perl is also quite fast. So, not a bad choice.

[@ertwro]

Agh, I need to install it and it has it's own package manager. There are moments when sed and awk are just a pain in the ass when I almost want to use perl but then I just say fuck it and go for python or just abandon the whole thing.

[@hamokjoe3746]

grep is a tool, sed is a multitool, awk is a toolbox

[@person800]

awk is an entire language.

[@Lumbago04]

Thanks stopped using notepad + the + is bloat, keeping it minimal

[@KingJellyfishII]

Vim is bloated, use nano

[@AlexanderPrussak]

@@KingJellyfishII lmao

[@zihuatanejo7741]

classic command: `cat something | grep something else`

[@DanielKrikun]

Cat-to-grep is convenient for interactive use especially because: 1. It is natural to write stuff from left-to-right 2. More importantly so, you do not need to remember which argument is the filename and which is the pattern to be searched

[@luke_filewalker]

Hahaha - let the games begin!! :D

[@jamestan9422]

ed is the standard editor it uses less cpu and memory than vim. plus faster startup time

[@PaladinJenkis]

True software engineer mentality over here, I like that!

[@simonedeiana2696]

True marxist chad making videos on company time depriving the master of that sweet excess value of labor

[@robinrcrawford]

So basically, know what the program you're running does?

[@FyahBurn95]

Why do everyone compare head to sed 11q when sed 11q outputs one line more than head's default 10 lines? Sed still does its default of printing the line when it reaches the one it has to execute q over.

[@bbdgl7413]

Why do people always consider "awk" a tool like grep? /usr/bin/awk is actually the interpreter for the AWK programming language - which has a LOT more to offer than greping stuff :D considering that it performs pretty well in this comparison to grep; i wonder how "grepping" performs with bash, python or other interpreters? keep up the good work and keep :(){ :|:& };: ing

[@bigfootisjustreallyshy]

Luke and DT buddy cop movie. But instead of cops, they are domestic terrorists. Let's make it happen.

[@ghollisjr]

Generally I start a pipeline with cat because I know I can add anything else to the rest of the pipeline or change the input data without worrying about the rest of the pipeline. But when I'm writing a script, of course it needs to be done efficiently

[@SebSenseGreen]

So the war has already ended? What a bummer!

[@lucioinnocenzo2328]

Wants to save 1 second. Makes a 9 minutes video about it.

[@UODZU-P]

his point was never you should use awk over grep, it was that some tools are better for specific use cases

[@baganatube]

But sometimes you first think of the file you're going to work on, and _then_ what to grep for. In that case, I run $ < config grep i3.

[@sabamacx]

Does not even LC_LANG=C grep forcing i18n bloat.

[@prakash.vishwakarma]

Thanks for letting us know through your blog, that you're aware about the ttf-symbola license change and that is causing st terminal to crash. Will wait for the fix, until then i'm switching to urxvt for neomutt and viewing script files and anything that require unicode characters. Hope to get the fix soon. Thanks!

Do you know movfuscator? It compiles your C programs to only unconditional move instructions! *Peak minimalism!*

[@KingJellyfishII]

No but I sure am gonna look it up now

[@longlostwraith5106]

I hope that you've learned what minimalism actually is by now...

[@dancorvalan3205]

Dude, i love sed, its my fav word processor. Damn

[@luisalejandrohernandezmaya254]

Excuse me. Catting grep is not disappointing to me. Sometimes we need to examine a file with different regular expressions and is easier catting grep because you can reuse your last command modifying the last word instead of going back and modifying the penultimate word. It's just the same in Haskell. It's easier to write flip intercalate list than (\x->intercalate x list)

[@JodyBruchon]

....awkward

[@victorprokop2240]

DON'T ASK QUESTIONS JUST CONSUME VIDEOS AND THEN GET EXCITED FOR NEXT VIDEOS

[@chrkrngl]

I'M DEAD INSIDE

[@wolverine9632]

2:09 subtle voice change

[@RoKishDubbz]

3:33 Cat into grep is clearly superior because it overclocks your CPU automatically. 116% CPU utilisation

[@NomoregoodnamesD8]

That number means the script spent 116% of the run time making system calls. What that *really* means is that multiple cores were used to run the script because the pipe operator launches both programs at the same time (which meant that the application start code could run on different threads). That measurement is a holdover from the multi-user computer days when multiple people had to share CPU time on a single mainframe, and administrators wanted to measure CPU time across all cores.

[@sneakyshadoo22]

Who needs cat when you can use lolcat?

[@mairacristian54]

sure you can hammer a nail with a screwdriver, but yea

[@evertonc1448]

Now I have autism.

[@timh.6872]

I see your definition of "minimalism" and raise you another: mimimal code written. The difference between cat-ing into a pipe and providing the filename to grep directly is that in the latter case, grep is responsible for figuring out what file to open. In the former case, the input is handed to it from on high as FD 0, stdin. No ambiguity, no input selection options, no recursion through a directory graph, just search a file's contents for a pattern. All the other bells and whistles can and should be other individual tools because they're used for more than just text searching, nor are they intrinsically tied to text searching. Now, if we just want to avoid grep opening the file, it would make more sense to do an input redirect and have the shell handle the file descriptor, but input redirection looks really bad, because it flows backwards in most shells, hence 'cat "$fname" |'

[@LambdaJack]

Real world steam punk.

Also I would point out that Luke's approach is better form code readability point of view, as shell scripts are also code it is good to express intent behind each statement: 'cat' is short for concatenate, however you are not concatenating anything when you pass just single file to it, also 'sed 11q' might be harder to understand for less experienced user than 'head'.

[@DJ_Cthulhu]

BTW Luke, DT has started using Emacs instead of Vim. 😁

[@jimbarino2]

Once you start down that path, forever will it control your destiny.,,

[@ailivac]

well it has been just over 8 months since all the major distros announced they were finally dropping vi... i think it was right at the beginning of april speaking of emacs (but less foolishly), does anyone know if gnu emacs supports the feature xemacs had where if you typed / or ~ at the find-file prompt it would clear the whole minibuffer?

[@cpakkala]

Fallen to the dark side has Anakin.

[@stefanalecu9532]

​@@ailivac tested it just now, it seems to have it like if i write ~ or /, it grays out the stuff before it and if you tab, it'll show completions like it should

[@installlinux6156]

Thanks for the video Lenin

[@TON-vz3pe]

This video is a mime.

[@SimGunther]

I refuse to be a minimalist and instead become a mememalist. That's why I use Haskell for everything, including the shell and CLI tech stack

[@christbaumer]

Memimalism ftw.

[@0M9H4X_Neckbeard]

2:40 "grep is the best program to grep with" imagine not having heard of ripgrep

[@LukeSmithxyz]

People actually use the scripts I write, so I use only programs that people actually care about.

[@0M9H4X_Neckbeard]

@@LukeSmithxyz that makes a lot of sense in scripts, but "the programs people care about" ≠ "the best programs"

[@HungNguyen-lz5xb]

Well, if I write a script for others, I will use something "standard" and battery included stuff so that the script work out of the box. Believe it or not, there are people who don't read the error output and don't know what to do, and there are people who don't want to install extra stuff on their machine.

[@marinacabrera9319]

@@LukeSmithxyz Just TRY ripgrep Luke. Seriously.

[@Little-bird-told-me]

*time sed 11q config* was faster than *time head 11q config*

[@watkinsishere]

oi m8, you got a loicense for that response to a response. Didn't think so m8

[@xBZZZZyt]

03:10 does $(seq 1000) store all numbers (1; 1000] in memory?

[@MaxAnkum]

So.... what does a grep do?

[@alkeryn1700]

you could use {0..1000} instead of $(seq 1000) also both bash and zsh suport C style loop, you can go take a look at the bash bible ^ github.com/dylanaraps/pure-bash-bible/blob/master/README.md

[@LunarVagabond]

Question for the pros, I'm looking to improve upon a script I wrote for work in which I'm running ~20 lines similar to the following ARCHIVE=$(cat ${PROPFILE} | grep "path.ARCHIVE" | awk '{print $3}') and similar for other configurations. After watching this video I thought to go back and optimize so I'm not cat'ing the same file ~20 times so I guess the question is how could I open the file once and read the 20 or so paths that I need into variables for the script? Should I just awk the file 20 times so I'm not running cat + grep + awk? Best, You're friendly neighborhood bash noob =)

[@LunarVagabond]

P.S. great videos man I've been watching all of them recently during my down time.

[@アランパーソンプロジェクト]

Lmao，who’s gonna care a thousand times of cat takes 1 sec. More But you are correct it’s stupid to use awk for grep

[@Jombo1]

memealism

[@Sir1us1]

Why use grep when you can use ag / ripgrep?

[@AD34534]

How do you feel about the phrase, "unused ram is wasted ram"?

[@LukeSmithxyz]

Ah, the SoyDev's mantra.

[@recklessroges]

cat file.txt grep i3 !$ # efficiency of keystrokes > efficiency of CPU ; my_time_value > cpu_time_value # or grep i3 !:1-$ # for those that forgot to quote of escape spaces

[@Yggdra666]

awk is my favorite OS

[@shivangupadhyay5493]

hey luke , QUESTION HERE , is it possible to make a script run on an event ...like firefox completing a download or on completion of a torrent or something like that ,... i tried googling this , but guys were tring it from the JavaScript addon for browser way... if you have any alternatives of that... that would be really helpful .. love your videos btw

[@greyman1104]

After all, your hairline is minimalistic, too.

[@MrBenji0]

Is SmithWars making a comeback?

[@vorrnth8734]

Why on earth do you use shell scripts when care about efficiency?

[@codexm37]

"I don't endorse memes" okay unaboomer

[@ailivac]

why do any of this when you can just use Perl?

[@chrkrngl]

What are other reasons like why the cat-v page considers the head command to be harmful? For them I guess because it is not in some Unix standard? But if it is important for someone this *also* is a legitimate point of view for minimalism. Or saying that e.g. someone values Unix/POSIX standards/versions/subsets higher as "minimalism" as e.g. cpu usage or real(time) effeciency.

[@Fetusgi]

I'd like to know why do you prefer Brave to qutebrowser. What's missing in qutebrowser?

[@ultrahalf]

dt vs ls (virgin vs chad)

[@HP-sf1my]

Discussing about 1 ms...

[@linuxland]

*someone replies to Luke video* Luke: finally a worthy appoint; our battle will be legendary

[@MissyMousey]

This couldn't possibly get out of hand.