This CLI Tool is AMAZING | Prime Reacts

Recorded live on twitch, GET IN
/ theprimeagen
Article: alexplescan.com/posts/2023/08...
Author: Alex Plescan | alexplescan.com/
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
Hey I am sponsored by Turso, an edge database. I think they are pretty neet. Give them a try for free and if you want you can get a decent amount off (the free tier is the best (better than planetscale or any other))
turso.tech/deeznuts

[@CommanderRiker0]

Former programmer and current linux/unix administrator I've beaten many of our in house Java/Python(many multi-threaded/processor) routines that massage data with parallel + Awk (or Mawk for even faster performace, given you fit the limitations). We crunch through massive amounts of plain text / hour. Nearly every time the younger staff are perplexed at how this can be (which is in turn perplexing to me how they think they will beat a nearly 40 year old highly optimized program with a few days of coding). Sometimes Mawk will beat my C++ implementations compiled with the most aggressive flags I can set.

[@BlackwaterEl1te]

I hear so many people mention AWK feels like i should spend a weekend with AWK and try it out.

[@CommanderRiker0]

@@BlackwaterEl1te If you need to manipulate text, CSV, or basically anything textual nothing beats the ease of use being able to isolate columns, rows, line numbers, regular expressions, etc. Having some of the worlds best computer scientists optimize its code doesn't hurt either.

[@BlackwaterEl1te]

@@CommanderRiker0 Cool thanks for the info, AWK sounds really useful especially in this text data heavy world.

[@CommanderRiker0]

@@BlackwaterEl1te I recommend the book "Sed & AWK" (Make sure its a later edition), the free pdf is floating around everywhere on the net (O'reilly publisher). Just skip the SED parts if you want. SED is a line editor for quick and dirty regex replacements (its basically built into VIM). AWK is a full programming language.

[@trejohnson7677]

Lol u have chatgpt now, awk is easier then ever

[@parnmatt]

the ::: _is_ parallel specific nothing to do with bash parallel is also written in perl

[@xdevs23]

Yea the ::: are just good ol' arguments to parallel

[@MelroyvandenBerg]

It's indeed written in Perl. Hence the stupid :::

[@ChadVanKlompenburg]

I use a lot of bash scripts not because I like bash scripts, literally any other scripting language would be better, but if written properly, it just works on almost on anything... Also GNU parallel is awesome.

[@harrytsang1501]

I work with docker images and sometimes even bash is too much to ask for. alpine falls back to sh

[@tapwater424]

More like works on almost nothing. Aside from built-ins, every command is an external program that may or may not be on the system. And the program may have different versions or implementations. Sometimes it's GNU tar, sometimes it's BSD tar. To make a platform independent bash script you need to find the common denominator of every target platform. Don't get me wrong, I use bash all the time, but it's because it's actually a very good model for composing programs together.

[@user-hk3ej4hk7m]

Parallels is great for anything you could do using multiple tmux sessions. Anything that requires joining and error handling gets overly complicated pretty quickly.

[@harsha1306]

> be primeagen > wax poetic about the benefits of gnu parallel > never run it on stream Por que Maria?

[@Palundrium]

Author of GNU parallel also has a KZbin playlist references in the manual: kzbin.info/aero/PL284C9FF2488BC6D1&si=lTOiasQ-llm9IwEB

[@theodorealenas3171]

A note on xargs: in POSIX it has no parallelism. I don't know how much that means though.

[@ThePrimeTimeagen]

That's good to know. I'm glad I use parallel for all my parallelism :)

[@ThePrimeTimeagen]

Also big fan

[@theodorealenas3171]

@@ThePrimeTimeagen you're a blessing. I gathered Reddit people now and "percentage viewed"s dropped.

[@marghidanu]

It’s written in Perl.

[@Alex-xl4xe]

I feel like there are so many amazing tools out there that you would never find in a google search if you search for a specific problem. Isn't there like a comprehensive list of those?

[@azratosh]

I wish I saw this live so I could spam "man parallel" in the chat; ::: is specific to GNU parallel

[@hiraginoyuki]

same here, it was just too frustrating to see all those comments getting ignored

[@azratosh]

@@hiraginoyuki Eh it wasn't frustrating imo, it was fun seeing Papa Prime try to figure out whether it was yet another weird BASH construct or not

[@Warflay]

Parallel is awesome, I even used it a few times where I didn't even need parallelization because it's argument parsing and hand handling is so much more powerful than xargs so I just used it instead

[@animanaut]

6:28 'brace expansion', if you want to look it up

[@driyagon]

i searched a few hours ago about gnu parallel and prime comes out with a vid, what are the odds Hmm Susge

[@UliTroyo]

It's like hearing about a new tech thing and that day seeing a Fireship "100 seconds of the thing"

[@dizz00001]

The odds are about 7/77

[@ayehavgunne]

Mmm sausage.

[@LordHelixe]

"$?" only stores the return code of last executed command

[@COneillCodes]

I had the same question about the ::: when I started seeing examples of the parallel command and did the same thing went crazy digging around perl, and bash and zsh and fish trying to figure out what this thing was and couldn't find anything but it is part of parallel its how it separates arguments or you can use -arg-file

[@dradic9452]

I surprised prime didn't know that I had to find out as well and found it in his book "GNU Parallel 2018 ,Ole Tange" section 2.1 'Input sources' it's the first thing he talks about. No, I wonder when it was introduced... dam I'll have to do some more digging.

[@MelroyvandenBerg]

Parallel is written in Perl >

[@Vegetoyesh]

Acadamia has knowledge about GNU parallel for a long time. This is what happens with big tech is so focussed on their own stack. They ignore the amazing open source contributions made by smarter people who know how to simplify things.

[@complexity5545]

Academia is about that money. Most of the tools used in college are used as advertisment. Most students end up with jobs that used those classroom commercial tools. I had to learn about parallel via perl in my sparetime and night job; I was my university's web administrator and saw it in a script that some guy in the 1990s wrote. Thank god, I stumbled upon the old timers' scripts.

[@Vegetoyesh]

@@complexity5545 Wasn't my experience. I got to use whatever I wanted. And so did my professor. I'm sure your point is valid in certain circumstances.

[@jlucsx]

Finally!!! Thank yooouu

[@ZacharyBetz]

Watched for the headline. Stayed for your narration.

[@MelroyvandenBerg]

Parallel written in Perl :)

[@minneelyyyy8923]

parallel is written in perl

[@complexity5545]

This is kind of surprising that he didn't know ::: is cartesian product (in parallel). Its basically data set math/stat like designs to create a argument list of commands. I n bash cartesian product is like `echo {a..b}{1..2}` and it'll print a1 a2 b1 b2. Its SQL like understanding too. Its used to reduce invocations and source code (especially in maths). He probably knew it though, but just forgot (or only saw it used a handful of times by the solo smart old-timers).

[@mitchelvalentino1569]

GNU Parallel and Tmux are the two tools I rely on daily for my work. Indispensable.

[@user-cc8kb]

yeah, he should have asked chat-gippity for the turbo fish thing :D

[@dr3d3d]

this episode was extra amusing as I am currently writing a bash script to auto play Cookie Clicker using only core linux/X11 tools, not just a clicker but full on buying,gardening, combo'ing etc

[@nibblrrr7124]

The most glorious 15,000 lines of Perl ever written.

[@limpiadora]

This could solve a lot of problem at my job, we have a bunch of e2e tests that are crazy, some developer that left try build a multilprocess version with node but it suck and it fail for no reason

[@vladimirreyes1938]

You're learning Spanish, I'd like to see how many native Spanish speakers we are ... greetings from the Dominican Republic.

[@MarcCastellsBallesta]

Do you want to count us with the likes? Or you want a reply from each one of us? 😂 😂

[@javierflores09]

There's quite a few from what I know. Greetings from Argentina

[@Markski]

Greetings, also from Argentina. Love el primo agen

[@juaninfante7000]

Dominican too

[@dots5641]

im going to write an assembler in bash. you cant stop me.

[@kvherro]

Parallel is a superpower

[@AloisMahdal]

Man, both `:::` and `{1}` are totally a GNU Parallel syntax.

[@CTimmerman]

Indeed, as zsh on MacOS says: ❯ ls {1} ::: {1..10} ls: 1: No such file or directory ls: 10: No such file or directory ls: 2: No such file or directory ls: 3: No such file or directory ls: 4: No such file or directory ls: 5: No such file or directory ls: 6: No such file or directory ls: 7: No such file or directory ls: 8: No such file or directory ls: 9: No such file or directory ls: :::: No such file or directory ls: {1}: No such file or directory

[@JoaoAntonioCardoso]

so... it seems that parallel is written in ... perl

[@Maraloon_]

read the doc, it's awesome

[@meowsqueak]

12 minutes later and I still don’t know what it actually does…

[@Diego-Garcia]

The name is Themultitaskgen

[@asdfasdf9477]

yeah, and when after 100+ attempts you finally get all the quoting/escaping/piping just right in your interactive bash session, you can begin fixing quoting/escaping/piping required for a script version

[@joelkorpela2706]

1080p is the new 4k!

[@noTmeDev]

The name is the parallelagen. XD

[@georgecarder4695]

Luke Smith sold me my first hit of GNU parallel.

[@JimWitschey]

dbrock losing his damn mind in the chat. love it

[@sealwithawkwardness3951]

Have you ever heard of VIM though?

[@apina2]

Ghanoo Parallel

[@mage3690]

As someone who has programmed altogether too many lines of Bash (all 10 of them were 10 too many), I know I've done a little too much Bashing (excellent name for it, BTW. It's called bashing because that's what I do to the keyboard with my head) when I see any control flow statements more complex than an "if". "While" is right out, and "for" gives me PTSD (dear God, may I never try to implement flags in Bash). Also, anything more than 5 pipes or escape characters on one line are heinous to read, and I've colorized my command prompt. Why does double quote string expansion escape single quotes, I'll never know.

[@Feedback406]

If you saw my zls colors regex for my completions you would trip out😂😂😂

[@shadyheadstash]

11:57 I'm serious

[@PinakiGupta82Appu]

Again, a suggestion for another mindless Rust-rewrite?😳

[@Tony-dp1rl]

Wow, it's like Powershell on Windows ten years ago.

[@GiovanniCKC]

Powershell got something right? I thought everybody hated it.

[@complexity5545]

Parallel was made in like 2002, back when SMP was being written in kernels, and google started promoting python instead of perl. During that time most programs were written to only run on 1 cpu. Parallel allowed you to take a 1 cpu program and use all 4 of your CPUs. Today we have 128-Cores and parallel will apply to all of them; stuff gets done quick. We'd use it in clusters and then Gearman was made. I think Powershell copied alot of that stuff.

[@H4KnSL4K]

Dude, Primeagen, you need to take a break and slow down sometimes. Take it easy and enjoy it, there's no rush!

[@blakebaird119]

At least the article didn’t call anything mesg

[@weeb3277]

bash or pass

[@martinhotmann7868]

Yeeeah finally this shit it out :) BTW: love your content ^^ I watch it all some days ;) Greetings from germany 🇩🇪

[@davidlee588]

found this on SO: Parallel processing makes sense when your work is CPU bound (the CPU does the work, and the peripherals are mostly idle) but here, you are trying to improve the performance of a task which is I/O bound (the CPU is mostly idle, waiting for a busy peripheral). In this situation, adding parallelism will only add congestion, as multiple tasks will be fighting over the already-starved I/O bandwidth between them. On macOS, the system already indexes all your data anyway (including the contents of word-processing documents, PDFs, email messages, etc); there's a friendly magnifying glass on the menu bar at the upper right where you can access a much faster and more versatile search, called Spotlight. (Though I agree that some of the more sophisticated controls of find are missing; and the "user friendly" design gets in the way for me when it guesses what I want, and guesses wrong.)

[@rawpointer]

I want to date the girl from algoexpert, is the only ad that I don't hate opposed to clement's mental picture.

[@333juniorb]

yeah C baby !!!

[@PinakiGupta82Appu]

😂😂👍👍

[@Han-ve8uh]

Anyone remembers which is the video on this channel where @ThePrimeTime spoke about gnu parallel being able to continue to the next command down the pipe without blocking? There were also some diagrams drawn in that video. Found it! thanks to youtube's good search algo, searching "parallel" led to kzbin.info/www/bejne/pnmtnqCKntt6Z80

[@harshgandhi100]

Stop paralyzing start parallelizing

[@elie3423]

Go sleep parmesan

[@Lemmy4555]

My eyes burn every time I read bash, pwsh and cmd. The technology is still not advanced enough for a comprehensible command line language.

[@CTimmerman]

Nushell and Fish have readable scripts.

[@KeinNiemand]

pwsh more readable then cmd or bash to me, still not great

[@Kane0123]

Or is it really gnu/Linux parallel

[@robervaldo4633]

better to skip parallel and just use ansible

[@ordinarygg]

agree, he is probably Junior dev

[@greyshopleskin2315]

Ansible to run tests? Wtf?

[@ordinarygg]

@@greyshopleskin2315 xD you probably didn't saw more then 5k test suites in your life in automated CI/CD lol

[@greyshopleskin2315]

@@ordinarygg no, but I don’t understand the use case for ansible. I used it to configure machines, that’s it. How do you use it? You have a cluster of n machines, and use ansible to pull latests changes from git repo and run tests across all machines? Or what?

[@ordinarygg]

@@greyshopleskin2315 pull from repo? This is security breach to give access machine to run tests to your repo, it should be synced outside. In general yes, but you missed: - collect the results - bootstrap env to run those tests if they are E2E - create a report If you use something like gitlab CI or even worse github actions imagine it will go down or change the syntax LOL. All kids that use free things don't understand one simple rule, if it's free then price is you and your code. Nothing is free in this world.

[@ordinarygg]

If you know Python forget about this tool and just use Multiprocessing module -> you will have queue, shared memory and more. Bash is very limited and hard to read in terms of just fundamental programming logic.

[@complexity5545]

Do you use it to invoke other programs?

[@ordinarygg]

@@complexity5545 read about multiprocessing, shell and async Pipes, you can open programs and async stream

[@thomassynths]

What's the point of this tool? Basically all modern scripting languages can do the same thing with the same amount of code. Except you aren't using a shell language anymore = Win.

[@samuelschwager]

gnu is not unix

[@bitti1975]

Your bash bashing is uncalled-for. Adhere your own advice and learn your tools!

[@ThePrimeTimeagen]

yes, i agree, but as i said, once i hit arrays, its time to use something better i still think this is a good piece of advice for bash

[@bitti1975]

​@@ThePrimeTimeagen Yes, there should be a threshold, and in the examples given using an array was unnecessary (that's why we have globs, e.g. `for test in potentially_flaky_*.sh; do...`). Still, there is also a threshold for switching to a 'real' scripting language, since interacting with CLI tools is so much easier in bash than in python, ruby and alike. And using an advanced feature can keep the script more readable if used appropriately.

[@bitti1975]

Btw. brace expansion (à la project/{src,dist}) I don't consider an 'advanced feature', since it's also very practical in interactive use (i.e. in particular when you don't write a script). And it's not even bash specific and supported by practically all shells (fish doesn't have range expansions though).

[@GiovanniCKC]

4:36 why not just use a makefile for this? feels like it would work well enough with the parallelization, but now it's more understandable, and I don't need to download a new utility. (maybe something like uhhhh) ```make # tests T=potentially_flaky_1.sh potentially_flaky_2.sh potentially_flaky_3.sh potentially_flaky_4.sh potentially_flaky_5.sh tests: $T potentially_flaky_%.sh: bash $@ ``` and then just exec the tests with ```sh make -j tests ``` and you can still get output in the correct order with just a couple tweaks to the makefile: ```make # output files O=pft_out_0 pft_out_1 pft_out_2 pft_out_3 pft_out_4 pft_out_5 tests: $O cat $O ; rm $O pft_out_%: potentially_flaky_%.sh > $^ ; bash $^ 1>$@ 2>&1 ```

[@papillon8i838]

love. nuf sed