How The 'awk' Command Made Me A 10x Engineer

[@user-uk9er5vw4c]

the narrative that lead to the tutorial is so cute as it never happened.

[@Bjawu]

I sure hope so, or someone needs to learn about spreadsheets.

[@elieobeid77]

it never happened indeed and a maximum of 4 lines of python can do this, that's why I avoid learning these tools

[@jazzochannel]

spreadsheet? is that like lasagna or vegemite or nutella,@@Bjawu ?

[@Bjawu]

@@jazzochannel There *are* some similarities with vegemite, now that I thing about it. You don't want to use too much.

[@ChimeraBlockhead]

I feel like people just don’t get this guy’s sense of humor

[@cthyau]

I used to use awk every day about 30 years ago. A nice walk down the memory lane with this video.

[@jazzochannel]

memory lane? why did you stop?

[@cthyau]

@@jazzochannelmy work now doesn’t involve coding and data manipulation any more.

[@tmanley1985]

I like how there was a narrative style to the video. That's really engaging. You've earned a sub!

[@mjscpr]

Great pace too.

[@Bluesourboy]

The awk command is now my favorite Linux command

[@catsupchutney]

I avoided Awk for 15 years, but then I saw the light.

[@cinderwolf32]

​@@catsupchutney I have a year left of avoiding it like the plague, maybe. Then I'll give it a shot.

[@bide7603]

Bro went from a 0x developer to a 0.5x dev

[@Dante-420]

10x0 is still 0

[@cleverhippo]

what a hater, the video was great

[@aimpizza6823]

It is but the title is wayy over the top

[@nufosmatic]

Many moons ago I helped maintain what we thought might be the largest awk script ever created - it converted between the proprietary FORTRAN code for compilers by our competitors into our own proprietary FORTRAN code. Of course, monthly, if not weekly, we'd find something it did not convert properly, and it would have to be changed. Not only was it possibly the largest awk script, but it was the most modified awk script...

[@artemiasalina1860]

Fascinating! Do you remember how many lines (roughly) it was?

[@nufosmatic]

@@artemiasalina1860 It was well over a thousand lines.

[@bide7603]

Sounds awk ward to work with…. I’m sorry

[@nufosmatic]

@@bide7603It's not funny, it's a way of life...

[@somebody-anonymous]

I once took a little boy under my wing who thought that text manipulation equated to social skills. He was a little awk ward

[@kellybmackenzie]

I enjoy using GREP and SED and I've been really wanting to learn how to use AWK. This video motivated me to do so, I'm gonna!! Also LOL, the end of the video made me laugh a lot. I love this!

[@Dad-ij2qy]

Thank you for describing the pattern-matching and default assumptions so thoroughly and illustrating each point with example code. Until today have never seen any awk tutorial that could get me up to speed so quickly as your video. Kudos! Showing how awk can replace both the grep and sed commands was icing on your cake. And your last sentence was a big flaming torch sitting atop the cake!

[@BytebroUK]

Get real- once you learn regex, and in my own world, specifically Perl regex, that stuff becomes fairly trivial. "Analysing log files from umpty-hundred jobs" used to be a chore but these days I can script a working solution in an hour or so. In Perl. Because that's the tool I'm familiar with.

[@adam872]

Utilities like awk are one of the reasons I love Unix. Everything is in a text file and the OS gives you the tools to parse and manipulate the contents.

[@weaponkid1121]

I get that awk is a useful tool overall, but with your given example, why exactly would writing a program be time consuming?

[@Temulgeh]

i think it was just a poor attempt at a motivating example and a joke at the same time

[@TheKuhtaMusic]

the whole premise was quite funny

[@jeanhaley3051]

It would take as long to write in python as it would take learning awk. At least pre AI, now AI could generate the command in moments.

[@Temulgeh]

@@jeanhaley3051 i just tried writing that python script and measured how much time it took me (without trying to rush it), it took 1 minute. it might be faster with awk in the long run? but 1 minute is good enough for me

[@utvikler-no]

Even less code with Perl. And this example is not recommended at all. What happens when the data is changed. It all fails.

[@TON-vz3pe]

Wow.. I know awk well, but what a presentation. This is an underrated video. Keep it up dude.

[@RobertElderSoftware]

Thanks, will do!

[@brokenregime]

The solution to the original problem was a bit of overkill, but this was a good example for some teaching of base awk functionality. This is a more direct and shorter solution, only changing the F lines and printing all lines: $ awk '$2=="F" {$1=sprintf("%.1f",($1-32)/1.8); $2="C";} 1' temps.csv (the single "1" at the end is the condition, which evaluates as "true", evoking the default action: print -- nice shortcut if you mean only to change some lines but print all)

[@ppokorny99]

Excellent. Glad to see someone else recognized and posted the solution I thought was more elegant And taught me something new with that “1” trick!

[@MohamedAli-dk6cb]

Finally and after so many years, I understand how 'awk' works. Thank you, Legend

[@PauloConstantino167]

Dude........ Your content is brilliant. You must become more known! We desperately need more Unix content on youtube.

[@BytebroUK]

'awk', 'sed', and 'Perl'. All I ever need for scraping from log-files!

[@GreyDeathVaccine]

head and tail is also useful for a redneck like me :D

[@obiwankenobe3962]

If you know Perl, do you still have a use for Awk? (I'm wondering if I should learn it, I already know Perl. From the looks of this video, I can do the same with Perl in maybe 20-30% more characters - not enough more to trigger learning it)

[@BytebroUK]

@@obiwankenobe3962 Sure. Back in the old Unix days you needed sed and awk to to text munging. Now I just use Perl for all of it. Perhaps more characters to type in, but much simpler syntax and easier to learn, IMO.

[@obiwankenobe3962]

@@BytebroUK thx for confirming my suspicion

[@shyflyf3772]

@@obiwankenobe3962 old comment but still, Perl was devised in part to be awk/sed killer, it has special flags for "read a line at a time and do stuff" and syntax for "write-only" one-liners. Perl is a Swiss Army Chainsaw, and that includes awk and sed functionality. It borrows quite a bit from sed and awk syntax.

[@Quarky_]

Amazing presentation! I've been avoiding Awk for close to a decade, so much so I default to Sed for this kind of tasks! After watching your video, I feel more confident to try Awk next time :) Thank you!

[@krozareq]

@4:10 "As you see, awk is not quite this smart. And here you can see how awk isn't quite as good as a CSV parser. " In GNU awk set the built-in FPAT variable. In your example: FPAT = "([^,]+)|(\"[^\"]+\")"

[@christopherweeks89]

This is video is great and demystified awk in a very easy to understand way

[@stillwrinkled]

so much content in just 10mins. amazing!

[@Zeioth]

"I was so unproductive!" → Proceeds to open the Gnome terminal

[@GreyDeathVaccine]

Alacritty FTW 🙂

[@lanternofthegreen]

This is why people hate the Linux community.

[@-aexc-]

using a better terminal does t make you more productive, just pipe the massive prints so you dont have to worry about print speed on compilation or something. (i say this as someone who uses a fast terminal (foot))

[@xyve6129]

I always thought awk was super hard to learn. 😭 Thanks for explaining it so well! 🙏

[@robbybankston4238]

At my past job where we used Linux exclusively, I used awk extensively along with bash scripts, sed, tr, od, etc. There are so many handy Linux commands.

[@john849ww]

I've never done more with awk than printing columns from a text file, like the 'cut' command. And for other things, use a perl one-liner. Loved the average calculation example in awk. Good to have a new tool in the belt!

[@plasticdub1477]

agreed - awk is awesome and i use it all the time - the most extreme example was a ~400 line script to analyze network files extracted from big integrated circuits but it’s great (and fast) at processing big text files - and it’s highly portable too since it’s installed on pretty much every unix/linux system dating back to the 1960s

[@FishKungfu]

Great video! So, it's 3 years later, I hope you got your juicy 1% raise!

[@TheEvertw]

I love how simple programs developed 50 years ago are still extremely useful. That is because the core of Computing hasn't changed much at all over that time, and UNIX did an excellent job of capturing that core. I pity the fools who know nothing outside of a GUI. Because with a GUI, you get a single tool that can do only what its designers allow you to do. But the UNIX concept is to have a load of little tools that one can combine to do exactly that what you want it to do. Perhaps I should write a GUI overlay for the shell, allowing the old UNIX tools to be configured and linked together graphically. That would be fun, and might help people discover the tools that are available at the CLI.

[@panossavvaidis6086]

BEST INTRO TO AWK. PERIOD!

[@multipontushd4626]

I used awk and sed to convert entire flight manuals into our dataformat that won us the deal with pilatus as we could convert it to our software suit.

[@doughale1555]

I challenged a guy who believes in awk, I let him define the problem. He could use awk, sed, … and I would use cpp, yacc, and lex. The challenge was on two fronts, dev time - first to start, and run time. He insisted that he may run slower than me, but he would clearly win dev time. Well, I ended up not using yacc, just cpp and lex. Beat him in dev time by nearly an hour. Mine ran in less than a second and his took a few minutes. To me awk is awkful with a silent K. It is a very useful tool, but it is just so easy to create your own specific, and much better tool quicker than using awkful.

[@pldcanfly]

This is such a great explainer of a tool i rarely used, but saw the potential, although i never dove in deep enough to learn it. Thank you!

[@BluesManPeich]

This is an amazingly instructive video. Thank you.

[@TheEvertw]

Excellent presentation of how AWK works and how it could benefit you. I haven't used it personally (I would use Python: slightly more complex to use, but MUCH more powerful), but I know people who use it often, e.g. to scan log files for certain situations. It is great for iteratively improving the filters, browsing through the data.

[@lifewater]

Ah the classic 10x engineer, Making 10x the work for everyone else. I too have earned this little.

[@moreirajesse]

Nice tips and examples, man. Greetings from Brazil.

[@chang112x]

The awk command is my favorite linux command. 😂😂😂 Great content! Keep going!

[@Neura1net]

This was an exceptionally good tutorial. Thank you.

[@pneumaticdetach]

Awesome explanation. Thank you

[@lldadb664]

It's been 3 years since you posted the video. I hope you got that raise. Gotta offset inflation somehow.

[@greob]

Thanks for sharing. Good luck with your 1% raise!

[@RobertElderSoftware]

Thank you!

[@NastyWicked]

@@RobertElderSoftware so... did you get the raise?

[@saaah707]

Humorous but informative, I'm no longer afraid of awk, nice video Don't even ask how many python scripts i wrote to do this kind of stuff, grappling with pandas csv reader and such

[@hermask815]

At a time when Excel had only 65000 lines, I used awk to condense big text data files to a manageable size used the associative array feature of awk. If you’re working with same files over again, I suggest to include one sample line for each pattern because you may not recognise the line from the pattern after 2 months. The Fahrenheit example I’d done in libreOffice. Import and so on. There was a time where you had difficulties getting an awk that works on windows.

[@test-rj2vl]

When I look at the command you ended up with I think if it was me it would have taken me less time to write it in python that I already know than to learn all that in awk.

[@prism223]

Real world examples IME look something like this: Some poorly documented program just spat out a noisy status message and produced a bunch of poorly named files. You want to parse the message to associate metadata with the output files, then rename the files with metadata as part of the name. First pipe message into awk to separate into fields, then do simple transform in awk or pipe into sed or grep, then sort file names for association with the metadata output. Then pipe both into awk again to generate a shell script to rename the files.

[@i.8530]

Its now 3 years later, he better have gotten that 1% raise

[@zeppelinmexicano]

The temperature data format is a good one for exercising awk. This is a data driven world.

[@BooleanOperator]

Great presentation style. I just subscribed, hoping to hear if you got that 1% raise.

[@RobertElderSoftware]

I didn't get any raise this year, but I've got a good feeling that if I work overtime for just a couple more years, someone will finally notice my hard work :)

[@first-thoughtgiver-of-will2456]

I really think we need better parsing libraries and at some point the most effective way to dev will be an editor that is easy to add features to which essentially will be coding from the command line. Even between awk and sed there are editing cases where I cannot cleanly implement my operation/macro.

[@wojciech-kulik]

Neovim?

[@e74av]

1.5% raise for the gentleman please.

[@RickTimmis]

Nice Video, this is a great explanation of awk, thank you for sharing.

[@jcdenton7914]

It's 3 years later, did you get that raise bro?

[@Blaisem]

I remember when I used to torture myself with arcane bash commands and construct a towering awk command for complex file parsing. The cryptic awk errors were chef's kiss. Honestly I just prefer any other language at that point.

[@ziranshuzhang6831]

writing a dedicated program takes forever? I agree awk does faster, but writing a python script will take at most half a minute...

[@hamesparde9888]

AWK is very cool. I wore a small program using it and Bourne Shell that was about 300 lines long, but it would have been a lot longer if I was using something else. I think people probably overlook it nowadays because it's old and it's syntax is a bit archaic, but it's very powerful!

[@DougDingus]

I love this video. Awk is one of the basic UNIX tools I always skipped over. Bad call, lol.

[@AWriterWandering]

Awk is a very powerful program indeed. The only thing that keeps it from completely replacing grep for me is the lack of a Perl mode for regex.

[@PotterPhoto]

As a data engineer I did a lot in VIM. But mainly just for small stuff like formatting sample files that I would test transform. And finding anomalies in huge files. Not as powerful as awk, but also a different interface.

[@yoitsgiook]

1% raise 😂 Thanks for the tutorial!

[@starterdev]

I'll never use IDEs and Vscode anymore to process texts! Thanks ❤

[@jozefsoucik3115]

I like your shorts "this is my favorite command", but the longer are more explanatory and better pauseable aswell. Thanx

[@elginbeloy9066]

0:30 dedicated Python code for that task is 6 lines. ``python with open('temps.txt', 'r') as input_file: with open('temps_all_c.txt', 'w') as output_file: temp, scale = line.strip().split() if scale == 'F': temp = (float(temp) - 32) * 5/9 out_file.write(f'{temp} C ') ```

[@TecnocraciaLTDA]

Today AWK isn't only a text-processing utility, it is a complete general purpose programming language! You can use loops, variables, regex, conditionals, functions, array and string manipulation, math lib, include files...

[@draakisback]

Awk is cool but it's just another scripting language. I don't really get the premise, I would have immediately just written a simple bash script to do it or used Python or lisp depending on what machine I was working on. Awk is useful for sure but it's literally the same thing.

[@asdf84421]

These elegant tools focused on practicality and performance are what made me fall in love with Unix (and then OSX). Windows feels like a cludgy toy OS in comparison.

[@josiahjack455]

I piped awk into sed once so I could graph certain lines of a csv dataset in Excel because Excel couldn't open the full file for me to filter it. Awk hey that worked, yeah that's what I sed.

[@aGj2fiebP3ekso7wQpnd1Lhd]

Writing a dedicated program would have taken about 7 lines of code and 5 min, not forever. But, I digress. Us Nix folks live by grep, sed, awk, cut, xargs, etc.

[@SpiroConomos-b1z]

Great vid thanks! @10:20 the "C" doesn't appear in the output. Should it be inside the double quotes? ie ` C" = "` be `" C = "`?

[@RobertElderSoftware]

Good catch, you're exactly right!

[@Misteribel]

Writing a script in C#, F# or Python to do this takes 1 minute and the code will be more understandable and can be copied into your project. Awk is Awesome in niche situations, but I'd only use it when other languages are not available.

[@andrefig822]

yeah given this is just a backstory case for a awk lesson. It still really depends the tech the dev is more used to. If the peson uses shell commands a lot everyday, it's easy to just awk, great to use what it offers. Otherwise is just no-brainer to write a small standalone app/script in your fav language (or any tech), hell it's most of the times just easier to google spreadseet or prompt into ai assistant (that appeared after this video being posted)

[@bromophenol2469]

The awk command is my favorite command.

[@alexonrails]

in 2024 it's counter-productive to rely on awk as it's missing libraries to work with common formats: multiline rows, binary (like parquet), csv with quotes, headers, footers, excel i.e. pandas.read_csv in jupyter/ipython would do the same (could have a one-liner if you wish) but it's compatible with binary formats and anything you could possibly think of. another option is excel if there is no need to repeat the transformation.

[@skyeplus]

There is also jq for json manipulation, but I've never used it in practice.

[@beginlinuxguru7354]

Very cool. I hope you don't mind, but I've just placed a link to this video in the awk chapter of the new shell scripting book that I'm currently writing. (It will be out sometime in the spring.) Many thanks!

[@RobertElderSoftware]

Thanks for the link, I feel honored :)

[@dylanh333]

If you're working in a Windows environment, you can use PowerShell in much the same way for data processing tasks like this. That said, I really need to start using awk in Linux instead of just contorting sed into doing things it wasn't made to do.

[@sadfox9294]

Writing a dedicated program wasn't an option, because that would take forever 1) I could write a simple Python script to solve this in minutes, maybe even under a minute. 2) awk commands are essentially little programs, except you are probably not very proficient with it and crafting the command will take more than writing the code in any other language you are familiar and this task was never going to be repeated again 3) I cannot count the number of times I thought I will not repeat a task, didn't save the solution and then solved the thing all over again later.

[@Hizashisam]

You are the great teacher

[@jiangxu3895]

Hello, thanks for sharing this knowledge. Interesting way of presentation and very concise and informative.

[@phillipscott1920]

Super Useful, and entertaining

[@takeshikovacs1081]

awk is amazing , I love it !

[@dexterplameras3249]

Perl was written because Larry Wall was frustrated with the Awk+Sed combo. Awk+Sed didn't do what he wanted it to do. Perl is still much better then the Awk+Sed combo, however very few people learn Perl but have reverted to learning Awk+Sed instead for text processing. Its a real shame because Perl is still hands down the best for text processing, whether it be on the command line or processing text files.

[@sabitkondakc9147]

yep, awk changed my habits either, thanks for sharing!

[@MickSanford]

someone has earned themselves a pizza party

[@OneManCanStopTheMotorOfWorld]

The Awk command is my favorite Linux command

[@RobertElderSoftware]

This is my favorite comment.

[@gargantooga]

It's been 3 years since you made this video. I really hope you got your raise.

[@RobertElderSoftware]

Not yet, but I'm still hopeful :)

[@petrus4]

Awk was pretty much the first scripting language; it was the ancestor of Perl. It's largely taken over shell scripting on Stack Overflow these days; the kids there love it because they're so paranoid about only calling a single process, and doing everything in a single line. I think its' syntax is a bit tricky, personally; but it does make it extremely easy to create mathematical filters that are hard to do with virtually anything else.

[@martiananomaly]

0:32 that is definitely not true. This can easily be solved using basic file reading, splitting and the C to F formula.

[@AllanKobelansky]

You can use libraries to extend AWK.

[@Criterion1990]

the awk command is my favorite linux command.

[@bultvidxxxix9973]

For reference, here is a PowerShell solution for that problem: Import-Csv .\temps.csv -Delimiter "`t" | Select-Object @{Name="temp";Expression={if($_.unit -eq "F") {"$([math]::Round(($_.temp-32)/1.8,1))"} else {$_.temp}}}, @{Name="unit";Expression={"C"}} | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation | %{$_.toString().Replace("`"", "")} However normally you wouldn't do that as one line in PowerShell. You can just save the whole table as a variable and work on that instead: $temps = Import-Csv .\temps.csv -Delimiter "`t" foreach($entry in $temps) {if($entry.unit -eq "F") { $entry.unit="C"; $entry.temp="$([math]::Round(($entry.temp-32)/1.8,1))" }} $temps | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation | %{$_.toString().Replace("`"", "")} Assuming you need it in exactly the same format again. I'd argue that using commas in a csv file and quoting the values would be better though.

[@kfletcher2005]

Awk is great. Been using it for 20+ years

[@johnsmith1953x]

@10:20, A Joshua Fluke like pun there? :D

[@Ashalmawia]

I was hoping there was a "but" to the "doesn't parse quotes" problem. it feels like I would always be running into the issue that I couldn't quite parse a file with 100% accuracy. (some line or other would be inconsistent somewhere.)

[@rehanga6944]

How about using Excel or Libre Office instead to open the csv file?

[@justawatchin2]

For just viewing a more complex csv, sure, but the transformations (especially more complex ones) are better scripted. Libreoffice beats Excel for me with imports as it defaults to UTF.

[@M.W.777]

Thanks -- You have a new Sub!! Take care

[@bugnar7]

Awk syntax is so convoluted and specific, how about writing a simple python script instead?

[@leonvolq6179]

Before awk I was living on the street and had no significant other. With awk I'm now living on a mansion and have two wives 🎉

[@essetee]

Here we work to live, we don't live to work. Social life is more important than work.

[@calmeilles]

Okay, so this was three years ago. Did you get that raise? 🤣

[@RobertElderSoftware]

I'm still waiting, but hopefully just another few years of overtime and someone will finally notice my hard work :)

[@chesshooligan1282]

Writing a simple script to do that task wouldn't take forever. I'm pretty sure I could write one in Autohotkey (using the StrSplit() function) or in Python (using the split() method) in under 5 minutes, which is probably about the same time it took you to figure out that awk command.

[@karthiksundaram544]

Are u covering sed awk

[@fanshaw]

Excellent explanation.

[@generalzugs6017]

Hmm, first time I watched this video, sound was behind the image, so you looked like Bruce Lee speaking in movies 😁 I thought it was intentional at first, then realized it was a problem in my browser. AWKward.