Learning Awk Is Essential For Linux Users

One of my favorite command line utilities is "awk" which is a text-processing program. It is mostly used for pattern scanning and processing. In this video, I will give examples of some of the basic awk commands, and show you some of the ways that I often use awk.
REFERENCED:
► www.gnu.org/so... - GNU Awk
WANT TO SUPPORT THE CHANNEL?
💰 Patreon: / distrotube
💳 Paypal: www.paypal.com...
🛍️ Amazon: amzn.to/2RotFFi
👕 Teespring: teespring.com/...
DONATE CRYPTO:
💰 Bitcoin: 1Mp6ebz5bNcjNFW7XWHVht36SkiLoxPKoX
🐶 Dogecoin: D5fpRD1JRoBFPDXSBocRTp8W9uKzfwLFAu
📕 LBC: bMfA2c3zmcLxPCpyPcrykLvMhZ7A5mQuhJ
SOCIAL PLATFORMS:
🗨️ Mastodon: distrotoot.com...
👫 Reddit: / distrotube
📽️ LBRY/Odysee: odysee.com/$/i...
DT ON THE WEB:
🕸️ Website: distrotube.com/
🐿️ Gemini Capsule: gemini://distro.tube
📁 GitLab: gitlab.com/dwt1
FREE AND OPEN SOURCE SOFTWARE THAT I USE:
🌐 Brave Browser - brave.com/dis872
📽️ Open Broadcaster Software: obsproject.com/
🎬 Kdenlive: kdenlive.org
🎨 GIMP: www.gimp.org/
🎵 Audacity: www.audacityte...
💻 VirtualBox: www.virtualbox...
🗒️ Doom Emacs: github.com/hli...
Your support is very much appreciated. Thanks, guys!

[@TecnocraciaLTDA]

AWK is not just a text processing utility, awk is a complete programming language!!

[@TheSulross]

yes, remove Perl, Python, and Node JS (notice no mention of Ruby - ah how far they do fall) - awk and bash is all you need

[@ziggyspaz]

@@TheSulross you need Perl to keep your sanity, you can remove the rest

[@gordongoodman8342]

Calm down. It's not that exciting.

[@TecnocraciaLTDA]

@@gordongoodman8342 it is, a lot!

[@tekvax01]

yup! I was going to comment the same thing. you could even use it as your shell.

[@SPSHOfficial]

At 8min, when you pipe uniq, note that 'uniq' does not detect repeated lines unless they are adjacent. You may want to sort the input first, or use 'sort -u' without 'uniq'. so it would be | sort | uniq OR | sort -u

[@fabrice9848]

indeed

[@joedunder4439]

quite

[@BrendoIsGaming]

rather

[@alex2143]

indubitably

[@sp10sn]

moist

[@JosephSaintClair]

awk is a must. So powerful. Btw. You actually did just square. Not square root :)

[@itsalivevideo]

Could have done the square root of i*i is i

[@baruchben-david4196]

I tried it with sqrt(i) and that worked. I kind of wonder how much math you can do with awk... Yes, exp(var), log(var), cos(var), sin(var) - but not tan(var); var**x gave exponent, as did var^x. Lots of different functions...

[@sachindeshpande1]

I have poorest knowledge of maths but I paused and rushed to comment section to check I am not wrong. But tutorial is very good.

[@BrutusHiatus]

This is a great tutorial about awk. I've learned quite a bit. The only thing that red flagged me was that NONE of those were square roots. They were squares.

[@edwardonsax9919]

I just posted this same thing, before seeing you posted it.

[@danieldevito414]

He just got the order backwards. "The square root of i*i is i". Anyone could have made that mistake.

[@OscarFURIOUS]

I just came to say that I can see that you listened to the comments and now you are sharing your knowledge on simple but technical stuff that's really helpfull. I apreciate that and also can feel that effort

[@ecavero1]

In the square root example, very time DT said square root, he means square! Great video, DT!

[@antoninperonnet6138]

Just swap the 2 fields and pretend you didn't see that: "10 is the square root of 100" is a completely valid statement 😃

[@ecavero1]

@@antoninperonnet6138 I could do that! 😉

[@KeithPeters]

It's AWK math

[@pleaseenteraname1215]

@@KeithPeters 🤣🤣🤣🤣

[@erics7004]

DT: AWK is important for linux users Luke Smith 1 hour later: Why AWK is useless for linux users

[@charlessmith5465]

_Awk is the predictable result of misguided Enlightenment philosophies._ 🤭

[@PS3PCDJ]

Brodie 2 hours later: Boomers fighting over usefulness of linux utilities

[@TarebossT]

_Why learning awk when you can learn grep_

[@blogattacker]

Luke Smith 1 hour later: "You can replace Awk with Emacs" 10hs later: "You can replace your wife with Emacs"

[@elderberry-hamster]

@@blogattacker 10.5 hours later...I did. 😂

[@jeffcauhape6880]

That is the most clear demonstration of AWK I have ever seen. Amazing.

[@driden1987]

Had to process a 120k line csv at work the other day. Awk was a god send

[@intrepidca80]

14:00 - /^[b,c]/ matches anything that begins with a "b" or a "c" *or a comma*.

[@Joe3D]

neat

[@InfiniteQuest86]

So true.

[@emeraldmasta7752]

Regex yeet

[@davidruedeman9990]

Learned more about awk in 20 minutes than in the previous 40 years

[@KeepItTechie]

Great tutorial DT! AWK is very powerful!

[@DistroTube]

Thanks, Josh!

[@KeepItTechie]

@Earthling-Z3R0 Thanks man. You're right.

[@tarekrahman1]

Glad you made this video. I was stumbling upon awk in everywhere and wanted to learn it but didn't have time to do so. Thanks

[@OpusFocus1968]

Thank you Derek I appreciate the time and effort you put into making these videos. You help us "middle users" take things up a notch. Middle as in we are no longer beginners but won't ever be sysadmins. We do however want more out of their Linux system.

[@gwgux]

This is one of the best explanations of awk I've seen! I'll save this one for some people I know that are still just learning Linux.

[@skirnir393]

Good introduction to awk. Back in 2013 i had to parse huge log files that had no field separator, even worst each line could be completely different depending on certain conditions. Awk saved my life.

[@4lc0h0l]

This is great, this video arrived just in time since currently I am on problem automating a lot of stuff with bash scripting and using awk for text-processing. Thanks for the content!

[@VexisMorlock]

Great TUT. I ran into an issue because I don't use uniq and sort often enough; uniq doesn't work if dupes are not adjacent so you need to sort first.

[@ourallabrador4023]

I have been using linux for years living on a few commands. I had only a faint knowledge of the command/programs such as awk or sed. Mainly because I had no ideas of their capabilities and use cases. Now I can see how useful they can be. I usually recommend your channel to my PhD students and observed a real improvement in their computer skills. IMO your channel is vital for linux wannabe and "familiar but not expert" users. So thank for your work.

[@ghodawalamoamanzahirabbas8996]

He also inspires me to learn about this powerful utilities.

[@milo_andrs]

Nice, thank you. I am beginning my journey on bash and this is awesome.

[@Desi-qw9fc]

Great video. An accessibility note: Your console’s active line is at the very bottom of the screen, which is where KZbin on Smart TVs puts the subtitles, so the subs block the thing you’re typing. It would be better if your terminal was sized so that the bottom 10-20% of the screen isn’t used.

[@h0ck3y66]

I love you man, you're the only one who was able to explain awk in a simple yet effective way. I finally understand how awk works!

[@JoshBenore]

I have always had trouble with awk because of all the syntax involved, this helped sooo much. When you pressed enter on the square root part then said that the output was correct, this is the task list I had to perform to get back on track: 1. Facepalm 2. Pull fire alarm 3. Crying session 4. Shower with clothes on 5. Call Mom 6. Eat some sweets 7. Finish the video

[@abarocio80]

Good video. Some of these, I didn't knew. An error in the video: it wasn't the "square root", but just the "square", or "power of two". The square root of 4 is 2, the square root of 9 is 3, the square root of 25 is 5, and so on.

[@christopherdavies7403]

i came to the comments expecting every comment to be this. I am highly disappointed in the YT comments shit posting today.

[@gordonbai5320]

@@christopherdavies7403 And this guy says 'I didn't knew', grammar mistake.

[@charlieoocharlie9466]

@@gordonbai5320 ok boomer

[@weege5.45]

@@charlieoocharlie9466 FLAME WAR! And you live in your mother's basement.

[@rochakgupta6116]

DT, you are amazing man. Yours are the only videos I have been watching on a daily basis. Thanks for sharing all the knowledge!

[@therealslimaddy]

I bet someone is already rewriting awk in rust for the memes.

[@alessandroferrari4699]

you mean for speed and safety? /s

[@rishirajsaikia1323]

@@alessandroferrari4699 people use rust because it's fast and easier to solve errors and even rivals c in speed while being easier.

[@TheSulross]

yes because obviously all these 40 year old Unix tools are buggy and broken after thousands upon thousands of users applying them daily over many decades

[@cgme7076]

@@TheSulross exactly 😂😂

[@arnoaldloukasscirpionis1558]

Well. People have already rewritten it in languages much less speedy than Rust - Go and even Common LISP. Knowing this, rewriting it in Rust is a much better idea !

[@RobertGuilman]

Sir, your a wizard with awk, i only knew half of this 😂

[@fabrice9848]

The simple fact that fish is your default shell deserves a like and a subscription.

[@pnavais]

Straight to the point & excellent as always. Keep up the good work DT ! 👏

[@mrdanymotion1903]

Hell yeah!!! AWK arrived at the right time!! Thank you Derek!

[@DaveRogersEsq]

Needed a quick refresher on awk as I had no used it in years. This was a great video, covering the essentials. Cheers.

[@Hexalyse]

awk is awesome, but if you just want the number of lines, using wc is less characters to type :)

[@TheSulross]

yeah, that was one example that did not resonate at all because wc will also provide totals when applied against multiple files - much less to type to just use wc

[@egor.okhterov]

The point is to show what you can do with awk =)

[@dragonsage6909]

That was some awesome awk-fu bro, thank you!

[@keebler6591]

Great video DT! Thanks for the awesome awk tutorial Except: s/square\ root/square 😀

[@Winnetou17]

I think a very good example of how good & useful is awk is the example of a log like the apache access log (in the default format) and properly filter the requests that responded with a 400 or 503 response, given that you can find 400 and 503 in the request address, or in the size or the time it took to complete. Also to filter all requests that are longer than a second for example and many other things like that. It's basically Excel powers in CLI over column-structured data/files (which are common in commands outputs or log files)

[@zainjadoon759]

I have been putting this off for a long time thanks for encouraging me to finally learn it.

[@wowgingermobile2123]

This is truly amazing, hope to have another new Awk video in the furure!

[@fargoflagrant7796]

This is really helpful and practical, thank you DT!

[@aldeezy1]

small suggestion, bring the bottom of your terminal window up a bit so the youtube progress bar doesn't cover up the command you're entering when the video is paused. Thank you for making this!

[@warhawk_yt]

I know in most window managers you can change the size of a window from left or right but not on the top or bottom at least from what I have seen unless you are in floating mode than you can change it to any size. Most of the space around each window is configured hard coded in your window manager config. You could change it but than would have to change it back after a video.

[@aldeezy1]

@@warhawk_yt clearly he just needs to increase the edge padding to something crazy like 48-64 pixels for the [ M A X I M U M R I C E ]

[@Soupie62]

A little bit of Awk can be useful. My standard example: I have a list of shares I track. The website I use lets me download data, but holds more data (open, close, max, min,volume ) than I want. Awk is perfect for filtering a single column (close values).

[@jacksongodsey]

This couldn't have come at a better time! I was seeing this command everyone in people's dwm configs and had no clue what it is.

[@TheBlueThird]

DT, this is an excellent video! Thanks!

[@baruchben-david4196]

Thanks for this video. It helps me understand how to use awk better.

[@bradleyeverson697]

Hey DT. Great review of the power of "awk". You cover all the basic functions for a new user. I use "awk" in a shell script to sort through my datalog's results that i need. Then dump it to a csv file.

[@necronlord52]

As I found out, you have to sort _before_ running the _uniq_ , otherwise it wouldn't work.

[@Lucas-md8gg]

My payment will be available in 2 weeks, then I'll be a patreon. Your content is amazing.

[@elad3958]

Great tutorial. I learned more in 20 mins w/ awk then in 7 years in IT.

[@datturns9947]

What did you do then in 7 years of IT? :))

[@Jerry-fy2gc]

Thanks buddy for such clear explanations. Now I'm comfortable to start write some crazy scripts.

[@rickbhattacharya2334]

Thank you professor DT for the lecture .

[@jeffg4686]

Thanks for the share. I used use text editors like Textpad to do these things with regex, but this is so much better since not just find/replace, but perform operations on each result, and being able to easy have in a shell pipeline of commands. Always too lazy to learn it for some reason, but now I'm motivated.

[@al1234]

This is the kind of stuff I love to watch

[@vn1384]

This is a great video on AWk..thanks

[@bitti1975]

The problem with awk is that most people know it but all seem to think its whole purpose is just to extract columns. This Video gives many practical example that it's so much more.

[@braelinmichelus]

A couple months ago I wrote a little shell script to gather system information such as CPU model, memory, computer model, shell, WM (using lscpu, dmidecode, etc) and I had to learn a little bit of Awk, Sed, and Grep to clean it up and make it look pretty. This video would have came in handy back then, instead of spending 30 minutes reading StackExchange articles on them!

[@ghodawalamoamanzahirabbas8996]

Could you share your script? I would like to use it!

[@rogernevez5187]

Awk ... very meaningful name !!! Looks like an assembly instruction.

[@TheJacov]

Yup, awk is cool. Just as an aside that was squares, not square roots. For square roosts you would need something like (n^(1/2)). AWK does have ^ which is raise to the power function. Raising a number to a fractional power is taking the root.

[@balochcode]

Excellent presentation!! to the point and straight forward! great job!

[@rolcarislew7761]

Yup, it's a complete language, as it is taught in CS degree courses (when taught). However this guy, otherwise, is a far more interesting teacher. Thanks, DT! This app/language confusion here is, like with Python, there is the language, AND there is ERIC, a free IDE that comes with Python. ERIC executes Python, but it's NOT Python. The AWK IDE is more invisible, but there. You can write independently, then execute.

[@zeppelinmexicano]

This is a great review of some practical usages.

[@addaline9645]

I learned so much more here in 20 mins then I did in my last two classes about this

[@anthienvo]

Heaviest 20 minutes in my life. Worth it

[@seansretroverse9082]

Nice Luther insult at 8:20! I for real laughed out loud on that one. I'm surprised no one called that out in another comment...😁

[@name1355_0ne]

Very nice introduction to awk features. Thanks for sharing this useful knowledge!

[@AWriterWandering]

I really need to push myself to use awk more. A lot of this can be done by just combining grep, sed, and cut, but it’s kinda nice having it in one process.

[@jacksystems]

Very useful info DT! I only knew how to print columns in awk.

[@christianalvarez8606]

Your audio sound 1000x better then before

[@JuhaLaiho]

First: a good video, thanks! But it might need a "part ii", going deeper into what makes awk special among other languages. Awk is what at least I call a "pattern-action" language. So, the thing before the braces is a pattern, a kind of conditional structure telling for which input rows to execute that given block of code, and the block of code is then what to do when the pattern does match. Pattern here refers to a much wider set of conditions than just regular expressions; it can e.g. be a comparison against some variable you set yourself within the program, based on some previous data, and so on. So, if your statement within the braces is just "if (some condition based on the input row)", you can just lose the "if" and move the condition outside the braces, to have the same effect. You also briefly show that the pattern can specify a range of rows, by having two expressions to match the first and last rows of the data to process for that particular action. This could be for example "/start-marker/,/^$/" to process all sets of rows that begin with a row containing a "start-marker" and end with an empty row. Whatever comes after the empty row before the next "start-marker" is ignored (by this action, but you can have other actions in the script to do something else for those rows). The patterns for the beginning and end of the range do not need to have any relation to each other. These together make ti possible to very rapidly build "state machines" for processing text with some more complex but still regular structure (IBM JVM garbage collection logs being one example of such data format; I've done some awk pieces that pull out numeric data out of those and then output that data in a "only numbers" CSV form that can be pushed out to a spreadsheet for producing graphs out of the data). It is of course possible to do the same with any general-purpose language, such as Python or perl, but they don't lend themselves quite as well to building such processing state machines as awk does, it takes much more time to have the equivalent Python/perl code look anywhere as neat and readable.

[@Theineluctable_SOME_CANT]

For pattern matching, try SNOBAL 😉

[@swarooprajpurohit110]

Hey, I'm a beginner. Can you tell me if I've to learn commands like awk, sed, etc or should I come back when I start bash scripting?

[@Theineluctable_SOME_CANT]

@@swarooprajpurohit110 well, it very much depends (on what level of learning and relevance to you personally too). Example: I know awkward, sed, etc exist. And if required, I could write a complete program in PERL, (which eats awk for breakfast, because that is exactly WHY PERL was devised...) to solve a problem. In reality, I'm usually solving problems at a much simpler, "high" level. So, you can position yourself in the "space".... I cannot overstate the relevance of experience. Not necessarily in coding, but in EVERYTHING.

[@swarooprajpurohit110]

@@Theineluctable_SOME_CANT Hey, I didn't get you. I meant what would your recommendation be for a beginner?

[@Theineluctable_SOME_CANT]

@@swarooprajpurohit110 just start by learning to completely understand the top display. top is like task manager. Then you could read the man pages for the main utilities. Learn a bit about how the cron demon works and what it is for. From there just install a few different distros and find your favourite.

[@umop3plsdn]

love both Sed and Awk

[@AkashAgrawal03]

Thanks for making this video! I've tried a couple of times to add AWK to my arsenal, but gave up halfway. This has inspired me to give it a more serious go. I'm generally pretty comfortable with grep and sed but looks like I can make this work for replacing those two for many use cases.

[@vkg2958]

Good Morning Sir, I just wanted to know which Distro and terminal you're using here?.. it's so dope! 🔥 Also that fun message after entering wrong syntax LOL that's where I lost it..

[@davidgomez79]

Great vid DT!

[@ivans3806]

Please do jq tutorial next: it's so awesome to manipulate json data, combined with curl it's very powerful for working with REST apis

[@ghodawalamoamanzahirabbas8996]

Is jq a GNU utility?

[@ivans3806]

@@ghodawalamoamanzahirabbas8996 it isn't (according to short duckduckgo search :) ) - but it's amazing nonetheless!

[@Merigold83]

Your example for the line count can be reproduced with the command "wc -l /foo/bar/file_a /foo/bar/file_b"

[@yatishs1071]

To print the last words or specific words we can use the cut -d "." -f3 or -f3- -f3 prints only 3rd word -f3- prints rest of the strings including 3rd word

[@bjazz0019]

Cut(1) would be my go-to tool over awk if only they'd do their delimiter handling properly. Until then, it is rarely touched. So sad.

[@eritert]

Awk is awesome, great vid I learned a lot

[@AndersJackson]

Great introduction of `awk` 3:30 can be done with 'getent passwd' :-) 13:45 you print the square of, not square-root of.

[@cassandradawn780]

3:00 `cat /etc/passwd | cut -d':' -f1` also works

[@mraviteja22]

I wish to have a series on awk,sed,grep from you dt just like Emacs and vim.

[@tekvax01]

Don't forget TR, HEAD, and CUT too...

[@gerarddelapatefeuilletee4063]

@@tekvax01 and tail, colrm, too...

[@tekvax01]

WELL! Cheers mate! I'd never used colrm before! How do you change the delim character?

[@gerarddelapatefeuilletee4063]

@@tekvax01 as far as I know, colrm just works in pure column mode, it doesn't deal with delimiter characters.

[@elderberry-hamster]

Great tutorial! I do prefer the WTF command. It covers all my confusion when trying to navigate the command line. 😜

[@juanmanuelcruz6683]

04:40 min. Pipe the awk to column command. Like: | column -t

[@user-ug4gy4bn1p]

Great video, helpful comments and even better resourceful and corrective people in the comments

[@adriancostin9138]

i always found it easy to use because i'm AWKward.

[@14u2ponder]

I am actually going through the famous Awk, the programming language for the last two weeks. And, I spotted a least 1 verboseness. When you are trying to format the output of your shells, you don't need the regular expression to look for slashes. Not sure why you did that. It might also be helpful to people to know that you are turning NF(which counts and stores the number of fields) into a $1, $2, however many fields are in each line, that is what $NF becomes. If it is 4 fields, then $NF becomes $4 and so on.

[@jnelly3426]

How on earth do you remember the AWK syntax? Each time I have to use AWK, I have to look it up again. Even when I am using it, it has been very basic. I’m not very good at reading the escape characters and anything in slashes. Thanks for making that clear.

[@akya]

Never needed awk in last 6-8 years or so... but thanks!

[@the_senthil]

Quality content 💯

[@sar3388]

For getting the last column, we could simply do : awk -F "/" '{print $NF}' /etc/shells. Why is pattern matching required?

[@gilbertzilla6770]

you dont need to put the double quotes. you can also do awk -F: '{print $1}'/ /etc/password

[@AndyDavis007]

Okay so nube here. I need to either awk on windows--what's the Linux on widows thing---or htf do I export *that* ou for awk to process a "computer accounts" move request the admin is clueless about

[@charlessmith5465]

if you want to list Windows users, it might be easier to use a PowerShell command. If you want to run Awk on windows, the [Git for Windows] installer can install a bunch of unixy utilities (Bash, Perl, Awk, etc), OR you could install Cygwin, OR I think the Linux compatability layer in windows is called "WSL", which you could use to set up a linux environment within Windows. Or use a VM, or use Docker... etc.

[@AndyDavis007]

@@charlessmith5465 wow lots of options. I have to check into the powershell one. Never waste an opportunity to work at a problem from different angles.

[@alexxx4434]

Missed an opportunity to show how to run calculations on numbers over different rows. With AWK you can basically have spreadsheet calculations.

[@nbensa]

Hi! You need a sorted input in order for uniq work correctly

[@gerarddelapatefeuilletee4063]

| sort -u does the job.

[@nbensa]

@@gerarddelapatefeuilletee4063 Thanks! I didn't know about -u

[@mehdimahdavi6619]

Hi, thx for ur great videos reagarding Linux! I have a question: r u still using trackballs?

[@calvinproject]

Amazing! Basically I can programming using awk. Thanks

[@DistroTube]

Yea, you can actually use "awk" as a scripting language. Might have been useful in past decades. But now, I'm not sure why anyone would script with awk when things like bash and python exist.

[@raushonmunni2633]

@@DistroTube maybe awk faster than python and more specific programming language bash or posix shell

[@c434rdd410]

thanks awk command that help me a lot in my project after watching this video ^_^

[@flesz__]

Thanks! What mic are you using? Also 7:40 , you could use awk -F "/" '{print $NF}' /etc/shells for the same result Your df command gives output equivalent to df -h , I suppose you configured an alias?

[@korleone72]

that is a very cool podcast, very useful. thank you

[@kirk0831]

Thank you for doing this video. Very helpful.

[@Euzahad]

Dude, plz do Udemy or any full courses, I came here from some lame udemy course, because there was no deep dive on awk command like here love Ur content, awesome !

[@lamprospitsillou6325]

For now I am more comfortable with sed, cut, tr . Surely awk will some day be in my tool belt , just can't find a reason to spend time learning it.

[@AlastairMontgomery]

I found that a lot of the cut, sed, tr commands I would pipe together could be done in just one awk command.

[@lamprospitsillou6325]

@@AlastairMontgomery That's why i said one day i want to learn it, i recognize that it's power full .

[@lamprospitsillou6325]

@Learn Linux I primarily rin dash as the default shell, i prefer to stay away from bash of possible, but yes bash has some really cool features, especially for variables

[@ismoiljonabdumajidov9260]

i really appreciate your effort , thank you for the great lesson MAESTRO .