The BEST CLI Tool

[@dnullify100]

I've been using jq for years. I do a lot of troubleshooting as part of my jobs, which means making a lot of ad-hoc API calls to JSON APIs and comparing them to SOT. JQ is so beautiful for wrangling API responses. Processing large JSON files. You can calculate stuff, reshape stuff, feed it to other stuff. Absolutely beautiful

[@Matter743]

I thought only i knew about this XD

[@magfal]

I used this to deal with a 9GB json file I was handed. Split it into 90MB parts before my database ingest and it saved me so much effort.

[@GreyDeathVaccine]

Do you want the Earth to collapse into a black hole under the weight of 9 gigbyte JSON? Are you nut?

[@wolverine9632]

Literally yesterday I was making plans for my first project using json, and I thought, "There HAS to be some CLI tool out there to make this easy. I'll look into that later." Thanks for saving me the trouble!

[@karaloop9544]

Look up one more character and search for jqp. Soooo nice. :)

[@JulianStokesIt]

If only I'd have known this a year ago. And it's been there on my computer for who knows how long! Very useful - thanks for sharing.

[@crispyybaconx]

This is actually pretty sick, I wish I would have known about this literally 2 hours ago where I had a big json file I wanted to get some info from

[@jackdeespadas]

for me was las week... fumbled like a mf...

[@ethanevans8909]

For those of you who are okay with non-posix shells, nushell is basically a whole shell built for this type of workflow and with all types of data. Its pretty pretty fun imo

[@AlecThilenius]

This! NuShell is truly fantastic, it's a shell built around the idea of structured data like this, except EVERY built in command (like ls) also outputs structured data to be transformed. Plus built in parsers for CSV, JSON, Excel and many more. Scripting is also just wonderful, with a Rust- inspired language.

[@mkwpaul]

I tried out nushell some time ago and I feel like its got the same problem that powershell suffers from, namely that dealing with objects is just as often an annoyance than it is useful. nushell definitely isn't as bloated as pwsh nor requires you to sell your soul to microsoft, but for that reason I could never use it as my main shell. Definitely a good tool for data querying/manipulation though.

[@conceptrat]

This sounds like my next tool time moment. 👍 Update: Awesome it's in the Termux repos 😸

[@4444kik]

Was about to comment the same. I haven't looked into jq much since I can do the same and more with Nushell without being limited to just json

[@alexstone691]

I am waiting for nushell to mature and stabilize, its awesome but changes too often and scripting just recently became a thing

[@JohnJones-pp3wd]

Wait what, this is the first time your hearing about this tool, just wait theres `yq` for yml

[@carlsjr7975]

The new golang yq is the bomb diggety. Old python yq was whack

[@replikvltyoutube3727]

There's also xq, but I prefer xmllint --xpath as it's pre-installed on most systems

[@retropaganda8442]

​@@replikvltyoutube3727 It's called xslt, and it's been around for 25 years. xsltproc is the cli tool.

[@colinjohnson5515]

The yq I have installed includes xq and jq. It makes shell scripting suck just a little less.

[@Sommyie]

YQ is a wrapper over JQ 😂

[@SanityAwry]

I use yq (the YAML equivalent) in my deploy jobs for a lightweight way of updating argocd applications in my IAC repos. Works great.

[@DobryWujaszekKun]

I knew and I used jq for quite a long time, but instead I didn't knew about I can pipe a vim buffer to different program xD

[@ragectl]

It's great to see jq to be maintained again. Great tool that deserves to be maintained.

[@AloisMahdal]

I once played with jq and suddenly I had test reporting system, where from every test (pretty complex tests covering automated system upgrades) I collected result in JSON, aggregated them into one big JSON file and then outside the pipeline I had a processor which could turn each of these JSON's into a series of static HTML files (one for each subset based on domain-specific filter). The whole processor thing was built entirely in jq, using its library system. It was.. great times. Seriously. Sure, the system had its limits but working with jq as a language was fun and since jq is a functional language I actually learned a lot by having to shift the paradigm.

[@lukeemhigh]

If you want to fiddle with the queries without having to type the same command again and again, there's a tool called jqp that lets you query json (and yaml) files and preview the results, and whenever you're satisfied you can save the query to clipboard, all without leaving the terminal.

[@aacolive]

Stamps: 1:35 - Reading a file 1:39 - Compact (-c) 1:45 - Specific field (.) 1:50 - Raw (-r) 2:21 - Showing keys (keys) 2:31 - New object 3:00 - Select not null fields 3:31 - has() 3:47 - Math 4:59 - Contains

[@Endelin]

When I use jq, sometimes it's really nice to use the crate "fltn" to pipe json to fzf, and then maybe even pipe my interactive selection back into jq.

[@mdjpurdon]

jq is life, it makes the AWS cli tool actually usable

[@dandogamer]

Hahah yeah been there before , the aws cli is a bit shit

[@itmeurdad]

Used jq with aws cli for years, got super comfortable with it but then took an azure gig and now their garbage json query language has broken my understanding of the syntax for both.

[@Drmonstereater1]

Pretty slick. I usually end up jamming together some array ops in JS in the browser console when needing to quickly get info out of a bunch of json objects.

[@RobnGG]

Still learning and only understand 15% of what you talk about but you’re still entertaining and motivate me to grind, that’s talent.

[@pesterenan]

My god. This left me speechless, I know NOTHING about CLI tools! Thank you Prime for showing us this!

[@TheStickofWar]

Everything useful was first a CLI tool. Nothing you can’t do.

[@barongerhardt]

@@TheStickofWar It always comes back to command line and config files.

[@RossDmoch]

Your wife's idea of a good tool, is different

[@henry-js]

nice nice nice nice ligma nice nice

[@darukutsu]

what's ligma

[@wowvain2989]

ligma balls @@darukutsu

[@caduhidalgo4996]

​@@wowvain2989 GOTTEM

[@gus2603]

@@wowvain2989 lmao

[@RatchetXJ0461]

Prime pushed me get more knowledgable at my cli one of the 1st things after neovim was jq, surprise that you did not know this earlier. Im actually using to pipe from a curl and into a xargs to perform queries in a row

[@demoncorejunior]

“what could this guy possibly so hyped about-oh yeah jq is really cool and i should remember it more often”

[@r00t3g]

jq is awesome. Yet there's more power if you dive into the nushell. It's like a shell with jq and rust-like typed scripting built-in

[@jonesherbert]

Before starting this video I thought what tool he could have discovered and nushell was the only thing I could think of to be excited about CLI-wise in the past few years.

[@abz4852]

In my course lab we need to use jq for querying json from a web api. I remember watching this video 3 weeks ago and ran over to revise things. Literally a godsend

[@FekuEntertainmentLtd]

4:43 - 4:58 Poor arch users. Prime went too harsh on them. 😅😅

[@blenderpanzi]

I have to use jq so infrequently, I always forget the query syntax and have to look it up again.

[@ScarabaeusSacer435]

That's my use-case also, and I don't use it all that infrequently, so I tend to use jet instead of jq, but that only sidesteps the problem if you use clojure on a regular basis.

[@wknight8111]

I love jq. I've been doing a lot of work recently with ElasticSearch and honestly jq is required kit for ES work. Combine jq with bog-standard bash pipelines and you can do some serious magic.

[@maximilianmoser2340]

it goes very well with jless!

[@ScarabaeusSacer435]

This is why I love the comments section. Now I have a new tool to pair with jq and jet.

[@brunoais]

I'm happy you finally found that great tool. I've been playing with it since 2020. I do even more than that. I also edit json files with it. I even flatten json files with it (not that it always works) but those stuff are great, yeah.

[@williamsloan7857]

After watching this video I decided to try jq. I had a pcap file that I need to process 10K packets. I exported the packets as json and was able to do a bunch of data reduction on the command line. I found it to be easier than using a Jupyter notebook with Pandas. Thanks for the info.

[@privatesocialhandle]

0:55 that’s a Quagmire pause during flight announcement. Actual tool is crazy.

[@bladeboles9167]

jq came in clutch today at work. Messy ass mongodb doo doo nightmare and it got everything filtered down by an obscure regex to a nice human readable form for troubleshooting. Thanks for the tip!

[@hackmedia7755]

two other text manipulating commands that are useful are 'awk' and 'sed'

[@shashantr.9380]

Nice one. You could consider making short videos for such cli tools. It's great motivation for using them

[@nicwhites]

It always warms my heart seeing devs learn more about the myriad of cli tooling available. Just wait till you see parallel

[@ristomatti]

It's been on some vid but just not featured as the main topic. He seemed to appreciate it.

[@kalyanoliveira3224]

okay, so I wanted to find an interactive version of jq, and the names I came across in that rabbit hole ended up being quite hilarious there's jid, which allows interactive json filtering than there's jiq, which also allows interactive json filtering, but uses jq as its backend then there's jaq, which tries to be jq, but fixing some of the things that jq does wrong we get to jnv, which allows interactive filtering of json files; they started by using jq as their backend, but now they use jaq as their backend oh, also jql exists, which is yet another json query language tool, but that does not try to be like jq, but which did inspire the author of jaq to make jaq, though jaq does try to be like jq, unlike jql

[@alexjenkins8026]

Man finally learns a useful programming language and is blown away.

[@justinrogo1415]

I actually hate the way jq is a whole new language and is pretty much essential today with how widespread (and useful) JSON is. I actually started using powershell on mac because its support for objects and converting from json is so much better.

[@pylotlight]

I use jq too infrequently to remember the syntax despite using it for years.

[@Gunzy83]

That opening was like poetry. jq is a vital tool.

[@twi57ed]

Im searching for that CLI thing all my life! Thank you!

[@hezuikn]

i made something similar for web scraping at one point but other than that i have no idea what you need this for so i want to ask what you do with it

[@NickSteffen]

Definitely great for parsing curl responses that give you a json token. I randomly came across this when trying to parse out an oauth token so I could stick it in a header for api testing

[@wille4986]

THANK YOU! I have been looking for something like this for the past 2-3 months to form a "lookup" tool against our ...struggling... database that houses non-sensitive document titles and tags; this will make life 100000% easier than figuring out per-user database queries!

[@macethablaze]

That's the kind of content I started to love you back in the days. Awesome

[@LucasSilvaChaves]

Mixing it with fzf makes even more awesome

[@Goshified]

This seems way better than my current tactic of opening a browser inspector with the console open and using a bunch of manipulation with JS to spit out what I want. Going to download it now, thanks!

[@VictorCaldo]

Been using this bad boy for a week and oh my God I've been missing out badly. More quick tutorials like these and how they work for you!!!

[@VictorCaldo]

I've been using nvim wrong too that !jq there opened my eyes

[@sohn7767]

I knew jq for a longtime just never played around with it this much

[@not_a_human_being]

using cli to visualise data in 2024... still... I didn't think we'll be here 10 years ago.

[@radikalZen]

4:54, you dont have to attack me like that . That personal bruh...

[@klamberext]

For those who don't know, then there are substitutes for yaml and xml also: yq and xq

[@Sluhrmz]

JQ? Whoa buddy, I thought this was a programming channel.

[@bigben8502]

lmao

[@rkdeshdeepak4131]

Lol

[@ever-modern]

I have never heard of Jake Weary till today... when I watched the retelling of "It follows". And that very hour Prime mentions him. Damn, these coinncidences are to stop me being atheist one day

[@tasonosenshikitsune1165]

I'm about to blow your mind. There is a different tool called yq. It does the same thing, but YAML. And JSON is a subset of YAML. It does both.

[@obviliontsk]

There is also xq for XAML.

[@rdca1587]

Well, congratulations Primeagen for being of one of the lucky 10.000s recently

[@BrainFunnel]

jq's the bomb, I've been using it for years - it's super helpful for processing kubectl -ojson output. As others have noted, yq is equally great when you need to process yaml...

[@dani-rybe]

And he said he doesn't like higher order functions... WELL LOOK WHERE WE ARE, its basically just a bunch of HOFs and HOF combinators. Reminds me a lot of optics from haskell as well.

[@kenamreemas3295]

this video is going to change my life in more ways than prime can imagine.

[@lbanca0113]

Did he try nushell? Basically does this to literally everything

[@jdal21]

been using nushell since early january for admin work k8s and its the best experience ive had since Im also a data engineer working with python, pandas/polars, sql

[@xXWelhoXx]

Never knew that Jquery had such an awesome command line interface!

[@FranLegon]

I use powershell to do this. Can not only read/select/filter json but also xml and csv

[@tom_marsden]

That’s cool. How does this work using PowerShell?

[@DrSinistar]

@@tom_marsdenPowerShell turns JSON into an in-memory object. The shell offers other commands for querying, mapping, and selecting any collection. Everything mentioned in this video is just standard PowerShell operation. All of the techniques you use to query JSON you could use for other structured data types or .NET objects.

[@itmeurdad]

​@@tom_marsdenpiping to convertfrom-json is the way. It just spits out a native powershell object/hashtable.

[@lcarsos]

btw, there's also yq if you happen to have a whole bunch of yaml that you want to parse through with the same syntax.

[@AR-jx1sy]

I use jq and yq every so often when I need it. However, because I use it infrequently, I need to go back and learn it again and again

[@chefaku]

9:20 "Tech humor: jq json jokes"

[@xyxytx]

have been using this for years. handy tool

[@theoryofmine7473]

There's nothing more satisfying than a well crafted jq

[@IulianMandacheYT]

You should also check yq (same as jq but for YAML) and tomnomnom/gron (it flattens the json keys so it's easier to find a key hierarchy and to grep the data).

[@Aras14]

You might wanna look at the article "let futures be futures"

[@uuu12343]

Jq was my best friend last year, last month, last week, yesterday, today as well

[@scheimong]

Honestly very surprised to see that so many devs don't know jq. If you've tried to work with any sort of web API in a shell script, jq is like the tool they teach in 101.

[@giriw206]

This is a cool tool, I didn't know it existed. I usually just write some basic python code to parse json files. But this seems much faster and easier.

[@fearmear]

I wish I had it yesterday when I had to remove some playlists from a massive Spotify backup json file.

[@flipperiflop]

My jq-fu is so weak - all these things that you could do with it makes me feel like a toddler trying to fit the square block through the round hole.

[@GrishaKirilin]

Wait until Primeagen discovers "jc", a dark brother of "jq", then it completely changes the interaction with the output of any core linux/bsd utility.

[@uuu12343]

This video didnt just teach me more jq than I already have been using, you taught me more about (neo)vim as well LEGEND

[@darthcabs]

So this is what it actually feels to see Primeagen talk about something that you know what he's talking about?

[@jgoemat]

Might have to check that out. Usually I just paste json into a browser console and use javascript. Handy to see it in a text editor though.

[@tornoutlaw]

That is handy, but right now I personally prioritize time into learning ksql to analyse data within kafka-topics.

[@GyanUjjwal-m4u]

The hair on prime while selected text

[@SeRoShadow]

I've been building a JS library similar to this CLI for a while to add to my repo as first project. Supports most things SQL does: Fields, Expressions, Aggregates, DomainAggregates, filtering, sorting. JOINS not yet Not sure if its worth finishing now as people might see it as reinventing the wheel. Should I continue working on it ?

[@джамп]

nothing wrong with reinventing the wheel if you learn while doing it

[@Jasonlhy]

I use dataweave which works for many things like XML CSV JSON, the only down side is it is not open source and I don’t think it is actively maintained too

[@AntoineamtoineStevan]

Prime would probably loose his mind if looking at Nushell ❤ JSON, YAML, CSV, TOML, SQL... all at once!

[@Starwort]

I personally prefer jaq, which is a RIIR that also fixes a few bugs and footguns of the original

[@davidwhitaker6489]

Sick bro squeal for jezson 😂 🎉🎉

[@ptf42069]

i like this jay-sawn query.

[@pieterrossouw8596]

I just always use jq for parsing out data from AWS secrets manager in a bash script... Used it for years but nice to see it can do way more.

[@NeetCode]

This reminds me of mongodb queries

[@ThePrimeTimeagen]

I'm almost offended by this

[@armynyus9123]

fz I find for my json digging sessions even more useful.

[@philadams9254]

I made something like this in PHP years ago but this looks much faster and has more options.

[@froggy3496]

Programmers when they reinvent the wheel again:

[@TheCardil]

You just pitched JQ to your wifey?! 😂 A legend ⭐

[@nublaii]

With these the commands you can do wonders: jq, gron and yq

[@jearsh]

knew about jq...but always struggled with the syntax. thx homie

[@LuisM_Santana]

This video was like "wait there's more"

[@HyperFocusMarshmallow]

This got me thinking about piping files in vim through nu-shell. Sounds nice.

[@Kynolin]

The best thing I've done with JQ is to get our platform guy to install it with the base OS image we put on machines at work. The rest of my JSON CLI needs are solved with JQ's magic. Nobody at my company uses JQ. They will spend weeks drudging through plain text hoping there's a way to make sense of it. (Hint: JQ)

[@sjfsr]

OMG, I wish I knew about this before. Nice tool

[@mauricioprado6395]

@theprimetime could you please describe what the hell were you doing before finding this out? were you like creating a new js app to filter things every time you needed to walk through data? or were you grepping and hopping?

[@daves.software]

I had a similar reaction to learning to use the find command.

[@rosendo3219]

hmmm also noticed @5:40 a drinking jar...shouting out to Cewpins? :)

[@powerranger3889]

4:50 As an Arch user, I feel personally attacked, man. (I use Arch btw)