The BEST CLI Tool

[@dnullify100]

I've been using jq for years. I do a lot of troubleshooting as part of my jobs, which means making a lot of ad-hoc API calls to JSON APIs and comparing them to SOT. JQ is so beautiful for wrangling API responses. Processing large JSON files. You can calculate stuff, reshape stuff, feed it to other stuff. Absolutely beautiful

[@Matter743]

I thought only i knew about this XD

[@aacolive]

Stamps: 1:35 - Reading a file 1:39 - Compact (-c) 1:45 - Specific field (.) 1:50 - Raw (-r) 2:21 - Showing keys (keys) 2:31 - New object 3:00 - Select not null fields 3:31 - has() 3:47 - Math 4:59 - Contains

[@magfal]

I used this to deal with a 9GB json file I was handed. Split it into 90MB parts before my database ingest and it saved me so much effort.

[@GreyDeathVaccine]

Do you want the Earth to collapse into a black hole under the weight of 9 gigbyte JSON? Are you nut?

[@JulianStokesIt]

If only I'd have known this a year ago. And it's been there on my computer for who knows how long! Very useful - thanks for sharing.

[@SanityAwry]

I use yq (the YAML equivalent) in my deploy jobs for a lightweight way of updating argocd applications in my IAC repos. Works great.

[@crispyybaconx]

This is actually pretty sick, I wish I would have known about this literally 2 hours ago where I had a big json file I wanted to get some info from

[@jackdeespadas]

for me was las week... fumbled like a mf...

[@ProxiedAi]

Wait what, this is the first time your hearing about this tool, just wait theres `yq` for yml

[@carlsjr7975]

The new golang yq is the bomb diggety. Old python yq was whack

[@replikvltyoutube3727]

There's also xq, but I prefer xmllint --xpath as it's pre-installed on most systems

[@retropaganda8442]

​@@replikvltyoutube3727 It's called xslt, and it's been around for 25 years. xsltproc is the cli tool.

[@colinjohnson5515]

The yq I have installed includes xq and jq. It makes shell scripting suck just a little less.

[@Sommyie]

YQ is a wrapper over JQ 😂

[@wolverine9632]

Literally yesterday I was making plans for my first project using json, and I thought, "There HAS to be some CLI tool out there to make this easy. I'll look into that later." Thanks for saving me the trouble!

[@karaloop9544]

Look up one more character and search for jqp. Soooo nice. :)

[@DobryWujaszekKun]

I knew and I used jq for quite a long time, but instead I didn't knew about I can pipe a vim buffer to different program xD

[@hackmedia7755]

two other text manipulating commands that are useful are 'awk' and 'sed'

[@RossDmoch]

Your wife's idea of a good tool, is different

[@henry-js]

nice nice nice nice ligma nice nice

[@darukutsu]

what's ligma

[@wowvain2989]

ligma balls @@darukutsu

[@caduhidalgo4996]

​@@wowvain2989 GOTTEM

[@gus2603]

@@wowvain2989 lmao

[@theoryofmine7473]

There's nothing more satisfying than a well crafted jq

[@lukeemhigh]

If you want to fiddle with the queries without having to type the same command again and again, there's a tool called jqp that lets you query json (and yaml) files and preview the results, and whenever you're satisfied you can save the query to clipboard, all without leaving the terminal.

[@bloody_albatross]

I have to use jq so infrequently, I always forget the query syntax and have to look it up again.

[@ScarabaeusSacer435]

That's my use-case also, and I don't use it all that infrequently, so I tend to use jet instead of jq, but that only sidesteps the problem if you use clojure on a regular basis.

[@RobnGG]

Still learning and only understand 15% of what you talk about but you’re still entertaining and motivate me to grind, that’s talent.

[@ragectl]

It's great to see jq to be maintained again. Great tool that deserves to be maintained.

[@demoncorejunior]

“what could this guy possibly so hyped about-oh yeah jq is really cool and i should remember it more often”

[@AloisMahdal]

I once played with jq and suddenly I had test reporting system, where from every test (pretty complex tests covering automated system upgrades) I collected result in JSON, aggregated them into one big JSON file and then outside the pipeline I had a processor which could turn each of these JSON's into a series of static HTML files (one for each subset based on domain-specific filter). The whole processor thing was built entirely in jq, using its library system. It was.. great times. Seriously. Sure, the system had its limits but working with jq as a language was fun and since jq is a functional language I actually learned a lot by having to shift the paradigm.

[@Drmonstereater1]

Pretty slick. I usually end up jamming together some array ops in JS in the browser console when needing to quickly get info out of a bunch of json objects.

[@brunoais]

I'm happy you finally found that great tool. I've been playing with it since 2020. I do even more than that. I also edit json files with it. I even flatten json files with it (not that it always works) but those stuff are great, yeah.

[@williamsloan7857]

After watching this video I decided to try jq. I had a pcap file that I need to process 10K packets. I exported the packets as json and was able to do a bunch of data reduction on the command line. I found it to be easier than using a Jupyter notebook with Pandas. Thanks for the info.

[@Endelin]

When I use jq, sometimes it's really nice to use the crate "fltn" to pipe json to fzf, and then maybe even pipe my interactive selection back into jq.

[@macethablaze]

That's the kind of content I started to love you back in the days. Awesome

[@RatchetXJ0461]

Prime pushed me get more knowledgable at my cli one of the 1st things after neovim was jq, surprise that you did not know this earlier. Im actually using to pipe from a curl and into a xargs to perform queries in a row

[@FranLegon]

I use powershell to do this. Can not only read/select/filter json but also xml and csv

[@tom_marsden]

That’s cool. How does this work using PowerShell?

[@DrSinistar]

@@tom_marsdenPowerShell turns JSON into an in-memory object. The shell offers other commands for querying, mapping, and selecting any collection. Everything mentioned in this video is just standard PowerShell operation. All of the techniques you use to query JSON you could use for other structured data types or .NET objects.

[@itmeurdad]

​@@tom_marsdenpiping to convertfrom-json is the way. It just spits out a native powershell object/hashtable.

[@pesterenan]

My god. This left me speechless, I know NOTHING about CLI tools! Thank you Prime for showing us this!

[@TheStickofWar]

Everything useful was first a CLI tool. Nothing you can’t do.

[@barongerhardt]

@@TheStickofWar It always comes back to command line and config files.

[@SeRoShadow]

I've been building a JS library similar to this CLI for a while to add to my repo as first project. Supports most things SQL does: Fields, Expressions, Aggregates, DomainAggregates, filtering, sorting. JOINS not yet Not sure if its worth finishing now as people might see it as reinventing the wheel. Should I continue working on it ?

[@джамп]

nothing wrong with reinventing the wheel if you learn while doing it

[@ethanevans8909]

For those of you who are okay with non-posix shells, nushell is basically a whole shell built for this type of workflow and with all types of data. Its pretty pretty fun imo

[@AlecThilenius]

This! NuShell is truly fantastic, it's a shell built around the idea of structured data like this, except EVERY built in command (like ls) also outputs structured data to be transformed. Plus built in parsers for CSV, JSON, Excel and many more. Scripting is also just wonderful, with a Rust- inspired language.

[@mkwpaul]

I tried out nushell some time ago and I feel like its got the same problem that powershell suffers from, namely that dealing with objects is just as often an annoyance than it is useful. nushell definitely isn't as bloated as pwsh nor requires you to sell your soul to microsoft, but for that reason I could never use it as my main shell. Definitely a good tool for data querying/manipulation though.

[@conceptrat]

This sounds like my next tool time moment. 👍 Update: Awesome it's in the Termux repos 😸

[@4444kik]

Was about to comment the same. I haven't looked into jq much since I can do the same and more with Nushell without being limited to just json

[@alexstone691]

I am waiting for nushell to mature and stabilize, its awesome but changes too often and scripting just recently became a thing

[@twi57ed]

Im searching for that CLI thing all my life! Thank you!

[@hezuikn]

i made something similar for web scraping at one point but other than that i have no idea what you need this for so i want to ask what you do with it

[@wknight8111]

I love jq. I've been doing a lot of work recently with ElasticSearch and honestly jq is required kit for ES work. Combine jq with bog-standard bash pipelines and you can do some serious magic.

[@privatesocialhandle]

0:55 that’s a Quagmire pause during flight announcement. Actual tool is crazy.

[@r00t3g]

jq is awesome. Yet there's more power if you dive into the nushell. It's like a shell with jq and rust-like typed scripting built-in

[@jonesherbert]

Before starting this video I thought what tool he could have discovered and nushell was the only thing I could think of to be excited about CLI-wise in the past few years.

[@kalyanoliveira3224]

okay, so I wanted to find an interactive version of jq, and the names I came across in that rabbit hole ended up being quite hilarious there's jid, which allows interactive json filtering than there's jiq, which also allows interactive json filtering, but uses jq as its backend then there's jaq, which tries to be jq, but fixing some of the things that jq does wrong we get to jnv, which allows interactive filtering of json files; they started by using jq as their backend, but now they use jaq as their backend oh, also jql exists, which is yet another json query language tool, but that does not try to be like jq, but which did inspire the author of jaq to make jaq, though jaq does try to be like jq, unlike jql

[@alexjenkins8026]

Man finally learns a useful programming language and is blown away.

[@abz4852]

In my course lab we need to use jq for querying json from a web api. I remember watching this video 3 weeks ago and ran over to revise things. Literally a godsend

[@rdca1587]

Well, congratulations Primeagen for being of one of the lucky 10.000s recently

[@shashantr.9380]

Nice one. You could consider making short videos for such cli tools. It's great motivation for using them

[@VictorCaldo]

Been using this bad boy for a week and oh my God I've been missing out badly. More quick tutorials like these and how they work for you!!!

[@VictorCaldo]

I've been using nvim wrong too that !jq there opened my eyes

[@mauricioprado6395]

@theprimetime could you please describe what the hell were you doing before finding this out? were you like creating a new js app to filter things every time you needed to walk through data? or were you grepping and hopping?

[@bladeboles9167]

jq came in clutch today at work. Messy ass mongodb doo doo nightmare and it got everything filtered down by an obscure regex to a nice human readable form for troubleshooting. Thanks for the tip!

[@wille4986]

THANK YOU! I have been looking for something like this for the past 2-3 months to form a "lookup" tool against our ...struggling... database that houses non-sensitive document titles and tags; this will make life 100000% easier than figuring out per-user database queries!

[@NickSteffen]

Definitely great for parsing curl responses that give you a json token. I randomly came across this when trying to parse out an oauth token so I could stick it in a header for api testing

[@nicwhites]

It always warms my heart seeing devs learn more about the myriad of cli tooling available. Just wait till you see parallel

[@ristomatti]

It's been on some vid but just not featured as the main topic. He seemed to appreciate it.

[@not_a_human_being]

using cli to visualise data in 2024... still... I didn't think we'll be here 10 years ago.

[@IulianMandacheYT]

You should also check yq (same as jq but for YAML) and tomnomnom/gron (it flattens the json keys so it's easier to find a key hierarchy and to grep the data).

[@CielMC]

What about NuShell with the nu though? What are your thoughts on that?

[@wtfdoiputhere]

i understand and agree with your excitement it's a God sent jewel to make my bash scripts less depressing

[@kenamreemas3295]

this video is going to change my life in more ways than prime can imagine.

[@maximilianmoser2340]

it goes very well with jless!

[@ScarabaeusSacer435]

This is why I love the comments section. Now I have a new tool to pair with jq and jet.

[@FekuEntertainmentLtd]

4:43 - 4:58 Poor arch users. Prime went too harsh on them. 😅😅

[@mdjpurdon]

jq is life, it makes the AWS cli tool actually usable

[@dandogamer]

Hahah yeah been there before , the aws cli is a bit shit

[@itmeurdad]

Used jq with aws cli for years, got super comfortable with it but then took an azure gig and now their garbage json query language has broken my understanding of the syntax for both.

[@jonesherbert]

Yes, but have you tried visidata and nushell?

[@Gunzy83]

That opening was like poetry. jq is a vital tool.

[@pieterrossouw8596]

I just always use jq for parsing out data from AWS secrets manager in a bash script... Used it for years but nice to see it can do way more.

[@Jasonlhy]

I use dataweave which works for many things like XML CSV JSON, the only down side is it is not open source and I don’t think it is actively maintained too

[@jgoemat]

Might have to check that out. Usually I just paste json into a browser console and use javascript. Handy to see it in a text editor though.

[@MythiPlayz]

I've been looking into jq at work, we have massive json files that we need to have search queries run on (35MB shite), problem is I need to wrap jq since I am using java :(

[@trejohnson7677]

lol fucker just install it and ask chatgpt to teach u sbout it over the weekend

[@lcarsos]

btw, there's also yq if you happen to have a whole bunch of yaml that you want to parse through with the same syntax.

[@BrainFunnel]

jq's the bomb, I've been using it for years - it's super helpful for processing kubectl -ojson output. As others have noted, yq is equally great when you need to process yaml...

[@xyxytx]

have been using this for years. handy tool

[@justinrogo1415]

I actually hate the way jq is a whole new language and is pretty much essential today with how widespread (and useful) JSON is. I actually started using powershell on mac because its support for objects and converting from json is so much better.

[@radikalZen]

4:54, you dont have to attack me like that . That personal bruh...

[@GertjanAssies]

Cool, using it often, btw did you know there's also a yq (same thing for yaml)

[@Goshified]

This seems way better than my current tactic of opening a browser inspector with the console open and using a bunch of manipulation with JS to spit out what I want. Going to download it now, thanks!

[@LucasSilvaChaves]

Mixing it with fzf makes even more awesome

[@pylotlight]

I use jq too infrequently to remember the syntax despite using it for years.

[@rosendo3219]

hmmm also noticed @5:40 a drinking jar...shouting out to Cewpins? :)

[@Twixmonster2]

Hey Prime, how come you use the web interface of ChatGPT? It takes no time to write a CLI tool that uses the OpenAI API, and it's a huge boost for productivity. I wrote mine in python, and I use it all the time in Emacs. By specifying a custom system prompt, my GPT writes directly in org-mode format, allowing me to do things like executing code blocks in a conversation buffer. I can't overstate how powerful this is, but maybe NeoVim just can't handle such power?

[@hypnogri5457]

The GPT-4 api is too expensive for me so I settled on the deepseek coder model with function execution

[@lbanca0113]

Did he try nushell? Basically does this to literally everything

[@jdal21]

been using nushell since early january for admin work k8s and its the best experience ive had since Im also a data engineer working with python, pandas/polars, sql

[@thisbridgehascables]

You can even pipe a curl to a rest api into jq that returns json.. awesome! Great for testing an api from the terminal.. thanks for sharing! If I could add jq to a bash script that’s part of a cron, you could do some interesting data extraction and save it to a new json file to be evaluated or transferred into a folder that an application actually can read or display through a web interface.

[@davidwhitaker6489]

Sick bro squeal for jezson 😂 🎉🎉

[@animanaut]

i use yq/jq to flatten out a file into a properties like format and pipe it through fzf for easy querys on the fly

[@ever-modern]

I have never heard of Jake Weary till today... when I watched the retelling of "It follows". And that very hour Prime mentions him. Damn, these coinncidences are to stop me being atheist one day

[@rosendo3219]

what is your setup on the terminal? i see at the bar below `1:nvim- 2:zsh*` some fancy stuff. can you scroll up your history with J, K buttons? ctrl + U ? what other fancy stuff you do with your terminal and how to set those up? make a tutorial please like you do for vim

[@darthcabs]

So this is what it actually feels to see Primeagen talk about something that you know what he's talking about?

[@uuu12343]

This video didnt just teach me more jq than I already have been using, you taught me more about (neo)vim as well LEGEND

[@tasonosenshikitsune1165]

I'm about to blow your mind. There is a different tool called yq. It does the same thing, but YAML. And JSON is a subset of YAML. It does both.

[@obviliontsk]

There is also xq for XAML.

[@a_maxed_out_handle_of_30_chars]

i thought it was just a formatrer 😶 so this tool is same as awk meaning it can do more than just print fields, will practise it more

[@steamer2k319]

I've used jq occasionally for years. Unfortunately, the syntax is not intuitive and I always spend like an hour trying to remember it's idiosyncracies. I hope someone does a rewrite with better porcelain at some point.

[@teejaded]

Still better than jsonpath /spit Way better than xpath /spit

[@steamer2k319]

@@teejaded If you're already working with XML, xpath really doesn't make it any worse.

[@fearmear]

I wish I had it yesterday when I had to remove some playlists from a massive Spotify backup json file.

[@GrishaKirilin]

Wait until Primeagen discovers "jc", a dark brother of "jq", then it completely changes the interaction with the output of any core linux/bsd utility.

[@chefaku]

9:20 "Tech humor: jq json jokes"

[@technolung]

*notification for new prime video* Speaking of getting excited for a tool

[@_PCode_]

Every time I get overly excited about some tool, I look at my wife with this look on my face that says 'I really need to tell you about how cool this tool is I found' and she Immediately knows I'm about to nerd out... 😂

[@daltonyon]

Jq in terminal is awesome, but jq with vim is incredible!! Installing now.... thanks Prime!!! ~ This video is 101% useful!! ~

[@josecapurro]

jq saved me many many times! The ability to select things is amazing!

[@sjfsr]

OMG, I wish I knew about this before. Nice tool

[@klamberext]

For those who don't know, then there are substitutes for yaml and xml also: yq and xq

[@Sluhrmz]

JQ? Whoa buddy, I thought this was a programming channel.

[@bigben8502]

lmao

[@rkdeshdeepak4131]

Lol

[@TheCardil]

You just pitched JQ to your wifey?! 😂 A legend ⭐

[@sohn7767]

I knew jq for a longtime just never played around with it this much

[@jearsh]

knew about jq...but always struggled with the syntax. thx homie

[@bryanenglish7841]

jq is pretty great, between that and jless I can view all my structured data files and manipulate them easily. Nice vid!

[@stannylou1636]

I’ve been watching you for the last couple of weeks and I love your energy first and foremost. What is your favorite language?

[@giriw206]

This is a cool tool, I didn't know it existed. I usually just write some basic python code to parse json files. But this seems much faster and easier.

[@flipperiflop]

My jq-fu is so weak - all these things that you could do with it makes me feel like a toddler trying to fit the square block through the round hole.

[@xXWelhoXx]

Never knew that Jquery had such an awesome command line interface!

[@samglassman1217]

🎯 Key Takeaways for quick navigation: 💻 The *video introduces JQ, a powerful tool for JSON query manipulation, highlighting its extensive utility for text processing.* 🗣️ The *creator's excitement for JQ is humorously contrasted with his wife's indifference, emphasizing the niche appeal of text manipulation tools.* 🛠️ JQ *is praised for its ability to handle complex tasks like sorting, compacting JSON files, and extracting specific data elements, showcasing its versatility.* 🎭 A *playful comparison is made between JQ and actor Jake Gyllenhaal to clarify pronunciation and avoid confusion.* 🧩 Demonstrates *creating custom JSON objects with JQ, illustrating the tool's flexibility in data restructuring.* 🕵️‍♂️ Highlights *JQ's capability to filter JSON data based on specific conditions, such as non-null values or arithmetic operations, enhancing data analysis.* 📊 The *video showcases how JQ can significantly reduce data sets by applying custom filters, demonstrating its efficiency in managing large JSON files.* 🏗️ Integrates *JQ with Vim, showing how to dynamically manipulate and query JSON within a text editor, exemplifying JQ's integration with other software tools.* Made with HARPA AI

[@StudioGuitarra]

i've been using also yq that's the equivalent for yaml, as a k8s heavy user is the best never happened to me in terms of file debuging

[@Frexuz]

its weird how his file is not an array of objects, just a line by line thingie. for use with actual json files, you have to use jq's map(), like jq 'map({name: .name})'