Merging JSON documents in the command-line - jq tutorial

[@J3rs3yM1k3]

I would love to see more of these.

[@szymonstepniak]

Challenge accepted! :) Take care and have a good day!

[@gitgosc7075]

wspaniała seria o jq, najwyższa jakość! Ogladanie tego to czysta przyjemność - chętnie zobaczyłbym podobna serię na temat narzędzia AWK. Doceniam ten ogrom pracy - nawet gdyby oprawa video była na niższym poziomie to dalej byłby topowy materiał ze względu na treść i przekaz!

[@szymonstepniak]

Dziękuję za ciepłe słowa, @GitGość! I dzięki za sugestię dotyczącą AWK - nie czuję się ekspertem w tym temacie, ale może przy okazji opracowywania materiału video udałoby mi się trochę podszlifować AWK-a :) Trzymaj się, miłego dnia!

[@obiwankenobe3962]

Szacun za super prezentacje (mam na mysli wszystkie odciniki). Swietnie intro do jq, dzieki!

[@szymonstepniak]

Dzięki wielkie za ciepłe słowa! Trzymaj się, miłego dnia!

[@horizonlimit89]

this is next level jq stuff! Ming Boggling :)

[@szymonstepniak]

Thanks for your kind words, Shitiz! Have a good day!

[@edwardishaq]

Wow, I never knew about all this stuff. Amazing JQ skills

[@szymonstepniak]

Thanks!

[@kyoungjinkim6394]

Great... thank you so much ~~~

[@szymonstepniak]

Welcome 😊

[@joanestradaramos1541]

AMAZING

[@szymonstepniak]

Thank you, Joan! Take care, and have a good day!

[@conor7083]

потрясающе! спасибо!

[@ering2898]

very helpful ! want to see more examples

[@szymonstepniak]

Thanks Erin! Have a good day!

[@AllanKobelansky]

Very good production value and content. Would you consider pausing for a few more seconds after you type the last character of a command? The transition from completing a command to hitting return is about 2-3 milliseconds. Also, would you consider putting the commands and the files in the comment section? I've subscribed.

[@szymonstepniak]

Thanks for your kind words, Allan! You're right - making pause for a few seconds before hitting the enter key would improve the user experience. I also plan to supplement upcoming videos with dedicated GitHub repo with all code samples and extra information. The video format has one significant downside - once it published, it cannot be edited. The external repo can help with keeping the information up-to-date, and it allows to add some extra information in future when needed. Thanks a lot for the very valuable input! Take care, and have a good day!

[@delio_pl]

Nice video. I have learnt much from this. I wonder if jq could be used to merge 2 objects that have complex type fields on the second level, third level and so on. Your solution seems to recognize field type (array/object) only on the first level. It may be my proposal of topic for the next part of this course ;) Thank You.

[@delio_pl]

I needed this solution to merge 'bookmarks' from my two browsers. Here is my code written in JavaScript pastebin.com/0zngrjqr

[@szymonstepniak]

That's an excellent point, thanks! The example I show in the video can be extended to support deep merge behavior by using a recursive function. Take a look at this example: gist.githubusercontent.com/wololock/238d7afe37808eaa3252cc09a4421b0f/raw/885d4a02b6e999167a19cbc7757cebb79b6f33e0/deepmerge.sh Here I extracted a function "f" that handles reducing a list of objects into a single object by merging key-value pairs. Whenever it spots that the type of a value on the right side is "object", it calls itself recursively on a list constructed from the .[$key] (left-side value) and the $val (right-side value). It's not very sophisticated, and it assumes that the same keys store the same type of value, so if you run into a problem like a["foo"] is an object but b["foo"] is an array, you may have to extend the if-conditions to take both types into account to handle it correctly. Hope it helps. Have a good day!

[@silent-ghost]

Hi Szymon, thank you so much for your guides. I was wondering if you could continue your JQ playlist and maybe include data explaration queries like max size of properties for example. So many times I had Azure fail due to values exceeding the target defined size but I had no ideea what column to adjust. Thank you so much!

[@szymonstepniak]

Thank you for your kind words, Vlad! And thanks for sharing ideas for more JQ related videos - I have a few in mind that I plan to record soon. I think adding a few more short and useful videos explaining some day-to-day tasks would be useful for the community. Stay tuned, and have a good day!

[@dayasagareddya.c.9547]

Impressive stuff it saves writing alot of code.observation the final query was working fine if I have an object 1 level deeper if it's at 2 level deeper merge was not working as expected.

[@szymonstepniak]

Yes, this is shallow merge. I explained recursive merge in this blog post - e.printstacktrace.blog/merging-json-files-recursively-in-the-command-line/ :)

[@dayasagareddya.c.9547]

@@szymonstepniak works like a magic..thank you.

[@aceninopis-an5336]

Hello,may i ask on what application are you using in the video?, I'm still new to these things

[@szymonstepniak]

I use Linux + ZSH shell + oh-my-zsh + Guake (terminal). Take care, and have a good day!

[@sudosnoop]

+1 👍

[@TJ-hs1qm]

It would be nice to also tag the source of a particular attribute in the final output ... subbed 🙂 Have looked into CRDTs ?

[@szymonstepniak]

Thanks for the suggestion! 👍 Nope, I haven't looked at CRDTs. Is it related to the video? Take care, and have a good day!

[@riteshm7466]

Szymon, Is there a way where we can use jq to compare 2 json files (only particular key:value) and give the diff of that?

[@szymonstepniak]

I'm not sure if you can use jq to compare two JSON files, but there is a way to use it in combination with the diff command. I can imagine that you have two JSON files that have a different order of fields you want to compare. I would do it in 3 steps: 1. Use jq to extract fields you want to compare in a specific order from the first JSON file and put the result in a new file (e.g. file_1.json) 2. Use jq to extract fields you want to compare in a specific order from the second JSON file and put the result in a new file (e.g. file_2.json) 3. Check the diff between the two files (e.g. diff file_1.json file_2.json) There might be an other way to do a similar thing, but I think this one is simple and straightforward. Good luck, and have a good day!

[@soundofmusic94]

hi man, thanks for your video. Question , how can I do this for a N number of json files, and inside every file is an array of objects?

[@szymonstepniak]

You should be able to do something like that based on this example. You can pass multiple JSON files as an input to the jq command (e.g. file-*.json pattern can match files like file-01.json, file-02.json etc.) As for the JSON document containing an array of objects - I don't have an out-of-the-box example, but I guess you can experiment a bit and come up with a good solution. Good luck, and have a good day!

[@kashyapkiran1729]

nice video. can you make some videos of make file

[@szymonstepniak]

Thanks for the comment, Kashyap! Could you give me some more context? Do you mean a video on Makefile? Thanks in advance and have a good day!

[@kashyapkiran1729]

@@szymonstepniak yes Makefile

[@szymonstepniak]

@@kashyapkiran1729 Hmm, I don't say "No", but I can't say "Yes" either, because I don't use Makefiles at all. If something changes in this area and I will find creating a video about helpful, I will definitely do it. Thanks for the suggestion, Kashyap! Have a good day!

[@xettabyte1]

pretty much amazing video sir, where can I send a Dm?

[@szymonstepniak]

Hi Ariful Anik, what type of question would you like to ask?

[@xettabyte1]

@@szymonstepniak about merging json file sir

[@szymonstepniak]

@@xettabyte1 Feel free to ask it here. If I won't be able to help you, maybe someone else will.

[@JohnOlat]

This was helpful, I have two json file Product and Prices which I got from post man. How do I merge them? The both have same Product: Bag SimpleCode#10290 Price: $100 SimpleCode#10290 How do I merge?

[@cya3mdirl158]

No to use on production

[@szymonstepniak]

Thanks for your comment, CamiL. Have a good day!

[@salazar2287]

too long

[@szymonstepniak]

Thanks for your feedback! I hope you could still learn something useful from that video. Take care, and have a good day!

[@conor7083]

how i can merge only two arrays. f.e. prices and tags? jq -s '.[0] as $a | .[1] as $b | $a * $b | .prices = $a.prices + $b.prices | .tags = $a.tags+ $b.tags' firstfile.json results.json - merge and another arrays.

[@conor7083]

jq -s 'reduce .[] as $item ({}; reduce ($item | keys_unsorted[]) as $key (.; $item[$key] as $val | ($val | type) as $type | .[$key] = if($key == "prices") or ($key == "tags") then (.[$key]+$val) else {} end ))' *.json and another objects merge too, but {}. how i can merge without {} ?!