How to Write Python Scripts to Analyze JSON APIs and Sort Results

[@coreyms]

We're covering a lot in this video. We're going to see how to request data from a JSON API, parse out the information we want, sort the data using a custom key, and a lot more. We're going to be using the Homebrew API, but you don't need to use Homebrew or a Mac to follow along. This can apply to many other APIs you'd like to analyze with Python. As I mention in the video, our final result _does_ have a little more functionality than what we can get from their existing API, so I believe it was definitely a useful exercise. For example, we can filter popular packages by a keyword in the description, which is a nice edition. Hope you all find this helpful!

[@Violet_Patrick]

I hope you make some videos on API development and talk about some of the fundamental concepts concerning API construction and design.

[@Violet_Patrick]

Corey, can you talk a little about naming conventions? Especially where things get a little more complex?

[@ammarhaider1530]

Can you make a video on Django REST framework?

[@nghuwei]

I just watched this while I’m Still learning on how to play with the data that I got from api call. Your video is so easy to follow along. I hope that you continue to create more contents on python .

[@kevindrew8347]

I know im asking randomly but does someone know a way to get back into an Instagram account? I was stupid lost the account password. I would love any assistance you can offer me.

[@arthurcaesar2200]

These are simply the best python coding tutorials on the world wide web anywhere. What a lifesaver.

[@Lamb4139]

Corey is the unsung hero of Python KZbinrs :)

[@hassamdaudi]

I'm a man of limit means... but YOU, you will be the first creator I support on Patreon. You are offering actual education which is helping me in my career. For that you deserve the respect and MAH MONEZ!!!!

[@djlatino101]

where was this video 3 years ago when i had to teach myself this??! better late than never. this is a massive help to future in the making python programmers

[@mr_jacklu8014]

extremely helpful. as always Corey explains everything in the cleanest manner possible. i think it's because Corey is Midwestern? Would you say that's the reason for your simplicity and clarity? I grew up in Indiana so that's how I feel about Mid-westerners.

[@jereziah]

These real world scripts you do are simply the best to see how “it all comes together”.

[@iChrisBirch]

This was one of your best videos to date. I love the longer format using a real world project that can be applied to many other tasks. I hope enough people are donating and supporting you to keep up this work.

[@jingyuchang1885]

No regrets to sponsor your channel at all! Good job Corey!! You videos really helped me a lot!!

[@benwalsh2825]

I really appreciate the courtesy you show other websites. I'm sure Homebrew was whacked last night with a bunch of JSON get requests--I know I tried it out--but I made sure to sleep each request between calls like you said. I am sure there's a lot of outside work that goes into making a great video--planning, writing, rendering, etc.--and I really appreciate the effort you put forth to produce such great content. Thanks so much for taking the time to put this video together.

[@coreyms]

Thanks! Yeah, I’m always torn on putting together videos like this because I don’t want to inadvertently hammer their system with a lot of people trying at the same time. That’s why I used my personal website for the web scraping tutorials instead of someone else’s. I believe their JSON files are static and aren’t making actual database calls, so I believe these were pretty lightweight requests. So hopefully they didn’t notice too much of a load.

[@Violet_Patrick]

I've been trying to get a job as a python developer and haven't had much luck yet. I tried messing around quite a bit with Django but after awhile realized that i was just covering material and not actually learning it. Now i watch your videos for joy and for your passion with programming, and its kept me going. Thanks Corey.

[@Violet_Patrick]

​@@GlennChambers Do you feel Java could complement Python experience? I was thinking of picking up JavaScript since most of my projects so far have been to do with Django and web development. Although, i very much enjoy scripting like this above video and 'server-side' logic.

[@EUU100]

@@Violet_Patrick have you been able to get a job doing python?

[@raymondmichael4987]

I just had to drop a little comment, to thank you. This is a massive knowledge and well taught. Stay safe Greetings from Tanzania 🇹🇿

[@백영래-u3x]

Your tutorial just made my day! It has been on my mind during several days how to get JSON files through API's by Python scripts but I could not find the right video for me on KZbin. Good job, Corey! Have a nice day!

[@archstampton5910]

A python mini-project : exactly what I needed. I will probably re-watch the video a couple of times, as there is a lot to digest. Thank you Corey

[@morethanprogrammer7594]

This is a great example of things you can do to practice Python or to have something on your CV. Thanks!

[@mahammadsameer9764]

Your literally the savior man 🙏🙏🙏

[@abhishekvasisht7241]

you are a lifesaver, your python tutorials are the best. You saved my day.

[@henryreith7721]

Huge thank you for this tutorial. I’ve been writing in PHP for years but more and more of the api’s I use have native Python SDK’s so makes sense to write in Python. This was very insightful to see where python is similar to PHP and where you can take a different route to get the same result. Really cool.

[@pplanetEarth]

Great video, thanks a lot for sharing, it answered my question that I was Googling. I also worked out how to write 30d-90d and 365d as separate columns, I resorted to list comprehension, there might be a better way. Looked through the comments but didn't see anyone mentioning this. On that note: I see a lot of comments saying like: I would be keen to see how you could bla bla, I think comments here should focus on the content of this video, rather than asking for various imaginations and new assignments, please!

[@davydwes]

thank you so much dude... i was 5 hours in a row searching for one of the solutions that you teach in this video.

[@CainanParker]

I would have to say you’re video tutorials are so easy to follow, I understand what I’m being taught and you explain each point so well!

[@gsb22]

You constantly reminding us not to DDoS homebrew tells me something about you, that you are considerate. :)

[@Neonlightxyz]

the way you explained is beautiful and awesome

[@v90cross2]

This is a really great video describing every single step required for JSON data process - Thanks a lot!

[@linkflap1342]

Just discovered your channel and i liked videos with practical approach. Thanks for your knowledge sharing and please do not stop creating more materials.

[@emylincon]

you are a legend out here!! I always learn something new watching your tutorials

[@ibnbattuta1304]

Thanks for the training! Just downloaded 4997 records today in 39 mins. No mods needed to your scripts.

[@sim9955]

did you apply his code to something totally different and it just worked?

[@letsberealq]

Amazing work. Clear explanations on whats going on . Thank you sir!

[@ericlambert7046]

Amazing skill as a teacher. Thank you - keep up the great work.

[@australianpopulism3490]

Your tutorial is really easy to follow and I learned a lot from you. Thank you.

[@ashwinmandaokar3620]

the superb explanation, it solves most of my problem, and thank you, keep educating us.

[@justin3594]

one of my all-time favorite videos.

[@mishrarp]

Amazing video on python. So much of stuff in a single video

[@anandmaurya3389]

Great video bro....it solved some of problem of my project

[@yeboahdominic1896]

Where have you been my mentor is been a longtime I have been waiting for you. By the way thanks for coming back.

[@okmidnight1716]

Thanks a lot sir..your videos always helped me to learn as a programmer...appreciate it.

[@muratsokucu5891]

You are the man! God bless you.

[@woody5012]

Really great video, so many potential use cases for this!!

[@prakash1171]

Awesome..... we really need that type of python script

[@akbarahmadi5066]

great teaching, great work

[@Dave-do3pp]

This is awesome. You are fantastic. Thank you

[@stevehageman6785]

Wonderful tutorials, thank you.

[@Bubbalubagus]

This was a great video, it really helped me. Thank you, truly, for taking the time to put this together - it is easy to analog this to other projects

[@aabidsofi19]

Really corey your tutorials are different they make my day.💕

[@anneest]

This was great Corey, thanks a lot for that! :-)

[@johanneszwilling]

Thank you so much! Best one on this topic I found!

[@nghiepcrypto7034]

I saw all video ads. Feel nice!

[@mdsegara101]

i've been watching this video for the 3rd times, just for a refresher, then come up to my mind that wonder if you could do a tutorial video to build that homebrew api, please.

[@ogoshi]

Thank you for your awesome videos, sir 👍

[@joeyzalman8254]

this is an amazing tutorial

[@prabhathkota107]

Response elapsed time & pref_counter was something new I learnt👌👏

[@indroindia]

Great video !

[@khaled-dz8357]

Hero as usual

[@TheKlajmaS]

awesome tutorial

[@muhammaddilshadkhaliq540]

love you keep it up sir......

[@fattanehameri448]

Thank you Corey!

[@a_maxed_out_handle_of_30_chars]

best spent 48:42 minutes of my life :)

[@fatbench]

22:48 - another option here instead of killing your program with ctrl+c would be to limit the for loop explicitly with an upper bound on the index: for package in packages_json[:5]: Sometimes these loops can bog down slower hardware and create lag before the program is killed. Or you can accidentally click outside the terminal and ctrl+c won't kill it until you click back into it. The above method removes all doubt about when the loop will terminate.

[@pplanetEarth]

Good point, I used a counter, limited it to 20 for testing

[@xionarxfl]

finaly some good fkn explainations

[@joiseystud]

47:10 great video as always. Can someone tell me why you don’t have to pass an argument to the install_sort() function or use parentheses for that matter?

[@mehdismaeili3743]

Excellent.

[@Nearco1000]

@Corey, deeps thanks , keep it up.

[@Jigyasu_RP]

Great work sir👍 Please make video on telegram bot using Python.

[@abednegonganga4927]

Hey Corey, I'm a great fan of your tutorials. I had a request though, would you probably do a series on ReactJS?

[@cylonscion5470]

Great video. One suggestion I have is to also say something about error handling in the script. For instance, what if response.get() had thrown an error for some reason (e.g. connection timed out). How should one deal with that in a script like this? How should one go about figuring out how to property specify a try: and except:?

[@coreyms]

That’s a great point. That’s an easy thing to forget.

[@rajath1964]

@@coreyms probably something for users to try out and learn about exception handling in python. An extended mini-work.

[@nurshah816]

You can always check the status code =200, with try catch block.

[@danielparada8736]

If you use the input() function at the end of the for loop the code will run one iteration at a time, waiting for you to hit enter before running the next iteration. It's usually how I do sanity checks on my loops.

[@briandreilly]

Excellent video!! I would be interested to see how a script like this would store financial information or data that changes periodically. Like timestamping, storing each json file individually, retrieving, and comparing them. Would you use a database to store the info instead of multiple json files? Many thanks.

[@gaylonalfano]

Amazing as always, Corey! Just curious how you would modify this to break up the request into a few parts, rather than making all requests and then storing the data at the end. Reason I ask is for slower connections if it errors out in the middle of then you lose everything that's been retrieved. My thinking is to use range() to break up the loop into smaller parts and then changing the subsequent writes to our package_info.json file to append new package data (rather than overwriting). Sound sensible? Anything I might be missing? Thanks as always for the amazing content!

[@drygordspellweaver8761]

Could even do threading

[@FIBONACCIVEGA]

I love your video. I'm trying to convert a json file into table format to be able to work more easily on it in python. It's just easier to understand and visualize for me in table format.but i can't get it. Any recommendation? I'll watch your video now about Sorting Lists, Tuples, and Objects.Thank you so much. It is a very useful video.

[@memeremor]

Hi Corey. Great video, again! Do you usually post the code on github or somewhere for further reference? While following along your code I was doing work on some other project using similar patterns.

[@kentkramer3866]

do u have to join the homebrew page for ssl access?

[@chikosan99]

Thanks a lot !

[@chrisbebek3192]

Corey what's the best way to approach this: Getting a certain key value when there's a regex match on another key value? Any suitable videos?

[@ejazshaikh1732]

Awesome tutorial, just one suggestion, you can use chrome extension for formatting and beautifying JSON in browser.

[@coreyms]

Nice. I’ll check that out. Thanks!

[@mmurray4043]

If you want to take over the world with python, i’ve got a video link in the description.... awesome video

[@GxDonaven]

You might possibly get a key error for the 'path-extractor' package since under the analytics section the name of the package is "path-extractor --HEAD" not 'path-extractor'.

[@lardosian]

Hey Corey, any plans to do something like creating our own api with Django Rest Framework, would be great to see how you would approach it, just started using it myself to render charts with Chart.js based on the data in the database and it was very quick to get off the ground. Used your django project as the base for my project so thanks a mil for that. I'm sure I will make some mistakes along the way so would love to see how you do it.

[@Shuubham_Harilal_Saroj]

Your videos are awesome. I had one issue with requests package, How can i fix the issue of SSL error on https requests without using verify=False

[@ahmedabushama4024]

God bless u …🙏🏽

[@ahmedabushama4024]

Am just wonder if u could make series in neural network with python…?

[@TheL337trance]

I have a couple Json dists and lists I need parsing through. You want to do a little freelance work?

[@frankconte2457]

Thanks Corey for another great video. I am following your tutorial using my Jupyter notebook. Unfortunately, the Jupyter Notebook server I am on stops sending out output (which is not to crash homebrew server). Can we simply download the file and then read it into your script? Please advise.

[@maidenfanusa]

when I include the break statement, it outputs the json file, but without the break, it doesn't output the json file. Why would that occur?

[@juanpablomtbxc]

Thanks 😃

[@gauravmishra44]

Respect!!!

[@daminilanje5523]

Hey i have one json file as a request body for an api..if suppose i have multiple json objects in that json file...for every object it finds in the json file the request should occur...how do we do this ?

[@JollyAZ]

How you dont have 1 million subscribers is just ridiculous

[@strange_facts_Of_the_world]

i need some help : how to comment out multiple python lines at once in sublime text ?

[@Nightizm]

What font are you using in your editor?

[@coreyms]

Source Code Pro

[@arwahsapi]

Could you please make a tutorial on how to combine/merge two (or more) sorted json files?

[@gerrelatanacio]

Hi, are you familiar with Tavern API? I hope you can make a tutorial about it.

[@GovindSingh08]

Hi Sir, I want to learn Python Django with Rest Api. Kindly suggest me your videos from where I should start learning.

[@mihaidinca2062]

I like very much your videos and i learned a lot from your videos. I am a little confused. how can this lib in my project ? if i can make a request to other site it does not work. now i am scapring a site and i dont figure out how can i format my data into a json format.

[@EUU100]

thanks for the video, really helpful! what do i do if the json list i have has certain keys missing sometimes. My loop worked twice but the stopped and gave an error KeyError: 'price' since the price is missing from the info for that item I am trying to take data from an Etsy API, do you have any advice on how i could get around this? i tried to put 'if price not in list: continue' in my loop but not sure if that is possible or if i have laid it out incorrectly. Sorry i am still new to Python, Thanks! any help would be appreciated

[@amshaprabhu9065]

Hi Sir, In line #13, - "install_sort" is being called and executed. As it takes one argument - "package", where it is being passed to it ? in line #13... I thought it will throw error, but it runs. Please respond.

[@mahalakshmi-fp9ze]

Hi!! Is it possible to create API using TKinter GUI?!? TIA!

[@v90cross2]

Is there any further instruction for how to visualize the collected data here?

[@coreyms]

Not with this specific data, but if you watch my Matplotlib series then you can apply that to this data to get some cool visuals

[@anthonysimonyan4763]

Hi there @Corey Schafer I have a question if for example you have two different json websites containing information information about the same product like one site contains name the other price after scraping how can I put them next to each other row by row?

[@silkogelman]

At 34:58 if you don't want to wait 25-30 minutes for the 1000s of requests to finish you can also do this: limit the for loop to 10, 100 of 1000 loops by truncating/slicing the packages list like the list at 46:25. Simply add [:10] or [:100] or [:1000] to the for loop code. for package in packages_json: then becomes for package in packages_json[:100]: You'll end up with the details of 100 packages in your package_info.json file.

[@drygordspellweaver8761]

You could also run multiple threads

[@PSY_JD]

Could someone explain to me how the sort function works because to me it looks like it's just an empty function and not even given any information?

[@jorgensenathomas]

got an error "#KeyError: 'path-extractor'" - tried for 5/10 min to find if google had a quick answer but no - made a try except. Great videos - keep it up :)

[@akashverma4197]

I am getting the same type of error. Please tell me how did you resolve this?

[@akashverma4197]

I am getting the error as "KeyError: 'residualvm'"

[@GordonAitchJay]

It sounds like the JSON the server is giving you doesn't always have a "path-extractor" key. Some of the items in the list might have it, but not all. Likewise with the "residualvm" key. You can handle this a couple ways. One simple way is to use the *get* method on the dict. So instead of something like json_dict["path-extractor"], you'd do json_dict.get("path-extractor", "No path given").