Always Check for the Hidden API when Web Scraping

[@hectord.7107]

I've doing this for years as a self taught programmer, there are some little tricks you did here that i didn't know, thank you for the video.

[@JohnWatsonRooney]

Glad it was helpful!

[@sworatex1683]

It's my first year in programming and there was nothing new actually. I don't even think that pain was worth it I'd just make the scraper in js and make it return a json string.

[@sworatex1683]

But I guess that would be useless for bigger projects. I'd just do it in js if I want like an actual product list like this.

[@gesuchter]

@D R lol

[@mattshu]

@D R how do you “block” scraping??

[@Davidca12]

This single-handedly cut the running time of my program from literal hours to a couple of minutes, cannot thank you enough!

[@JohnWatsonRooney]

Brilliant, thanks!

[@abc.2924]

I've been using this trick for a while now, and I've learned it from you, so thanks. Amazing work man

[@JohnWatsonRooney]

That’s great 👍

[@leow.2162]

I'm not a very experience programmer, I've been doing it recreationally for like 2 years on and off but I did a lot of webscraping and this is just a really neat piece of knowledge that I wouldn't have come across own my own. Thanks.

[@zenon1903]

Please ignore my first comment. I checked out your first video in this series and learned about using scrapy shell to test each line of code. With that I found the bug in my code. The code worked PERFECTLY as advertised. Your the man! Much thanks!

[@freekdl6648]

I rarely praise anything, but this tutorial was SO good! Well explained, no filler. In 7 or 8 minutes you guided me through finding the hidden information I needed, which tools I need to use and how to automate it. This tutorial gave me enough confidence to try to write my first Python script! Within hours I built a scraper that can pull all metadata for a full NFT collection from a marketplace. Without this video it would have taken days/weeks to discover all of this

[@JohnWatsonRooney]

That awesome! thank you very kind!

[@channul4887]

"In 7 or 8 minutes" More like 11

[@freekdl6648]

@@channul4887 Nope! I had different goals so no need to follow the full tutorial

[@MelonPython]

I even added it in my playlist. Great video. Definetely starting to love API's more and more.

[@danielcardin9241]

Because of this video, I was able to start my own rockets and satellites company. In only four hours, I started the company, launched thousands of rockets, and now I have my own interplanetary wireless intranet from which I can control the entire galaxy! Thanks again!

[@transatlant1c]

Nice video. It’s worth noting as well that many APIs will pageinate, so rather than checking how many total results exist and manually iterating over them - you just check to see if the ‘next page url’ or equivalent key exists in the results and if so, get that too until it doesn’t exist anymore, merging/appending each time until the dataset is complete 👍

[@JohnWatsonRooney]

Yes you’re right thank you!

[@ianroberts6531]

In fact you can see at 05:33 that this particular API does just that - there's "nextPageURL" and "totalPages" at the end of the response JSON.

[@shivan2418]

This is, no joke, the most useful video I ever saw on KZbin!

[@tikendraw]

I just want you to never stop creating such informative video. For god sake.

[@angestellterderantifagmbh]

My man, this is EXCACTLY what I was looking for. Had to do some extra steps, but a little try-and-error and basic understanding of HTTP was enough to solve my problem. Thank You!

[@Kralnor]

This is a true gold nugget. Thanks for demonstrating how to easily view the request in Insomnia and auto-generate code!

[@drkskwlkr]

Loved everything about this video! Great delivery style, production quality and interesting topic for me. First time visitor to this channel and not a Python user (thanks, KZbin, for your weird but helpful predictive algorithms).

[@JohnWatsonRooney]

Thank you! I’m glad you enjoyed it

[@brockobama257]

bro, you're a game changer and i love you. if i ever see you in person ill offer to buy you a beer, or lunch, coffee whatever

[@gleysonoliveira802]

This video was the answer to my prayers! The next best option was to watch an one hour video and hope they would teach what you taught... In 10 minutes!!! 👏👏👏

[@JohnWatsonRooney]

Thank you glad it helped!!

[@anduamlaktadesse9284]

Hidden API is by far the easiest way to scrap a website!!!! Thanks bro!!! Big Clapping for you!!!!! I've followed all your procedures &finally i did it.

[@wp4297]

HUGE! I've been looking for this info for 2 days. 12 mins of your video better than anything else, by far. Thumbs up and thank you so much

[@JohnWatsonRooney]

Thank you !!

[@wp4297]

@@JohnWatsonRooney you saved me a lot of time. I'm new to the topic, next days I'll take a look at you channel

[@Jacob-gy9bg]

Wow, thanks for this excellent tutorial! I just spent all this time writing cumbersome Selenium code, when it turns out all the data I was looking for was already right there!

[@JohnWatsonRooney]

Great! That’s exactly what I was hoping to achieve with this video

[@fuad471]

really nice and helpful tips in an actual topic with a sight-pleasuring recording quality, thank you for your time and efforts.

[@ЦветанГергинов-с7ю]

There is always something new to learn. I’ve been spending hours to grind such an information by hand-writing the whole program to get my result ;D Thanks!

[@joshuakb2]

This video came into my feed just a couple days after I used exactly this method to collect some data from a website. Very good info! This is much easier than web scraping. Unfortunately, in my case, the data I could get out of the API was incomplete, and each item in the response contained a URL to a page with the rest of the info I needed, so I had to write some code to fetch each of those pages and scrape the info I needed. But much easier than having to scrape the initial list, as well.

[@JohnWatsonRooney]

Thanks! I’m glad it helped in some way. I often find that a combination of many different methods is needed to get the end result

[@ERol-du3rd]

Awesome advice, a lot of people skip checking the requests when building scrapers but it can save a lot of time when it works

[@michelramon5786]

I was like "hm, okay, yeah" to "HOLY SHIT, THATS THE DOPEST SHIT I'VE EVER SEEN" I'm starting to get into this niche and I intend to learn more Python and SQL (you know, Data Analysis stuff/jobs) and I'm doing a project to scrape NBA statistics but there are always some errors and it ends up taking a long time. BUT THIS IS GOLD CONTENT, KEEP IT UP

[@marlinhicks]

Been using python for a couple years now as a picked up language and I really appreciate getting to see how someone experienced approaches these problems

[@judgewooden]

I like how you regularly start sentences with 'you might think' assuming we are all idiots. I approve, glad smart people, like you, make time to explain to us plebs how the world works. Apprecated.

[@JohnWatsonRooney]

Hey, thanks. I do my best to explain things how I would have wanted to be taught

[@GLo-qc8rz]

OMG man, was searching for 3 hrs how to extract javascript data w/o complicated rendering and your vid gave a 3 second solution. thank you so much man

[@huisopperman]

Thanks for sharing! This has helped me a lot. After struggling for weeks with selenium, I was able to apply this technique fairly quickly, and am now using it as source to scrape ETF-composition data to feed directly into a PowerBI dataset. Much appreciated!

[@sheikhakbar2067]

I always come to your channel for these excellent time-saving tips and tricks! Thank you!

[@JohnWatsonRooney]

Glad you like them!

[@sajayagopi]

I was struggling with selinium to extract a table from javascript website. This video saved so much time. Thank you

[@BIO5Error]

Very, very interesting - I'm going to give this a go myself. Cheers for another great video John.

[@ThijmenCodes]

Nice video! Used a similar method to collect European Court of Human Rights case documents since there is no official API. Glad to see such methods gaining popularity online, it’s so useful!

[@mujeebishaque]

I love you, John. You're awesome! Thanks for being unique and producing quality content.

[@BOSS-AI-20]

This video is really amazing I learned web scraping from your videos thanks

[@vinny723]

Great tutorial. My screen scrapping job went from 4.5 hours to 8 minutes!!!!!

[@Oiympus]

nice tips, it's always fun to poke around and look at what data webpages are using

[@krahkyn]

This is such useful content that shows how much value experience gives - thank you for the straightforward and realistic tutorial!

[@mattimhiyasmith]

I have used the inspect with network method but wasn't aware of the copy as url method, thanks for that tip will save me a lot of time!

[@JeanOfmArc]

You have shown me the light. Thank you for stopping me from making more web scripts that load up web pages in browsers to click buttons.

[@JeanOfmArc]

I have tried this method, but sadly the site I am trying to scrape from returns "error": "invalid_client", "error_description": "The client credentials provided were invalid, the request is unauthorized." Am I out of luck?

[@glitchinLife]

Nice tutorial on scrapping, some tricks I have been using myself, and some others never heard of until now thx for sharing!!! Small adjustments if I may (please don't take this as criticism) I think you don't need to loop over each product to copy it to your res, you can use extend instead, also I think the header didn't change so you can take it out the loop over pages

[@eakerz5642]

Tnx :) Went from 1 hour scraping with Selenium to 1 minute just getting the JSONs.

[@Rob-ky1ob]

Instead of looping over the list and doing an append of each individual item, you can do list().extend(list()) which extends the list with the new list. The result of this is 1 list of dictionaries (basically an identical result to how you did it) but with less and cleaner code.

[@ScottCov]

John Great Video...Thanks for taking the time to do this!!!

[@klabauter-ev4ix]

That was incredibly helpful and exactly what I needed today. Your presentation is very clear. Thank you!

[@davida99]

Wow! I just found a gem of a channel! Love your content!

[@JohnWatsonRooney]

Thanks appreciated!!

[@phoenixflower1225]

This is seriously high level content right here

[@ExtraterrestrialSilhouette]

GREAT VIDEO!!!! i was having a difficult time looking up different elements and keeping an eye on DOM tree so this is such a game changer Thank you for sharing

[@aidanomalley8607]

Thank you, your videos has automated my job. All i need now is a AI cam of myself

[@rameshks5281]

Easy to understand and very neat & clean narration. Keep it up 🙂

[@JohnWatsonRooney]

Thanks a lot 😊

[@Josh_r7]

Thank you so much for this tutorial. It helped me a lot on my project. And i learn a lot of new things that i didnt know. Thank you!

[@muhammadrehan3030]

Thank you for such a wonderful videos. I learned a lot from you. BTW your video quality and background are always very beautiful.

[@JohnWatsonRooney]

thanks! it's nice of you to mention video quality and background, i do my best!

[@phoenixflower1225]

Thank you so much - this is so insightful and educational. Really helped me understand so many things in so little time.

[@unknownuser993]

Wow that ‘generate code’ feature is super useful. Thanks!

[@lucasmoratoaraujo8433]

Greetings from Brazil! Thank you! I just had to adjust some of the quote marks on the header (there were some 'chained' double quotes (like ""windows"")), making some of the header's strings be interpreted by python as code, not text. Just had to change inner double quotes for single quotes (e.g. "'windows'") and it worked perfectly!). Can't wait to try your other tutorials! Once more, thank your very much!

[@JohnWatsonRooney]

Hey! Thank you! I’m glad you managed to get it work

[@isaacmartinez442]

Wow I love this! I was able to do it!! I did have to adapt to my own situation but still. Thank you so much

[@milosZcr]

Great content, very useful now that I am learning about this subject. You earned a new sub here

[@irfanshaikh262]

John, you make scraping interesting and motivating simultaneously. Good that I found your channel P. S. I lost myself it at 0:10😂

[@pascal831]

Thanks John! You are a lifesaver sir!

[@Jiloh5]

It worked like charm! I really needed this. Thanks

[@ninjahkz4078]

Lol, I hadn't thought of a possibility to get an api like that until now haha ​​thanks a lot!

[@davidl3383]

brillant, i start to do that and it's very effective. Good chanel and good job. Thank you John

[@mrklopp1029]

Thank you for those videos. They're extremely helpful. Keep up the good work! 🙂

[@JohnWatsonRooney]

Glad you like them!

[@karthikshaindia]

Very nice... I did earlier the same on an another site, that was bit tricky. This is very straight forwarded site. Meanwhile , Insomnia reduced more works ;). Thanks for an another great video

[@JohnWatsonRooney]

That’s great. Yes I picked this on because it was very easy, I think it helps people understand the core concepts better

[@alexcalish]

Wow, I think this 1 tutorial will do the most to up level my scraping than I could have ever imagined. Bye bye selenium.

[@Алексей-ъ3л2х]

Thanks for the video! What is the name of the VSC theme you are using?

[@JohnWatsonRooney]

Thanks! It’s gruvbox material

[@bigstupidtree3771]

This has saved me hours today. Genuinely, thank you. 🙇‍♂️

[@JohnWatsonRooney]

That’s great, thank you for watching!

[@mattmarkus4868]

what browser is that? What are you using to allow you to right-click a response and do copy as curl cmd ? What tool are you referring to?

[@JohnWatsonRooney]

hey, this is firefox, right click the page and go to inspect, network tab, find the request and rightclick, copy as curl

[@voinywolnyprod3046]

Quite interesting! Thank you so much for showing such nice tricks, gonna get familiar with Insomnia.

[@Night_Sketching]

John, a specific video about how to scrape React website would be nice. It uses a mix of html and JSON data on pages...just an idea. Keep up the good work loving it.

[@Moiludi]

Thank you! It provided a new way of thinking at the issue of collecting data. 🙏

[@RatoCanguru_Lucas]

Man, this is gold. Thanks for sharing!

[@edgarasben]

This is amazing! So many things I didn’t know.

[@inspecteurbane5666]

Thanks a lot, very interesting video, i learned so many things that i didn't know. I will come back for sure!

[@uboxtech]

what to do if we got cors error?

[@JohnWatsonRooney]

make sure you copy all the headers from the browser and send with your request

[@uboxtech]

@@JohnWatsonRooney i was trying to do the same thing with instagram, a normal js fetch request, but i didn't get any response.

[@RomuloMagalhaesAutoTOPO]

Great. Please, which software do you use to get python code from API response, on time 2:14 of this video? Thank you very much.

[@JohnWatsonRooney]

It’s called Insomnia api client - you can also use postman too, they are very similar

[@RomuloMagalhaesAutoTOPO]

@@JohnWatsonRooney thank you very much.

[@im4485]

Hi John. Amazing content as always. Do you think I can skip learning scrappy for now? Can I do most of the scraping tasks just by using BS and request html?

[@JohnWatsonRooney]

Sure you can. If it works for you then carry on!

[@nadyamoscow2461]

Clear and helpful as usual. Thanks a lot!!

[@tobias2688]

Hi John, I loved the video so much that I had to join Patreon to subscribe there to you. Thanks!

[@JohnWatsonRooney]

Hey! Thank you very much!

[@lokeshchowdary7487]

Thank you for making this awesome tutorial.

[@techkhid4836]

Thank you soooo much.....I needed THIS!

[@bronxandbrenx]

My master in data extraction.

[@JSaretin]

Instead of looping over the results again, you can use res.extend(products)

[@JohnWatsonRooney]

Yes absolutely, thank you for sharing

[@codydabest]

This was nearly exactly my job back in 2014/2015 for a giant e-com shoe company. Was always nice when you'd come across a brand that included their inventory count in their API. But yes selenium/watir all day lol

[@JohnWatsonRooney]

I’m often quite surprised how much info you can easily get!

[@rayjenscode3501]

nice. I wonder if it can also check website monthly traffic?

[@alphaO27]

Great video, how about cases where the data of interest is not present in the XHR tab of network ?

[@JohnWatsonRooney]

check to see if its in plain html in the source, or if there's a script tag with a load of JSON data in. Failing that some sites hide the XHR requests and obfuscate the JavaScript code. it's possible to unravel this but its much more complicated!

[@ericbwertz]

Nice video -- perfect level of detail.

[@JohnWatsonRooney]

Thanks

[@Rapid1898]

Thanks for the amazing video - this is a great new method for me scraping data. One question regarding the regarding the formatting in the chrome-browser of the json-file. When i am clicking in chrome on the GET-request and saw the full json-information (around 1:27 in your video) - in my chrome the whole json-information in shown in one single line and not well formatted like in your video. Is there some chrome-option to change this? Thanks!

[@JohnWatsonRooney]

Hey! Sure I forgot to mention, it’s a chrome add on for viewing json, if you search the chrome extensions for json you’ll find a few to try. I think this one is “json formatter”

[@Rapid1898]

@@JohnWatsonRooney Tried it with the json formatter extension - but still all the data in one line in the response window

[@Rapid1898]

But at the end now big issue at all - simply copy the long string to vscode and format the json code

[@JohnWatsonRooney]

@@Rapid1898 i think i misunderstood, you in the dev tools (inspect element). have you got the "raw" button ticked?

[@giovannimilana6428]

Huge thanks this video was a game changer for me!

[@EmanueleCannizzaro-c5q]

John thank you for the videos. How do you deal when in the network tab xhr you have a graphql object not a Json one?

[@TheEkkas]

Such a nice vid, if there was a VPN add, I didn't even notice it!

[@techlabingenium3440]

Great video!! Is there other way to double check if this method is available in a web site or just the way you show? I'm new to this. Thanks!

[@JohnWatsonRooney]

Hey thanks you! It’s really just this way, you can also get the data by loading up a chrome browser with playwright but it’s only useful in really specific situations

[@techlabingenium3440]

@@JohnWatsonRooney thanks for your reply, I tried this with a few sites and it works perfect! I found one where it uses the POST method to obtain the json, tried to copy it in insomnia and postman but got a 403 forbidden error even tough I sent a get request with the required headers. this will only work when the site sends the get request, right?

[@elliotnyberg9332]

This is amazing and will help me alot. Thank you!!

[@tubelessHuma]

This is very useful trick John. 💖

[@lagu1ful]

thank you for the information that you have explained, this is very helpful for the research I am doing

[@lagu1ful]

thank you so much

[@randyallen8610]

Great content. Thanks for this video

[@im4485]

Hi John, I can't find your video where you used Postman for this. Do you still have that video on your channel?

[@AutomaticTrashCan]

What if the webpage hides the xhr? I have been trying to find the xhr for so long and just never find it

[@tnssajivasudevan1601]

Great information Sir, really helpful.

[@vladgryban8746]

what browser or extension are you using? your data is in very nice format, whereas mine is just unreadable json

[@JohnWatsonRooney]

Hey, I don’t think I’m using any extensions - I think this is just default chrome. I do use Firefox too though which I believe displays Json nicer

[@vladgryban8746]

@@JohnWatsonRooney thanks

[@jawadch8723]

This feels illegal. And I love it!

[@JohnWatsonRooney]

Haha

[@chadgray1745]

Thanks for this - and other - videos, John. Super helpful! Regarding the cookie expiring, can you suggest a way to use playwright to programmatically generate the cookie used on the API request? I am assuming that cookie isn’t the same as the cookie used for the request of the html but maybe that’s wrong?

[@joseluisdiaztorres825]

Thank you so much for the tutorial. I have a question, how to get a Authentication value that include the header, can I do automatically and without selenium? In this moment, I get it manually in the network tab, further, the authentication value expire after of a time.

[@JohnWatsonRooney]

I don't think you can no - what I do is load up the page once with selenium, grab all the headers and cookies and use them in subsequent requests using this method.