Web Scraping Made Easy Using this Method.

[@OfficialMazLi]

I learned this trick in one of your older videos, it saved me a lot of times, i even made a discord bot using a website api by doing this trick, pretty useful. Thanks for the great content sir.

[@Rbm726]

Thanks!

[@JohnWatsonRooney]

Thank you very much!

[@letsexplore7977]

You are providing very valuable materials to community thanks

[@JohnWatsonRooney]

Glad you think so!

[@AhmadRaza-rh3pg]

Hello John, I am doing web scraping of an e-commerce website using its backend APIs. When I check the browser's network tab, the API returns data, but when I use it in Python requests with the same cookies and headers, the server doesn't respond correctly and doesn't return data.

[@graczew]

Good stuff as always. Maybe you should do something from devops site of scraping. Maybe how to automate/schedule your scrapers on virtual machines etc?

[@JohnWatsonRooney]

thanks! yeah... thats a good idea, i'll see what I can come up with

[@graczew]

@@JohnWatsonRooney I'm currently try top use aws ec2 instance to run scrapy and it's works ok only thing witch don't like me is crontab but I use tmux and you can connect on any device at anytime 😉

[@paradix847]

I'm getting a 403 when calling the request URL :( Any idea how to go about it?

[@uygarsaroglu4233]

hello, I want to extract the data of the traders on the bingx site. But I can't get the data. Can you help please.

[@yogeshbhamre-q5l]

sir, i am from india,please could you tell me, why does not allowed some website fetch and xhr in network of inspect

[@AmodeusR]

Could you tell me the name of the theme you're using in your IDE? Thank you in advance

[@JohnWatsonRooney]

I think this one is called Kanagawa

[@brute505]

what is the point of scraping if they give you an api to use??

[@killianrak]

Hey I’m trying to scrap vinted API unfortunately there’s delay between the data is created and sent to me I have good connection so It’s not because of my network Some people success to bypass this delay but I don’t know how do they do Have anyone any idea of how can we bypass this delay ?

[@JohnWatsonRooney]

hey I haven't done vinted but a quick check shows it to be fairly standard, is it a delay in getting a response?

[@killianrak]

@@JohnWatsonRooney yes, when I’m scraping articles, I print the time from when they are created : this time is always between 30s and 2min, while some people get articles instantly

[@pypypy4228]

As always, excellent stuff. On 7:27 i would add a parameter indent=4

[@JohnWatsonRooney]

Thanks - yes good idea 👍

[@kidmen279]

What is your OS?

[@JohnWatsonRooney]

ubuntu with i3

[@123arskas]

Beautiful

[@ferilukmansyah249]

thanks for good content

[@alfakih7247]

Hey champ, please share more real world project

[@humansaremortal3803]

Have you tried scraping the dark web? I have a friend do some scrapping for his master thesis and god lord, everything is slow and it breaks. I don't know how to go around it.

[@JohnWatsonRooney]

no i haven't, it snot something i know a lot about im afraid