How To Scrape (almost) ANY Website with Python

[@JohnWatsonRooney]

Grab IPRoyal Proxies and get 50% off with code JWR50 at iproyal.club/JWR50

[@bakasenpaidesu]

Can you make a video about how to use it properly?

[@JohnWatsonRooney]

yes will do

[@bakasenpaidesu]

@@JohnWatsonRooney thank you

[@triott6497]

I signed up for IPRoyal after watching your videos but couldn't get the static proxies to work. It returned timeout error. I tried changing the network settings but could not solve the problem. Do you have any idea what could cause such issues? Thanks.

[@edboss36]

You are the webscrape master

[@devpala]

Hey, the video is really really helpful. Thank you very much for it! You are the go to channel for me whenever I wish to research on any topic related to web-scraping. You're doing a great job man! Also, in the end of the video you said that this is not your preferred method for scrapping infinite scroll dynamic websites. So which one is your preferred method, which is also scalable?

[@JohnWatsonRooney]

Thanks I’m glad you’re enjoying the content! The way I mentioned is by reverse engineering the sites backend api and making requests to it - I have a few videos on my channel that explain the basics of this idea!

[@devpala]

@@JohnWatsonRooney Oh alright. Yeah I’ve gone through those videos of yours and they indeed made my task a lot easier. So thanks for that too! XD

[@tomermolnar6927]

Brilliant! love your attitude, admire an out-of-the-box thinker! Keep up the good work buddy!

[@JohnWatsonRooney]

Thank you!

[@stewart5136]

Another great video! Thanks for showing both methods. 💯

[@JohnWatsonRooney]

Thanks mate!

[@abdelrhmanabbas7228]

You are a great tutor, and I suggest a video discussing and comparing all of these tools, why and when we could use them what is the best compo great work keep making tutorials

[@Drtsaga]

I watched the section between 4:30 and 5:00 (roughly) so many times. The off-by-one space there was extremely distracting as well as satisfying when fixed. Cheers

[@podcaste4437]

This is so timely for me @John, as I was literally building a scraper yesterday to scrape a website that used XHR. Top content! Additonally, would it be possible for you to share the java script "code" that was used in the PageMethod function?

[@JohnWatsonRooney]

Thanks for reminding me I've added it to the description now!

[@successroutines]

Great video John

[@successroutines]

I'm wondering if there is a way to use playwright with scrapy's shell? For me scrapy shell just seems to open the browser at the url and then block the scrapy shell from opening.

[@ishandandekar1808]

Need you to make a nvim setup video because thats cool af

[@giftcp82]

can you please do a video on your neovim configuration

[@TimCollins]

Excellent tutorial! Thank you

[@JohnWatsonRooney]

Glad you enjoyed it!

[@malwaredev33]

Great video content about webscrape. Your doing amazing bro.

[@StefanFlorescu-ur8uv]

Thanks for another great video! This method seems so easy and wanted to try it myself but unfortunately, it seems that scrapy-playwright doesn't work on windows. Some sort of Linux emulation (WSL) is required. Also thanks for the iproyal discount. I was looking for such a service and your discount comes just perfect, will use it after NYE party :) PS: Everyone, a Happy new year!

[@doodelinux]

Great video John, which Editor are you using ?

[@JohnWatsonRooney]

It’s neovim, with the basic ide config by chris@machine

[@doodelinux]

@@JohnWatsonRooney Thank you bud, I've been looking for an alternative to PyCharm and vscode for a while now

[@greyngreyer5]

I'm literally just getting started with python and need a fast study done for my thesis so I decided to study word usage on reddit. Should go through with it? Idk if i need any special stuff :/ I don't even have python installed. Cheers

[@yBlade05]

Off topic, but will you do a video on websockets?

[@JohnWatsonRooney]

Good idea I’ll add it to my list

[@jeroenvermunt3372]

This is nice, but my problem with using playwright is that it the twisted reactor always leads to issues when I want to run my spiders using python scripts

[@JohnWatsonRooney]

you could try using Splash? It hasn't been updated in a few years but may still work. Or create your own scraping/render service separate and use that?

[@casual_gamer1413]

Which is the best for JavaScript rendered websites? Selenium or puppeteer or playwright?

[@JohnWatsonRooney]

I use playwright mostly now

[@casual_gamer1413]

@@JohnWatsonRooney but I'm selenium expert, Should I use playwright instead of selenium or stick with the selenium according to your expertise?

[@JohnWatsonRooney]

@@casual_gamer1413 selenium! as you say you already know it, they do the same thing for web scraping purposes

[@casual_gamer1413]

@@JohnWatsonRooney thank you❣️❤️

[@karthikb.s.k.4486]

What is the IDE and theme used for this?. Nice explanation

[@JohnWatsonRooney]

It’s neovim and catpuccin theme

[@ervankurniawan41]

instead of playwright, can we use splash for any projects? which are recommended for web scraping?

[@JohnWatsonRooney]

You absolutely can - it’s not as easy to setup and use well in my opinion but fits well into a specific use case. However I don’t think it’s been updated for a while and I’ve had some people tell me it hasn’t been working for them recently. Give it a go and if it works for you then great

[@MrTurt99]

Is that turle neck from uniqlo? It looks 👌

[@JohnWatsonRooney]

hah thanks - its Indicode brand.. but i got it from tk max

[@vishalsugandh]

Hey, how can we scrape PDFs that are embedded to be viewed by chrome pdf preview? I think they use javascript.

[@vinubalank]

Hi John.. What options do you suggest if I have to save screenshot of webpage as jpeg or as html itself.. Is it possible to with Scrapy

[@JohnWatsonRooney]

Yes! Where I have the pagemethods - you can add in another “screenshot” that will do it

[@vinubalank]

​@@JohnWatsonRooney Thanks John

[@_manasikara]

As a newbe... Does anyone have some experience with a PUP - a command line tool for processing HTML? Is there any way to import it to the Playwright project the same way as the HTMLParser? Thanks.

[@AK-Star007]

Do you think chatgpt will put many of us out of commision??

[@JohnWatsonRooney]

Not anytime soon no

[@Kalter_int]

How to run a script from playwright in jupyter notebook?

[@manuelalejandrosalazargome1047]

yes

[@vahsek7488]

Hey how to scrape data from an Android application.

[@JohnWatsonRooney]

not something I've done before, but i know that the app will have a backend server/api that it makes the requests too, you;'d need to find this and reverse engineer it. or it might be possible to emulate the app on a pc/through browser?

[@jesseroeleveld5430]

@@JohnWatsonRooney most apps have ssl pinning security so we can not intercept it, to bypass this we can use nox player and man in the middle proxy to intercept from nox

[@gabrielkcgamox]

hiii, where is the github codes that you are using

[@comfortzonegames2131]

Next button is hidden for me. I am stuck

[@StrifePulse]

Any way to get around PerimeterX?

[@ramzan1813]

why you don't use selenium?

[@JohnWatsonRooney]

I used to! It’s great.. I just spent more time with playwright and find it a bit easier to use

[@ramzan1813]

@@JohnWatsonRooney I use selenium it's powerful but some time it's some modules not working properly and thay makes me angry 😅 and I think I have move to new solutions but then I reminded myself that I have use proxies. But I don't like use of proxy I don't know why but I scares from using proxy. Is their any free proxies?

[@kattamaran]

I would really be interested on how to find api‘s 😊

[@JohnWatsonRooney]

i have a few videos on it on my channel! basically using the network tab in your browser to look for requests when pages are loaded

[@kanwaradnan4849]

@@JohnWatsonRooney YEAH BUT IT DOES NOT WORK ALL THE TIMES.

[@jesseroeleveld5430]

@@kanwaradnan4849 for api usage it’s important to look at thenpayload that is sent. Is this a data form or a json payload? Also look good at the headers, this will fix 9/10 of your issues. Still doesn’t work? You probably forgot to fake some cookies :)

[@kanwaradnan4849]

@@jesseroeleveld5430 yeah i found it buy watching video on this channel

[@kanwaradnan4849]

@@jesseroeleveld5430 the reason I told the above statement is because I created a react app and it doesn't show my api using old tricks

[@shakhauathasan9555]

brother, i'm in a big problem. last 20 day's i'm trying to scrape one of site. but i failed all the time. I watched 100 of videos. but i failed. can you scrape a site for me. if possible plz reply to my comments. this is my final year project. you just scrape me some data. my final year defense in knocking my door. plz brother if possible reply my comments.

[@skshaheen7506]

If you're looking for a challenge, then try to scrape 9anime, It will be an interesting challenge. And also great content to watch. 🍿🍿

[@techlogger]

My friend did it with just requests.

[@skshaheen7506]

@@techlogger hmmm yes it's possible 🤔, but you have to find the API.

[@techlogger]

@@skshaheen7506 yes. Not only api.. you have to solve heavy Java obfuscation too. It's more complicated than normal scraping

[@skshaheen7506]

@@techlogger so after some digging with fiddler I have found the api and also able to get the video embedded url but couldn't get it to stream, since I have little JavaScript knowledge and unable to use the devtool (because they blocked it) this is as far as i can go for now, will try it later 😌.. and also as you mentioned this site use Obfuscated JavaScript.

[@techlogger]

@@skshaheen7506 you can unblock devtools, stream links ( m3u8 links ) are mostly restricted.. you have to pass proper headers, payload or in some case a decryption key. And yes that website is heavily obfuscated. But it's doable.

[@bakasenpaidesu]

Can't say I'm first 😂

[@ramonmijangos1091]

John, Is it possible to scrapy an Android or iOS app

[@heaton922]

are you using lunar vim?

[@JohnWatsonRooney]

No but it’s the basic ide by the lunarvim author, it’s great