Scraping with Playwright 101 - Easy Mode

Playwright is an incredible versatile tool for browser automation, and in this video I run thorugh a simple project to get you up and running scraping data with PW & Python
Join the Discord to discuss all things Python and Web with our growing community! / discord
If you are new, welcome! I am John, a self taught Python developer working in the web and data space. I specialize in data extraction and JSON web API's both server and client. If you like programming and web content as much as I do, you can subscribe for weekly content.
:: Links ::
My Patrons Really keep the channel alive, and get extra content / johnwatsonrooney (NEW free tier)
Recommender Scraper API www.scrapingbee.com?fpr=jhnwr
I Host almost all my stuff on Digital Ocean m.do.co/c/c7c90f161ff6
A rundown of the gear I use to create videos www.amazon.co.uk/shop/johnwat...
Proxies I use proxyscrape.com/?ref=jhnwr
:: Disclaimer ::
Some/all of the links above are affiliate links. By clicking on these links I receive a small commission should you chose to purchase any services or items.
:: Chapters ::
00:00 - checking site
02:18 - start code
05:24 - detail page
11:13 - pagination
16:30 - summary and run

[@bgriffin5447]

That split move was nice

[@Sharedbook]

This is awesome!! As an API Security Specialist, I always start by looking at the HTTP calls, searching for an API call that might have that same info. Saving me time from scraping the page. Most of the time I’m having success with that approach, especially when dealing with solid companies/websites/platforms.

[@alexanderkomanov4151]

Great one! I think that using pytest-playwright package can save several lines of code in the initialization part, because you can just use the page:Page fixture

[@graczew]

Good content as always. Enjoy your Easter break 😉👍

[@robertramirez2167]

I like that image blocking tip!

[@Extrey]

Nooooo waaaay, i just found schema on another websites, nice trick anyway, but i find it more efficient to read the info from the category pages. Thanks for your videos, they always inspire me!!!

[@elu1]

Thank you John for the teaching. I seem to have issue with Xvfb for running 'headless'. Any suggestion or resources that I can learn from?

[@fredde7356]

Hey John, can you please continue the scraping livestream with your test site? 😃 Would love to see how to handle the drop-down menus, Java script and how to handle stricter cloudflare rules Would be happy to hear about some news! Enjoy easter :)

[@munchcup]

On cloudflare One idea is usually using undetected chrome driver to avoid cloudflare and you can put delay while logging in to solve the captchas the first time and save the cookies. After that you no longer need to solve captchas it will be automatic.

[@IshaqKhan010]

sir can you make a video how to deploy playwright script on google cloud function / vpc please

[@user-wu4ip7mp3z]

I'm following this exact code in VSCode and only the initial web is opened, it doesn't open the subsequent pages that direct to each of the product, no idea how to fix this...

[@user-wu4ip7mp3z]

nvm, fixed it, turns out the data-selenium=...GridView... has been changed to [data-selenium='miniProductPageProductNameLink']

[@carloiurcovici]

Thank you John, I've been really enjoying your videos recently and applying everything at work where it comes in really handy. Would you consider creating a python/scraping course on Udemy or a similar platform?

[@JohnWatsonRooney]

thanks for watching. I have thought about creating a course but no serious plans yet i;m afraid

[@carloiurcovici]

@@JohnWatsonRooney thanks for the reply, if you change your mind you got my money 😂

[@badrenanna3961]

can you please start talking about some difficult cases : - scraping a website that has cloudflare protection against bots (even using proxy rotation it didn't work) - scraping website that have captchas protection .. Thank you

[@munchcup]

One idea is usually using undetected chrome driver to avoid cloudflare and you can put delay while logging in to solve the captchas the first time and save the cookies. After that you no longer need to solve captchas it will be automatic.

[@danueecitizen]

can this work with amazon ? 🤔

[@alexdin1565]

Thanks john, but now days most websites don't allow you to open links like you do they will block you after 3 or 4 pages open in same time another question If you can make a video on how we can use playwright inside a docker with proxy to make many requests at same time it will be very nice sorry for my English, I'm not a native speaker

[@s6yx]

Can’t you just do viewpoint for setting a screen size and header and run it headless with no issue

[@archiee1337]

why not headless?

[@mohsinhassan88]

Omg why the white editor??

[@РНТ]

Exactly. When I saw it I immediately remembered this video: kzbin.info/www/bejne/jp3Koo2bmtSCqqs 😂

[@tendosingh5682]

For some its easier on the eyes. MY eyes cant stand the dark themes.

[@mohsinhassan88]

@@РНТ exactly how I felt. And specially since John usually has amazing videos and everything is so perfectly balanced in terms of theme and ease on eyes. I was a super shock

[@pkavenger9990]

Your content is good but i think you should engage with your audience more instead of speaking like you are talking to yourself. You will see that you will get much more views. Take Gotham chess channel for example he is not a Grandmaster of chess but His channels have more views and subscriber than Hikaru and Magnus because of his communication skills.

[@JohnWatsonRooney]

Fair point thanks for the advice