3 Ways To Scrape Infinite Scroll Sites with Playwright

[@hadjuse2.87]

This is exactly what I was looking for because it matches perfectly with Instagram scrapping

[@silkogelman]

Interesting to get the product data as JSON data that way! Thank you John. 🙏 And Playwright is so nice to work with, really cool.

[@JohnWatsonRooney]

Thanks Sil

[@MrZinchyk]

I Scraping this site, you can do it through requests, it's good to get json there. In json, get the total number of positions, divide by 24. So we get the total number of pages. sorry for my English

[@juliopaniagua8723]

Hey John! great videos! Could you make a tutorial for scraping aspx pages? Ive been struggling to find any good tutorials on this. Cheers!

[@Valentin439]

Thanks for the information John! Really useful

[@villageidiot8718]

Thanks for another arrow in the quiver

[@lindafitriani]

You're a legend! thank you so much for this

[@joseniltonandrade5353]

Great video, John. Thank you a lot!. Is there a way to do this using requests? I have some code to do this scroll using selenium, but it's taking too long to scraping.

[@AdamArmstrong-nh5xs]

Thank you! This came at the right time

[@janmarc132]

What is that editor? I would love to try it.

[@JohnWatsonRooney]

its neovim !

[@jyorko721]

Is it nvchad or you running your own custom. Would love to know the keymap for your terminal

[@janmarc132]

@@JohnWatsonRooney A video about that would be nice. Or even just a short.

[@abdullahsahin1083]

Can you share your development environment. I think you using to vim so if these possible you can share your plugins, vimrc file, etc. :) Thank you so much John :)

[@itzcallmepro4963]

Thanks alot , i didn't know about the event part although i used playwright alot , is there anysource to get all good feature and practices in it ?

[@JohnWatsonRooney]

Everything I’ve learned has come from the official documentation, it’s really good and covers python well

[@Osegbuvalentine]

Do you have a complete tutorial on playwright?

[@drac.96]

Have you tried Crawlee before? Really interesting.

[@JohnWatsonRooney]

I haven’t I’m afraid

[@drac.96]

@John Watson Rooney Also, I've used this for crawling sites with infinite scrolling as well. Makes it as simple as one function call `infiniteScrolling()`, and that's it. Sure, it doesn't beat doing it manually, but it works. I've done exactly what you've described in the video: scroll down the page and collect the incoming data on a different site with this. It works great!

[@rexsybimatrimawahyu3292]

Idk if you will reply to this, but i want to ask if its possible to scrape infinite pages with scrapy? If its possible can you guide me how to look into it? Im kinda new to webscraping. Thanks before

[@JohnWatsonRooney]

you can if you use scrapy-playwright or scrapy-selenium. with the browser control you can scroll down the page before rendering it. But its best to see if you can find the API calls that happen each time a new set of data is loaded and try to copy those urls into your code and request it directly

[@rexsybimatrimawahyu3292]

@@JohnWatsonRooney thanks for the help.after thinking through about it, i will just use scrapy-selenium. Im not ready yet with API calls and stuff

[@tippapanchuechamnan1419]

Hello, I encounter an issue that page keep scrolling up and down during searching for selector, is there any way to make the page stay still and just react to that selector? Please help

[@ruasrr]

Hi John, amazing videos, thank you very much! I'm having an issue maybe you can see the solution quickly. I'm scraping a website which have "load" button after the products so I have a for to get all products, then click load, get again, load... but I'm always getting stuck after some amount of products, near 300... is possible that's memory or any limitation which is generating that?

[@tomahocbc8228]

can you make a video on how we can integrate ScrapingBee with playwright ??? i try it but when page reload or open new tab it not change my IP (the website detect Im not from the country allowed )

[@JohnWatsonRooney]

Let scrapingbee do the playwright part, you can just use requests and ask it to render the page for you or execute JavaScript

[@Tiagol343]

Is there any way to get data from a site that is already open in the browser without having the playwright open the browser again?

[@Shajirr_]

Getting this error: AttributeError: 'PlaywrightContextManager' object has no attribute '_playwright' So far found no way to fix this.....

[@muhammadirshad7497]

dear can you make one video on scraping zoopla website scrape with beautifulsoup

[@bakasenpaidesu]

❤

[@herehere-k8e]

ดีมากๆเลยครับ