How I Scrape Data with Multiple Selenium Instances

DISCORD (NEW): / discord
Selenium Grid first look for web scraping concurrently with headless chrome
Patreon: / johnwatsonrooney (NEW free tier)
Scraper API www.scrapingbe...
Donations: www.paypal.com...
Hosting: Digital Ocean: m.do.co/c/c7c9...
Gear I use: www.amazon.co....

[@Septumsempra8818]

Yes!!! My scraper system has grown exponentially and it's a bit too much to handle. This is exactly what I've been looking for

[@anushibinj]

I wish all tutorials were as descriptive and straightforward as this one. Immediately subscribed ❤

[@irfanshaikh262]

I never experiment anything on my own in actuality. I just wait for your innovative solutions to come through so that i learn and implement them. Hope there are more sessions based on selenium grid of just not scraping but with operations like populating a form on a webpage concurrently. Thanks John for being an amazing teacher

[@matth3wss]

Just what I needed to watch, thank u very much

[@sviatkey]

I am working on remote server and had no time to check how grid works. I do know now. Geeez. This is what I was looking for. Thumbs up 👍

[@JohnWatsonRooney]

Thanks for watching !

[@TheJFMR]

Amazing John Watson, this exactly was an issue i was struggling with. And there arent so much information.

[@JohnWatsonRooney]

Thanks! Appreciate it

[@anishpillai]

This is very useful. Hope you make more tutorials for selenium grid, especially running in a cloud environment.

[@JohnWatsonRooney]

Yes more coming

[@TheJFMR]

Another thing you can do its use a browser as a service (like an API) And you connect to that browser through API requests.

[@pascal831]

Awesome work as always John! Thanks brother!🎉

[@JohnWatsonRooney]

thanks!

[@rick-hoekman]

Very cool! Definitely going to try to set this up myself and test it with multiple scrapers.

[@JohnWatsonRooney]

Please do and let me know how you get on, I’ve got some more stuff to test like running grid over multiple severs via docker swarm

[@rick-hoekman]

We'll do.. Running scrapers over multiple instances would be very interesting to see how you would set that up!@@JohnWatsonRooney

[@chandrasekaran2429]

I was very New in Web scraping but definitely I can try different ways 😊 thanks for sharing this information Your video 😊

[@JohnWatsonRooney]

great thanks for watching!

[@chandrasekaran2429]

@@JohnWatsonRooney i was regular followers

[@CodePhiles]

Thank John for this video and illustration, it was new for me to know about this feature, which is awesome, I remember I did multiple instances int he past of webdriver to run simultaneously, but also seems to be sequential !! as it was a bit of hassle but it was working, but now with this feature it will be more easier.

[@soul_maestro]

as there is a selenium-arm built for docker you can also run that on raspberry pi or even a pine64 without a gui-OS installed on it like i do. btw, it's still a browser that's spooled up and it's not headless, as you can vnc into those instances by clicking on the camera and see the browser open and close... just like you did on your desktop. so those instances aren't headless, they just open inside docker which can be running on another host.

[@JohnWatsonRooney]

thanks for the clarification about the headless you are right. I need to look into the rpi arm version!

[@123arskas]

Amazing content. Would love it if you could create a Docker Crash Course.

[@JohnWatsonRooney]

I’d love to however I’ve still got a lot to learn about docker!

[@jiaqint961]

Thanks so much for the sharing of knowledge.

[@AllifIzzuddin]

I think that's kind of similar with playwright with persistent, browser new context, different tab/instances with different cookies, headless

[@JohnWatsonRooney]

It spawns multiple instances rather than reusing the same with extra pages. I think there was a time when you could connect playwright to grid. I’m gonna explore the playwright options tok

[@technicalking4711]

Can you please make videos on Docker with these kind of experiments, that would be awesome..

[@JohnWatsonRooney]

Yes sure there are more like this coming

[@alexdin1565]

thanks jhon for this amazing video like every time please i have a question about selenium i try the code in your last video and i want to add a chrme profile but i can't

[@MohsenHussien-t6v]

Thanks John,How I can pypass cloudflare capatcha?

[@JohnWatsonRooney]

have a look for cloud scraper and see if that helps you

[@kanwaradnan4849]

As i deployed that to the cloud i couldn't get any response from the Amazon site, but for every other site it worked well.

[@GusMD84]

A tutorial on how to set this with aws lambda would be amazing!

[@technicalking4711]

Amazing

[@JohnWatsonRooney]

Thanks

[@41v47]

I know my comment might seem off-topic, but I really like your color theme. It looks so soothing and easy for the eyes. Could you please share the name of your color theme? Thank you.

[@JohnWatsonRooney]

No problem sure, it’s called everforest

[@Optimusjf]

Excelent

[@jpeca13]

What are the advantages of using Selenium grid instead of Playwright async?

[@richiestark4921]

What about this grid or multisession with the non headless browser, the chrome extensions and docker. It's challenging to setup together.

[@dobcs3236]

❤

[@CrazyFanaticMan]

John quality work as always, i have a question mate related to Neovim, bows your experience with it been? It seems like everyone these days have jumped on the bandwagon I use default IDLE text edutor for quick scripting and VS Code with Emacs key bindings for more complex projects I really love my Emacs key bindings, is learning Vim a requirement for Neovim or can I also use Emacs keybindings as well?

[@JohnWatsonRooney]

thanks mate. Yeah I'm loving Neovim but yes its all vim keybinds. I guess you could create your own keymap but I dont think that would be worth it. I never learned emacs so once i got the basics of the vim movement, copy/paste and some basic motions it really clicked for me. I'd say if your happy with what you've got don't worry about it. Nvim fits my flow really well and i feel faster than i was in vs code/pycharm. if i use vs code now i used it with vim bindings too.

[@MDAbdurRahimcs50]

How Can We Add Proxy with Remote driver?

[@nizarfathurohman486]

John i can't follow you on java command things. Hope you make detailed video about selenium grid.

[@lordlegendsss7776]

How can i use these type of script in mobile python

[@iamshiva003]

Hello I needed some help in scrapping Amazon website please reply

[@AmodeusR]

Why use Selenium when there is Playwright?

[@JohnWatsonRooney]

I normally use Playwright, but Selenium 4 is pretty good too and has Grid.

[@AmodeusR]

@@JohnWatsonRooney Wait, did Selenium just got updated? I don't remember such a functionality and being so easy to import to use it :0

[@JohnWatsonRooney]

@@AmodeusR selenium v4! (welcome to the discord #101 ;D)

[@salamandralw]

where is github code ?

[@bakasenpaidesu]

.

[@kaistai]

Thank you soooooooooooooo much~