Selenium - Real World Web Scraping - Challenges and Solutions

In this video, you will learn how to solve a real-world scrape using Selenium, including export to CSV.
See the same site scraped using Scrapy: • Python and Scrapy - Sc...
Source: gist.github.co...
Visit my site for free course on Scrapy: coderecode.com/
~-~~-~~~-~~-~
Please watch: "Making Scrapy Playwright fast and reliable"
• How to make Scrapy Pla...
~-~~-~~~-~~-~

[@cadbane9072]

omg. that stale element error has driving me nuts the past few days. Thank you! You explained it reallllllyyyy well. you got a fan!

[@mikewaters6980]

You’re a great teacher- one of the best I’ve seen.

[@codeRECODE]

Thank you, Mike. Your words mean a lot!

[@meamitacharya]

Please upload tutorials on XPath selectors

[@codeRECODE]

Putting on the list :-)

[@engineerbaaniya4846]

Please upload more videos

[@Song-dc5hl]

Hi codeRECODE, thanks for your video. I have an issue with the headless mode, I am trying to scrape another website, and I can get the info with normal mode. But when I apply headless, it does not return any info when I use the xpath selector. The reason I want to use headless is that i want to run my code on linix server. thank you again for your video.

[@arpitakar3384]

jai hind sir loved to follow today

[@haideralihassan5053]

Sir If we open freelancing platforms sites at same time while running all this will it any affect profiles as Chrome is opening and closing automatically? Please answer this before I start.

[@codeRECODE]

Ideally, this is what you should do - write the code and run it yourself to see what happens. Less ideal answer to this question is that selenium uses a different profile. Your other browsing sessions will not be affected. Furthermore, if you set to run in headless mode, you will not even know when it opened and closed.

[@haideralihassan5053]

@@codeRECODE thanks sir.

[@haideralihassan5053]

Hi sir I have created Google maps scraper with selenium and used technique to save data like in this video. And it's all working well. But problam is that if script get any error just before it ends correctly the data will be not saved. Tell any technique that data save in Csv accordingly as script runs. Just like scrapy. If script stops any where in middle then we get data that script collected till that time instead of getting nothing.

[@codeRECODE]

Use try-except-finally and write the data in the finally block.

[@KhalilYasser]

Thank you very much. As for the code on GitHub in the line `school_name_selector = '.school-title ~ h1`, I think it should be `school_name_selector = '.school-title h1'`

[@codeRECODE]

I wanted to check this before reply but I haven't got to that yet. Thanks anyways :-)

[@codecafe_bits]

Please make a video on using Scrapy with Selenium

[@codeRECODE]

Good idea! Adding to the list.

[@robertsmith4019]

It's much less trouble to use WebDriverManager.

[@codeRECODE]

Yes, that does make me things easier.

[@manimaranr7509]

Please do the video , which has more hyperlink pages in java script websites...

[@codeRECODE]

Do you have any specific problem that you are facing?

[@shabbirahmed9734]

Thanks a lot!!!

[@codeRECODE]

Happy to help

[@muhammadrizwan-tz9oz]

After printing the third school getting: Index out of bound exception

[@rangabharath4253]

awesome sir

[@abukaium2106]

raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: how i solve the problem?

[@codeRECODE]

Look into Explicit and Implicit waits in the documentation. selenium-python.readthedocs.io/waits.html

[@stalluri11]

What's your email address?

[@codeRECODE]

coderecode.com/contact-us/

[@daddyrichie]

Very Nice and Very Helpful. Thank you ! May I ask if you have any plan to make a tutorial that includes a site login, navigation and monitor of XHR traffic once on the correct page. (Your $1 course "schools" video has some of the elements I'm referring to.) What I'm trying to solve is logging in, getting to the correct page gracefully with natural pauses and lags like a human does when clicking through. An example could be a live auction where a current bid on an item is served and visible on the UI and on the XHR stack in real time along with other info flowing in. The purpose fulfilled would be to capture time, date, item, sale price and build a "comparable sales" database. To be a bidder in any auction, one most likely has to login and navigate to the specific auction channel of interest, sit there and watch the items come and go and manually record what an item sold for.