Web Scraping Multiple Pages with Python and Selenium + CSV File

[@ahmetturk5675]

Hey! I was checking for something like this , Thank's a lot

[@jawadkazim6610]

You're so underrated , You should have millions of subscriber 😌

[@734833]

Finally what I searched for.

[@manpreetkaur-dk5mm]

Thank you sir, you this tutorial helped me so much.

[@athakur33]

Hi Alex, Could you pls make a tutorial on linkedin profile scrapping which can scrap Name, job title, location, Education, experience will be much appreciated

[@marinuss6004]

Underrated content creator LX_schlee

[@TheMilind222]

you're amazing love you sir

[@dawidlorek363]

fantastic tutorial! ❤

[@informationtoallj]

Thankks again

[@temblux2449]

Sir, I have completed your two courses. Web Scraping through APIs and Selenium. Sir, I’m fan of your teaching. Your way of teaching is easy and comprehensive. In Selenium, I’m facing issue regarding pagination. Structure of pagination is not being built by me. I saw this video but that’s complicated for me because method is different to udemy course. Sir, Please help me regarding to pagination structure regarding to udemy course for Selenium Thanks😊

[@carachen4082]

Thank you so much for the tutorial! I have one small question about writing csv file that I wrote exactly same code with you but the output is not clearly categorized as yours. In my file it was all in one cell even though i split with semicolon. May you advise why it happened? I'm using VSCode.

[@amrasfoor6269]

thanks for this video may I ask you how to handle it when there is no id attribute for search bar

[@diegocamelomartinez5566]

Hi :) I actually have a page where the lengths of the lists are not the same always. I am scrapping over 35 pages and on page 12 there is an item that doesn't have one of the attributes (e.g. description). Which ends up messing up everything. Do you have any suggestions on how to approach that?

[@NihumDe]

maybe go with a "try: " and "except:" approach?

[@marcogelsomini7655]

woow good stuff thanks man. writerow() function is maybe more handy?

[@praveenhosamani2274]

Hey !!!!...please make a tutorial on web scrapping of store locators such as cars dealers information,two wheelers, etc using both beautiful soup and selenium...!

[@justwowtv4998]

Great but what happens when you have a 1000 pages website? You will loop through all pages one by one? This will take days...

[@informationtoallj]

THANKS

[@architmishra015]

How and where are you writing the xpath? I can't find any such options in inspect

[@sowmiyasivakumar2425]

Hi, Your demonstration was really helpful. I have a doubt...how to click on those jobs eg: Software Engineer and nav to the other tab? actually i need to do it for the entire listed jobs?...is there a way to do it?

[@neroplus-it]

there is a way to do that but it was not covered in the video. If there is a demand i will plan to create a tutorial which explicitly shows how to navigate to a separate link and grab the information from there. I actually use BeautifulSoup for this since its a quite straight forward process. Thanks for your question and the feedback!

[@sowmiyasivakumar2425]

@@neroplus-it yaa am expecting for your tutorial. Thanks for your quick response

[@andrewmosola4013]

@@neroplus-it Thanks for the tutorial. I am also wondering the same as @Sowmiya above. I hope that there can be a demand. Thanks

[@diegocamelomartinez5566]

Also would be interested on seeing that

[@ProjectSkillsQMUL]

Thanks for the tutorial! I am new to coding and web scraping. What code would need to be written if you want to scrape all pages possible until you can’t click the next button instead of just 3 pages? Also, I need a include a wait function for each page since they are so JavaScript heavy.

[@marinuss6004]

time.sleep(1) will wait for one second ( remember to import time )

[@friendsforever715]

how to import selenium library in Jupyter Python? I'm getting an error prompt " No module named 'selenium' " once I run the script.. thanks..

[@neroplus-it]

"pip install selenium" in your CMD, or "!pip install selenium" directly in your notebook first

[@azadsaifi5919]

Hey, In my laptop, there is no option to search here is only option filter, so How can I search xpath or anything let me know please :)

[@technoscopy]

Hello can you scrape the Jsf formats based website please

[@Chariotzable]

getting this error.. else: ^^^^ SyntaxError: expected 'except' or 'finally' block

[@dark_legions2227]

Hii... Your content is awesome.. I'm from India your help I'm scraping Imdb top 250 movies bt I want scrape all pages data..how to scrape data from next page becoz next is button

[@deepak7317]

just write the x path of the next botton as a click and iterate the for loop to your desired page

[@deepak7317]

I ahve done this to scrap the sunglasses data from flipkart for first 120 glasses #Empty lists to store the attributes brand_name=[] prod_tags=[] price_v=[] time.sleep(3) start=0 end=3 for page in range(start,end): p_tags=driver.find_elements(By.XPATH,'//a[@class="IRpwTa"]') for i in p_tags[0:100]: prod_tags.append(i.text) next_button=driver.find_element(By.XPATH,'/html/body/div/div/div[3]/div[1]/div[2]/div[12]/div/div/nav/a[11]') next_button.click() time.sleep(1) for i in range(len(p_tags)): break brand_tags=driver.find_elements(By.XPATH,'//div[@class="_2WkVRV"]') for i in brand_tags[0:100]: brand_name.append(i.text) next_button=driver.find_element(By.XPATH,'/html/body/div/div/div[3]/div[1]/div[2]/div[12]/div/div/nav/a[11]') next_button.click() time.sleep(1) for i in range(len(brand_tags)): break