Web Scraping with Python: Ecommerce Product Pages. In Depth including troubleshooting

Follow along with me as I scrape data from an online store. I loop loop through pages to get product links, then each of those links to get information from each product page, scraping data only available there.
User Agents:
developers.whatismybrowser.co...
Hosting: Digital Ocean (Affiliate Link) - m.do.co/c/c7c90f161ff6
Scraper API I use: www.scrapingbee.com/?fpr=jhnwr
Proxies I use: proxyscrape.com/?ref=jhnwr
Gear Used: jhnwr.com/gear/ (NEW)
Patreon: / johnwatsonrooney (NEW)

[@fatimaelmansouri9338]

This is excellent content. I've been browsing for hours looking for a clear and detailed explanation and was lucky enough to find your video. And only 20 mins long! Thank you for sharing!

[@barzhikevil6873]

Thanks John, that was a very helpful video. As an economics major, I really need to be able to gather lots of data and process it efficiently, so web scraping was just a natural thing to learn. Keep up the good work!

[@m1sti_krakow]

Easily the best video on web scraping in Python I've ever seen. Only 20 minutes, but it has more content than many 1hr+ tutorials. Also you've explained many useful cases (f.e. what if we don't have some element). Thank you!

[@JohnWatsonRooney]

Thank you!

[@goodkidnolife]

Fantastic video, not only is it easy to follow along but explanations afford a genuine learning opportunity, rather that just a simple copy and paste. With myself being new to python, a big thanks is in order!

[@djuzla89]

Never subscribed so fast, your explanation during work is priceless, and the speed if just perfect

[@Neil4Speed]

Great tutorial, just went through. An excellent progression from the last one as most of the scraping that I have wanted to do involves "digging in". I feel that I am finally learning as I noticed the issue with the rating as we were typing it through!

[@JohnWatsonRooney]

Excellent glad it’s helped you improve!

[@ellisbenm]

Really valuable stuff. First web-scraping vid I’ve seen that goes into building a database with the scrape contents.

[@JohnWatsonRooney]

Thanks!

[@chartalized.9533]

Cant thank you enough for this mate! So helpful. Love the clarity!

[@JohnAtkinson-ww8qe]

Nice! These methods are the exact reason why I started my journey today in learning python

[@phantomsixtrading7094]

Awesome video. Very thorough instruction. Thank you for going slowly and speaking clearly. Easy to read your screen. Overall phenomenal video.

[@vvvvv432]

Another great video, thank you so much ! Your scrape videos are much better and to the point than online training platforms. I was watching a web scraping from pluralsight yesterday and I learned 2% from what I learned here.

[@JohnWatsonRooney]

That’s great I’m glad you’ve learned from my video!

[@bipinsartape683]

no words to thank-you. you made BS4 so easy

[@pandharpurkar_]

thanks John. Stay healthy..! Good concept clearing skills you have

[@daveys]

Getting the classes, divs, spans is something that I find quite confusing but I think you explained it well here. Thanks for posting!

[@ephraimmotho887]

Always enjoy such practical tutorials... Thank you so much for your efforts💯❤

[@marcin7992]

Youre really the Best mate One and only with web scrapping who managed me to scrap my data. Of course there is a need to adjust to own needs or Web needs but in the end you translate it perfectly. Very, very intuitive. Thank You.

[@JohnWatsonRooney]

Thank you for the mind words! I’m very happy I can help

[@deepak7751]

Finally after browsing for 3 hours I found someone clearing doubts beautifully. Thank you for sharing such a nice video.

[@JohnWatsonRooney]

Thank you!

[@deepak7751]

@@JohnWatsonRooney Sir Can i get your email id i am having a query for which i need your help. Thank you

[@Kicsa]

I have only been watching tutorials but this is really inspiring since you used it in a real website, thanks for the great video!

[@SeanWilston]

Thank you John. Very clear and useful information

[@SadamFlu]

Bro... You're the man. That was so well explained! You don't fuck around, you just hit it!

[@JohnWatsonRooney]

Thanks!

[@lukerobertson1000]

Love it!! Thank you mate, very clear and simple to understand.

[@gisleberge4363]

Very useful...thanks for putting it all together in such a clear way and easy to understand!

[@JohnWatsonRooney]

You're very welcome!

[@business5707]

John very valuable content. thanks to share with the community

[@victormaia4192]

very insightful, was nice to follow, now I'll try to do something similar with my projects to extract infos from ads, thanks!

[@rich-xf3sh]

You deserve way more subscribers, great work, hope you keep posting!

[@JohnWatsonRooney]

Thanks!

[@archanjd4463]

Excellent stuff! Straight to the point

[@adarshdessai6752]

Amazing 😻 thanks bro. You have made a scraping lot easier.

[@aksontv]

Thank you sir, and please add more advance tutorials in this playlist

[@alejandrofrank7900]

Oh man, this is insanely good, keep it up!!

[@JohnWatsonRooney]

Glad you like it!

[@fumanchuyn]

This video is the kind of video that you think people wont share this knowledge for free, it is hope yet!!! Amazing video m8 keep doing it

[@irfanshaikh262]

The concurrent.futures applied to this has accomplished the task like a wonder. Thanks again John

[@RodrigoLobatorodrigo]

Simply awesome!! Really good job :D

[@studywithrobin2715]

A few hours later, and your tutorial's already being implemented in my everyday web scraping / data cleaning work at the office. +1 Subscribed! Edit: Do you think you'll make a video combining the stuff in this video with using Jupyter Notebook to fill a CSV/Excel file?

[@salahgouzi2458]

Cant believe you got 1k subs only, that was insanely informative, thank you

[@JohnWatsonRooney]

Thanks glad you liked it

[@salahgouzi2458]

@@JohnWatsonRooney thank you too, keep the good quality up!

[@finkyfreak8515]

Wow John, you have a new fan here :D. Super helpful!

[@JohnWatsonRooney]

Thank you!

[@im4485]

Very nice... Straight to the point

[@GFXGreg]

Excellent tutorial, thanks!

[@HabibKhan-kj8um]

You're fucking amazing ! Kudos for such an awesome explanation. This is what I was looking for. Hats off to you

[@DATA_MACHINE22]

very beautiful and from scratch.👏👏

[@JK-iu4wl]

Great video. Thanks for sharing.

[@sourabhrananawareyujfestbw9858]

Best Video ever on web scraping ....#liked #commented #Subscribed #Love From India

[@nathannagle6277]

Great video thanks! You should do a part two where you scape the pictures and export the CSV onto your own webpage.

[@ariwansrisetya5714]

Thank you mate! Very helpful.

[@bagia1000]

at last! this is what i am looking for

[@leleuf3]

Fantastic tutorial Man, thanks a lot!!!!

[@JohnWatsonRooney]

Thanks for watching!

[@gomesgomes8206]

What a great training video. Thank you John. You are a great instructor. Explained well, easy to follow, clear and uses a real life example (real life challenges one would come across). Lots of ahaa moments that I had been struggling with including the what if an element is not present how to keep your program running (using Try: Except:) as opposed to your program coming to a complete stop. How easy was that? Only took me 6+ days of searching.

[@JohnWatsonRooney]

Glad i could help Gomes!

[@mikeaheirwe7157]

This was a very useful video, thank you! I would have like to see how you handled the 'in stock' element because some products didn't have it listed. I tried to use your idea ("try:" and "except:") to overcome this but some of the products which didn't have the "in stock" label were actually in stock

[@user-dk6fu4oz7f]

Amazing!! Sooo helpful!!

[@phillfairclough]

Great stuff, been following for a few days now. How do you manage to trim or replace any urls that you scrape? Some of the sites I have been trying leave whitespace but have a %20 in the actual URL

[@amith_1923]

Just joined your channel, hope to learn more and thanks for the video

[@jonathanfriz4410]

Very good one! thanks bro!

[@NeraJima]

Thank you. Really helped

[@shreyasdeodhare2574]

John, your video was fantastic! I appreciate the clear explanation, but I'm curious: will your approach work for any website? Looking forward to your insights!

[@alexlytle089]

I really love your videos bro. For scraping webpages do you prefer beautiful soup or selenium ??

[@talhajavaid3020]

very helpful , thanks!

[@kel78v2]

Just started learning python and found this video. Easy enough to understand your workflow and steps. Can I just ask what additional steps would be required if the products are behind a login access? Mind helping?

[@athulyesudas]

best web scraping video.. keep it up bro

[@JohnWatsonRooney]

Thank you very kind

[@Msr94145]

amazing tutorial!! thank you!

[@EllisGerrard]

Thanks for such a well explained video! Especially for beginners. I have one (noob) question - when I print(whisky) - I get back (u’OUTPUT’) - which I have found out is Unicode formatting. Is there anyway I can configure Sublime to get an output the same as yours? Thanks again.

[@reymartpagente9800]

Thankyou for a simplyfied content as always. We love your videos I hope you can make video also on how to scrape json data under javascript tag. I encounter more websites like this. This is a bit advance than your previous videos

[@JohnWatsonRooney]

Thank you for your feedback. Yes I am planning more advanced scraping soon including JSON and script tags

[@shubhamkumar857]

@@JohnWatsonRooney how do i scrap the random td without any class or id

[@pepecopter]

Great tutorial thank you

[@RainyRun]

Hi John. Thank you for your work/ Can please suggest some software which will scrape (do the same thing as in your code ) ? I am not familiar with pyton installing and using it

[@Nafke]

When you program web scrapers for work do you wrap the different parts of your code as functions and then call them from a main function like: get_Links(), get_products(), etc or just leave it as a long script because this simple enough? Also thank you so much for your content. I’m not a stem student but I was able to learn enough to build my own dataset for school even though I never programmed before. Thank you so much for taking all this time.

[@MuhammadFarhan-jr3cg]

hey bro thats amazing video

[@stevennagliati6959]

This was really useful and clear, thank you! I'm just getting started with web scraping and I see you have multiple videos and playlists on the subject - which ones would you recommend I'd start with? Cheers from Spain!

[@JohnWatsonRooney]

Glad it was helpful! Try my Modern Webscraping playlist I think you'll find some useful things in there - kzbin.info/aero/PLRzwgpycm-Fio7EyivRKOBN4D3tfQ_rpu

[@stevennagliati6959]

@@JohnWatsonRooney brilliant will do that, thanks!

[@AV-nh8mp]

Thanks for all John, you are the bets! I'm new in this area and I'm really happy from learn to you. I'm stuck with a paga that carge again and again untill the end, the page don't have number of pages that change ...could you help me? I saw another of your videos but couldn't find one that match ... thanks again

[@intesartaieb8590]

amazing video thank u

[@farhadkhan3893]

Awesome,, Thank you

[@adityadixit2816]

Great tutorial ... I also wanna ask you a question like When I write product schema markup for product its very lengthy and time consuming to write schema for links when there are 1K or more product are present is that possible to automate this process

[@sandeepkumargiri9609]

Nice one sir,

[@spearchew]

excellent video. Better even than corey schafer and that's saying something. Subbed for sure.

[@TelstarB]

How do you choose the best library for the scraping? Time? Complexity? Grat video btw

[@lofieros]

Best Ever codes

[@gonzaloalarcon4926]

te amo, me salvaste la vida

[@durgeshkhade2417]

Good man!

[@jasonkesterson2402]

I was able to get the rating by splitting on the and taking the index of [0] I'm not sure if that s the best way but it worked. :)

[@phantomsixtrading7094]

Can you post a screenshot please?

[@lieza2171]

Thank you so much!!!

[@jorgemarques2585]

Hi John, great tutorial, followed by the letter, but in my case I have a list with same class with 3 items, by using 'li' and the class, I can only retrieve the first item and not the 3 of them and get the 3 together if I use 'ul' which is not ideal, how to go around this? Thank you very much.

[@Blr046]

This is really nice. Thank you so much for sharing your knowledge. I want to build a python web scraping project GUI based where the user will select which web site to scrape and initiate the scraping job on click of a button. Please let me know how to achieve this.

[@davidwachie2518]

wonderful

[@rolf8107]

hello, I had a question, which packages do you use within preferences because I get all kinds of error messages when using your code.

[@novarahman5049]

Thanks so much for the tutorial, I'm attempting it with a similar website but the review number is placed within a tab to click placed as a button id, any idea how to pull the review number from there?

[@waeeeezeeeee]

Thank you 🙏

[@samman5980]

hey I see you're using lxml instead of html.parser. How do you grab a href link with html.parser instead after defining the a tag and class?

[@ashishtiwari1912]

This is what I was looking for. Most youtubers just made a video about how to scrape the first page but didn't tell of how to fetch the data for each product and then do pagination. Now it's very clear to me. I am getting one error:" 'NoneType' object has no attribute 'text' " after 30 or 40 iterations. I wonder what does that mean? I tried checking the solution to it on stackoverflow but the code shows the same error. And yes this is very useful for beginners to Intermediate. Keep making such videos. I have subscribed to your channel.

[@JohnWatsonRooney]

thanks for your kind words! it sounds like maybe you reached the end and got all the pages? see what happens when you go to the last page in the browser

[@ashishtiwari1912]

@@JohnWatsonRooney Yes. I have got all the pages. I have one more question to ask. I am putting down the link below. I am trying to extract the company details such as Name, Telephone number etc. The html code shows that it is with a list tag and then within each list tag, there is span tag with itemprop. I am trying to use span and itemprop but I am not getting the result that I want to. idn.bizdirlib.com/node/5290

[@daddy_eddy]

Thank you. Why I get errorr: name = soup.find('h1', class_='product-main__name').text.strip() AttributeError: 'NoneType' object has no attribute 'text' I wrote "r.text", but it didn't help

[@_RamjiG]

need some help here! I executed until 4.30 time stamp but only got empty list but you got the link for items.

[@tablesawart2728]

I applaud you for your clarity. At 8:50 I run the program and get only this: '[ ]' (the two square brackets for a list)... Why??

[@Sara.Saavedra]

me too I get the same. did you solve it?

[@dhruvipatwa4050]

This is so helpful. I am literally a beginner in Python. Would you recommend any other videos that I can watch learn basics of Python. Thank you so much!

[@JohnWatsonRooney]

Thanks for your kind words. If you are completely new at Python check one of the complete courses on YT on freecodecamp channel to help you get started. You don't need loads of knowledge to start web scraping

[@dhruvipatwa4050]

@@JohnWatsonRooney Thank you so much! Also, the website I am trying to scrap have product images nested in product images like this: . What should I be using for productlist = soup.find_all('li', class_='ssMAPPriceCheck'). I could be completely wrong at this but just gave a shot based on what I understood. Can you please help me? Thank you.

[@user-kn4ud5mf3o]

Looks like I get rate limited(getting a 403 Forbidden) ->(in the second loop)after looping over each link to get the name, reviews, and price. It got me all the links but I get a forbidden when looping over each link to get the data. Any tips?

[@aaronramsey4922]

Please, tell me, what to do if several elements that come one after another have the same class and style? When I parse, only the first element appears, and instead of the next elements. 2nd question! Let's say, first and second elements have the class and style, but beginning with the third one no class or style, and it stops appearing giving error. What to do?

[@ahomes6329]

Hi. I'm new here. When I run print(productlist), I'm getting back an empty list. What could be the problem?

[@mukulbahuguna9553]

thanks for the info.. i have a question can you tell how to do scraping of e-commerce using java?

[@gpcamillo]

Thanks for that. Great video!

[@atsource3143]

Hi John, just wanted to know is there any way to scrap hidden div tags/elements using playwright, beautifulsoup etc? Thanks

[@fernandodaroynavarro4231]

Hello @atsource3143, did you find the answer to this? I have the same problem about scraping hidden tags.

[@im4485]

Hi John, close to the end of the scraping process i keep getting this error: ConnectionResetError: [Errno 54] Connection reset by peer Any idea what this is?

[@Troglodyte2021]

User Agents blocked me. I think I have to come back to your video again when I need them. Salute!

[@nathannagle6277]

They block you if you use a VPN

[@Troglodyte2021]

@@nathannagle6277 Thanks!

[@yummywithali]

Thank you it was so useful. I have a question. I want to crawl product data and at the same get product description that is a link on another page. how we can crawl product description when it is on another link?

[@JohnWatsonRooney]

Hi, sure, scrape the url for where the product description is and request that data from there within a for loop and add to your data

[@roberttuttle4284]

Thanks for the lessons. I have a website that requires me to either input dates, then I get a list of links. I must click on each one and scrape a json file. I am doing this with Selinium because I am a beginner and Beutifulsoup doesn't seem to work. But its super slow. I think the site uses JavaScript. Is there a better way to do this than using a headless browser. Do you have a video that might help me.

[@devawratvidhate9093]

is there any tutorial for scraping multiple product reviews form amazon eg:- i want to extract multiple mobile reviews from 1st page of amazon as u showed above