Web Scraping to CSV | Multiple Pages Scraping with BeautifulSoup

In this tutorial you will learn how to scrape multiple web pages using BeautifulSoup and export the results to a CSV files using Pandas Library.
We use books.toscrape.com as the web scraping playground and get access to the alt attribute of the images and the name of the classes.
------10 Web Scraping Project Ideas ------
pythonology.eu...
------Support Pythonology------
www.buymeacoff...
------Best Online Resource for Python------
Datacamp: The best online resource to learn Python, Web Scraping, Data analysis, and Data Science (Affiliate link)
datacamp.pxf.i...
------Sourcecode link------
github.com/Vid...

[@skye0613]

Finally, a video i can understand and doesn't make me feel dumb. Thank you good sir!

[@FIBONACCIVEGA]

same for me.

[@abdulwali4920]

literally the best video on webscrapping....i have watched hundreds of videos but this is the best.

[@Pythonology]

Thank you very much Abdul Wali for your nice words. Very encouraging :)

[@mohammedtofa431]

I have been searching for a video like this forever. Thank God I landed on your page. Really wonderful and amazing video showing step by step. You are a living legend. Just subscribed as well.

[@knoxholst67901]

I just wanted to let you know that I really enjoyed this video. I was feeling like learning python was stupid. Then I found you doing a cool project and it was easy to follow. I am inspired again thank you.

[@satishm5260]

This is the most on point tutorial I ever watched. No bullshit, no jargon, Just pure knowledge. Thank you Sir, I learnt a lot from this small video.

[@appleestrada6810]

This is by far THE best and easiest to understand explanation I’ve heard about using python to scrape data. Thank you for your effort in creating this video. You got a new subscriber!

[@SandeepGurjar-ko5ju]

Your work needs to be appreciated man. The way you explain things in a calm, composed and soothing voice. The simplicity of the tutorial indicates your grasp on the web scraping. Thank you.

[@Pythonology]

Appreciate it, Sandeep.

[@zedascouve2]

Absolutely jaw-dropping the power of web scraping. Congrats for the wonderful and comprehensive video. Waiting for more!!!

[@marcoschwarz4640]

I watch a lot of videos about programming and most of them are really good. However, this really is a standout piece. The way it combines theory and practice is second none. Well done,sir

[@Pythonology]

Wow, thanks!❤

[@samuelebojielu855]

Thank you so much for this video. I have watched several web scraping videos but this is absolutely the best so far.

[@nox6687]

Wish I could like this twice, I had a web scraping class that didn't explain this as well as you did in half an hour

[@homeoffice3524]

Finally! Really clean and easy to follow scraping video.

[@mattbaker6475]

Great video! You're not getting enough credit for how well this is made.

[@hirrog3865]

After so much searching I finally get a video that is so easy to grasp on scrapping from multiple pages. Thank you

[@SkillsGear]

Finally, a video that puts paid courses to shame! Hats off to you for the great tutorial! You just did not explain, the way you went back and forth helped me understand a lot. Kudos! Could not resist the urge to hit the like and subscribe button. Will definitely visit your channel for more guides and tutorials! ♥

[@muhammadmubeen8452]

Thank you very much sir I was watching many tutorials and was getting confused to understand the html structure then I found your video you explained everything beautifully I completed my project successfully thanks a lot sir❤

[@wanangwalumbaningwira3871]

ooooooooooooooooooooohhhh I really love this video you saved me big time. This is really outstanding, well detailed and you explainations are very logical and clear

[@lesotv1056]

OMG, I am so impress thank you so much for this wonderful lesson. I cant believe I got this for free . God bless you.

[@extropiantranshuman]

wow - best tutorial so far on beautifulsoup! Thank you!

[@sundy93]

This video is such a relief ,absolutely the best material about scraping ! Thank you so much!

[@everyany4161]

Thank you so much for this video! It's literally an answered prayer for me. 🙏

[@randomperson-gp2wd]

Thanks a lot. With some basic level in python and 0 background in programming, I was able to successfully do a project for my master thesis related to media coverage about certain topic.

[@jeinnerabdel]

Great video! I wrote the code while you were explaining it and I kinda grasped the idea behind what you were doing. The only thing I don't understand is about the indentation and how it affects the for structure. In other languages, you end the for with some code and nest them like any while-do or if-else-endif type of stuff. I also thought that Python was like Javascript where data would automatically be translated on each variable based on it's content Var1 = Here you go (text) or Var1 = 12 (num) but as I saw on your example, you have to transform data into numbers even if they are actually numbers already. Interesting!

[@Mr.Rahul-s-18]

Such a GreaT Explanation Dear JusT Love it😘 Love From India 🇮🇳NamsTe🙏

[@KamleshShirsath-n7b]

Thanks a lot for this detailed video. Hoping to see more video's like this.

[@AbsSak]

I just loved it . I used to think web scraping is too hard but when I saw your video it is like so simple that even 10 years old can also understand. Simply great job 👏

[@rambhupal8870]

Its just really awesome and very easy to understand also and i have submitted this as a mini project. Thank You brother.

[@mohamadehtheshams2527]

Great Sir Today I learned how to do web scrapping .. Nicely Explained 👍. Please make more content

[@Pythonology]

Glad you liked it

[@catchiemitakata2142]

The best tut on web scraping. Very beginner friendly. Keep it up

[@georgewilliams9422]

Thank you! Very clear and useful!

[@ephraimmotho887]

Great tutorial yet again... This channel is so valuable for people who want to learn programming but do not have the money to go to school for it... Are there any other similar channels on youtube or outside the platform (websites etc.) that offer such great value but may not be popular? Please reply if you even have one suggestion. It is really helpful

[@nairchandnichandran7503]

Wonderful!. Simple and concise🥰

[@Raulenz]

Thanks, great video. Excellent explanation and great english.

[@BenjaminBalaj-yp8zk]

You explain everything very clearly. Everything makes sense now!

[@glowtechvisuals6176]

I like the way you teach while talking. make me understand. Tank you very much

[@osoriomatucurane9511]

So awesome! Consice & crystal clear! You are absolutely a legend.❤

[@pannusingchauhan6883]

this is just crazyyy. loved the tutorial

[@netacklee]

Wow, this video is so helpful, thank you!

[@David-c3h3o]

You really did make a great video.Thank you

[@raedkm]

Thanknyou for this clear and easy to follow video.

[@janp3648]

Omg .. this is such a perfect, informative, easy to understand explanation ! Thx a lot.

[@ghashanti]

This was great content. You made web scraping super easy.

[@bashirahsalami3408]

Thank you very much Pythonology. This was well-explained and very easy to understand.

[@dawnS33ker]

This video is an absolute gem. Thank you for this..

[@NikhilPawar151]

Great video. I really like the way you explain the concepts. Everything working fine and easy to understand

[@Pythonology]

Thanks Nikhil

[@richardharris-to3mc]

top notch. I managed to follow this, so thankyou!

[@MKOladapo]

Great lesson...Very resourceful

[@debasishrajbehera]

thank you so much sir....................i learned a lot .....its so helpful to me..🙏

[@joelngige5776]

Fantastic video on web scraping

[@Birwi]

Really helpful, thank you!

[@sabrinaazwa759]

thanks alot for the detailed tutorial!!!

[@mavisadjei4192]

Thank you very much. Is there a good book you can recommend?

[@czr372]

Thanks a lot for all this web scrapping tutorials! Ill try to do my own scraps now!

[@izainonline]

Great Tutorial can we scrap the secure or can say not allowed scrapping text.

[@pasindumadushan474]

Thank You So much SIr!

[@abdullahattia2491]

thank you so so so much very helpful! liked, and subbed

[@aspirant9511]

Very well explained ...thank u..

[@payamsarwar5611]

Very productive, thank you

[@AdamDoe-br5tw]

Great tutorial Thanks, now what if the pages have different/variable names like site/brand/VariableBrandname and I only have a list of the pages How to set the "i" variable to look into a set of "variablebrandname"?

[@newperspective9]

I saw what I need to saw, Thank you!!!

[@dimi_ga]

Thank you very much, you helped me a lot with your vid. 🙏

[@gadomix3989]

Thank you for a great video realy its cool project ever seen

[@svenGPT]

Just found it and love it. Thank you!

[@danielomoregie7603]

This is the best web scrapping video on the internet

[@CedricBrown-wy7kq]

You are the MAN!

[@saharyarmohamadi9176]

Thank you very much, very good and detailed explanation

[@Pythonology]

khahesh mikonam

[@swatisingh-yw1fw]

it was very helpful video, keep on making such video.

[@DSTtools-p4j]

This is very cool Keep.it up bro...

[@annicesullivan]

Nice tutorial on scraping multiple pages to CSV with BeautifulSoup! Any tips on reliable proxies for handling large scraping jobs like this? Heard Proxy-Store offers specialized scraping packages, anyone tried them out?

[@lesteravecilla77]

Super Thanks for this video. It is very clear and useful for people who like start web scraping like me. Good job and keep it up! 👏🙂

[@AmanKumar-xw5bs]

wonderful sir ! Learnt a lot

[@mynshk]

Many thanks for your demonstration! :D

[@ryansmithza]

Exactly what I was looking for! Thanks!

[@gur0004]

lovely stuff. I thoroughly enjoyed it.

[@khaledhamdy9944]

Great my friend

[@ibrahimaboelsoud7881]

24:53 what a vim move 😄.

[@chiedozieehirim5659]

Thanks a lot. It helped solve a problem. I have a question though. How do u handle 403 and 503 status_codes errors when scrapping a website?

[@Pythonology]

403 and 503 status code errors, indicate that the server is refusing to fulfill the request. To handle these errors, you can use the requests library to make the request and check the status code. One way to handle these errors is to use try-except blocks to catch the error and handle it appropriately. For example, you could include a sleep function to wait a certain amount of time before trying again, or you could implement a retry loop to keep trying until the request is successful. Another approach is to use a library like requests-html which has a built-in support for handling these errors and retrying failed requests automatically. Also, you can use a User-Agent in the headers to make the request appear as if it was coming from a browser instead of a scraper, as some websites block requests from known scraper IPs and user-agents.

[@leibaleibovich5806]

Thank you for the video! It is helpful, indeed.

[@ihebbibani7122]

Thanks a lot. Subscribed.

[@Fisayoxx.x]

Thank you so much for this. Thank you

[@fpldaily5172]

how do i webscrape the page and the content in the page eg, your video extracts the title, price etc but lets say I also want to extract the book page and the content after the book the book page like some e-commerce sites show the products name, price etc but when I click the page it shows decscriptions, reviews , and more pictures of the product how do I extract that aswell? Thanks mahn I like your work!

[@olubukola107]

Thank you 🙏 so easy to understand and helpful

[@mister_maxwell]

Very very good, I learned so much new and interesting stuff.

[@PP-pd3pf]

Thank you for the great content.

[@tundebadmus7702]

You explain really well.. keep it up

[@AsmaaMGaber]

very informative video, thank you for your efforts. I use jupyter notebook and I wrote the exact code, yet it doesn't scrap all pages and it scrap only the last number in range. do you have any idea what could cause this error?

[@dreamer14492]

you pulled the name from the image alternative tag sometimes image alternative can be anything instead pulling the title from the h3 tag and title attribute will be better in my opinion

[@fredquiemby8024]

Thanks

[@adebisiabiodun3378]

Amazing video

[@QasimAli-hd7pd]

Well explained

[@evoplayz7136]

Would you please make a video on how to scrape the data inside of each link

[@ayoubcharbaji884]

hi sir, well i have a question about the page numbers, if i'm working with for exemple three websites and i don't know how many pages they've got so what should i do to make my code scrape all the products ?

[@littleboyspace8940]

title = article.find('img').attrs['alt'] star = article.find('p')['class'] Could you explain why we need .attrs for title when you can ditect access it like you did with the class tag? title = article.find('img')['alt'] star = article.find('p')['class'] I have try this and it work the same. Is there benefit to use attrs? Thank You

[@deniskarani9953]

thank you, sir!!! I have really learned a lot from you.

[@aryanpatel8140]

Nice video and very well explained❤ Just wanted to know how do I print the genre of every book next to it?

[@abrarbaltii]

How to store these data into a database like mongodb ...kindly make a video on it also it would be a great help.

[@AnulithaVardhanMarripeddiVijay]

I have the prices in tags and soup.find ignores it all together. Any idea how to handle that?

[@matteos9783]

Thanks, but what if I want follow the subpage of every book and extract the informations in these pages? I mean before I extract the informations in the first page, then go into every subpage of the books and finally grab the pages informations

[@deepakr229]

Thank you 🎉

[@jungjaehyun8625]

Multiple pages start at 21:20