Let's Build a Python Web Scraping Project from Scratch | Hands-On Tutorial

Join our upcoming 20-week data science boot camp: www.jovian.ai/data-analyst-bo...
💻 Web scraping is the process of extracting and parsing data from websites in an automated fashion using a computer program. It’s a useful technique for creating datasets for research and learning.
🔗 Resources used in the workshop:
- Rough notebook: jovian.ai/aakashns-6l3/scrapi...
- Final notebook: jovian.ai/aakashns-6l3/scrapi...
- Web scraping project guide: jovian.ai/aakashns/python-web...
- Web scraping tutorial: jovian.ai/aakashns/python-web...
In this workshop, we’ll use Python and its ecosystem of libraries to scrape information from a website and create a dataset of CSV file(s).
Here are the steps we’ll follow to build a web scraping project from scratch:
✅ Pick a website and identify the information to be scraped into a CSV file
💾 Use the requests library to download web pages from the site programmatically
💬 Use Beautiful Soup to parse and extract information from web pages
📝 Create well-formatted CSV file(s) with the extracted information
✍ Document and share your work online in the form of a Jupyter notebook or blog post
⌚ Time Breaks:
Introduction 00:00
Problem Statement 7:03
Setting up Jupyter 15:09
Fetching pages with requests 25:26
Parsing pages with beautifulsoup 33:27
Saving data to CSV files 1:01:41
Scraping another page 1:03:10
Defining functions 1:11:28
Putting it together 1:30:35
Documentation 1:56:04
Publishing your notebook 2:26:13
Q&A 2:28:22
🎤 About the speaker
Aakash N S is the co-founder and CEO of Jovian - a community learning platform for data science & ML. Previously, Aakash has worked as a software engineer (APIs & Data Platforms) at Twitter in Ireland & San Francisco and graduated from the Indian Institute of Technology, Bombay. He’s also an avid blogger, open-source contributor, and online educator.
-
Learn Data Science the right way at www.jovian.ai
Interact with a global community of like-minded learners jovian.ai/forum/
Get the latest news and updates on Machine Learning at / jovianml
Connect with us professionally on / jovianml
Subscribe for new videos on Artificial Intelligence / jovianml

[@JosepOriol24]

That was an excellent tutorial, as always! In Jovian we trust!

[@chetanvgoudar9079]

I saw many vid on youtube about scraping but this one is a lit.. thanks jovian for this awesome content and crisp explanation.. Waiting for more videos on data science ❤️❤️

[@janicejose5809]

Such a precise explanation ,amazing content ,really impressed!!

[@sakata_gintoki007]

Honestly this is one of the best KZbin Programming tutorial. You make it from scratch and discuss it elaborately. Hope you can make more awesome teaching videos in future.

[@jovianhq]

Glad you liked it!

[@myelifek]

Thank you Aakash and Jovian team for the webminar. You guys rock!

[@jovianhq]

Glad you found it useful :) Help us spread the word!

[@web_angle]

Very good tutorial on web scraping. I learned a lot from it. Thanks!

[@happyjohn1656]

I just can't trust a man who doesn't use a dark theme when there is one available

[@MohammedAli-kl6dw]

Thank you. This project was very detailed and so very usefulful for beginners like me. You put in a lot of efforts for this. Thank you once again for your genuine efforts sir. I wish by the grace of God you grow by leaps and bounds and along with you we also catch up.

[@devendrakhanolkar423]

You are Absolutely methodical and good that you have revised and summarise the topic. Your voice and ability to teach is impressive:-)

[@norsen5517]

Best Tutorial on Web Scraping !

[@muthupriyadharshinim9033]

Thank you for this wonderful lecture.

[@sachinsoni5044]

One of the best tutorial ever followed🙂. Great work Aakash sir🎉

[@sahuanvesh5683]

best video on web scrapping till date brother..

[@channaakhil9426]

This is an excellent video 🙌🏻

[@joepomroy1499]

This is a super great tutorial! Thanks.

[@jovianhq]

Glad you enjoyed it!

[@VKBMath]

This is a great tutorial

[@vincent3542]

sir i just can't thank enough for this briliant stuff, love from indonesian bro!

[@abdullahateeq1236]

Its an amazing... I'm really impressed with your problem solving mind...🤯🥰

[@nehagarg899]

Thank you for this project and explaining it line by line. It really helped me a lot to do web scrapping project.

[@jovianhq]

Glad it was helpful!

[@prosen9w]

Great video for absolute beginners. Thanks a lot

[@jovianhq]

You're very welcome! Do check out our other tutorials as well

[@nathanalabi5842]

I must say this is Top notch! Thank you for this sir!

[@jovianhq]

Thank you kindly!

[@tanujkumar5894]

Thanks aakash for this free webinar. Your tutorial are best 🔥👍

[@jovianhq]

Glad you liked it!

[@ajitkulkarni9771]

Thank you for this wonderful tutorial. Found it very helpful.

[@jovianhq]

You're very welcome!

[@hemantvardani1436]

After watching this video, I became fan of your teaching . Thank you Aakash and Jovian Thanks for this beautifully curated video.

[@jovianhq]

You are most welcome

[@athanikarammy7476]

Great tutorial Aakash sir....please do more ...your way of methodologically describing the things and explaining them are great.

[@jovianhq]

Stay updated! More courses coming soon....

[@shobhitbishop]

Hi, This is the first video i watched on your channel and it is amazing👍

[@jovianhq]

Glad you enjoy it!

[@sameergoilkar9956]

Best video on scraping

[@math8964]

Great thanks 🌸.

[@pratikkamble2776]

It was very great session . The explanations was up to the mark.

[@jovianhq]

Glad you liked it

Redirected here through Microsoft Copilot recommendation, what a content sir. Thanks alot!!!!

[@mayankbhatt5604]

What a BEAUTY, what a beauty, the best web scraping video on whole youtube

[@jovianhq]

Glad you like it! Thanks for watching.

[@zain5251]

You are a legend , thank you

[@Afri_Data]

As a beginner struggling with web scraping for over a week. after watching the video, i've been very confident with beautiful soup and also been recommending it to my college on our Data Science community forum. Thank you

[@jovianhq]

Thank you for sharing!

[@futuregootecks]

WOW, this is amazing!!! I've been stumbling along trying to learn python and scraping over the past several months, but this really simplified a lot of it. I really appreciate learning good habits along the way such as using a jupyter notebook for everything. Thanks for all you do! Looking forward to more of this!!!

[@jovianhq]

Glad it was helpful!

[@Lotaristo]

I really like this guy. Although I'm sure that he's experienced one, he's doing work from the perspective of a total beginner - and that's very good option. Although the are many tutorials where people could make similar stuff in much less time, they mostly have a lot of details which hard to remember, and more appropriate for more experienced users, which could do at least basic or intermediate stuff without any problems and which just need to learn some new syntaxes and use it - but for beginners there are a lot hard-to-understand things, and I'm sure that to show them as many details as possible, simplify things and repeat even most basic things time after time (even if it takes more time) is the most appropriate way to learn something new. Also this recommendation not only to write code, but also to create a plan and move according to it - I'm totally agree with it - if you are new with any tool and don't know what and how you should work, planning will really help you to move forward much more smoothly (alas, almost noone makes such good plans - if there are comments in a code it's already nice thing to have).

[@jovianhq]

Thanks for the kind comments! We're glad you like our content.

[@marchanselthomas]

100% quality content

[@webscrapingfreelancer76]

Wow, thanks man

[@aabharaut327]

Thank you for the nice content and it's very well explained.

[@jovianhq]

Glad you enjoyed it!

[@ShivamS01]

Thank you so much sir.

[@vijayabhavani6103]

thanks for the tutorial very informative and clear for beginners

[@jovianhq]

Glad it was helpful!

[@gauravrai5784]

I understood everything, thank you

[@jovianhq]

Glad it was helpful!

[@ai.simplified..]

Thx, good one

[@sonukumarjagylan]

superb...it was very confusing for me earlier but after watching your video regarding scraping i have completed this project in one go...carry on plz upload more videos

[@jovianhq]

Great 👍

[@PabloSanDev]

Thank you Genious!

[@jajatisahoo3831]

Awesome ...your way of explanation is very good.

[@jovianhq]

Thanks a lot 😊

[@durgesh1227]

Sir you are really a great teacher. Thanks for the course

[@jovianhq]

Thanks for the kind comments.

[@RamandeepSingh_04]

Thank you so much Jovian Tutorial was great and will surely help in my career i am grateful to you for spreading your knowledge to everyone for free. if i succeed in my career, i will surely spread knowledge for free

[@ayaongaming7670]

TNice tutorials man is so good at explanation ...since I have been watcNice tutorialng soft soft tutorials , tNice tutorials is my first ti to really understand it . I love soft

[@Pure_Hadith_Moments]

Really great work.

[@jovianhq]

Glad you think so!

[@eugene5907]

You are the best!

[@maamounhajnajeeb209]

very good job

[@chsaarth]

I heckin love this guy!

[@jovianhq]

Thank you. We love you too!

[@mohammedmunavarbsa573]

awesome

[@nisittripathi7863]

Thanks for your videos sir

[@jovianhq]

You're so welcome!

[@raghavendrasaisurisetti4745]

I didn't get the div tags(in my case) output as u got for p_tags

[@riahjaklich5691]

i am trying to scrape the techpowerup gpu and cpu specs list. all the specs in the table have the same "td" tag, with no class. How do I find them?

[@Matter_Of_Facts484]

even after following all the steps and syntax correctly for me the length of topics tags is showing zero. requesting you to help me in solving this

[@mkgeidam]

Thank you Aakash for the video. Keep it up!!!

[@jovianhq]

Glad you like it!

[@kamalgopalsingh244]

I am doing the same as u did but not able to save page_contents in webpage.html ,showing error ie cant encode the character

[@sanjubell]

Hi,I have an project to Crawl(URL) and Extract new entries from HTML ie(Title,Subtitle,Abstract and Download time) and store these entries in a database and the crawler should be triggered to run every 15 minute,finally During reruns no duplicates to be entered.can someone sujjest?

[@user-ev2ye6qe3j]

What to do when I have to web scrap multiple urls?? Please tell me.

[@rahulbaksariya1768]

Sir instant data scraper se bhi kr skte hai web scraping...

[@NSASANAPURIKAVYASRI]

sir, why we are using --upgrade and --quiet while installing libraries?

[@akashpd2113]

It upgrade to hightest update available and quiet will not produce any output while using pip

[@priyanka76pa]

I am getting an error, pandas has no attribute as DataFrame, can you help me with it ?

[@RohanOnBike]

To all the guyz who are thinking of running the scrapers on a scheduled basis, then this most probably wont work ( Jovian team please correct me if im wrong) you will need to setup a cron job, now that will work, but thats not a production level solution. You can create a django project which can schedule the tasks with celery beat, and save the data to an rds. Celery flower will provide you with a dashboard through which you can see the results of scheduled tasks. And you can obviously use any BI tool to analyse the data you scraped. We have implemented this, have more then 25+ scraper servers running 24x7, works like charm.

[@jovianhq]

That's right!

[@minutespeace]

Great 👏👍

[@jovianhq]

Thank you! Cheers!

[@mysterious_minder]

Sir can we get PHP code of a website by web scraping

[@curiousful]

Very nice session..I am trying this for IPL teams.. thanks for this Effort..👍

[@jovianhq]

Glad you liked it!

[@tommycravens8161]

Jupyter notebook keeps deleting all of my work. I lost like 2 days worth of work and I am unsure how to retrieve it. I was hitting save and check point after every line, but I don't know what happened to everything.

[@CricketGuruji15]

Nicely Explained but very Confusing

[@TheWorpler]

Nice tutorial. I just started a project where many page or data items are missing, quite a challenge.

[@jovianhq]

You can do it! 👍

[@ByteAdvocates]

Run on binder is not working

[@sumanabiswas3091]

Thanks for the nice tutorial. Just want to know, does the procedure applicable to scrap data from a web chat platform.

[@jovianhq]

Mostly no! As big chat platforms are highly secured, you might have to use some API or gather the data directly from the chat platforms

[@dannybuntu]

I'm just following the web scraping tutorial. Logging in / Signing up to Jovian, makes me "Start an Application" for a DA and ML course?

[@jovianhq]

You can click on sign-in on the top right of the website and sign-in with your Email. For more courses, check out: jovian.ai/learn

[@thenewway416]

Nice👍

[@jovianhq]

Thanks ✌

[@kdalan894]

This video is really helpful! I am doing a webscraping project in another website by using this video as reference. When running the final function, some topics are scraped perfectly but some topics showing IndexError: List index out of range error. Could you please help on this!?

[@jovianhq]

Seems like there are some missing values in some tags. I guess you are making a dictionary of lists at the end, you can make a list of dictionaries instead. Find each block of the item/product you are scraping, save the values in a dictionary, each dictionary will have details of one item/product. Make an empty list and append the dictionaries in the list. By creating a dictionary you will be able to track whether you have any empty values in an item or not.

[@kdalan894]

@@jovianhq Thanks for your response😊 It helped. Now I can able to scrape all the topics successfully

[@pythonenthusiast9292]

Sir , can beautiful Soup do everything that a scrappy can do ? or is there some use cases where scrappy is the only option? if so , then can you give one example?

[@AakashNS]

If you want to download an entire website with many different types of pages, then Scrapy makes more sense. You can do it with beautiful soup, but it will require you to write much more code. On the other hand, if you're just looking to download one page or a few pages of the same type, you can do it easily with requests and beautiful soup. Using scrapy is probably overkill in such a situation. In general though, don't worry too much about which library to use, just start working on a project and you'll figure out everything along the way.

[@jovianhq]

You should also look into Selenium for web scraping. We have a tutorial on it on our channel

[@noyesamuel4410]

thanks Jovian, have issues defining the functions TypeError: 'builtin_function_or_method' object is not subscriptable. this is error it's giving

[@jovianhq]

Hey, can you please post the error on Jovian Discord server with screenshots of the error and code. Jovian discord server link: discord.gg/xHGCtbUypy

[@pankaj2253]

I m trying to scrape article details like author comments, votes and etc but some of them are missing so can you explain plz how to handle missing values

[@jovianhq]

You can make a dictionary like {'name': value, 'author': value, 'votes': value, ...} for each data(books) and make a list of dictionaries. Wherever there is a missing value, you can add None to that. This way you will get a list of dictionaries where each dictionary contains data of one row. Now, you have to convert the list of dictionaries into a DataFrame. If you face more issues please post your issue on jovian.ai/forum.

[@abhishekchandra6229]

Hi Bhaiya, I got the tag, by adding one more .parent topic_title_tag0 = topic_title_tags[0] div_tag = topic_title_tag0.parent div_tag.parent.parent This code is giving the a tag for topic link. :) 53:30

[@shyamss2338]

Hi can you perform EDA analysis using python on this scraped dataset?

[@jovianhq]

Yes it can be done, but you would need to pull more data out, be aware that you're not getting blocked by the website you're scarping.

[@Heyitsvaishnavi]

Hello sir I really loved this video and your explanation is perfect. But i just have one error in the 'GETTING INFORMATION OUT OF A TOPIC PAGE' section For some reason the len(repo_tags) is showing the output as 0 Due to this i am not able to proceed further. It would really be helpful if you could help me solve the error. Looking forward for your reply. Thankyou 😊

[@lylabata6813]

I am having same issue....did u solve it?

[@elisapfau8563]

Thank you so much for uploading this and for all of the effort you put into teaching this subject! It was very, very helpful. One odd thing: When I committed my code, it took just a split second and then all it said was "Committed successfully!" ...but my .CSV files never got uploaded to the 'Files' page on Jovian. Do you know why this might be? Thanks again for everything you've done here!!

[@jovianhq]

Glad it was helpful! To upload files in Jovian, you'll have to manually enter the file name: jovian.ai/docs/user-guide/attach.html

[@manojstatistics9600]

HI Nice video, can you please do a video of extracting the released tags with dates. It will be great helpfull for me. I have tried but full of errors. Please help me with this topic.

[@jovianhq]

Hey, can you please post your doubts on Jovian Discord server: discord.gg/xHGCtbUypy

[@heyrobined]

Hey Akash sir, Very nice video today is my first day as a scraping student so I am scraping a website the page has this kind of tag >>>>>>>>>>>>>>> NORAD ID : 48832 >>>>>>>>>>>>>>> here I understood that by selecting the tag I can get the "Norad ID" but I also want the value after it that is ":48832" the value doesn't have any tag I have successfully scraped other things but these values doesn't have any tags how to solve something like this?

[@dailydozze]

Hi, I want to extract data for specific keywords, like from LinkedIn or Instagram. How to go ahead with that.

[@pythonenthusiast9292]

+1 also how to know if scraping some websites wont lead to any misuse ? how to ensure that ?

[@AakashNS]

Try to come up with a strategy by following what we did in the video. If you face issues, post them here and we'll help you out.

[@AakashNS]

@@pythonenthusiast9292 Check their "Terms of Use" page and write to their support email. In general, scraping for commercial purposes is disallowed.

[@jovianhq]

Scraping LinkedIn might be difficult without logging in.

[@yashnenwani9261]

One query sir, In my project i need to search for a particular thing and then scrape related data. How can i implement the search-bar of a website.

[@jovianhq]

You cannot scrape Dynamic websites using Beautiful Soup. You have to use other libraries like Selenium or Scrapy to scrape Dynamic pages.

[@jovianhq]

Also sometimes it can be done using the correct key in the urls, For example in googles site you can search using the key /search?q=text "www.google.com/search?q=what+do+I+want" . In a website if you find a key like this you can search by changing the key value easily.

[@abhijeetvighne4452]

Hi, Can make tutorial on self build web scrapper i.e without using beautifulsoup so we can understand how beautiful soup works and how to code for it

[@jovianhq]

Beautiful Soup is pretty easy to use and understand, it just parses the downloaded HTML page into a well-decorated format, the codes are self-understandable. Please go through some tutorials/documentation to understand what the code does. You should be able to understand it without much effort.

[@Afri_Data]

Please i have a question is it only the class that can be specified?

[@jovianhq]

No there are multiple ways to find an element, class being the most commonly used. 1. Find by tag: 2. Find by attribute: 3. Find by class: 4. Find by Id 5. Find by CSS selector: 6. Find by text content: 7. Find by using a function: You can pass a function as an argument to filter elements based on custom conditions. 8. Find by chaining methods: You can chain methods to find elements nested within other elements. Please do google search to know more about each of the above method.

[@Afri_Data]

@@jovianhq learnt all possible way except for chain method. Any reference as to that

[@ricardoparedes4080]

How’s it going?

[@sathwikaburgula8816]

Where is the Documentation.

[@sahasp.c206]

Can you do a tutorial to scrap data using Jsoup java library

[@jovianhq]

Sorry, we don't cover Java on this channel. But you can check out our tutorial on Selenium.

[@gurpreetsingh-lu3ob]

Sir, my name is Gurpreet Singh and my question is that i want to scrape a web data. The particular data is in Base64PDF format and the language of data is PUNJABI.Is it possible?

[@jovianhq]

That might not be possible with the tools covered here.

[@_newideas]

@@jovianhq Sir could you suggest any other method or tools?

[@technicalabhi7683]

How to scrap mobile app data

[@chandranbose5804]

Hi Aakash sir u r amazing tutor. i need to join data science project oriented course

[@jovianhq]

You can check the courses offered by us, all the courses have a Final Capstone Project at the end of the course. The courses offered by us have helped many to build/improve their skills. The courses offered by us are given below(More coming soon) Data Analysis with Python: zerotopandas.com Deep Learning with Pytorch: zerotogans.com Data Structures and Algorithms with Python: pythondsa.com Data Analyst Bootcamp(6 Months & Paid): zerotoanalyst.com/

[@chandranbose5804]

@@jovianhq last course too costly.an ordinary indian man cannot afford this much

[@rahulsahni1258]

Is it necessary to have the knowledge of html to do this web scraping project?

[@jovianhq]

Yes, a bit. If you know what are tags, ids, names, attributes and DOM you should be good to go.

[@shatakshivishnoi916]

what could be the future scope of this project?

[@jovianhq]

There could be many, first and foremost can be doing EDA on the data you've collected. You can also save it for future and gather enough amount of data and do some ML problem on it.

[@keshavgarg9559]

sir i am doing web scraping of xillow but getting verify you are a human what to do?

[@jovianhq]

You might have to use selenium for this, please see this tutorial: kzbin.info/www/bejne/fJS6XnSOqM6leKc

[@keshavgarg9559]

@@jovianhq thank u

[@hammadfaheem5979]

Woww, Very good tutorial, Love from Pakistan

[@pawanaryan93]

1:04:14 explanation