Easy Web Scraping in Python using Pandas for Data Science

[@KenJee_ds]

I didn't know about this pandas functionality! Great video!

[@DataProfessor]

Wow, it's Ken Jee! Thanks for the comment and kind words! I also subscribe to your channel, great content by the way, especially the 6-part DS project from scratch series.

[@KenJee_ds]

@@DataProfessor Thanks! I am loving your stuff as well. I need to start using colab more. Keep up the good work, the tutorials are very helpful!

[@karthiavenger4577]

You great bro Down to earth

[@HVjugo]

I used this before, but I didn't knew that you can select the table using the brackets, awesome! Thanks for the video!

[@DataProfessor]

Glad it's helpful, thanks for watching!

[@monicadesai7928]

Great Explanation of each step....right from opening file to end....because sometimes as a newbie we find difficult to which file to use from github also.....Thank you ....Great Video!

[@DataProfessor]

Wow thanks for the encouraging words, glad you’ve found the video helpful 😊

[@givansot4581]

thanks a lot. I am doing a machine learning project and do web scraping in the same code...thanks this is better

[@TcRiverrat18]

Excellent work breaking this down. I have only used R, but this seemed incredibly intuitive. Thank you!

[@soufianelamsiah4337]

what would be best for comparing prices between competitors?

[@usmanafridi9668]

Amazing! I am totally new to web scraping. I tried to scrape the website using beautiful soup library for 4 days now, but I can't get past the basics. You have extremely simplified it for me. For instance, I just scraped data from Wikipedia about the list of countries and their population and got the whole table in the first attempt. Thank you so much! I wonder if this can be used for other pages like LinkedIn, Glassdoor data collection? Because there are no tables there. Professor, thank you so much once again!

[@DataProfessor]

Glad to hear that the video was helpful! For non-tabular pages you may have to use beautifulsoup and/or selenium

[@mj7146]

Great content ! Any idea on how I can scrape data for example from linkedin Jobs Postings. I found Octoparse for this, any ideas?

[@DataProfessor]

Thanks Mert for the kind comment. pandas works only for tabular data from webpages. For linkedin posts, we'll probably have to use beautiful soup for that. I might make a future video about that, will put it into the to-do list.

[@mj7146]

Data Professor thank you 🙏

[@DataProfessor]

@@mj7146 A pleasure!

[@oguguaonyinyechi4980]

@@DataProfessor Hi Data Professor, we are still expecting this :grin:

[@shwetaredkar734]

Informative.

[@DataProfessor]

Thanks Shweta for the kind comment!

[@badraboufirasse433]

Very helpful thank you!

[@DataProfessor]

Thanks Badr for the kind words!

[@jojushaji3010]

Ure awesome sr

[@DataProfessor]

Thanks for the kind words

[@Troglodyte2021]

A great tutorial!

[@DataProfessor]

Thank you!

[@cllim80]

Thank you for the clear explanation !

[@DataProfessor]

A pleasure! Thanks for watching 😃

[@kalyanprasad4069]

How do we deal when we encounter the error "HTTP Error 403: Forbidden" while reading url with Pandas? How should we proceed in this case? Kindly advise.

[@blankmedia01]

Hey I tried using the code on Wikipedia to scrape tables on Wikipedia. When it comes to scraping on place with loads of other data and i just want to pull the table alone is there a method for that? As with current code im pulling whole page. And I just want the playoff stats... i think I'm supposed to creat dictionary then assign it to a dataframe but I dont know how when it comes to urls and websites.

[@aniwahidaabdulrahim2538]

Hello Professor, I would like to suggest you to publish a video about RSelenium which use with Selenium Webdriver for automation system testing :D Hope it may benefits others. This is just my humble suggestion.

[@DataProfessor]

Great suggestion! I have played around with Selenium for Python and have found it pretty powerful. What I made so far was a short script that can take screenshots of my youtube channel's page (or any webpage).

[@randyluong6275]

this tutorial gets my subscription. Thank you Professor. :)

[@DataProfessor]

Wow, glad to hear that, welcome aboard 😃

[@rogerwprice]

Fabulous - it's soooo easy when you know how!

[@DataProfessor]

Thanks for watching Roger, absolutely agreed with that 😃

[@da_ta]

Great well explained clear and excellent quality of sound. Thanks for doing this keep it up!

[@DataProfessor]

Thanks for the encouragement 😃

[@nourarifi2642]

thank you for your video my question if there are many tables in so many pages (20000 page) what should I do ???

[@DataProfessor]

The pandas read_html function is suitable for a simple webpage with relatively few tables. For more complex and large volume of pages I would recommend to look into beautifulsoup and selenium.

[@luciferkhusrao]

Awesome work by the hero! Keep teaching like this

[@DataProfessor]

Thanks for the encouragement 😃

[@pauloreis8868]

Hi, Professor! Thank you for the contents you brings to us, it really helps! \o/ Lately, I've been asking myself: How important is web scraping for a data scientist? How often do you web scrape? I just started learning it, I'll keep going and I wanted to know your thoughts about its relevance.

[@DataProfessor]

Hi Paulo, webscraping comes in handy when you want to create your own dataset from available data on the internet. For example, you want to analyze the salary of data scientists from glassdoor database then you can do that with webscraping. Hope this helps 😃

[@kennykern6292]

This helped thanks!

[@DataProfessor]

Glad it helped!

[@muhammadjamalahmed8664]

Please don't stop making videos. These videos really helps alot.

[@DataProfessor]

Thank you, glad it was helpful!

[@vyacheslavgorkunov3790]

Thx for the video, was really helpful. I wish u more subscribers, man ;)

[@DataProfessor]

Thanks for the support! 😃

[@salikmalik7631]

Really awesome.. Data Professor

[@DataProfessor]

Salik, Thanks!

[@melshae8630]

Wow your video is the best , it took me forever to run this .This video helped me in 5 min. Thank you !!!

[@nickolaisimmons4638]

Wow this is a great video! Very well organised!

[@prashant381]

A query, in row 12 , why are we using .index along with df.drop ? why wouldn't df.drop work without it ?

[@kwanpakshing]

The video is great. But the screen text us way too small to read. Suggest that you can enlarge the font or reduce the white space in the screen to make the video no e readable

[@DataProfessor]

Thanks for the suggestion, greatly appreciate it, yes in recent videos I have increased the font size.

[@engr.inigo.silva2000]

Bravo Data Professor, nice lecture!

[@shankaricharan510]

Thanks a lot - this helped a lot.

[@fazlaynur4509]

Thanks bro, for your nice tutorials

[@DataProfessor]

It's my pleasure

[@manishabheemanpelly3580]

Thank you so much for this concept it was really time saving one!

[@DataProfessor]

Glad it was helpful!

[@amoahs7779]

Hi professor I truly enjoy your videos and have learnt a lot may God keep you successful in life. A question that's been on my mind is what laptop do you use as I really like the keyboard sound when you type unless you are using a external keyboard. Is it possible for you to show us a set-up of your desk ? Kind regards

[@DataProfessor]

Hi, I'm using a MacBook Pro (2016) and yes the keyboard feel is good on this laptop although being a bit flat which is a good thing as it allows minimal effort in moving from one button to the next.

[@danniliu2544]

Hi Data Professor, thanks for this video. It's very helpful. I'm a newbie starting out in data science and web scraping. Just wondering can you use pandas functionality for scraping data that are not laid out in table? and how would you do that? could you perhaps create a video on scraping non tabular data if you haven't already?

[@DataProfessor]

Great question, to web scrape non-tabular data you can look into using beautiful soup and also selenium libraries for Python

[@danniliu2544]

@@DataProfessor thank you for the pointer, much appreciated!

[@Panucci75]

Exactly the question I was gonna ask. Thanks.

[@sangpark7656]

Hi Professor does the original data need to be a html file to start with? Does the original data always need to have a table to extract data?

[@DataProfessor]

Yes to both questions, that’s the limitation of this approach. Other than that selenium + beautifulsoup is a good combo to look into.

[@sangpark7656]

I see. Thank you very much for the guidance!!@@DataProfessor

[@Moonlight-jx2sj]

Amazing! your video helped me with my 1st homework in Data Mining. And also thinking to jump into data science, so Thank you so much! Like and Subscription!

[@DataProfessor]

Glad I could help! And welcome to Data science!

[@wisjnujudho3152]

this is exciting. i love pandas

[@legacylifey182]

Thank you so much for this concept it was really helpful respect !

[@spacebird9430]

hey professor, thankyou for the content. but i was wondering when we are scrapping by just passing the link how does it know to only read data from the table and not any other information.

[@DataProfessor]

Hi, the function will detect HTML syntax. The syntax for tables in HTML is and the read_html() function finds these to figure out that they are tables and extracts the data.

[@lolsucks3599]

Is there an api for sports results? or you have to do it via web scraping?

[@lucianodomingues2290]

Great video Professor!

[@DataProfessor]

Glad you liked it!

[@RyanLoh]

Can you also use df2019(df2019[‘Age’] == ‘Age’) to find the ages containing the word ‘Age’?

[@narongtumsri-ubol1737]

thank for knowledge

[@DataProfessor]

A pleasure, thanks for watching

[@raphaellutz2693]

Very nice video

[@DataProfessor]

Thanks :)

[@markslima1557]

very cool thanks!

[@sanjj_1]

f strings are more readable compared to the .format() method

[@tannyamishra9291]

Can you please explain how to read all the retrieved urls

[@vaasudhfp2874]

not working for other sites i did it for tripadvisor nothing came

[@priyalshah8869]

How do I keep the url that the coloum tm has in my dataframe?

[@nowdevoted1649]

Superb, let me bring you some more guys to your channel

[@DataProfessor]

Awesome, welcome to the channel!

[@nowdevoted1649]

@@DataProfessor 🙏

[@ekoatm1914]

Matur nuwun sanget sedulur....

[@sameermehdi3143]

Thankyou so much sir

[@AmitKumar-hm4gx]

Do you know if we can use this to scrape sites built with dynamic JS, and how do we do this if we have to login ?

[@piyushyadav7162]

Hi! ken jee, I try your code of web screping on kaggle but I'm getting RLError: error. i try to solve but i cannot resolve ...please give me your suggestions

[@DataProfessor]

Hi Piyush, The pandas library allows scraping webpages that have tabular data such as from Wikipedia. It is really limited to those with a predefined table format. To scrape webpages I'd recommend looking into selenium and beautifulsoup

[@argiepoul7457]

What are the prerequisites to watch this tutorial? I know some python, is this ok?

[@DataProfessor]

Yes, beginner’s level of Python is sufficient to follow along.

[@XoreLP]

Why did you use string.format instead of String concatination

[@Papiii_benz]

Thanks !

[@DataProfessor]

Thanks for watching!

[@harshitsharma8131]

what if there is no table on a web page ??

[@moatasimashraf6818]

(ImportError: lxml not found, please install it) I got this error. what is the solution?

[@DataProfessor]

Hi, you can install lxml via pip install lxml

[@moatasimashraf6818]

@@DataProfessor Done it, thank U

[@lyhuutai3339]

how to save df to excel ? please

[@saulo_foot]

Every link turns into a df. How can I concatenate all the dfs?

[@DataProfessor]

Hi, dfs can be concatenated using the pd.concat() function, you can play around with axis=0 or axis=1 depending on how you want to combine the dfs (side by side or stacked on top of the other)

[@mootaz3944]

i try it on ur channel ( just for testing lol )

[@qi8983]

Awesome

[@tareqmahmud3902]

You look like jomatech's big brother :O

[@DataProfessor]

Haha, I get that a lot. Joma and I should do a collab video 😆

[@tareqmahmud3902]

@@DataProfessor But Sir I learned a week's lesson from one of your 10 minute video. I can't be more grateful to you. Thank you.

[@DataProfessor]

@@tareqmahmud3902Thanks, glad to hear that they’re helpful! 😊

[@alexwatson6370]

Don't name your variables str or you will shadow the string builtin

[@DataProfessor]

You're right, many thanks for pointing that out, why did I do that. I've changed it to url_link now.

[@ishpandey7886]

Is this useful for every situation? I am trying to fetch data from glassdoor but this method is not working Link: "www.glassdoor.co.in/Job/bengaluru-data-analyst-jobs-SRCH_IL.0,9_IC2940587_KO10,22.htm"