Beautiful Soup 4 Tutorial #1 - Web Scraping With Python

[@dariyababumalapati7144]

The 'text' argument is changed into 'string' in Beautiful Soup 4.4.0.

[@DetectiveConan990v3]

yes thank you

[@IanWeingardt]

thank you so much, I was very lost when I got the "DepacrationWarning"

[@parvpaigwar2925]

@@IanWeingardt It appears that the content might be dynamically loaded by JavaScript in amazon website, which means it might not be present in the initial HTML response

[@jimstand]

SO I am writing some software to start a business. I am scraping 25 web pages. I hacked through the first 20. The last 5 were difficult so I tried using BS4 with this video. Using BS4 made the last 5 easier than any of the first 20. Thank you Tim!!

[@adnanpramudio6109]

I started learning python few months ago and chose web scraping as my specialization. Your selenium playlist is fascinating. Thanks Tim

[@mihailmilenkov6223]

Hey how did you progress?

[@AliAhmed63708]

r u currently freelancing webscraping ?

[@alex59292]

@@AliAhmed63708 i am

[@hjvela1907]

@@alex59292 So where can I reach you for some webscraping freelancing.

[@japhethmutuku8508]

@@hjvela1907 hello do you still need a web scraping freelancer?

[@Recklessness97]

Subscribed. The last 4 minutes of the video is exactly what I needed. The Soup tree structure part, specifically dissecting the price out of the HTML code. I could get the price on my own web scrap script but it also came with a bunch of other "junk" that was apart of the "tree". Thanks for pointing me in the right direction and explaining how it works!!!!!

[@dbstudio7859]

def amazing(): while 1: print("Thanks Tim") amazing()

[@unpatel1]

I was puhsing learning web scraping for some time now and finally jumped in today and watched my first video on this topic. I like Tim's videos because they are simple and easy to underatsnd, so I decided to go with his video on this topc. Thank you.

[@kristaandrews3405]

I'm using Anaconda, so had to use different import information. You explained this better then any video I've watched.

[@七人の侍-b1q]

"Dummy html file" The html file who is trying his best: 😿👍

[@MrBobman82]

Tim I just started scraping with BS4 THANK YOU!

[@sampsondzameshie-sb3ek]

Hi, l love all your videos boss. Thank you very much. I do not have an IT background but fell in love with your videos and started studying Software development in school right now.

[@tanmaypatel4152]

Man I was literally looking for a good tutorial on Bs4 and guess what Tim read my mind. Thank you very much Tim :)

[@BB-si6cz]

And I started with web scraping like 2 days ago

[@tanmaypatel4152]

@@BB-si6cz Oh that's cool !

[@Damientrades]

Deffo KZbin AI reading your mind maybe it was Alexa

[@tanmaypatel4152]

@@Damientrades I was already subscribed to Tim so I got the notification :)

[@melodyparker3485]

I'm pretty sure that Corey Schafer also has a good tutorial about beautiful soup.

[@Mallan_]

Many thanks. I was struggling with scraping some links from a page but couldn't until I watched this video.

[@garybenhart]

Unfortunately, the code mentioned in the video at 13:15 no longer seems to work, probably because NewEgg no longer allows a Python script to download the htlm from web site pages. It seems to me that most web sites are being "bot protected" today, a problem that is specifically mentioned by Tim in the video at 11:25. This points to a very significant problem when you consider using a tool like Python to web scrape, because using standard Python code is not ever going to work. Finally, when you do get lucky and get your Python code to web scrape, that code that works perfectly today will probably not work very long.

[@AsuGhimire]

real, its a struggle to learn when you're trying to debug and its just privacy policies in your html files xD

[@igordc16]

Straight forward, simple explanations , easy to follow. Thanks Tim! You're a excellent teacher, keep up the great work you're doing here on youtube.

[@wlqpqpqlqmwnhssisjw6055]

I am good in Bs4 But I just came to give you like .For your work

[@khiryshank4930]

Anybody else having problems with bot protected sites? I finally got it to read on Wikipedia, but other websites return an empty string.

[@matrix26uk]

1 quick point to add about BS4 not installing. Sometimes being connected to a VPN can stop modules being installed. Try dropping off the VPN and running Tims install commands

[@hydrocrazynik76]

Such a great tutorial! I usually don't comment but this was absolutely spectacular. Thank you so much!

[@nightwind132]

god that 3080 price gave me stress of when I was hunting down my own. Great tutorial btw it's been a great help!

[@BonVoyageWorld]

you should have more than "just" 1,18m subscribers. thank you Sir!

[@acutisnasus7217]

8:26 Oh nooo,... you're in the matrix. You glitched!!! Top tutorial!!!

[@namename-cl8kk]

Finaly the best timing ever i was waiting it plz speedeun that series

[@Spleed7887]

Dude, I think you should do more C++ tutorials. They're really good!

[@elpython3471]

I second this. Those tuts are good!

[@loisvallee7291]

need this to access my uni's timetable more easily, thanks man !

[@TechWithTim]

Glad I could help!

[@philippededeken4881]

Great video. Thanks to you, I'm starting a new business in the tyre industry.

[@oskarwallberg4566]

Beautiful video man! Just realised how pedagogical and well dispositioned you videos are.

[@keifer7813]

8:25 It's always fun seeing bloopers mid video lol

[@GeneralCA-k9l]

i show this video after two years thanks pro❤❤

[@mmbaguette1520]

Hey Tim, can you make a video on how to get a programming job? 👋

[@selo2410]

THANK YOU, I've been waiting for you to make a tutorial on this for some time now, thanks again.

[@neroplus-it]

your videos on web scraping motivated me to create my own video-series about this topic(s)! as always, great content! thanks for sharing your knowledge.

[@davevanemmenes27]

Congrats on your 1 million, All the best

[@Said664016]

The best tutorial ever! You're saving my life!

[@PeterPankowski]

Excellent done for a first example! Amazing explained!

[@derelictmanchester8745]

Love your channel Tim, the best tutorial ever..

[@toshitsingh7270]

As always your tutorials are super educational and also thanks teaching it for free, it really helps.

[@as_below_so_above]

Great video and great timing to put it out! I had to use BeautifulSoup for the first time just last week and this was great at solidifying everything I learned!

[@prodigyprogrammer3269]

8:23 did you forget to edit 😂😂 love your videos BTW ❤️

[@popey747]

Wonderful to be learning Beautiful Soup with Kermit

[@anwar587]

Web scraping is very useful trust me and of course beautifulsoup is the best library for this

[@FreAcker]

hey, just updating. find_all(text=) is deprecated switch to string= method instead;)

[@abdulkadirosman2816]

thanks, but it still doesnt work for me

[@Khyreemlb]

Amazing stuff man. You got yourself a new sub. Thank you for all of the content and hard work. I've been benging all of your videos like I was watching Netflix lol

[@tieutantan9562]

This serial is my need. Thank Tim!

[@thec-m]

This was a really useful tutorial and it was clear to understand, unlike some of the other videos I found. Thank you! I'm sure there are many people out there like me that find themselves trying to slightly improve their code, resulting in learning how to use some new massive python library like this. Back to the video: I think it would have been good to replace the URL at the end of the video with another NewEgg listing to show the same code extracting a different price (assuming the tags are the same). Also, looks like you forgot to edit out the part at 8:24.

[@ChrisOfTheOutdoors]

Anybody know why I would be getting "IndexError: list index out of range" on line 10 - "parent = prices[0].parent" at the 15:29 minute mark in the video? I've copied the whole code exactly.

[@abssdabss]

make sure your url is correct

[@markslima1557]

Thank you this video is so straightforward I think I finally got the hang of this

[@prof.code-dude2750]

I wanted to create a BS4 project 😀 and you made a tutorial

[@friday8118]

How do we input the html or the website we want to scrape? Great video, thank you.

[@tildesarecool7782]

I was following along with this video and couldn't get it to work. Actually I was following along but with my public "all games" steam library page. I couldn't figure out why it wasn't work. I was losing my mind. Then I finally saw in the source this JavaScript block with formatted data for all my games. It's "DB Query" and also the JS appends the data to the DOM programmatically. So indirectly this video taught me why Beautiful Soup couldn't find the tags I kept searching for on the steam library page. Side note, anyone want to scrape their steam library for some reason (instead of using steam db or whatever) it's all there on that page as some kind of JSON. Good video btw.

[@extropiantranshuman]

the camera angle alone is increasing my intellect

[@wege8409]

This reminds me of how some nights Grandpa and I would eat melty cheese in the mudroom. We laughed so much as cheese dripped down his face. I can still remember his laugh. It sounded like a hundred murders of crows filtered through a ring modulator. RIPO Grandpa please stop haunting my dreams.

[@ayaanp]

I think Tim can read our minds 👀

[@rahulxdd]

Thank you Tim. I always wanted to learn Beautiful soup for personal projects but never did. Today is the first time I watched a tutorial on this topic. Anyway, how long will this series be? Can't wait for the next part.

[@AgGh-c5s]

Nice video but please, can you also make another video teaching how to scrape app or apk?

[@renecro1007]

Thank you! Now I can grab dad jokes from 20yo websites and send them to my friends!

[@RominiyiOmodona]

Fellow man of culture

[@proxyscrape]

Great tutorial Tim! I appreciate the clear and concise explanations you provided.

[@Zydres_Impaler]

Tim, please make a series or video fo the "requests" library.

[@mghostdog]

So when I run the script looking for the "$" on the site I'm parsing, I get an empty list [ ]. Does that mean that the website is preventing me from seeing that particular item/price?

[@intelblox7354]

im getting a index error when i put prices[0]

[@ethicalhacker9720]

I think it is the website. I tried another website and it worked.

[@ezekomaugoo5569]

Quite a very concise course and informative. Thanks for this guide.

[@hmodexl]

ur explanations are very clear,thank for ur effort.

[@Tim_Brueggen]

For me doc.find_all(string="$") doesn´t work. I have to search for the full string in the div. Can someone help?

[@Knuddelfell]

exactly needed this

[@AmirRTR]

best guy on yt

[@keifer7813]

8:09 Isn't nesting tags in HTML impossible? This part got me confused Also at 16:12, couldn't you just use parent.strong instead of parent.find("strong") ? Great video by the way

[@josepholiver5713]

I am running into this same exact issue. Not sure what to do and can't find a stack overflow forum that's helpful

[@jacobfuller5643]

super helpful for a project I am working on, thanks!

[@tomasoon]

Great video, but the most impressive thing is when you did this video the video cards price was $2613, and now it's $1549 in less than a year xD

[@mijimelon]

mr suhardi sent me here

[@b07x]

Thanks, this was easier than I thought

[@ScriptureFirst]

outstanding walkthru, as usual, ty... I like the chapter divisions, concise talking, maximized screen, text size :)

[@greening6904]

Tim you wont believe i was working on a meteo app and needed a parser thx

[@softicecreamer]

im looking for graphics cards and tim mentions them. wow its like u read my mind

[@Popcorn_and_funny_moments]

hey tim great work , i need to learn how to do using python columns and boxes in visual studio code thanks very much .

[@simple-security]

well played sir...well played.

[@JanBadertscher]

Tried 3 BS4 tutorials, on 2 completely fresh environments (one native py3 the other one a jupyterlab environment) and find_all() always returns empty. Any ideas why this happens?

[@ivanyosifov2629]

If find_all returns empty array that means what you're looking for is not in the document

[@Xero_Wolf]

@@ivanyosifov2629 I have the same issue and what I'm searching for is in the document. Even when I test with a simple html.

[@ivanyosifov2629]

@@Xero_Wolf It might depend on the editor you are using. For some editors you need to give the file path as */index.html* or *./index.html*

[@camplays487]

@@ivanyosifov2629 For me, the .find("strong") returns NONE even though the print statement before it clearly shows strong tags, any idea what could be causing that?

[@_n1c0l4s]

I am using the find_all(text="something"), and it also returns and empty array... I know that what I am looking for it actually is in the document. Could the problem be something of how the html file is structured?

[@chukwudifrancisawulor883]

Thanks Tim 🎉

[@RandyWatson80]

As always, this was super clear

[@jalepenofatty6704]

great video , hit a bunch of roadblocks with the imports and environment %PATH% changes i had to make, and then the openssl issue, but yeah took me a day to get thru this and finish, i appreciate the hardwork.

[@unpatel1]

Glad that you finally solved your problem! Me too, had a hard time with the %PATH% thingy!@$$. Time to time %PATH% problem appears from no where and eats up lots of my time. I have worked a little with R and I found it relatively simple and easy in this aspect, package installation and management etc...

[@andrews9168]

@@unpatel1 another workaround is to use pycharm

[@unpatel1]

@@andrews9168 Thank you for your suggestion. I do have pycharm but not using it, just use VS Code all the time. I will definitely try pycharm.

[@julianaschmidt1059]

So useful! Thank you so much!

[@lucaskellerlive]

Do you have an availability if I paid for you a Zoom call? I watch your videos all the time and I'd really appreciate if I could hop on a Zoom at your hourly rate to answer a few specific questions. Thanks for everything!

[@lucaskellerlive]

Do you have an availability if I paid for you a Zoom call? I want your videos all the time and I'd really appreciate if I could hop on a Zoom at your hourly rate. Thanks a million!

[@lucaskellerlive]

Yes indeed. Have email?

[@arjix8738]

just do "pip install bs4" its a shorthand

[@learnwitharbia3477]

Thank you so much for such valuable content

[@WasimAkram-of9iv]

Hi, This is very interesting, Thanks for sharing. I have one question, Can we scrape website industry? Like Any site which belonging to any category like, Automotive or healthcare etc.

[@romanv4519]

Awesome tutorial. New to this channel, but I like your style Tim. Thanks a lot, very well explained!

[@SnedzTheBricklayer]

Well that's my Wednesday night planned out 😅

[@zawadahmed5484]

Keep on your beautiful contents

[@SmeeUncleJoe]

I tried your code but on a different webstore website. Definitely has "$" signs on the page and i was, with modification able to print the whole HTML code out via prettify, but I get this error : "IndexError: list index out of range" on the line "parent = prices[0].parent". Any ideas ?

[@pranavchopra_0075]

I am having the same problem, I even inspect element copied the $ sign from site incase there was unicode problem but it still didn't work

[@CrawdadSoftware]

maybe website have bot protection?

[@Will-fh9fj]

Nice, Tim. I mean, nice.

[@siamahmed8287]

Can you make tutorial how I can scrape a dynamic web page? Like built with react.?

[@rog_shakhyar6171]

it would be same

[@omarciano42]

That would only be possible with Selenium, which Tim has a series on, just search it

[@mahirrahman4253]

So... Selenium vs Beautiful Soup which one would you go with?

[@anwar587]

Beautifulsoup is the best and easy to use but if we want to talk about selenium is for automation not for scraping but u could still use it... in my opinion of course

[@hades5444]

Go with selenium if you want to automate stuff. If you just want the data then BS is a fine choice

[@mahirrahman4253]

Thankyou for the replies :)

[@filmedbyjulia124]

I liked this video, good content.

[@THISISCHARISMATIC]

Absolutely great videos, I’m new to python and coding in general. Your content is really great and easy to follow, would this web scraping method work for finding stuff like meta data for songs ?

[@mayem2915]

i keep getting IndexError: list index out of range

[@almaghror1]

Thanks Tim

[@RonaldPostelmans]

Hi Tim, nice video, need stuff. have you any links to a tutorial of you or else someone, who has scraped websites that block scrapers?

[@AC-he8ln]

8:28 who else rewinded there

[@guy6567]

Thanks Tim! :) awesome and helpful

[@gvikram18]

Could you do a video series on pywinauto for automating windows applications?

[@scottdol2099]

Great stuff as usual! What's the schedule for the next episodes?

[@pokedreadhead6089]

So sick thanks for the video!