Beautiful Soup 4 Tutorial #1 - Web Scraping With Python

[@dariyababumalapati7144]

The 'text' argument is changed into 'string' in Beautiful Soup 4.4.0.

[@DetectiveConan990v3]

yes thank you

[@IanWeingardt]

thank you so much, I was very lost when I got the "DepacrationWarning"

[@parvpaigwar2925]

@@IanWeingardt It appears that the content might be dynamically loaded by JavaScript in amazon website, which means it might not be present in the initial HTML response

[@Recklessness97]

Subscribed. The last 4 minutes of the video is exactly what I needed. The Soup tree structure part, specifically dissecting the price out of the HTML code. I could get the price on my own web scrap script but it also came with a bunch of other "junk" that was apart of the "tree". Thanks for pointing me in the right direction and explaining how it works!!!!!

[@jimstand]

SO I am writing some software to start a business. I am scraping 25 web pages. I hacked through the first 20. The last 5 were difficult so I tried using BS4 with this video. Using BS4 made the last 5 easier than any of the first 20. Thank you Tim!!

[@adnanpramudio6109]

I started learning python few months ago and chose web scraping as my specialization. Your selenium playlist is fascinating. Thanks Tim

[@mihailmilenkov6223]

Hey how did you progress?

[@AliAhmed63708]

r u currently freelancing webscraping ?

[@alex59292]

@@AliAhmed63708 i am

[@hjvela1907]

@@alex59292 So where can I reach you for some webscraping freelancing.

[@japhethmutuku8508]

@@hjvela1907 hello do you still need a web scraping freelancer?

[@dbstudio7859]

def amazing(): while 1: print("Thanks Tim") amazing()

[@unpatel1]

I was puhsing learning web scraping for some time now and finally jumped in today and watched my first video on this topic. I like Tim's videos because they are simple and easy to underatsnd, so I decided to go with his video on this topc. Thank you.

[@tanmaypatel4152]

Man I was literally looking for a good tutorial on Bs4 and guess what Tim read my mind. Thank you very much Tim :)

[@BB-si6cz]

And I started with web scraping like 2 days ago

[@tanmaypatel4152]

@@BB-si6cz Oh that's cool !

[@Damientrades]

Deffo KZbin AI reading your mind maybe it was Alexa

[@tanmaypatel4152]

@@Damientrades I was already subscribed to Tim so I got the notification :)

[@melodyparker3485]

I'm pretty sure that Corey Schafer also has a good tutorial about beautiful soup.

[@sampsondzameshie-sb3ek]

Hi, l love all your videos boss. Thank you very much. I do not have an IT background but fell in love with your videos and started studying Software development in school right now.

[@kristaandrews3405]

I'm using Anaconda, so had to use different import information. You explained this better then any video I've watched.

[@MrBobman82]

Tim I just started scraping with BS4 THANK YOU!

[@garybenhart]

Unfortunately, the code mentioned in the video at 13:15 no longer seems to work, probably because NewEgg no longer allows a Python script to download the htlm from web site pages. It seems to me that most web sites are being "bot protected" today, a problem that is specifically mentioned by Tim in the video at 11:25. This points to a very significant problem when you consider using a tool like Python to web scrape, because using standard Python code is not ever going to work. Finally, when you do get lucky and get your Python code to web scrape, that code that works perfectly today will probably not work very long.

[@AsuGhimire]

real, its a struggle to learn when you're trying to debug and its just privacy policies in your html files xD

[@igordc16]

Straight forward, simple explanations , easy to follow. Thanks Tim! You're a excellent teacher, keep up the great work you're doing here on youtube.

[@Mallan_]

Many thanks. I was struggling with scraping some links from a page but couldn't until I watched this video.

[@hydrocrazynik76]

Such a great tutorial! I usually don't comment but this was absolutely spectacular. Thank you so much!

[@七人の侍-b1q]

"Dummy html file" The html file who is trying his best: 😿👍

[@namename-cl8kk]

Finaly the best timing ever i was waiting it plz speedeun that series

[@selo2410]

THANK YOU, I've been waiting for you to make a tutorial on this for some time now, thanks again.

[@nightwind132]

god that 3080 price gave me stress of when I was hunting down my own. Great tutorial btw it's been a great help!

[@neroplus-it]

your videos on web scraping motivated me to create my own video-series about this topic(s)! as always, great content! thanks for sharing your knowledge.

[@wlqpqpqlqmwnhssisjw6055]

I am good in Bs4 But I just came to give you like .For your work

[@Khyreemlb]

Amazing stuff man. You got yourself a new sub. Thank you for all of the content and hard work. I've been benging all of your videos like I was watching Netflix lol

[@BonVoyageWorld]

you should have more than "just" 1,18m subscribers. thank you Sir!

[@philippededeken4881]

Great video. Thanks to you, I'm starting a new business in the tyre industry.

[@oskarwallberg4566]

Beautiful video man! Just realised how pedagogical and well dispositioned you videos are.

[@davevanemmenes27]

Congrats on your 1 million, All the best

[@derelictmanchester8745]

Love your channel Tim, the best tutorial ever..

[@Said664016]

The best tutorial ever! You're saving my life!

[@GeneralCA-k9l]

i show this video after two years thanks pro❤❤

[@loisvallee7291]

need this to access my uni's timetable more easily, thanks man !

[@TechWithTim]

Glad I could help!

[@as_below_so_above]

Great video and great timing to put it out! I had to use BeautifulSoup for the first time just last week and this was great at solidifying everything I learned!

[@PeterPankowski]

Excellent done for a first example! Amazing explained!

[@toshitsingh7270]

As always your tutorials are super educational and also thanks teaching it for free, it really helps.

[@Spleed7887]

Dude, I think you should do more C++ tutorials. They're really good!

[@elpython3471]

I second this. Those tuts are good!

[@ChrisOfTheOutdoors]

Anybody know why I would be getting "IndexError: list index out of range" on line 10 - "parent = prices[0].parent" at the 15:29 minute mark in the video? I've copied the whole code exactly.

[@abssdabss]

make sure your url is correct

[@prodigyprogrammer3269]

8:23 did you forget to edit 😂😂 love your videos BTW ❤️

[@acutisnasus7217]

8:26 Oh nooo,... you're in the matrix. You glitched!!! Top tutorial!!!

[@thec-m]

This was a really useful tutorial and it was clear to understand, unlike some of the other videos I found. Thank you! I'm sure there are many people out there like me that find themselves trying to slightly improve their code, resulting in learning how to use some new massive python library like this. Back to the video: I think it would have been good to replace the URL at the end of the video with another NewEgg listing to show the same code extracting a different price (assuming the tags are the same). Also, looks like you forgot to edit out the part at 8:24.

[@markslima1557]

Thank you this video is so straightforward I think I finally got the hang of this

[@keifer7813]

8:25 It's always fun seeing bloopers mid video lol

[@tieutantan9562]

This serial is my need. Thank Tim!

[@lucaskellerlive]

Do you have an availability if I paid for you a Zoom call? I watch your videos all the time and I'd really appreciate if I could hop on a Zoom at your hourly rate to answer a few specific questions. Thanks for everything!

[@anwar587]

Web scraping is very useful trust me and of course beautifulsoup is the best library for this

[@matrix26uk]

1 quick point to add about BS4 not installing. Sometimes being connected to a VPN can stop modules being installed. Try dropping off the VPN and running Tims install commands

[@andrealcantara1437]

I'm trying in a different website. I can get the HTML, but when I try to look for specific texts it doesn't work, I always get an empty list, even though I can see that there is that text in the page.

[@labscience8271]

Same problem. Did you find a solution?

[@hamzayunusa2224]

@@labscience8271 did u find one?

[@abdulrahmanal-saadani8769]

I have the same problem but if you noticed in the video he said that some websites may block you when you try to script their html page so maybe the is the reason why you get an empty list

[@DauvO]

@@abdulrahmanal-saadani8769 I have the same problem.. but I think that if the html can be seen in the console in the previous steps, that means the robots haven't done any blocking? I would think if you can see the data that's gameover once you learn how to manipulate it.

[@AnibalDellagiovanna]

For me it only work if you look for the hall test in the element. For ejemaple The full text" will not work for "full" or "The full". It only work if you search "The full test". You can test it with a local HTML file. Is not the web filtering it.

[@popey747]

Wonderful to be learning Beautiful Soup with Kermit

[@prof.code-dude2750]

I wanted to create a BS4 project 😀 and you made a tutorial

[@proxyscrape]

Great tutorial Tim! I appreciate the clear and concise explanations you provided.

[@khiryshank4930]

Anybody else having problems with bot protected sites? I finally got it to read on Wikipedia, but other websites return an empty string.

[@chukwudifrancisawulor883]

Thanks Tim 🎉

[@Knuddelfell]

exactly needed this

[@AmirRTR]

best guy on yt

[@ezekomaugoo5569]

Quite a very concise course and informative. Thanks for this guide.

[@mmbaguette1520]

Hey Tim, can you make a video on how to get a programming job? 👋

[@jacobfuller5643]

super helpful for a project I am working on, thanks!

[@julianaschmidt1059]

So useful! Thank you so much!

[@rahulxdd]

Thank you Tim. I always wanted to learn Beautiful soup for personal projects but never did. Today is the first time I watched a tutorial on this topic. Anyway, how long will this series be? Can't wait for the next part.

[@tildesarecool7782]

I was following along with this video and couldn't get it to work. Actually I was following along but with my public "all games" steam library page. I couldn't figure out why it wasn't work. I was losing my mind. Then I finally saw in the source this JavaScript block with formatted data for all my games. It's "DB Query" and also the JS appends the data to the DOM programmatically. So indirectly this video taught me why Beautiful Soup couldn't find the tags I kept searching for on the steam library page. Side note, anyone want to scrape their steam library for some reason (instead of using steam db or whatever) it's all there on that page as some kind of JSON. Good video btw.

[@friday8118]

How do we input the html or the website we want to scrape? Great video, thank you.

[@pokedreadhead6089]

So sick thanks for the video!

[@keifer7813]

8:09 Isn't nesting tags in HTML impossible? This part got me confused Also at 16:12, couldn't you just use parent.strong instead of parent.find("strong") ? Great video by the way

[@josepholiver5713]

I am running into this same exact issue. Not sure what to do and can't find a stack overflow forum that's helpful

[@simple-security]

well played sir...well played.

[@hmodexl]

ur explanations are very clear,thank for ur effort.

[@ayaanp]

I think Tim can read our minds 👀

[@romanv4519]

Awesome tutorial. New to this channel, but I like your style Tim. Thanks a lot, very well explained!

[@RandyWatson80]

As always, this was super clear

[@wege8409]

This reminds me of how some nights Grandpa and I would eat melty cheese in the mudroom. We laughed so much as cheese dripped down his face. I can still remember his laugh. It sounded like a hundred murders of crows filtered through a ring modulator. RIPO Grandpa please stop haunting my dreams.

[@filmedbyjulia124]

I liked this video, good content.

[@AmbiNerd]

wooo wooo thanks TIM huge help!

[@FreAcker]

hey, just updating. find_all(text=) is deprecated switch to string= method instead;)

[@abdulkadirosman2816]

thanks, but it still doesnt work for me

[@ScriptureFirst]

outstanding walkthru, as usual, ty... I like the chapter divisions, concise talking, maximized screen, text size :)

[@extropiantranshuman]

the camera angle alone is increasing my intellect

[@JanBadertscher]

Tried 3 BS4 tutorials, on 2 completely fresh environments (one native py3 the other one a jupyterlab environment) and find_all() always returns empty. Any ideas why this happens?

[@ivanyosifov2629]

If find_all returns empty array that means what you're looking for is not in the document

[@Xero_Wolf]

@@ivanyosifov2629 I have the same issue and what I'm searching for is in the document. Even when I test with a simple html.

[@ivanyosifov2629]

@@Xero_Wolf It might depend on the editor you are using. For some editors you need to give the file path as */index.html* or *./index.html*

[@camplays487]

@@ivanyosifov2629 For me, the .find("strong") returns NONE even though the print statement before it clearly shows strong tags, any idea what could be causing that?

[@_n1c0l4s]

I am using the find_all(text="something"), and it also returns and empty array... I know that what I am looking for it actually is in the document. Could the problem be something of how the html file is structured?

[@Will-fh9fj]

Nice, Tim. I mean, nice.

[@jalepenofatty6704]

great video , hit a bunch of roadblocks with the imports and environment %PATH% changes i had to make, and then the openssl issue, but yeah took me a day to get thru this and finish, i appreciate the hardwork.

[@unpatel1]

Glad that you finally solved your problem! Me too, had a hard time with the %PATH% thingy!@$$. Time to time %PATH% problem appears from no where and eats up lots of my time. I have worked a little with R and I found it relatively simple and easy in this aspect, package installation and management etc...

[@andrews9168]

@@unpatel1 another workaround is to use pycharm

[@unpatel1]

@@andrews9168 Thank you for your suggestion. I do have pycharm but not using it, just use VS Code all the time. I will definitely try pycharm.

[@greening6904]

Tim you wont believe i was working on a meteo app and needed a parser thx

[@learnwitharbia3477]

Thank you so much for such valuable content

[@alagappank1242]

Superb...🤩

[@zawadahmed5484]

Keep on your beautiful contents

[@b07x]

Thanks, this was easier than I thought

[@softicecreamer]

im looking for graphics cards and tim mentions them. wow its like u read my mind

[@mghostdog]

So when I run the script looking for the "$" on the site I'm parsing, I get an empty list [ ]. Does that mean that the website is preventing me from seeing that particular item/price?

[@intelblox7354]

im getting a index error when i put prices[0]

[@ethicalhacker9720]

I think it is the website. I tried another website and it worked.

[@guy6567]

Thanks Tim! :) awesome and helpful

[@itssuperbaby2979]

Amazing tutorial, but one question - in 9:09 what does the [0] do? I tried running the code without that and there was a bug, but with it it worked perfectly fine. Im just wondering what the function of that is

[@Mmmkay..]

He accessed the first tag in the html file using index [0]. Supposed he used index [1] he would've accessed the second tag in the html file. He did something similar again @15:25 when he was locating the first parent tag of the price value. Hope that helps !

[@itssuperbaby2979]

@@Mmmkay.. Thank you so much!

[@thesocksv2483]

Thanks you a lot, you're the best.

[@Zydres_Impaler]

Tim, please make a series or video fo the "requests" library.

[@jamiemorrissey2858]

Nice, good video, learned a lot

[@mousemeister]

nice editing job and content ofc thx

[@laurasasso8798]

Perfect ! Thank you

[@hollowr9953]

Interesting video, as always

[@beratsamil]

thanks Tim! :D

[@Popcorn_and_funny_moments]

hey tim great work , i need to learn how to do using python columns and boxes in visual studio code thanks very much .

[@CarlosPerez3dArt]

Super cool you are so helpful

[@fuadpalchayev7269]

Thank you very much!

[@DGHere12]

thx for this tutorial, tim

[@philippao8078]

Hello, top. Thanks!

[@romaintisserand8921]

Nice, thank you ^^

[@tomasoon]

Great video, but the most impressive thing is when you did this video the video cards price was $2613, and now it's $1549 in less than a year xD

[@83yWasTooShort]

Really useful, cheers

[@siamahmed8287]

Can you make tutorial how I can scrape a dynamic web page? Like built with react.?

[@rog_shakhyar6171]

it would be same

[@omarciano42]

That would only be possible with Selenium, which Tim has a series on, just search it

[@THISISCHARISMATIC]

Absolutely great videos, I’m new to python and coding in general. Your content is really great and easy to follow, would this web scraping method work for finding stuff like meta data for songs ?