Better Web Scraping with Mechanical Soup - Build a Database of Cat and Dog Photos in 10 minutes

[@tylersnard]

I appreciate that you keep things simple and colloquial. Many teachers try to sound smart and they just end up confusing their audience. You are a good teacher.

[@jeremyjamero199]

Awesome! You make my small understanding of Python seem like I've been coding for years!

[@subhamsubhasispatra]

This is really an awesome tutorial . Please make a playlist on mechanical soup . ❤️❤️

[@PythonSimplified]

Thank you so much Subham! May I suggest you another very cool web scraping library, even more powerful than Mechanical Soup?? Have you tried using Selenium? It scrapes information out of dynamic language websites (Mechanical Soup can only do HTML & XML) and you can log-in and out of any web account - in short, it's the most powerful scraping library I've ever tried! If you want to learn how to learn how to web scrape Instagram images, checkout my tutorial on Medium: medium.com/analytics-vidhya/web-scraping-instagram-with-selenium-b6b1f27b885 Or checkout the video version (I still recommend the article though, it's more accurate and less controversial 😄): kzbin.info/www/bejne/n3uqp4x4fZ6gmbs

[@namastemukesh]

Please keep posting projects more frequently. That makes a lot of sense for us. More realistic and more usables.

[@PythonSimplified]

Hi Mukesh, I'm gonna try to do my best to post videos on a weekly basis. The feedback I'm getting is incredible, I didn't expect it when I started this channel and thank you very much for commenting and sharing your thoughts! :) With that said, I'm currently trying to figure out how to film in 4k as I got a new camera. It might take me a few extra days, but I think it's worth it for the long run. Once I have it figured out, I'll advertise an organized upload schedule so that everybody knows what/when to expect new videos ;)

[@namastemukesh]

@@PythonSimplified You have the skills to make people understand easily. I do have ideas about small projects, if you want to discuss.

[@chiranjeebroychowdhury7759]

Easy Peasy, Lemon Squeezy! Nice breezy video yet again! I have used Scrapy and Beautiful Soup a lot. Never worked with Mechanical Soup. This was easy. Also I usually download and save files using urllib. Thanks again Mariya!

[@webscrapingwithandy2110]

⭐️ Timestamps ⭐️ 3:00 - Search for a term on google 3:45 - Extract source HTML from the web page 6:29 - Navigate to the new page 7:08 - Look for all the image elements on the new page 9:01 - Make a local directory for the images 10:19 - Download and save all the images

[@PythonSimplified]

Sorry Andy, somehow I missed your post, thank you so much for the timestamps - you rock!! ❤

[@CdaPlays]

Hi, New Subscriber here..from Facebook python group

[@PythonSimplified]

Thank you so much for subscribing! Welcome to the channel! :)

[@CODTALES-KILLSTREAKS]

I love this channel! I’ve watched many many python videos and I am in love with this channel

[@ankanmazumdar5000]

Thank you dear...you're most beautiful data acience creator in this universe ❤️

[@jamalhasanzakarneh9837]

Thank you very much. It is very nice and helpful one. I used the same code but I got 20 images however you got 3057 images; could you please explain why?

[@siamahmed8287]

You explain things really simply. That's why i like you. But would you mind if i give you a suggestion that will you change your wallpaper? Something fancy? 😇

[@PythonSimplified]

Hi Siam, this video was filmed almost a year ago! 😁 Not only do I have a new wallpaper now, but I actually have a new computer! (and much much better videos about webscraping by the way 😉) Checkout both of my Selenium bots: Instagram Scraper: kzbin.info/www/bejne/n3uqp4x4fZ6gmbs Facebook Scraper: kzbin.info/www/bejne/iaS7lKylmtuejJI An additional Linkedin bot will be released on my channel tomorrow 😉

[@siamahmed8287]

Hey I know. I've seen your videos. I don’t miss your videos

[@Scal]

Hello ! EDIT : I did a new scraper thanks to your tutorial on Selenium and it worked !!! Thanks a lot !* Sorry I am doing same kind of thing on a page with a form but I got a problem. I think my problem is the same as "how can I go on page 2 from cat search results 1st page?" I would like to fill in a form on a 1st, submit it, and move the browser to a 2nd new validation page. the 2nd page is just summarizing what I inputed on the 1st page, and asking me to validate with a checkbox and a submit button. I get the new 2nd validation page url correctly with browser.get_url(), but when I do browser.launch_browser() I m still on the 1st form page response = browser.submit_selected() print(browser.get_url()) new_url = browser.get_url() browser.open(new_url) print(browser.get_url()) page = browser.get_current_page() print(page) browser.launch_browser() both print(browser.get_url()) are showing the 2nd page url, but browser.launch_browser() still display the 1st page. the print(page) still show 1st page html and when i try to select the form with the checkbox on the 2nd page it can't find it. How can i make the browser go to the 2nd page and be able to check the checkbox on the 2nd page? I think my problem is the same as "how can I go on page 2 from cat search results 1st page?"

[@sagarsriva]

watching your video for the first time. you should make more such videos! great learning

[@thebuttnakedpodcast3006]

You make everything feel so lite and easy, thank you. #best teacher ever

[@PythonSimplified]

Thank you Harold! 😃 Keeping it simplified!

[@richardmodad7166]

this was a really good scraping tutorial

[@PythonSimplified]

Thank you Richard, I'm glad you liked it! :)

[@Orion3000k]

🤯🤯🤯 completely awesome. Great explanations. This is what we devs dreamed of. Thanks tons.

[@essetee]

Nice tutorial, but i don't get the url printed, but instead he starts my browser and show the url's. Also better to change code for creating directory in : #create the directory if not os.path.exists(path): os.mkdir(path) to avoid errors if the dir exists.

[@faceprofesor]

Very neat. Thanks for the tutorial. I have a question: how many images are being downloaded over the scrapping process? Can one set the number of images downloaded?

[@JohnWatsonRooney]

I’ve not heard of mechanicalsoup before look interesting I’ll give it a look thanks

[@baconsledge]

Your videos are excellent. How about recommendations for packaging stand alone Python executables?

[@TrevorHaagsma]

Auto-py-to-exe is awesome

[@digigoliath]

From now on. I will be drinking Mechanical Soup!!

[@PythonSimplified]

hahaha good luck surviving this! :D

[@digigoliath]

@@PythonSimplified LOL

[@mrsuncure1259]

TY! for teaching me this. And also thank you Rohit & Benny for the hint.

[@Grinwa]

What if the image is not stored in the img tag on the website...? I faced this problem this morning the The website i wanted to scrap store's image url this way that kind of websites uses css And using sop the regular way can't scrap images from websites that uses this method I have managed to find a way to bypass that, am planning to release the code on GitHub I spent the whole day working on it and learned a lot to be honest

[@shuvrodas6576]

Your videos are great. You can teach so much in a few time that amazed me. But one thing bothers me that I cannot see the screen clearly as the font size is too short to understand. I only understand it by hearing from you. Would you mind increasing the font size? Although, This was a complete tutorial. Thank you for the video.

[@overcomermindset]

Woman. You are a beast !!

[@TSerumula]

Wow . Thank you. You are really good at teaching.

[@tradewithdani122]

how to give different headers (user-agent)with mechanical soup to the website?

[@PythonSimplified]

Hi Danyall, this should do the trick: import mechanicalsoup browser = mechanicalsoup.StatefulBrowser() browser.open("my_website.com/") browser.set_user_agent('my_agent') Let me know if it worked! :)

[@pavanghai]

Simple N sweet, thanks for sharing specially using Jupiter , I prefer using google Colab

[@PythonSimplified]

Thank you Pavneet, I just saw your comment on FB, I'm glad to see you here tool! I'm usually a huge fan of Colab as well! The free GPU option is incredible! however, because this video involves a local directory I thought it will be much easier to do with a local interface... I guess mounting Google drive is not really that difficult, but Jupyter is a bit more comfortable on that aspect :)

[@akaabdullah]

Please make a video on how yo bypass google recaptcha with selenium or any other automation tool with python that will be great thank you in advance

[@PythonSimplified]

Breaking a Captcha is no job for Selenium 🙃 It much more complicated than accessing the DOM! I'm actually about to test a really cool tool (ScraperAPI if you heard), it should be able to break captchas in no time! 😁 I'll definitely film something about captchas soon, stay tuned! 😉

[@tothandras8160]

thank you for your demo. (what I would not do is to re-assing the value of the same variable with different meaning though; it is a bit difficult to read the code later for me.)

[@rahulkmail]

Nice tutorial

[@aHardReset]

Cool, I suggest add page scrolling and bigger versions of images :)

[@PythonSimplified]

Thank you for the feedback Aaron! :) The images are small for a purpose, when you feed them into a dataloader for training it is common to resize/crop them into a size of 28x28 pixels (like the famous MNIST datasets for Machine Learning). But if you're not using these images to train models - it's definitely best to get their bigger version. It just involves a bit more steps, because each of the images is located on a different website, so it makes it more difficult to automate the process. If you end up coding this, please share it with me, I would love to see what you came up with! :)

[@devakinandanrakoti1891]

It will be better watching if you can zoom in more. Some commands are not clear while watching.

[@PythonSimplified]

Hi Devakinandan, have you seen any of my newer videos? I zoom in quite a lot there, the fonts are huge! I've filmed this video almost a year ago, so many things have changed since 😉 Checkout my newest web scraping tutorial with Selenium, I think you'll find it very interesting: kzbin.info/www/bejne/jpWphqN8iqmnbpo

[@kostas6915]

Hello a technical question: can we web scrap an html table when a login is required for access to that page? (assuming we have the credentials ofcourse)

[@user-mf8gg8dj4x]

ты моя умница

[@kennethstephani692]

Great video!

[@fazlaynur4509]

love your tutorials

[@PythonSimplified]

Thank you! 😁

[@iansjackson]

HELP.... using VS Code. After browser initially launches it sticks on the search page with 'cat' sitting there in the dialog box. If I press enter then I get my cat images, but automation halted at that point. Any ideas?

[@mehdismaeili3743]

Hi thanks for this great video.

[@digigoliath]

🤩 Nice. Soup is Good!

[@alixaprodev]

Marya keep it up

[@PythonSimplified]

will do! :)

[@madslug7139]

YOU ARE A DYNAMITE 😎🔥🧨🙏🤘!!!

[@PythonSimplified]

Thank you very much Mr. Mad Slug! :)

[@mockydigital]

hello... I love your tutorials. Quick question though how can we make the images scrapped HD.

[@faisalee]

Awesome!!!

[@susmitbhamare]

Waiting for more frequent videos.

[@PythonSimplified]

Thank you for your feedback Susmit, I will try to do my best to post videos on a weekly basis! :) I'm currently figuring out my new 4k camera and how to provide better quality videos for you guys, it might take me a few extra days, but in the end I believe it will really improve your viewing experiance. Once I figure it out, I promise to advertise an organized weekly upload schedule and push the channel forward :D

[@susmitbhamare]

@@PythonSimplified Awesome 😍. Eagerly waiting for the content🥺🥺🥺

[@mytoptechs]

Google must have updated their code as the URL won't return anything back. I just get output of the url that was used and not the page contents.

[@basheersoft7118]

thank you

[@mapachu6959]

nice!!! I love your tutorials

[@haydnrayturner1383]

Can we set he minimum size of the image to download?? And can we increase the number of images we download?? Complete noob to coding so any help would be amazing!! Thank you!!

[@rverm1000]

great video. but wasnt able to resolve the error. attrib error partially initialized module. mechanicalsoup has no attrib statefulbrowser( likely due to a circular import)

[@albertgutierrez886]

Nice tutorials 👏👏

[@alixaprodev]

Thank you so much ❤️. It really helps me

[@PythonSimplified]

You're welcome, I'm really glad you found it helpful!! 😀😀😀

[@TrevorHaagsma]

I wasn't able to use Mechanical soup for NTML AUTHENTICATION, any ideas how this could be done?

[@gero8049]

Oh my god. Why didnt i find this before. Instead of Indian guys I much prefer this beautiful ladie explain me things.

[@mushinart]

Try requests-html .... This one is truly nice

[@PythonSimplified]

Thank you for suggesting Nasser! I'll take a look shortly (I'm familiar with the get requests, but requests-html is brand new to me), Thanks again! 😃

[@mushinart]

@@PythonSimplified sure thing. You will like it , its amazing. I teach python and specialized in coding training and cyber security. Please check my website : www.dotslashhack.io Love your channel and content. Truly amazing and i really appreciate the effort. All the best. And if there is something I can help you with regarding creating a good content in python, I'll be glad to help. Also i co own an arabic blog about python: www.pythonat.com Please check us out. Love and respect N

[@darrickhatten7046]

Ty

[@funtooz9513]

Hi need help? can you please tell me how can i find or extract [ href target="_blank"] links ??

[@PythonSimplified]

Hi Yuvraj, you can use some of the syntax from this video to target the blank anchors, try using the following commands: page = browser.get_current_page() all_anchors = page.find_all("a") And once you collected all the links on the page, you can only include those who have a target="_blank" with list comprehensions. blank_anchors = [link for link in all_anchors if link.get("target") == "_blank"] I hope this helps, let me know if it worked! 😁

[@funtooz9513]

@@PythonSimplified Hey Thanks a lot, it works.. Thanks again.... !!!!!!!!

[@justqtip]

very cool

[@user-hd3pz2ow1b]

thank u

[@Planet_Xplorer]

Good work

[@janduplessis1357]

Love the tutorial, love all your videos btw You have a great way of explaining Python. What do can we do to download more images and not just the first 20 or 30. I know Google loads them dynamically - is there a feature in MechanicalSoup that allows for this. Thank you

[@janduplessis1357]

Found the answer in your reply to another user: Selenium!!

[@basheersoft7118]

رائع

[@ebq8281]

Amazing ❤️😘😘😘

[@DonEdward]

"Dabble-yew, get" 🤣😂😂

[@manitthatrimontrichai8446]

Cool !

[@gabrielfernandez9660]

Ella tan hermosa la amo !!

[@abhishekjoshi4091]

Can I do this in gitpod too or online jupyter notebook ?

[@dejohnny2]

@Python Simplified that is a big ass laptop! 😁

[@rangabharath4253]

Awesome

[@PythonSimplified]

Thank you! :)

[@bedramtamang1439]

Awesome video

[@PythonSimplified]

Thank you! :)

[@bedramtamang1439]

@@PythonSimplified If you got time I request you to make video on infine scrolling, logged in system, dynamic content like javascript content etc.

[@BeAPythonDev]

Subbed from FB. I'm going to need to make sure I'm not releasing the same vids as you ;)

[@PythonSimplified]

Even if you do it's ok, the same concepts can be interpolated differently by different people, some will find your explanations easier and some will find mine. The more information we release put there in the web - the better :)

[@desarrollojava]

👏👏👏👏👏👏👏👏👏

[@invinciblenoman]

why does it save only 20 images only?

[@PythonSimplified]

To include more images, you can scroll down with: browser.execute_script("window.scrollTo(0, document.body.scrollHeight);") If you'd like to scroll down multiple times, try looping over a range: for j in range(0,10): driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") time.sleep(5) //wait 5 seconds that way you can include as many images as you'd like, depending on the number of iterations. I have a video premiering on it in a bit, checkout "scroll to the bottom of the page" in the timestamps 😀 kzbin.info/www/bejne/iaS7lKylmtuejJI

[@invinciblenoman]

@@PythonSimplified this is so great ❤️❤️❤️

[@PythonSimplified]

@@invinciblenoman no worries! 😁 And sorry for the long wait, it was an older video of mine so I don't check it very often for new comments 😅

[@invinciblenoman]

@@PythonSimplified yes I know I was searching your email id to send you this question in writing 😝

[@PythonSimplified]

@@invinciblenoman hahaha I'm actually getting bombarded with questions on my public email from Github, so your best bet to get a reply is here on KZbin (on recently uploaded videos, I guess)😉 I'm also chatting live each time I run a Premiere, so I'm available for questions then. And soon I'll start streaming live sessions, so that all questions would be answered right away (just waiting for my new monitor and keyboard to arrive before I start doing that 🤩🤩🤩)

[@shuvrodas6576]

You should zoom your screen a little bit.

[@hinomuratomisaburo4901]

Cool 👈 content, cute teen 👍😎

[@omkardalvi1588]

Mam please make a gui chatting app using socket and tkinter please

[@umer.on.youtube]

You were still on Win8 in 2020 ??? 🤨🤨

[@RedStefan]

Next video is on Chicken Soup, right?

[@muhammadzaakirmungrue3146]

nice outfit 😉

[@rohitkumarrajak2529]

Will you try to do one more this. Using machinicalsoup try to login into Instagram. If you read this message then like this.

[@PythonSimplified]

Hi Rohit, absolutely! I'm going to create a video for webscraping of dynamic language websites with Selenium. I wish Mechanical Soup could help with Instagram, but it is not supporting REACT, only HTML and XML unfortunately :( Selenium is the way to go if you want to try it on your own in the meanwhile... filming this video might take me a few weeks, I'll give you a shout once it's ready :)

[@rohitkumarrajak2529]

@@PythonSimplified I tried many times with Selenium but it's not working at all. Some errors always comes. I got a pip library which is working perfectly that's "Instagram-scraper" in github that working perfectly without any loging process if you want to make a video on that try to use "Instagram-scraper" code. I hope you are going to make video as soon as possible. Thanks

[@PythonSimplified]

@@rohitkumarrajak2529 Let me look into it, I'll send you a solution before I film the video, I don't want you to get stuck waiting for me to design pretty graphics and write a fancy script - I could probably send you an implementation in code before the video i even in the making ;) Give me a few days to figure it out, I've seen people scraping instagram so it must be do-able!

[@rohitkumarrajak2529]

@@PythonSimplified i have one more question. if you want help me then i will ask. and what about insta scraping

[@PythonSimplified]

Hey @@rohitkumarrajak2529 you'll be able to login to Instagram with the following code: colab.research.google.com/drive/1S_MxiCixGACGGAxkI7P114akwoZwMesv?usp=sharing If you implement the same methods of obtaining the React code, you can figure out the rest... if not, just wait for the video ;)

[@mattmovesmountains1443]

When you had searchword=cat, and then added the 's' and said "I'm going to concatenate" I started laughing at con 🐈 enate

[@PythonSimplified]

Hahahahahahaha that's awesome!! I didn't notice it 🤣🤣🤣

[@lovedale28]

Why don’t you use a web scraping tool instead of writing all those code😊

[@feliuswyvern7189]

Well 1, because people like to code and do things themselves. It's also more rewarding when you code it yourself. 2, because you gain more experience and knowledge of how it actually works, and learn how to code this app yourself. And add in your own edits/tweaks to the program vs downloading what someone else has already made

[@jeanmi8184]

Oh une canadienne

[@slayerification1]

I wish one day I can have a girlfriend like you!

[@jpongthep]

thank you

[@PythonSimplified]

Youre welcome, I hope you found it helpful! :)

[@hiseggcelency]

Hi! I have a problem, so after i do browser.get_current_form().print_summary() nothing happens, when i try to print it it says "none" i'm using visual studio code, any idea?

[@PythonSimplified]

I can see how something like this would happen if you didn't include the following lines prior to this command: browser.get_current_page() browser.select_form() However it's hard for me to tell if I can't see you code, do you mind sending me a screenshot? you can reach me at [e-mail] :)

[@hiseggcelency]

@@PythonSimplified Thanks for the fast response! I'll send you the code! I do have browser.get_current_page() browser.select_form() above it since i followed your guide, thanks!

[@RedStefan]

Next video is on Chicken Soup, right?