Beautifulsoup vs Selenium vs Scrapy - Which Tool for Web Scraping?

[@xpathservice2179]

3:28 you got the wrong title there... I guess it should be SELENIUM not SCRAPY

[@JohnWatsonRooney]

Oh man, you’re right. thanks for pointing that out.

[@rudisygo6804]

Mr John sir. I would really liketo thank you for the service you have given us online. YOU ARE TRULY VALUED THANK YOU FROM SOUTH AFRICA. You seriously are become a role model and you teaching styles are awesome!!!!!!!!!!

[@JohnWatsonRooney]

Thank you very much! Hello to South Africa 🇿🇦

[@celerystalk390]

The best video comparing web scraping tools hands down!! Thank you for another extremely useful video John!

[@travis.gooden]

Wish I had watched this before choosing Selenium for a scraping project. Really feel you hit the nail on the head. Great video!

[@ayeshavlogsfun]

Why ? In not selenium best?

[@travis.gooden]

@@ayeshavlogsfun Selenium is really good for automating tasks such as browsing, interacting with elements on a page, etc. I had more trouble setting it up purely to scrape results, as that isn't it's primary focus.

[@ayeshavlogsfun]

@@travis.gooden which Module are you using for scraping now ?

[@affanahmed9318]

most of the time what I do is use selenium to get me where I want then extract what I want by making soup of the page using beautifulsoup extracting specific tag info then afterwards using pandas to save the list data in data frame and exporting it out as a csv or excel .

[@NivAwesome]

Thanks for the information! I found another method which is not very efficient but worked for me on a small dynamic website page. I ran selenium in the background, sent keys (ctrl + a, ctrl + c) then pyperclip.paste() into a variable. Then I used re module on the string to take the information I needed. I used split method as well on the new lines to convert the string into a list with strings.

[@JohnWatsonRooney]

Cool idea - if it works for you then that’s great!

[@Berenduinelbardo]

Thanks for your videos. I can understand you very well, thank you for taking care of your pronunciation, I am Spanish and we do not have Spanish-speaking channels as good as yours. Keep it up.

[@JohnWatsonRooney]

Thank you!

[@alexgarov9164]

Si encuentras alguno, por favor comparte! jajaja un saludo

[@mattmovesmountains1443]

Also, have you thought about a patreon? Your videos are consistently helpful - I'd join it around a $5/mo

[@rrahll]

Really interesting theme! Thank you for your tutorials and good luck! Really useful content!

[@harshgupta1999]

best explainer video on youtube about this topic

[@Kralnor]

Thanks for a great rundown of the options available for web scraping in Pyhton. There were a few that I was not familiar with.

[@omkarpatil9234]

Scrapy for me is like second to none. Its a total beast if you are able to use it to its full potential

[@JohnWatsonRooney]

Yes I agree, have come around to it and using it a lot more now

[@omkarpatil9234]

@@JohnWatsonRooney So now i am finding that to extract text and parse html 'beautifulSoup' is really good than Scrappy which also confirms what you said in your video ! So right now for my task i am using beautifulSoup (for parsing) and Scrappy(for running a headless browser), Which works fine but I am curious if you know any more easy techniques to parse html using scrappy especially to get text. Please let me know if you have any thoughts . Thanks you !

[@RichPortah]

Another A+ video..... After I watch your videos I find a website to scrap just for fun

[@omidasadi2264]

gist of the tech in web crawling ... nice and easy got summarized. thanks my friend

[@AmirAnsari-wh6gt]

you are the god of scraping :) thank you sir

[@GelsYT]

I have been using requests along with bs4 & I did heard about scrapy and I agree it isn't good for beginners so I was a beginner that time it really was daunting. But now I think it's time to scrapy time

[@JohnWatsonRooney]

It really is very powerful when you understand it!

[@ProjectXH]

Greatly informative, thank you.

[@Actanonverba01]

Very useful for beginners

[@lautarob]

Another very educative video of yours. Thanks!!. Question: what would be the best method to scrape a site that has a huge table that is spread across many pages, in a way that (for example) only 30 rows are shown in the first page (say 1 - 30) then the next 30 on the 2nd page (31-60) etc. etc. all the way up to over 10,000 rows or more. Can I move to the next page using Scrapy of BS4 or I need Selenium for that purpose?.

[@JohnWatsonRooney]

That depends on the way the page is getting the data. I suspect it’s via Ajax - check out my newer video on scraping JavaScript tables that might help you

[@MirGlobalAcademy]

Facing the same problem

[@ericxls93]

So useful! Thank you.

[@dreamtolive4564]

Really good and informative

[@hudsona4004]

You mentioned that selenium sends information about itself to websites being scraped, so that websites could detect that selenium is being used. I'm curious if you know more about this and any workarounds?

[@hubertcombomarketing2693]

Hi John. Thank you for your great work. Could you please make some short video about parsing html tables with colspan inside?

[@e4rohan]

great video thanks for the summary

[@JohnWatsonRooney]

Glad it was helpful!

[@ankushgaur9367]

Helpful. Subscribed!

[@ivy4372]

Thanks for the whole content! I got a question and it would be very helpful for me, if you can support: I have to scrape a dynamic website. If I scroll down, more objects are loading to this page (always 50 new). When I look in "developers" of my browser, I find the Data I need in the folder "XHRs" and with every scroll for new 50 objects, there is a new file called "730" with the new 50 objects in json-format. I need all the 730-files. do you know how to scrape them?

[@JohnWatsonRooney]

Sure check out this video I did here it covers how to get that information Always Check for the Hidden API when Web Scraping kzbin.info/www/bejne/eqLXnYVmrpWmq9E

[@Hyuts]

Great video thank you. Thoughts on AutoHotkey?

[@JohnWatsonRooney]

I've never used it sorry!

[@cnsnipon]

I liked a very didactic explanation

[@WaleedShreef]

Hi John, I missed you lately because I got busy. I want to tell you Happy New Year. And I want to ask how I can benefit financially from web sides scraping Regards Waleed

[@JohnWatsonRooney]

Happy new year to you too. To start I’d say try to get some paid work scraping data that people need, then try to build something useful with data you scrape and charge for the service

[@roberttuttle4284]

I am new to scraping. If you have a dynamic website that requires you input dates or numbers and click on buttons, what else besides selenium works? Does Beutiful Soup work? Very interested.

[@YamekDrope]

Can we get more information on the method mentioned at the end? How can we simulate requests?

[@abidhossain5527]

Can you make a video about recent scrapy-playwright bug about implementing scrapy-playwright setting implementation and some books or resources to learn scrapy.

[@mohitsharmagarg]

what would you recommend for creating scraping tool ?

[@vitorrochatech]

Hello John, Thanks for the video! I have two questions: What the best tool for scrapper a website with login or autentication? And, when the website use a api with autentication, what can i use?

[@JohnWatsonRooney]

Easiest option is to use selenium or playwright, but it can be done with requests too - you’d need to find the login endpoint and send the credentials over.

[@thulasirao9139]

I had a confusion which one should I choose for scarping?. Thank you.

[@arjay_2002]

Please tutorial on your command terminal on your windows

[@JohnWatsonRooney]

Sure, I could do a setup video

[@arjay_2002]

@@JohnWatsonRooney thank you!! It looks really neatt

[@Klausi-uq4xq]

It depents on the Site and what infos i need..at the time in prefer Scrapy..

[@nachoeigu]

There are some things which I can't understand. What if I have to scrape a website that use Javascript (if I make a request, I receive a part of the content of the page)? The unique solution is to use Selenium or, with Scrapy, you can handle it without problem?

[@JohnWatsonRooney]

You’ll need to use something to render the JavaScript out for you, and return all the page data as html to parse. That thing is a browser - but in some cases a smaller lighter version of a browser that runs headless (we don’t see it). That’s like splash or puppeteer (that’s what requests-html uses) or it could be selenium that we can control

[@helovesdata8483]

Overall I think one should take the time to learn Scrapy if you need to web scrape for a job. It will be worth it in the long run. What do you think? (as I move over to your scrap for beginners video) LOL

[@JohnWatsonRooney]

I think once you have the principles down, or if you are already proficient with Python, then definitely learn Scrapy

[@md.sharifulislamshanto2103]

What are you thinking about 'ParseHub'? Is it well enough to use in professional field?

[@adamchurchwell4019]

do you have any videos on scrape masking?

[@-__--__aaaa]

bro what if captcha v2/v3 in some website . How to get rid of that

[@-__--__aaaa]

@Neo No dude i need to do manually

[@ShivamSharma-if1oh]

I want to scrape data from infinite scrol website which library should I choose

[@d8nnii_]

How do I scrape from reebonz.com? They added a layer of protection from a vendor (which I can’t remember) that renders their site almost impossible to scrape.

[@JohnWatsonRooney]

look for the api the site is making requests too (network tab of inspect element) and start to make them yourself from your code

[@DashaZakella1001]

what tool i can use to bypass blocking websaits if it understands that I m using automation tools?

[@renatosardinhalopes6073]

Wait, isn't the last method just requests with beautiful soup? What you mean when you compare beautiful soup against scrapy then?

[@proribrajokproribrajok7789]

Is node js scraping is like python scrapy? which one better performe?

[@JohnWatsonRooney]

The principles are the same, you make a request and receive data. I don’t know JavaScript well really to comment - but as far as I’m concerned you can’t go wrong with Scrapy - it’s built specifically to scrape data after all

[@FishingWorlds]

I was trying to get things done last 20 minutes with BeautifulSoup but I have to press an accept button on wozwaardeloket.nl and the site is made in JSP so that mean's BeatifulSoup will not be able to post form data to one page and than post another form data to the new page right?

[@jjeffery129]

It seems easier to use selenium to scrape google map by searching different zipcodes for gas prices but it’s too slow. Can scrapy be able to interact with the website like searching different inputs, or it is better to just use google api?

[@ahmedchokri3240]

Hey bro, i need some help, I'm working on a project, a part from it is getting some data from Instagram and put them Into my Web app of course they must be always updated.. In this case i think selenium is low but i need it to connect to Instagram account also i need http request... So plz advise me..

[@lurkagurka]

So using Scrapy won't get u blocked as from using Selenium?

[@higheringai68]

Who might be using IBM Watson Discovery?

[@the1gofer]

Thanks!

[@manishdaga]

Greate Video

[@sachinshavi2243]

Sir I getting an error while running scrapy project.. error is Scrapy 2.4.1 -no active project Unknown command: crawl Use "Scrapy" to see available commands

[@darqlite6780]

you mentioned that if you needed to click a button or input in a field than selenium could be what you're after, does that mean that you _can't_ accomplish that with, say, scrapy and some addons?

[@JohnWatsonRooney]

Well, yes you could - depending on what it is. You can write LUA scripts for Splash that can simulate that, or if you are find a way around having to actualy click something, like getting the data elsewhere, or by finding the url that the data comes from you could get around it. There are some libraries that allow some control over these things but are all based around a browser somehow, like Mechanical Soup

[@ahmedmando9502]

thanks bro

[@mattmovesmountains1443]

Do you have a video that goes over the best scraper/tool for websites that have a constantly changing text element? Stock prices being the most well known example of this. I'm making something to scrape a "freefall auction" (price drops until someone buys one, or until it hits a predetermined low) and gather the lowest prices reached for multiple auction lots. I love using requests-html but it seems that it only captures the initial state of a rendered page, rather than any updates that occur once loaded. My guess is do basic gathering info with requests-html, then grab prices with selenium, which is my current approach, but wanted to check with the expert!

[@JohnWatsonRooney]

Sounds like a cool project - if you want to email me the site I can have a look and see what I think? Email on my KZbin main channel page

[@mattmovesmountains1443]

@@JohnWatsonRooney Will do - thanks!

[@NivAwesome]

@@JohnWatsonRooney May I email you a similar thing as well? Found it difficult with scripts, couldn't reach the page source with python (I think they rejected me because it's headless) and couldn't render it with requestsHTML...

[@0xmatriksh]

what about creating a video on this?

[@ss877S]

@@JohnWatsonRooney Sir can we use Beautiful soup for web scraping stock prices .Please help

[@depenz]

requests + bs4 VS scrapy?

[@alihusham1560]

So scrapy is the best

[@sachinshavi2243]

Sir how to take paragraphs one by one .?

[@Kig_Ama]

Great.

[@mastersdubai4729]

Video very interesting,,plz advise how I can scrap following , followers profiles in Instagram ,,plz

[@sachinshavi2243]

How to get unformatted data into format data .?