Get Started with Scrapy - Python's Best Web Scraping Framework

[@aefgshwrthwrhnwrhnbw]

You're absolutely nailing everything I look for in a how-to: Hands on material, very informative, fast enough pace that you have my attention the whole time. And it's very refreshing that you don't have 30 seconds of fluff to fast forward through in the beginning. Thank you for these videos, they're really really good.

[@JohnWatsonRooney]

Glad it was helpful!

[@abdelrahmannawar2137]

I really appreciate your channel, I feel like you are the only one on yt who can do web scrapping. I hope you get the amount of subscribers you really deserve.

[@tubelessHuma]

Thanks for video on Scrapy. looking for more tutorials on this framework. 🌹

[@edsonmatheus7976]

Another great masterclass! Thanks John.

[@Analyse_US]

Great video as usual. Your tutorials are compulsory watching for me..

[@Pr0filer4]

Great video as always John, i would really like to see an Image Pipelines tutorial also so that we can also grab those images along with all the other data scraped. Thanks in advance and keep up the great work.

[@JohnWatsonRooney]

Great idea, I just did a database pipelines one - I will look at the image pipeline too

[@gustavopoa]

Thanks John

[@priverAbde]

thanks a lot for this tutorial😀

[@koosschutter1675]

What if webpages has random amount of fields and I would like to create them dynamically from a dictionary. The dictionary is built using a loop for each page naming the key str("Col"+str(x)), but each page has different amount of fields, then I don't know how to alter items.py. I would also appreciate a video on pagination while having to navigate through multiple links on each page. I can only find videos that deal with pagination or navigating categories, but not both together. The webpage I scraping doesn't have a next button.

[@vishyr9578]

Very well explained tutorial. Being new to Python and Scrapy, it helped in understanding much. Quick clarification, some of products had a price range for example in page 2.. so I tried not using TakeFirst(). I item returned seems to be a list with price but I am not able to concatenate with with "-" by using Join or split.

[@CarlosWong54]

Great video thank you! 🙏

[@hayathbasha4519]

Nice video Can u pls let me know how to wait until the page is loaded/element is visible

[@cameronvincent]

set -loction: - position paramater cannot be found that accepts argument scrapy. Please help

[@GauravGurjar]

Can we use scrapy where we need to put location name for scraping products from a particular location... like I want to scrape products data are available in Oklahoma state?

[@davorm1430]

Hey John, i have a question if you dont mind answering. Is it legal/detectable if im using selenium to keep a single or multiple live streams open and just pull viewer count. Data is to be sold aswell, thats my concern. Am i even making any requests in this case?

[@JohnWatsonRooney]

Hi Davor. There will be some grey area around that, I think generally if you sell your service and not specifically the data it might be better as you technically don’t own that data, even though it’s freely available online.

[@davorm1430]

@@JohnWatsonRooney thank you for your answer sir :)

[@hariharans9408]

Amazing video😊Sir plsss make a video for how to use pipeline to expert scraped data to google sheets in scrapy project.

[@JohnWatsonRooney]

Sure I can look into that

[@hariharans9408]

@@JohnWatsonRooney tq sir 😊 you helped me 👍lot in scraping and one more request sir, make more videos in scrapy framework😊

[@lonapress2359]

Thanks a lot John, I am struggling with a "Site" that prevent most of my requests using my own "User-Agent" from scraping the site even though i have UA configured in settings.py, i tried scrapy fake user agent but out of 20 pages i scrape 4~6 pages. Downloads\Requests limited every 20 Seconds. any hints ? cheers

[@JohnWatsonRooney]

Seems like maybe the site is stopping you, try enabling the auto throttle in scrapy - just use the default settings and see if that helps!

[@MuhammadHassan-se9xv]

Try changing IP using proxies.

[@uchihaobito723]

can any one help me. i am having trouble while installing scrapy. ERROR: Could not build wheels for twisted-iocpsupport, which is required to install pyproject.toml-based projects .

[@hariprasad-zq2bp]

Pls make a video on scraping data from svg line charts using selenium python

[@uniquezone1596]

dear sir i am new learner i am learning scrapy from you but when i going for scrap something it show Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) envery site show same even this video website also show same in my pc

[@ashwinramachandran3889]

I want to scrap contact details (mobile number and email id) from websites. Is it possible to scrap 1000+ websites daily?

[@SecurityTalent]

Great

[@JohnWatsonRooney]

Thanks!

[@_rebellious_666]

Are you related to Wayne Rooney?

[@JohnWatsonRooney]

Nope 👍

[@carlosaugusto-mo2hb]

hi man, may I ask ya something ? how do I deal with this statement ? for each "LI" tag there is this line and I just can't loop the UL tag. can you please help me ?