Crawl and Follow links with SCRAPY - Web Scraping with Python Project

[@hmak5423]

The guy who left a dislike think you misspelled "scrape" as "scrapy"! :) Otherwise, who is that dumb enough to dislike such informative content? Thanks for all your help, Mr Rooney. You are the reason why I passed an interview just a few days ago!

[@JohnWatsonRooney]

Haha thanks, well done on the interview!!

[@hmak5423]

@@JohnWatsonRooney Thank you. Means a lot! By the way, I know you've a busy schedule, but I've a request for you which you can look into when convenient. The following might be long, but I thought I'd just put it all in one message... www.bearspace.co.uk/purchase?page=1 The interview comprised of a task that stated the above link. Upon opening the link you'll find art pieces which are essentially links that go to the art piece's page showing the price, title, dimensions and the "Add to Cart" option. The task was to go into all of the art pieces links to extract the price, title (name of the painting) and the dimensions. I encountered a problem with loading more pages (Load more) as there is no limit on the page numbers that can be queried - meaning that even typing 'page=1234' in the url link will load a page even though the art elements themselves stop at 200 total (essentially at page 10 or 11). I want to see how you would do the extraction/scraping that results in a dataframe of values including title, price, dimensions. Respectfully yours, Mak

[@JohnWatsonRooney]

Hi Mak. I think generally for that site I would keep a list of urls and check the length of the list after each page, and if it is the same as the previous page break out the loop.

[@tubelessHuma]

Well done john. Need more Scrapy Tutorials.🌹

[@mokolokoa3288]

The best scrapy thing i ever seen. Great Work !

[@JohnWatsonRooney]

Thanks 👍

[@vickysharma9227]

I was looking for this kind of video, executing with selenium for this task took more time. Thanks for the video Man.

[@dickyindra4923]

WELL DONE!!!! GREAT VIDEOS SIR JOHN!!!!

[@JohnWatsonRooney]

Thanks very much!

[@omidasadi2264]

great, thanks for sharing it.... John how could deal with pages without pagination? I mean pages with a policy of scrolling down and appearing continuous products.

[@vishalverma5280]

Hey John ! I guess you missed some thing here , what about pagination how should I apply in such code ?

[@bitarddrag60]

Hi John thanks cool video! i have a question is it possible to somehow make scrapy go check whether this category has subcategories and go through them and parse and if not then parse what is there if you know then explain how to do I will be very grateful and apologize for the language barrier

[@umarsaid6340]

What if the categories spans to multiple pages? So I want to follow the next button and for each page found, I want to follow the links to each detail item. I can use CrawlSpider, but I wonder how to use Spider only.

[@mushinart]

Thank you bro , cool stuff as always

[@JohnWatsonRooney]

Thanks!

[@GelsYT]

literally answered every question i have

[@GelsYT]

THANKS

[@samjane6187]

this is insane work man keep it up and smile more please lol :D

[@JohnWatsonRooney]

Haha

[@CenTexCollector]

Site I'm looking like doesn't have all the CSS just HTML it looks like. Do you have a tutorial video on getting responses for that?

[@ansuapillai2785]

Exactly wanted video.... thanks

[@JohnWatsonRooney]

Thanks glad you enjoyed it!

Awesome John! I have a big question for you: is it possible to get data out in parse (not just the links), join them with what you get in parse_categories, and yield everything for each item? I managed to do it by getting data in parse, passing the parsed data as arguments to parse_detail, calling the Item loader, and yielding there. Not the best solution nor the most efficient, but worked! Hope you can do a video on CrawlSpiders.

[@AwB]

I second this question

[@GelsYT]

It sounds a great solution

[@hayathbasha4519]

Hi, Please advice me on how to improve / speed up the scrapy process

[@janyawn6923]

Hi John, looking at the logs and output - do I understand correctly that scrapy did this asynchronously?

[@JohnWatsonRooney]

Yes it’s asynchronous, scrapy is built on the twisted library. Although it’s async - it’s not async and await which most people associate with async

[@ulfgj]

so, when we want to follow links in ALL THE underlying pages that has a specific url pattern - not just one level down - this isn't it, right?

[@tnssajivasudevan1601]

Thank you Sir Great video

[@JohnWatsonRooney]

Glad you liked it

[@RenatoEsquarcit]

Hey guys... I'm wondering one thing. Is there any market for web scraping services? Every time I check fiverr I get frustrated big time seeing people doing this for 5$.. I mean mastering scrapy alone needed a decent amount of programming skills...

[@Kyosika]

Hmmm, in SEO this is sometimes wanted. I have been playing around with Python scripts and it is slowly making my life a lot easier for myself!

[@RenatoEsquarcit]

@@Kyosika I love python too, but very complex stuff sold for few dollars is strange :-/

[@artabra1019]

me too indians make web scraping work garbage i submit proposal to upwork the client is have budget 60$ to do extracting data from website after few day i look again that project it was given in indian guy and its only cost 10$ they make technical skills garbage but programming is really hard to master.

Actually it is 5 dls per quantity. If you need a bunch, it'll cost you some dough!

[@RenatoEsquarcit]

@@artabra1019 absolutely agree...!

[@amineboutaghou4714]

Thank you John !

[@reddatosteam3721]

Great content! Following your video , what if the categories also have pages? A kind of multi level page. For example, follow the category, the product, then pagination, once all product were scraped, go back to the next category and so on. I mean instead of having a bunch of start URLs. Is that possible? I tried to mange one and I was able to get only the first page from every category 🤔

[@shalinisrivastava7250]

Yes, I also have same problem

[@amritanshusharma950]

Hi, i watched your video and it's awesome, so i am also doing some project just like you did on this video but in different way and i stuck. Here is what i want, first i want the data from each column then open the link and then extract some extra data from that link . I know it sound crazy but that's the project i am doing right now using scrapy. I check scrapy documentation and i didn't find anything similar to what i want.

[@palashpathak21]

hi John, i am getting 503 error when i do scrapy shell 'url', i have tries setting the user agent, still same, can you help please

[@vladimirantonov4506]

Thanks a lot! Very interesting and informative!! :))

[@youvanced6593]

What are the benifts of using this instead of the Requests library?

[@tomstalley3179]

Thank you!

[@rafewm151]

Thank you that was really useful

[@rostranj2504]

How do you follow links in a website with nested pages? I'd like to grab the links in a map, then follow them, scrape that page, look for another map with links follow and scrap until I reach the end of the nested pages.

[@victormaia4192]

Thanks! very easy to follow

[@sixstringscoder8412]

I don't understand why using a parse function with a callback would be needed in place of just using a parse function minus the callback. You show the code in the video, but it's unclear why you couldn't just use the code in parse_categories, rename it as 'parse' and run it just like you've done in past videos about Scrapy. Can you explain the advantage of using the callback? Thanks for your videos and your instruction! Highly appreciated the contribution you make!

[@melih.a]

Is there a way to have the default website a link from playwright after logging in?

[@RicardoPorteladaSilva]

great tutorial!. Any tips to get freelance jobs remote, I'm from Brazil - Sao Paulo!. I already try plataforms as upwork, workana, but doesnt work for me!. Anyway thanks in advance!

[@ggboostedtop483]

Thanks bro!!

[@azizaalkuatova9527]

Thank u for the video! How to select with CSS the class with spaces in it? dots doesn't work and as in your example the class name was unique but in many websites they are like parent-child and the beginnings are the same, if anyone can help with that?

[@ssshwanth]

what if how to extract note there is no json

[@مستقبل_مشرق]

thank you you are the best

[@Iglum]

Great stuff! Gonna check out all the videos :) i want to learn how to scrape sites like texture.ninja etc for getting me some public domain textures. 👍❤

[@artabra1019]

i cant get project. in webs scraping

[@களவையும்கற்றுமற]

I need a video about scraping google search results without getting banned. Please do it. Suggest some affordable API or proxy service for that. Thank you

[@Don_ron666]

Why does he use a virtual environment

[@Scuurpro]

How would I do this if I were too combine this with parse json like in this video kzbin.info/www/bejne/rpvMloWMo9qDmbM. Basically I'm trying to create a crawl spider to scrape products then retrieve the Request URL to get the json data. I don't really know where to start. i was able to build my first two bots just using playwright and css selectors.