Following LINKS Automatically with Scrapy CrawlSpider

[@JohnWatsonRooney]

You can also generate a CrawlSpider in the commandline using: "scrapy genspider -t crawl name site.com"

[@gleysonoliveira802]

Every time you release a new video, it always deals with something I'm going through at my work. So, thanks a lot for sharing your time and knowledge with us.

[@JohnWatsonRooney]

You are very welcome

[@tubelessHuma]

Getting deeper into Scrapy. Thanks for this video. 💖

[@AliRaza-vi6qj]

Thank you so much john for sharing your knowledge with us I become your fan after watching this video and expect you to make more and more videos on web crawling,scrapig

[@baridie2002]

thanks for sharing your knowledge! very interesting the CrawlSpider, your videos are great! greetings from Argentina

[@dipu2340]

Thanks for sharing the knowledge ! Videos are of high standards. Could you please make a video on the best approach for using scrapy for pages which contains dynamic items(like picking from a drop down list where URL does not change).

[@0x007A]

Always mention that the terms and conditions and/or legalese is verified not to explicitly disallow webscrapping or similar restrictions. Additionally, document data sources and any licensing, terms of service/use, and copyright restrictions whenever scrapping data.

[@ahadumelesse2885]

Thanks for the great walk through. Is there a way to follow links of link ?? ( extract a link and follow that, and extract another link and follow, and so and so on .. )

[@MrSmoothyHD]

Hey John, great to know how to follow links to subsites. Is there a way i can tell my spider to parse&write the whole Site-Content into my file/s?? - What i want to do is make a full export of a forum and i want to save the front- aswell as all subsites, files, pics and css files (to be fully able to navigate through the forum in the offline html/xml file)

[@ataimebenson]

Great Video as Usual. Thanks

[@JohnWatsonRooney]

Thanks!

[@reymartpagente9800]

Hi John, Can you make a video using regular expressions? And it would be very practical also if you can use it in real projects like scraping emails or contact numbers in particular websites for example. I'm you old fan from the Philippines.

[@JohnWatsonRooney]

Hey! Nice to have a comment from you again, one of the originals - thank you! Yes Regex, of course that is a good idea I will add it to my list.

[@adnanpramudio6109]

Great video as always john, thank you

[@JohnWatsonRooney]

Very welcome

[@tnex]

Hello John, thanks for doing an amazing job. I'm new to python, but thanks to you I'm really getting good at it. I followed you all the way until i got stuck at the "scrapy crawl sip". When i execute the process i get an error message "SyntaxError: invalid non-printable character U+200B". can you please, don't know where the error is coming from. how can i share my work with you

[@RS-Amsterdam]

Great video John and thanks for sharing. I have a bit off topic question if I may. I want to scrape a photographers web site/page with images. I set up a basic scrip like you taught us in the past. Now the images on the page have an img link to another domain where the images are stored. The images on the photographers website are the full res images (no thumbs) from that other domain only cropped with width 200px When I put my mouse on the img src link it gives a pop up with : rendered size + dimensions (around 200px) and intrinsic size + dimension (around 1300px) However when I run the script it will download the rendered size image (small) , quite strange IMO. Any idea how I can make it work so it will download the intrinsic size (big) of the image Greetings RS

[@serageibraheem2386]

Thank you very much

[@mrmixture3155]

informative video, Thank-you SIr.

[@codetitan5193]

btw vscode theme look nice? which one is it?

[@JohnWatsonRooney]

Sure its Gruvbox Material theme

[@spotshot7023]

Hi John, I am trying to take user input using init function and put it inside rule extractor but the spider is not scraping it. If I pass hardcoded value and pass it to rule extractor where I don't have to use init function then it is able to scrape the page. Any solution for this?

[@JohnWatsonRooney]

Hi - I think you’ll need to use the spider arguments for this, you can find them in the docs and I’ve got a video on them. This is what I’d try first

[@stephenwilson0386]

I'm getting a TypeError: 'Rule' object is not iterable. Only difference I'm seeing in my code from yours (besides the page and dictionary I'm scraping) is that I only set up one rule with one allow parameter. What am I missing?

[@JohnWatsonRooney]

I'm not 100% sure but if you have only one rule and include it like I have try adding a comma to the end, i think its expecting a tuple still

[@stephenwilson0386]

​@@JohnWatsonRooney That did the trick! Gotta love a simple fix. Love your channel and style of showing this stuff, it really makes it more approachable. You should consider making a course on Udemy or somewhere if you have the time, it would be a big hit!

[@TheEtsgp1]

You have any videos showing how to use pandas data frame for start URLs and output scrapy data to a pandas data frame instead of a csv

[@umair5807]

The scraped items are not in a sequence. They are randomly added. Why this happened John?

[@muhammahismail1843]

Hi there, how we can add 3rd url and scrape data from the 3rd url.

[@nelohenriq]

Can i use this method with headers and cookies on sites that throw a 403 error when not using them? I can only scrape if i have the request headers but how can i implement them here? Thanks in advance

[@graczew]

like as always ;)

[@JohnWatsonRooney]

Thank you!

[@usamatahir7384]

how can we also add category heading in it

[@raisulislam4161]

Does CrawlSpider work with Scrapy-Selenium and Scrapy-Playwright? Is it possible to render JavaScript?

[@JohnWatsonRooney]

Yes it does, as it still uses the same scrapy request that can in turn be used by playwright

[@raisulislam4161]

@@JohnWatsonRooney thanks. I will try it today. What a relief ☺️

[@emanulele4162]

As ever Amazing video, I' ve Watched almost all your videos and they are all very specificly. I wanna ask you a video that talk about scraping but in addition to Kivy (or python frameworks like It). Is It possible? Thank you from Florence

[@JohnWatsonRooney]

Thank you I'm glad you like my videos! I've not used Kivy but I tihnk you mean creating an app or similar that can scrape data? If so then yes! I am working on some stuff like that now!

[@neshanyc]

Great Video John, I'm working on a scrapy project and I'm looking for a mentor. Is there a way to contact you? :)

[@MrTASGER]

Please create video about spider templates. How create my own template.

[@JohnWatsonRooney]

sure I will look into it!

[@MrTASGER]

@@JohnWatsonRooney ohh sorry. About PROJECT template. I want create project with my settings file.

[@NaughtFound]

hi. beautiful theme! please tell me your theme name. tnks