This is a Scraping Cheat Code (for certain sites)

Scrapy keeps on giving, the sitemap spider automatically extracts links from XML sitemaps and yields requests based on a given rule set.
This is a scrapy project using the sitemap spider, saving the data to an sqlite database using a pipeline.
Join the Discord to discuss all things Python and Web with our growing community! / discord
If you are new, welcome! I am John, a self taught Python developer working in the web and data space. I specialize in data extraction and JSON web API's both server and client. If you like programming and web content as much as I do, you can subscribe for weekly content.
:: Links ::
My Patrons Really keep the channel alive, and get extra content / johnwatsonrooney (NEW free tier)
Recommender Scraper API www.scrapingbe...?fpr=jhnwr
I Host almost all my stuff on Digital Ocean m.do.co/c/c7c9...
I rundown of the gear I use to create videos www.amazon.co....
Proxies I recommend nodemaven.com/...
:: Disclaimer ::
Some/all of the links above are affiliate links. By clicking on these links I receive a small commission should you chose to purchase any services or items.

[@BrentBrewington]

Curious why you prefer that text editor (forget the name…you mention it early on) - instead of an IDE like VSCode or PyCharm?

[@jaimesandoval7435]

Hello John, thank you for share your knowledge. I am a newbie, I am starting my learning in Python and Scrapy. In this same example, how would you approach it if each product page is a dynamic page with JS?

[@JohnWatsonRooney]

I would use either scrapy-selenium or scrapy-playwright (on Mac/linux) to render the page, they are easy enough to integrate and work well

[@return_1101]

Mr. Rooney - its awesome!

[@mayurbhirud7998]

❤