Website to Dataset in an instant

1000 items in one API request... creating a dataset from a simple API call. I enjoyed this one, there will be a part 2 where I clean the data with Pandas.
This is a scrapy project using the sitemap spider, saving the data to an sqlite database using a pipeline.
Join the Discord to discuss all things Python and Web with our growing community! / discord
If you are new, welcome! I am John, a self taught Python developer working in the web and data space. I specialize in data extraction and JSON web API's both server and client. If you like programming and web content as much as I do, you can subscribe for weekly content.
:: Links ::
My Patrons Really keep the channel alive, and get early content / johnwatsonrooney (NEW free tier)
Recommender Scraper API www.scrapingbe...?fpr=jhnwr
I Host almost all my stuff on Digital Ocean m.do.co/c/c7c9...
I rundown of the gear I use to create videos www.amazon.co....
Proxies I recommend nodemaven.com/...
:: Disclaimer ::
Some/all of the links above are affiliate links. By clicking on these links I receive a small commission should you chose to purchase any services or items.

[@stevenlomon]

Super neat!! Also as a Swede I chuckled at "this is a pretty standard e-commerce site" when talking about Sweden's most valuable brand haha

[@JohnWatsonRooney]

haha! yeah huge brand..! thanks for watching

[@theonlynicco]

you are a bloody animal mate, love your work a ton!

[@shubhammore6332]

I never comment on youtube videos but this has been so helpful. Thank you. Subscriber++

[@superredevil12]

love your video man, great content!

[@cagan8]

Just followed, great content

[@mattrgee]

Thanks! Another really useful video. What would be the best way to either remove unwanted columns or extract only the required columns then output a json file containing only the required data? This and your 'hidden API' video have been so helpful.

[@JohnWatsonRooney]

thanks! you could remove the keys from the json (dict) in python before loading to a dataframe, or if you are going to use the dataframe remove them there buy dropping columns

[@LuicMarin]

I bet you can't make a video on how to avoid cloudflare websites, not simple test cloudflare website but proper ones where cloudflare detection works properly

[@graczew]

Good stuff as always. I will try use this with fotmob website. 👍😉

[@matthewschultz5480]

Thank you very much John, great series - I am a bit stuck between this video and the cleaning with Polars video in taking the JSON terminal output and converting for use in Polars. Is there a def and function I can add to the code to output to csv (or JSON)? I considered importing csv and json libraries and creating a def and print but unsure on this step. Many thanks again

[@theonlynicco]

check his playlists and go to the one about web scraping for beginners, i think vid number 4 he covers conversion to CSV and JSON quite well.

[@RyanAI-kk1kv]

I'm currently working on a project that involves scraping Amazon's data. I have tried a few methods that didn't work which led me to your video. However, when I loaded amazon and looked through the JSON files, I couldn't find any of them that included the products. Why is that? What do you recommend I should do?

[@jayrangai2119]

You are the best!

[@ying1296]

thank you so much for this! i always had the issue of trying to scrape data from sites which paging is based on "Load More"

[@JohnWatsonRooney]

Glad it helped!

[@TheJFMR]

I use polars instead of pandas. Everything improved with rust will have better performance ;-)

[@mohamedtekouk8215]

Kind of magic thank you very much 😭😭😭 Is this can be used on scraping multiple pages ??

[@rianalee3138]

yes

[@negonifas]

not bad, thanks a lot.

[@schoimosaic]

Thanks for the video, as always. In my attempt, the website's response didn't include a 'metadata' key. Instead, the page restriction was specified under the 'parameter' key, as shown below. Despite setting the 'pageSize' to 1000, I only received a maximum of 100 items, which suggests a system preset limit by the admin. I'm uncertain about how to bypass this apparent restriction of 100 items. params = { ... ... 'lang': 'en-CA', 'page': '1', 'pageSize': '1000', 'path': '', 'query': 'laptop', ... ... }

[@JohnWatsonRooney]

there will be a restriction within their API, I was surprised the one in my example went up so high, 100 seems about right. you will have some kind of pagination available to get the rest of the results

[@milesmofokeng1551]

How long had u been using linux or archlinux distro would you recommend it?

[@JohnWatsonRooney]

3 years full time, dual boot/on and off for 10+. I use Fedora at the moment, seems to be a good mix. Unless you rely on windows specific software for work, or play games, 100% linux. Only thing I don't do on linux is edit videos, and that's for convenience.

[@heroe1486]

@@JohnWatsonRooney Most games are more than playable thanks to proton now tho, the only drawbacks are the ones with really intrusive AA like Valorant's one.

[@JohnWatsonRooney]

@@heroe1486 yeah its good to see, last thing i played was PoE and that was absolutely fine

[@viratchoudhary6827]

I discovered this method three years ago🙂

[@EmonNaim]

😘😘😘