Create Your Own Scraper API with FastAPI and Python

[@Christian-mn8dh]

Just came across this channel and am so surprised that you don't have at least 70k subscribers. Such quality content! keep at it.

[@JohnWatsonRooney]

Thanks, will get there some day I hope!

[@samman5980]

I was literally just thinking about doing this. Are you some how web scraping my brain?!?

[@JohnWatsonRooney]

Haha, that would be quite a skill to have!

[@techamusement6714]

It's damn truth

[@pythonantole9892]

Just blown away at how simple this is.

[@celerystalk390]

Another great and super useful tutorial! Thank you and keep up the good work John.

[@JohnWatsonRooney]

Thanks mate!

[@loverboykimi]

It was smooth. Thanks.

[@c.caratti2848]

Great intro tutorial to FastAPI with scraping! Thanks!

[@techamusement6714]

Thank you sir you just provided that what we wanted.Absolutely exquisite

[@ricksanchez7077]

Thanks for sharing 😀

[@s6yx]

dope. I was meaning to just use bs4 and flask for my old webscraping code but decided on fastapi because of this video

[@hayat_soft_skills]

I love the content that you are sharing . your teaching method is awesome! kindly make more videos on hosting scraper on digital ocean like selenium script , scrapy script Thanks!

[@zacky7862]

Great tutorial. I'm gonna need this.

[@chitranshjain5931]

Thanks for sharing this ! Really like your stuff and it makes my life easier. 👍

[@yusufrumi1626]

Happy new year John.

[@JohnWatsonRooney]

Happy new year!

[@paulmiller591]

Cool stuff John

[@JohnWatsonRooney]

Thanks Paul

[@adityaraj1284]

i have watched this video, its a fantastic video i got all i wanted except one that how to deploy it online so that i can use api for my android app projects. Please make a video on it as this is really gonna help me and i would be very thankfull to you.

[@MuhammadAbdullah-fy6sg]

man you need to start streaming. Honestly there very few programming streamers on twitch.

[@bighneswar98]

Nicely explained mate. Unable to use fastapi along with scrapy. Throws internal error everytime and on command line it shows twisted reactor already running. Any fix?

[@JohnWatsonRooney]

I’m that case I think you’d need them separate - use scrapy to save to a database then fastapi to display the information

[@bighneswar98]

@@JohnWatsonRooney So no way to run scrapy scrapers directly like request html ones using fastapi?

[@thekarthik]

Great tutorial, thankyou. I was just thinking about doing this haha. Will this work with multiple endpoints too?

[@JohnWatsonRooney]

Thanks! yes it would, we you just need to code them in

[@DM-py7pj]

Have you done a video on setting up a virtual environment in python?

[@waliullah3321]

please create an api for Turnitin (plagiarism checker) using fast api or flask or Django

[@artof-war]

How can you scrape more than one tag ?

[@alfakih7247]

How to update negative reviews on Google

[@hassanrahmani4764]

I need to become expert in web scrapping what should i do😁

[@-__--__aaaa]

maybe not to use bs4 to scrap everything & also u should try using regex btw

[@hassanrahmani4764]

@@-__--__aaaa hmmmm okay i ll see

[@thekarthik]

@@hassanrahmani4764 honestly, have a look at this channel. It's all web scraping and he explains in an easy to understand manner

[@mariabazueva7204]

@@thekarthik do you know where he works now?

[@karthikb.s.k.4486]

Nice tutorial. What is the command prompt that you opened in windows OS for running fas api. Please let me know.

[@hassanrahmani4764]

Its is wsl sub system for linux u can get it from windows store on win 10😊

[@karthikb.s.k.4486]

@@hassanrahmani4764 Thank you 😊

[@hassanrahmani4764]

@@karthikb.s.k.4486 welcome am.thinking to make a web scrapoing and programming channel should i make it like will it grow? Freelancing related stuff and programmming

[@karthikb.s.k.4486]

@@hassanrahmani4764 sorry I am not sure on this

[@valuetraveler2026]

What do you think about cacheing these FastAPI requests? I am using render() within the scraper function so it is slow. How would you handle dynamic data which changes frequently?

[@JohnWatsonRooney]

Since making this video I’ve found that separating it out is much better - having a separate script for scraping and adding to the database freeing up the api to serve the data itself. If you want to have a more scrape on request service you really need to think about using something like celery to manage the tasks and return the data when the scrape is complete

[@paul_devos]

I've been scraping websites for 10+ years as a hobby data scientist. bs4 & requests. Rare occasion if I have to, use Selenium. You have the most interesting content I've seen in some time on KZbin or the web in general. Using tools I hadn't even heard of. requests_html, selectolax, httpx, etc. I had used Scrapy before (found it too bloated for my needs). That said, regarding FastAPI + web scraping -- I hadn't ever really considered FastAPI as I mostly just make "packages" (e.g. Reddit API wrapper like praw) for any websites I scrape regularly. I'm guessing the added advantage of this is you can sell/expose the API instead of the package... otherwise, is there any inherent advantage to using FastAPI versus just creating the aforementioned package wrapper?

[@JohnWatsonRooney]

It's very use case dependent. Things like this can be useful integrating into applications, and having a simple API to access your data or run scraping jobs is quite useful but you are right, most are just standalone scripts/projects that run and have a specific task.

[@forbidden_lion]

Been trying to upload this thing on Heroku for the past 3 hours, no solution! Can you please make a video on deploying FastAPI Scrapper on Heroku?

[@kannadastocktrader3369]

Can I webscrap Amazon website for the product details and price??

[@JohnWatsonRooney]

yes, i have videos on my channel to show you how

[@kannadastocktrader3369]

By creating api

[@kannadastocktrader3369]

I have more than 1000 pages to be scrapped, it is taking 2 hours for me to scrape , but I wanna automate this and also reduce the time

[@tpag20]

I wish learn Python with FastAPI with Graphql with Strawberry.

[@ahmd09]

I just want to know how you got JSON formatted in the browser in such a nice way. I hate unformatted JSON data

[@Klausi-uq4xq]

in firefox it is built in.. for chrome you have to install an extension like JSON Viewer

[@JohnWatsonRooney]

I think I use “json formatter” chrome plugin

[@ahmd09]

@@JohnWatsonRooney thanks

[@ahmd09]

@@Klausi-uq4xq thank you

[@willyle23]

Hello John, I have a question. I was able to create everything like you have shown but I have used render() for a JS website. When I go to host it on fastAPI, it is telling me to use the AsyncHTMLsession instead. Have you ran into this problem?

[@JohnWatsonRooney]

Hey! Did you put the render part inside an async function? If so change it to a regular sync function with just “def” and it should work

[@willyle23]

@@JohnWatsonRooney Thanks for your reply, No I did not. What I did was create a scraper class then created a function called load_rls which takes in all of the pages available on the given URL ( by looping through r.html and calling a helper function scape_other_pages to scrape data from all of the pages and add to a list.) . Both of the function uses HTMLsession class and the r.html.render(). This was able to run through an entire javascript page and return a response in the python terminal, however when I try to post it on FastAPI, it gets an error. I did what you suggested and I am getting a 404 not found error. Thank you for the help! edit: I found out why it was giving a 404 error, the parameter cat I takes in converts / into %2f,. I changed my function to just take in 1 parameter and when running it, I get runetimeError: There is no current event loop in thread "AnyIO worker thread"

[@tpag20]

awesome!

[@DanishKhan-ob3xh]

how to remove watermarks using python fastapi

[@theinstigatorr]

I have to confess I don’t really know what an API is still, particularly in the context you’re using it. I thought an API was a way for people hosting data to provide outsiders access to data without being able to access everything carte blanche. To me this just looks like you’re creating a scraper project. What am I missing?

[@faldofajri6796]

MATURSUWUN SANGET MISTER

[@randyallen8610]

how can i contact you

[@ferilukmansyah3037]

great tutor thanks john

[@acikast]

Thank you John for sharing your knowledge!

[@JohnWatsonRooney]

Sure thing!

[@NotBeHaris]

I have question not related to this video sir how to scrape web which are build with vue or react means with javascript because there behavior is different from simple html sites.

[@rameshks5281]

Hi sir, can you create one video about import multiple URLs present in external files like excel and import those URLs individually and loop through and scrape and append the scrape data to the same file .