HTTPX is the ASYNC Requests I was Looking For

[@samibenhssan3121]

Surely the best youtube channel for web scrappers. I learned a lot from You. But i don't understand why you don't focus more on scrapy framework. It is surely the best tool. My salutations from Tunisia

[@JohnWatsonRooney]

Thank you, very kind. I have more scrapy videos planned, i think a lot of my audience is beginners though that may not want to try scrapy yet so I try to cover as much as I can

[@aflous]

I second this. Please more content about Scrapy!

[@return_1101]

Very interesting. I started a course about Web Scraping and Web Automation. Love very much this topic.

[@dennistanui7085]

Awesome. Thanks for keeping us updated

[@per-henrikjonsson6893]

Hey, thanks for introducing me to HTTPX. Very interesting. I like what you presented, but if I may suggest an improvement it would be to not fidget around with the mouse and scrolling up and down all the time. It was really distracting. But, you still have a new subscriber now. Thanks! :-)

[@JohnWatsonRooney]

Thanks! Fair point taken aboard thank you!

[@jonathanjayes]

Thanks John you're the best man, has anyone ever told you that? It's true!!!!

[@kevinz1991]

great video and super clear as always. thanks a lot!

[@tomkirbygreen]

Great video, thank you for this.

[@tubelessHuma]

Thanks John it really helpful. 💖

[@hahabobblehead2059]

hey mate, I'm quite new to Python and coding in general. Love the content on this channel! Any chance you could put your videos in beginner, intermediate and advanced playlists?

[@JohnWatsonRooney]

Thanks! Yes I’ve been meaning to reorganise my playlists I will get this sorted as soon as I can

[@yaksvk]

That is a great example and thanks for the video. Two questions, slightly off-topic and maybe a bit nooby: 1.) The "results" global list. Isn't it a suboptimal idea to append to global variables and wouldn't it be possible and a better idea to pass a reference to it to the get_episode function? 2.) I see type hinting is used in the get_episode function, but not for log_request and log_response functions. Intuitively, type hinting would be especially helpful for these cases where the argument is a non-trivial object (so that it can be enforced/suggested by my IDE) - https.response/request. Is this practice common?

[@wylde780]

Very interesting thanks for posting it

[@JohnWatsonRooney]

Thanks for watching!

[@Zale370]

What is the difference between requests-html and httpx? On the surface they seem similar.

[@recepcakir4866]

I send the requests but some responses are missing, the number of responses are fewer than requests, what to do here?

[@GhaithAlMasri-cv3dg]

thx alot can i post multipart/form-data with httpx???

[@JohnWatsonRooney]

Yes! If you check their docs it will show you how

[@hengkyariputra4795]

Any recomended library for scraping dynamic web asynchronously?

[@TruthSeeker7101]

Thank you for the video, I have a question,I created automation with the Batch file, you can open all the applications and websites you need in one click, Can we do this in python? Thank you

[@ameurchabane6001]

Thann you John you are my teacher , i 'm steak learning python i have juste face thread and proccessing ans multithread un python .. How we van use it To accelerating our scriptd .. Sorry for this bas english i do my effort

[@JohnWatsonRooney]

Hey! The best way to speed up http requests is using async - the time lost is in waiting for a server response and async requests will help here a lot. I have some videos on my channel about this already search for async!

[@augustonunesfarias4062]

Hi, Im trying to scrap a yahoo finance historical data soo when I request the html to extract the historial prices table the requests send me a different html what seems to be a javascript document

[@JohnWatsonRooney]

The page is being rendered by JavaScript, I have a video on JS tables in my channel which should help!

[@augustonunesfarias4062]

@@JohnWatsonRooney oh thank you man, Im sure that will help alot

[@jeroenvermunt3372]

Isn't there an api for yahoo finance? Edit: maybe it's only for historical data though

[@augustonunesfarias4062]

@@jeroenvermunt3372 ya, exists. but the job application what I was doing asks for scrapping the data without a api

[@untitledmonster]

Could you do an async on GraphQL?

[@JohnWatsonRooney]

You could but I don’t think it would help, as you’d just eat your credits, or you’d be better off building larger requests. I suppose if you wanted to make 10x 100 request sizes you could async it assuming you have the available costs. I’m not totally familiar with graphql though so this may not be totally accurate

[@untitledmonster]

@@JohnWatsonRooney typically, we apply the post method and pass the GraphQL query as the json argument. I just can't happen to figure out how to async that. your channel is a gold mine! do you do one-on-one counseling/career advising?

[@angelinaioanna5980]

httpx with client is 3 times slower than requests with Sessions

[@Scratchmex]

You are not reusing the client on each request, you are creating one for each so you don't benefit from it in any way