Performance tuning is usually an appendix in books and courses and the usual suggested solution is either timeit() or re-writing the critical part in c/c++. Thanks for showing a better approach.
@bva03 жыл бұрын
This!! There's a book on Cython that right in the beginning shows how important it is to profile the code before attempting anything more elaborate, so as to not waste effort on something that isn't a bottleneck
@dadestor3 жыл бұрын
well profiling is basically timing and if you need high speed python won't cut it anyways no matter what you do with it
@ApiolJoe3 жыл бұрын
@@dadestor What if you can reach something fast enough with some refactoring instead of rewriting in another language? Via profiling you can see quite easily if the program is spending a lot of time in functions that should not take much time, in which case you know what to refactor. Profiling can also help you see which parts of the program need to be particularly well written in the next language and which ones are not too critical on performance. Not exactly sure why you made your comment sound like you consider profiling as a waste of time.
@dadestor3 жыл бұрын
@Andrii Shafar v8 is good, but it's still an interpreted (or JIT -ed) language
@blueman3333 жыл бұрын
All those clickbaity websites and articles NEVER really talked about such a wonderful library that actually is a part of standard library. Today I learnt something new which will be actually helpful.
@mCoding3 жыл бұрын
Thank you so much! Glad you enjoyed!
@THEMithrandir093 жыл бұрын
Just to make it clear, what makes async so powerful is that it allows a cpu-python-process to parallelize waiting, in this case for responses from the network card. It is useful for data I/O to other devices, but it usually won't speed up stuff like costly operations on huge data-arrays. For that, multiprocessing is the way to go. If you happen to not like async for some reason, threading does basically the same thing, but as a different concept.
@bva03 жыл бұрын
Thank you for the post, I didn't know of async, just multithreading and multiprocessing. Do you reckon it may be advantageous to use async when scheduling workers in a master/slave paralellization paradigm? Then again, I think OpenMP does this "by default" doesn't it? 🤔 (Bare in mind my understanding of these things is rather superficial, since OpenMP can be used in Cython via a fairly high level api)
@THEMithrandir093 жыл бұрын
@@bva0 Since the trend away from monolithic software to microservices is huge right now(and for good reason), whenever I hear keywords like master/slave or multiple workers, I always check if microservices make sense over classical concurrent processing. The additional networking overhead is usually neglible thanks to great tools like gRPC and once you have the architecture set up, scaling from one server to entire datacenters, the roadmap is suddenly just a (complicated) recipe you follow instead of a monstrous engineering problem. So if you run multiple slave workers anyway, I'd rather run more single process workers that are ephemeral instead of dealing with concurrency in monolithic software. There's the 1% of cases where this won't go well, e.g. training neural nets, but for those few cases there exist great compute frameworks already, e.g. tensorflow and pytorch. But sometimes the solution/problem is small enough and you really only want to parallelize waiting and/or are limited by some device I/O, then doing async or threading to do that can be a great choice for sure! HTH
@bva03 жыл бұрын
@@THEMithrandir09 Thanks for the reply! I mainly use sparse direct solvers for optimization problems in mechanical engineering. Though I've been trying to migrate to sparse iterative solvers for HPC. I'll spend the next hour or so googling some terms in your answer, before asking further questions! Lol Edit: and yes, I use Tensorflow for some neural network problems! Though I find it difficult to fully understand the equivalence between certain numpy and tensor functions in order to fully get the benefits of operating in graph mode.
@mCoding3 жыл бұрын
I'm thinking of having a "when to use async vs multithreading vs multiprocessing vs nothing" video but first I need to introduce all the topics :)
@filipbartos75843 жыл бұрын
@@mCoding Thats great idea!
@BakkarGraphics3 жыл бұрын
I never expected to learn such a thing, this channel is full of stuff that i never expect to learn, thanks man :)
@mCoding3 жыл бұрын
Happy to hear that!
@gokublack48323 жыл бұрын
exactly, that's why I love this channel, lots of beginner content out there, but not as much intermediate/advanced stuff and that's what we get here
@Julie90093 жыл бұрын
Thank you so much for this very timely video. I had an app that I presumed was slow because I had some inefficient SQL queries. After profiling, I discovered that the slowdown was actually in a regex that I was running against every returned record. I now only run the RegEx against records that contain certain text. It was taking 38 seconds to run. It now runs in 9 seconds, 8 of which is rendering the Jinja template. This video saved me what would have been many fruitless hours trying to improve the SQL, which turned out not to be the problem.
@mCoding3 жыл бұрын
So great to hear your success story! I'm glad my video helped!
@onedez9 ай бұрын
Essentially, in the slow example, you're doing every get request one at a time, waiting for the first to finish before moving on to the next one. In the asynchronous example, you start all of the requests at the same time, doing them simultaneously, then wait for all them to finish before moving on.
@benixmo3 жыл бұрын
I was using async requests when needed but the profiling tool is a game changer, that was something I did not know existed.
@mohammadzuhairkhan86613 жыл бұрын
Can you make more videos on intermediate topics? This is my first time learning about async and it already seems so helpful!
@HypnosisBear3 жыл бұрын
OMG I've landed on a gold mine. This channel is awesome 👍👍👍
@mCoding3 жыл бұрын
Thank you so much I'm glad you enjoy!
@eric-seastrand2 жыл бұрын
Been using python 2 years and never knew it could do async/await. Thanks for this!
@arisweedler47033 жыл бұрын
Icicle graph … flame graph ! Wow, python makes that easy. What a wonderful developer experience. It even serves it in your browser for you 😳😱😍
@josephlyons33933 жыл бұрын
Hey, I have a degree in CS and am employed as a Python dev. I fully understand threads and processes, but never use coroutines/ async-await. Would love an in-depth video in the differences in programming and usefulness, as well as what’s going on under the hood.
@bernardcrnkovic37692 жыл бұрын
basically it relies on user-space context switching which is cheaper than kernel/thread-space context switching done in multithreading. you could do the same with multiple threads but overhead of switching thread contexts would be larger than event loop contexts.
@DaVince212 жыл бұрын
I was struggling to find good, simple but practical examples of the advantages of async and await but this has really cleared it up for me.
@Chaosman883 жыл бұрын
The timing of this video is perfect, I need to profile a GUI that runs a little bit slow, and I have to investigate why :) Also I'm looking forward for more profiling videos, this could be a series
@mCoding3 жыл бұрын
More to come!
@raymond-andrade3 жыл бұрын
Awesome content. This literally comes as I had issues with time it took me to process large amounts of data. I spent hours trying to look up videos to speed certain operations up only to find it to have little impact. Can't wait to try this out.
@vijaybaskar76353 жыл бұрын
In a world of beginner tutorial, here comes a saviour with intermediate topics. Thanks a ton mate
@mCoding3 жыл бұрын
Happy to help!
@simonthor75933 жыл бұрын
Pycharm also has a builtin run configuration called "profile code" or something like that, so you do not have to write the boilterplate cProfile code. This also shows a nice visualization
@Mekuso83 жыл бұрын
I prefer kernprof which can shows your code's execution speed line-by-line which can really help in some situations
@bva03 жыл бұрын
Cool, first time I read about this
@aqualung14662 жыл бұрын
This was a terrific video - I just cut a script from 45s to 3s using your technique. Thank you James!
@mCoding2 жыл бұрын
Awesome! Great to hear and glad that you were able to use snakeviz to go use!
@FireOnYouTube4 ай бұрын
My code went from taking 47s to 8s then when changing the max concurent requests number I got it down to 4s then finally to 1.4s. 33x faster. Perfect video.
@JohnDoe-wq9pr3 жыл бұрын
This is great info...good practical tips, presented to the point, with examples, and without all the time wasting filler content. Even IF it the purpose was to just make the video longer, the additional content was ACTUALLY useful. I'm sure there are plenty of videos on cProfile, but the bit about snakeviz was above and beyond. A lot of other channels that produce instructional videos can learn from this channel. I always look forward to more vids from this channel, even if they just randomly pop up on my stream. Time will spent!
@mCoding3 жыл бұрын
Thanks for the kind words! Glad you enjoyed!
@TheAmPm1233 жыл бұрын
Today I learned that if you delete the 2 second sleep it takes 2 seconds off the time... Seriously though, great video, thanks :)
@mCoding3 жыл бұрын
You'd be suprised how easy it it is for a debug sleep or print statement to end up in a prod release ;)
@niconeuman3 жыл бұрын
Excellent video! Profiling is a super important tool! Your explanation and examples were very clear and useful!
@leotaku52163 жыл бұрын
Great video! I'd also really like to see you do a whole video on how to best make use of asyncio. I roughly understand the API, but rarely know how to best make use of the different features. An introduction to your favorite async-powered libraries (e.g. httpx) would also be really helpful.
@lepsycho36913 жыл бұрын
Even though I knew just from the code that the web requests were the slowest part, I will definitely use the profiler in my future projects! Thx for the amazing video!
@mCoding3 жыл бұрын
You are welcome! I know in this case the answer was obvious, but usually it's less obvious in a real project, I just wanted to keep it simple for this little video. Glad you enjoyed!
@vishalmishra7018 Жыл бұрын
Thank you for making such a high quality video. Amazing.
@SavitskyVadim Жыл бұрын
Masterpiece! Thanks for your effort, I managed to get my hands to keyboard from other side of sofa just to write this comment.
@transistivehq2 жыл бұрын
Man! You create amazing content! Thanks for all.
@philippmatten43202 жыл бұрын
Holy fuck, dude! How do you come up with all these ideas for new videos?! I so enjoy your content! Please keep up the good work! ;)
@rtxmax82233 жыл бұрын
Advanced stuff with so much new stuff to learn about python modules.
@nadavgolden3 жыл бұрын
More on async in python please!
@PrashantKg19963 жыл бұрын
Another great video. Can you please make an elaborate video on async and parallel processing? It would be really helpful for people like me who don't have much understanding of such topics.
@mCoding3 жыл бұрын
It's been in the works for a while... hard to get right!
@Muhammed.Abd.6 ай бұрын
Woah! Thanks a ton Any idea on profiling GUI or flask applications? In GUI we have different threads for GUI and one for behind the scene works. I need to profile the backend task threads
@davidblake86123 жыл бұрын
I quite like the profiling function in the Spyder IDE that comes with the Anaconda distribution. You can just hit F10 and it profiles your code and shows the output.
@FutureAirify3 жыл бұрын
so many interesting videos to learn things here and there, thanks a lot
@alexanderschafer89792 жыл бұрын
15 Iterations with a two second sleep between each one in 12 seconds!! Man, this guy is really fast!!!
@mCoding2 жыл бұрын
I believe the sleep was outside the loop!
@alaapsarkar3 жыл бұрын
2:12 very very relatable I wish I knew about this a year ago when I was working on my thesis lol, this will be very useful, I always get to learn something new on this channel.
@mCoding3 жыл бұрын
There's always a next time!
@ryanking92173 жыл бұрын
This is brilliant, thank you for sharing this. The profiling is going to be really useful at work.
@ggsap2 жыл бұрын
Could you make a video explanining asynchronous and parallel programming technically? I see many others just give a broad overview of how to use the library, not explain what it does under the hood. Great content!
@mCoding2 жыл бұрын
This is on the horizon, but if you are looking for an existing video, David Beasley has a great 2 hour ish video that might go into the detail you are looking for.
@youngzproduction74982 жыл бұрын
Nice tricks you put here. Thanks a lot.
@MSemih-dk6xp3 жыл бұрын
KUDOS!! Sincerely, I want you to thank you for your great videos!! I quite appreciate the knowledge you're sharing with us
@mCoding3 жыл бұрын
Thank you very much!
@matiasmoglia3 жыл бұрын
Amazing content and very well delivered, thanks mate!
@mCoding3 жыл бұрын
Thanks so much!
@benlong10622 жыл бұрын
This is a great video. Wow. Thanks!
@mCoding2 жыл бұрын
Thank you for your kind words!
@JohnDlugosz3 жыл бұрын
How about benchmarking the same task done with Perl 5?
@tahini2453 жыл бұрын
For asynchronous HTTP requests, the two main go-to libraries are either this one (httpx) or aiohttp. But is there any speed difference or advantages using one library over the other? Or do they do the exact same thing and it's simply up to personal preference?
@tissuepaper99622 жыл бұрын
That's a question that you would have to answer yourself by reading the parts that you're actually using in a given project and/or profiling your code using both libraries.
@Talon_243 жыл бұрын
A python video that includes async that is not a) replace a print("Yes") with an async call to a function that calls print("Yes"), or b) Send requests to a webserver that takes exactly 1 second to reply by using socket and make the little change of completely rewriting the entire program to have it use async? In my lifetime? I'm SO going to hit that like button an odd number of times! Thank you! Also great to see snakeviz promoted, that was a lifesaver when i found it; If that's helpful for someone, here's a powershell function for calling the script with the profiler: Function pyprofile { python -m cProfile -o $env:TEMP\python_profiling.prof $args snakeviz $env:TEMP\python_profiling.prof }
@JamilBousquet3 жыл бұрын
Was not aware of httpx, fully expected you to talk about aiohttp. Thanks for sharing this video it's really helpful. Would love to see your take on multiprocessing in python if you've not already done so.
@mCoding3 жыл бұрын
My pleasure! Been working on an async/threading/multiprocessing video for a while, it's hard to get right!
@tobiasbergkvist45203 жыл бұрын
line_profiler is nice - and way more intuitive to use than a flamegraph.
@voinywolnyprod30463 жыл бұрын
Very clear and detailed explanation! Thank you very much!!!
@mCoding3 жыл бұрын
You're welcome and thanks for watching!
@khalilrouatbi63452 жыл бұрын
beautiful! i like the content of this channel.
@sinakarimi82732 жыл бұрын
Very well, but i couldn't understand how actually it can speed up code with one thread and processor? Can you suggest a book about coroutines?
@tk36_real3 жыл бұрын
What's the main reason not to multi-thread this?
@_Zabamund_3 жыл бұрын
Very useful and nice clean examples. Thank you.
@mCoding3 жыл бұрын
You are welcome!
@emifro3 жыл бұрын
That looks very useful, ill definitely use it when I need to.
@RayHorn5128088056 Жыл бұрын
I wanted to see a comparison of both versions of the errant function but that wasn't done.
@chinnku3 жыл бұрын
Keep up the great content mate! Really helpful. Cheers!
@mCoding3 жыл бұрын
Thanks, will do!
@mriganknalin35193 жыл бұрын
Can you tell me what extension you use so that your brackets turn yellow when you place your cursor there?
@mCoding3 жыл бұрын
This is a setting in pycharm. "Highlight matching delimiters". It should be on by default.
@MarvinTurner3 жыл бұрын
Amazing and straight to the point. Thanks
@argsahoo3 жыл бұрын
This was really helpful
@mCoding3 жыл бұрын
Awesome, glad you think so!
@DecimasoN3 жыл бұрын
Out of interest why did you choose httpx over aiohttp ?
@stifferdoroskevich18093 жыл бұрын
AMAZING! Thanks for sharing!
@mCoding3 жыл бұрын
You are very welcome! Glad you enjoyed!
@darkchoclate2 жыл бұрын
Funny thing is, i made a project that does the 3n1 check for a long range of number, and save the step and result in a file... I wrote in both c++ and python, surprisingly python took half as much time... i guess python is better in writing optimized c++ code than me😅😅
@SodaWithoutSparkles3 жыл бұрын
I have a simular situation. My intention is to get links from a webpages and download the files from said links. My solution is to use a loop to browse through webpages and get the links, then download them. Would it be better to write all the links to a file first then downloading it?
@mCoding3 жыл бұрын
It doesn't matter much whether you write the links to a file (though you may want to do that for your records), the important part is that you download the files asynchronously just like we download the pages themselves asynchronously in this video. Just keep in mind that automated downloading of files may be against the terms of service of a site even if you would have access to download all of the files by hand.
@BlackHermit3 жыл бұрын
snakeviz is awesome! Thanks James!
@mCoding3 жыл бұрын
You bet! Thanks for watching again!
@alexwhb1223 жыл бұрын
Yet another fantastic video. Thanks for this. I learned a lot! Do you ever use the profiler in PyCharm?
@mCoding3 жыл бұрын
I use the free version of PyCharm :S so no profiler for me.
@alexwhb1223 жыл бұрын
@@mCoding I actually did not realize that was a difference between the pro and free version. Makes sense. I'd go free as well if my job did not pay for my license. I also did not know about the asyncio lib... so that's also a handy tip. 👍Keep up the good work. Love your videos. It's nice to find more technical content... in my experience most python videos I see are either way to basic or a good level of complexity, but the explanations suck or the production quality is bad... so you hit a really nice balance.
@falcao_g3 жыл бұрын
this was really helpful, thank you!
@mCoding3 жыл бұрын
Glad it was helpful!
@NostraDavid22 жыл бұрын
And to answer "yes, but is this still slow?": think about how much data you're processing and how long it takes. That gives you the MBps, and THAT can indicate whether your code is still slow. If you have a super fast machine and super fast internet and the data is only a few megabytes, I would say it's still slow, because modern machines can handle gigabytes of data per second. Just a little rule of thumb to know whether your optimized code is still slow.
@dsenese3 жыл бұрын
Please, help me with this. You needed to await the gather function because the further process need to calculate all of it. But what if I need to do some processing individually for each website as soon as it completes? I can't do that because the interpreter always tells me to put an await somewhere. How can I do that?
@hzi882 жыл бұрын
You really should have also shown profilehooks. It is just so much more convenient to add a decorator to a function call. It is easy to do and easy to remove then adding all the with ... blocks in the code. It has its limitations but to profile a single function at a time it is quite good and fast to use.
@simonmatveev2 жыл бұрын
Thanks for video, it seems me really useful
@CritiKaster2 жыл бұрын
@mCoding Love you videos and explanations! I'm a big fan and hope to see more of your content. Very interested in this example - however I notice that the resulting http/https counters are way higher that the synchronous function - many of the asynchronous responses are just empty - is this a bug or a feature? Would love to understand this better. Thanks in advance!
@CritiKaster2 жыл бұрын
Ah found it - turns out the website URLs allmost all served a 301 redirect - fixed it by adding `follow_redirects=True` to the client.get line, so the full line is: tasks = (client.get(url, follow_redirects=True) for url in urls)
@aaronbrant61492 жыл бұрын
Thank you so much for your videos, by the way , Can you tell me which software did you use to record the video? How to put the speaker in the video, I also want to use. Thank you so much.
@mCoding2 жыл бұрын
I'm using obs studio to record, and I have a grewn screen behind me so I can use the chroma key filter to make the background transparent.
@tushargupta7643 жыл бұрын
How do you get to know these stuff ? Suggestions for resources where I can find them ( apart from open source projects ). Something like good blogs etc. Where they break down things a little for you effectively ?
@filipbartos75843 жыл бұрын
Great video! This was kinda obvious how to speed it up for devs already in bussiness, can you do some tricks you use to speed up your code?
@nitsanbh2 жыл бұрын
Opinion: JS has got its async/await much better & easier to use. This is the equivalent JS code for the htmls: const htmls = await Promise.all( urls.map(async (url) => { const res = await fefch(url); return await res.text(); }) );
@matercomus2 жыл бұрын
nice, very usefull!
@guillermoalcantaragonzalez65322 жыл бұрын
Hey, Im curious. If you retrieve you URLs using map instead. Would that also take so long?
@АртёмЕфимов-о6н3 жыл бұрын
Such a useful channel. Thanks a lot.
@CemKavuklu3 жыл бұрын
Fantastic stuff. Thank you very much.
@mCoding3 жыл бұрын
Glad you enjoyed it!
@wduandy3 жыл бұрын
I love it! Just love it!
@wduandy3 жыл бұрын
Like + Fav on it!
@mCoding3 жыл бұрын
Thanks so much 🙏
@josephlyons33932 жыл бұрын
Does asyncio.gather() actually allow for them to complete in whatever order they complete in? I feel like in my testing, it blocks in the order of the tasks in the list. I know that asyncio.as_completed() was made for that purpose.
@Kingofgnome2 жыл бұрын
Could you compare async requests against a simple multithreading Pool like from multiprocessing.dumy, doing the requests with Pool.map? I'd guess threading has more overhead?
@brunorcabral Жыл бұрын
What are the advantages of SnakeViz over KCacheGrind?
@juanpabloguerra95125 ай бұрын
How can you profile when it’s an api such as fastapi?
@TheJFMR3 жыл бұрын
I need to study to understand better async await, even though good video!
@ZakariaHABRI3 жыл бұрын
Hi ! Dumb question. Is it really fair since you run the profiling after get requests ? Thanks for the video!
@mCoding3 жыл бұрын
You are welcome! What is this fairness you speak of? In the real world you may write some code that is too slow. This is not an idealized competition to see how many cycles some operation takes, it is a practical tool to diagnose and fix why your real code is slow. If your real code involves get requests, then you need to profile the get requests along with everything else.
@angelcaru5 ай бұрын
I blame the compiler.
@dimitrisproios18603 жыл бұрын
Would threads make it parallel or concurrent? Assuming a much bigger number or urls to visit would threads scale worse than asnycio? What if instead of http pages we had ws connections? Great videos thank you for your analysis extremely insightful
@mCoding3 жыл бұрын
Too many hard questions to answer in a comment! Parallelism vs concurrency is not about using threads/processes/async, it's about whether tasks can be completed at the time time and whether tasks can cooperate. I'll have to defer to this popular stackoverflow answer: stackoverflow.com/questions/1050222/what-is-the-difference-between-concurrency-and-parallelism. Again with websockets it depends how you use them.
@dimitrisproios18603 жыл бұрын
@@mCoding Thanks for the resource and certainly not easy to elaborate in comments. I just think that explicit use of multiple threads couldn't optimize it further because of GIL. And For tasks with frequent use of resources eg big for loops, websocket with frequent non IO calIs maybe it could be a bit difficult to optimize without going for separate processes, it is just that compared to other languages this is a major issue that is not easy to bypass imho. Nonetheless thanks for mentioning coroutines. I look forward for a deeper dive it future videos.
@nickheyer2 жыл бұрын
So when this sends out an async request, it's like this?: send request > check for response > send request > check for response > send request ... instead of: send request > wait for response > send request > wait for response ...
@mCoding2 жыл бұрын
It's more like you send all the requests and then ask the the operating system if you have any responses. Every once in a while the os says hey you have a response from this request. Under the hood the os itself could implement this with a loop polling each buffer one at a time but there are clever ways it can avoid that most of the time.
@nickheyer2 жыл бұрын
@@mCoding I appreciate the response! All requests are sent and then the os is asked for responses.
@ROMVN952 жыл бұрын
Add threading or multiprocessing and boom 💥
@nopens3 жыл бұрын
Thank you!
@didotb012 жыл бұрын
how different will httpx async be compared to multiprocess requests?
@DendrocnideMoroides2 жыл бұрын
well asynchronous programming is very different from multiprocessing asynchronous programming means that when some process for example connecting to a website in the meantime some other part of your code can run multiprocessing means that a program can use all the cores/threads in your computer to do its job so if your computer has a ton of cores/threads then multiprocessing is going to be better but if your computer has only 1 core/thread then multiprocessing will make no difference at all
@AndiAbrudan3 жыл бұрын
Came here for the profiler, left with httpx
@aminebouaita92023 ай бұрын
THANK YOU SIR
@syedmokarromhossain48672 жыл бұрын
I always use threading using loop requests
@ethankremer56942 жыл бұрын
Hey there,have you heard of line_profiler? It's an amazing package that will document exactly how much time each line in your code takes. I use it similarly to these features, but it is far easier to implement. (Just import at the top and add a @profile decorator to any functions you want to be profiled)
@arbaazshafiq3 жыл бұрын
Could you please create a video on asynchronous programming in python? That'd be great.
@Mrjulle33 жыл бұрын
Httpx ftw!
@mCoding3 жыл бұрын
I know, I can't wait for it to hit 1.0!
@ShivamJha003 жыл бұрын
Very informative
@mCoding3 жыл бұрын
Glad you think so!
@helicopterpilot4810 Жыл бұрын
After running it with my project I've found out that 95% of the time is spent on ~:0(). I.e. it doesn't analyze the code run in the threads.