Python Multiprocessing Tutorial: Run Code in Parallel Using the Multiprocessing Module

  Рет қаралды 809,616

Corey Schafer

Corey Schafer

Күн бұрын

Пікірлер: 946
@coreyms
@coreyms 5 жыл бұрын
Hey everybody! I hope you find this video helpful. I'm finally moved into my new home, so I should be back on a schedule of regular releases very soon. Hope everyone has a great weekend!
@deadfool6105
@deadfool6105 5 жыл бұрын
Please make a video on asyncio too..
@param8378
@param8378 5 жыл бұрын
@@deadfool6105 yes yes
@beastgenv
@beastgenv 5 жыл бұрын
Congrats on the move
@wizard-28
@wizard-28 5 жыл бұрын
Corey, can you make jupyter notebooks on these tutorials also. By the way congratulations!
@admintest9291
@admintest9291 5 жыл бұрын
Bless you man!
@vicstan5983
@vicstan5983 5 жыл бұрын
Why do people give thumbs down to someone who teaches for free? And a small detail ... the man is a grandmaster! Thank you sir for doing this. Greetings from the Netherlands
@bartekz88
@bartekz88 5 жыл бұрын
There are jealous because they're programming in Cobol.
@darwinvinci7744
@darwinvinci7744 5 жыл бұрын
I think it's the Australians. You see, they have everything upside down there, when they thumbs up, it's thumbs down for us. Corey is the best.
@Bubelaschka
@Bubelaschka 5 жыл бұрын
I think it's some kind of obsessive compulsive disorder.
@josephsagotti8786
@josephsagotti8786 4 жыл бұрын
~0.78% of people who voted, thumbed down down.
@kyleyoung4974
@kyleyoung4974 4 жыл бұрын
.. because its not working for them.. so they wasted their time getting 'taught' something that didn't work. .hence thumbs down.
@ginadi9733
@ginadi9733 4 жыл бұрын
There are courses that take thousands of dollars for teaching what you're teaching. you're teaching very well and you give it free. Respect.
@gytisbl
@gytisbl 2 жыл бұрын
31:12 Mr. Schafer finishes with the theory and goes into the real-world example with processing images 35:37 implementing multi-processing to single thread/process operation 40:04 easy switch from Processes to Threads Love Your videos Mr. Schafer. Especially Your communication skills. No BS, just clear communication. I also love that You don't edit out mistakes, and summarize after several steps. Thank You for putting the effort and sharing.
@nguyennguyenkhoi9988
@nguyennguyenkhoi9988 2 жыл бұрын
when i watch your vidieos i belive that if you can't explain something easy to understand, absolutely nobody on this world can do that. thanks you alot Corey, you make my life easier.
@anhurtjv
@anhurtjv 3 жыл бұрын
I just started an internship in Germany and you have no idea how helpful these classes series are to young people like me that want to succeed in the professional world of today :)
@spacecowboy2483
@spacecowboy2483 Жыл бұрын
This video is gold, every second of it. You are very talented at teaching complex concepts, you can't even begin to fathom how much I appreciate your work. Thanks!
@IrshadKhan-vk8rn
@IrshadKhan-vk8rn 5 жыл бұрын
Thnx sir I got placed in a MNC all bcz of you learned python a lot and still learning a lot whenever I have any doubts stack overflow and you r my mentor thnx again preparing for higher studies now..
@venkatkumar8672
@venkatkumar8672 5 жыл бұрын
i pursuing a college degree and I'm learning python in different resources and i learned the basics , will you please recommend me a good resource for mastering python language and to go to next level??
@eduardolpz386
@eduardolpz386 4 жыл бұрын
Great video! If you are running this in Windows and a later version of Python (I'm running 3.7), you need to run all multiprocessing under main(), then run if __name__ == '__main__': main() Make sure the do_something() function is declared outside main().
@yoyoliyang
@yoyoliyang 4 жыл бұрын
current
@caseymcmahan4834
@caseymcmahan4834 3 жыл бұрын
Thank you! Just the fix I was looking for. This is apparently true of macs as well
@Svkenapps
@Svkenapps 2 жыл бұрын
You fixed my Programm🤝
@drygordspellweaver8761
@drygordspellweaver8761 2 жыл бұрын
Only the code that spawns the subprocess needs to be guarded.
@thomasmayne2629
@thomasmayne2629 2 жыл бұрын
thank you so mutch
@ronnetgrazer362
@ronnetgrazer362 5 жыл бұрын
It's one of those tutorials that has you reaching for the like button several times during the video, and once more after it's ended.
@krystianfeigenbaum238
@krystianfeigenbaum238 4 жыл бұрын
Well, then let's hope this urge posesses you an uneven number of times or it would be unfair to the author.
@tebbythomas
@tebbythomas 5 жыл бұрын
Another wonderful video. Thanks to your videos I've gained an understanding of how to build and deploy a Django web application, using matplotlib for data visualisations, python multi-threading, python-multiprocessing, exception handling, python unit test cases and more. Your content is consistently excellent because of how practical and well thought out your videos are. Extremely useful! Please keep making these videos! Thanks a lot! You're helping a lot of people with this high quality content!
@shawnalexander5704
@shawnalexander5704 5 жыл бұрын
Of all the resources out there that I have looked to get a start and grasp on this subject (some cost money), this video is on point!
@erik-martin
@erik-martin 4 жыл бұрын
Corey's channel has done more for me in 2 weeks than any other resource I have tried over the last 10 years.
@raghav-cx3qq
@raghav-cx3qq 4 жыл бұрын
I am not in a position right now to donate through patreon. All I can do is watch your videos without skipping ads and give it a like.
@coreyms
@coreyms 4 жыл бұрын
That helps a ton too! Thanks!
@dhurda471
@dhurda471 3 жыл бұрын
This has been incredibly helpful for me and my team. I'm a network engineer working on multiprocessing SSH connections and this made everything make sense. Definitely subscribed after seeing this.
@theteleportercell6749
@theteleportercell6749 2 жыл бұрын
I have been looking for videos like this for like over 1 year, and today I randomly saw your video. Very helpful, easy to understand, thanks a lot Corey!
@world_of_hein
@world_of_hein 4 жыл бұрын
@Corey Schafer, Thank you very much for taking the time to make this video. I have written my first-ever parallelized code and the speedup is invaluable. For years I was apprehensive about learning parallelization methods because they seemed very complicated. You, my friend, have opened the flood gates. This work puts me one step closer to completing my Master's.
@facefamousee
@facefamousee 5 ай бұрын
love the honesty, i feel like i've understood the concepts even more.
@thedrei24
@thedrei24 5 жыл бұрын
oh man, you don't even know how long I was looking for a video on multiprocessing.
@dinhluongnguyen9758
@dinhluongnguyen9758 8 ай бұрын
when getting to learn the idea of data parallel and distributed data parallel, they stuck me a lot, but after watching your 2 videos, they help me a lot. Thanks sir, you're a life saver !
@jordanbyrne4628
@jordanbyrne4628 3 жыл бұрын
Just came here to say thanks for this. I just completed my first ever paid work programming, made a simple image slicing and joining script. Client mentioned he wanted it to run though 20000 images in under an hour after I built a demo. I rebuilt it using concurrent.futures and your video saved me hours of work! Plus it runs super quick. Thanks so much!
@drygordspellweaver8761
@drygordspellweaver8761 2 жыл бұрын
Curious how much something like this pays? I wrote a similar but much more complex script for myself recently
@renatoaraujo6108
@renatoaraujo6108 4 жыл бұрын
Nice explanation about paralel processing! Thank you! Nice explanation about paralel processing! Thank you! Nice explanation about paralel processing! Thank you! Nice explanation about paralel processing! Thank you! Nice explanation about paralel processing! Thank you!
@RomanKozin
@RomanKozin 2 жыл бұрын
Still. After 3 years - it's still helpful. Even counting that I need to test this out - nothing that much has changed in Python for MP. Thanks for such an excellent explanation!
@snoopyjc
@snoopyjc 5 жыл бұрын
Thanks Corey! Next please do async!!
@AnShett
@AnShett 5 жыл бұрын
Yes, this is concept I haven't used but would really like to learn about.
@Skaxarrat
@Skaxarrat 5 жыл бұрын
Yes please, async and when to use async/asyncio/threading/multiprocessing.
@user-be8ud2qd2d
@user-be8ud2qd2d 5 жыл бұрын
Yes would strongly appreciate this
@folorunso
@folorunso 5 жыл бұрын
Yes, async will a perfect next. Thanks Corey for the handwork.
@ashfaquechowdhury1343
@ashfaquechowdhury1343 5 жыл бұрын
Yes please and then about locks,race conditions and queues as well. Thanks a lot.Loved your explanation.
@kkloikok
@kkloikok 2 жыл бұрын
By substituting a recursive Fibonacci call as the process you can really get a handle on how this helps. ``` def recursive_fib(n): if n == 0 or n == 1: return n return recursive_fib(n-1) + recursive_fib(n-2) ``` pass ``args=[55]`` and watch your CPU utilization hit 100%. Very satisfying (even if it is useless).
@nadik4531
@nadik4531 4 жыл бұрын
Brah i was stuck at work tryna do this. Searched your name and braaaah! Your videos are so helpful.
@woddenhorse
@woddenhorse 2 жыл бұрын
same
@nikitapetrenko1236
@nikitapetrenko1236 2 жыл бұрын
Even these years, this video is really informative about that topic
@chinpokomon_
@chinpokomon_ 5 жыл бұрын
the longest deja vu in my life so far
@sensonpan7994
@sensonpan7994 4 жыл бұрын
same, lmao
@saikhamhein
@saikhamhein 4 жыл бұрын
Same LOL
@1OJosh
@1OJosh 4 жыл бұрын
I swear
@KhoiThinh
@KhoiThinh 4 жыл бұрын
I needed to check github repo first to confirm that I was not watching the same tutorial yesterday ;))
@frederikwaltherliberothchr757
@frederikwaltherliberothchr757 4 жыл бұрын
Hahaha! Same
@8koi139
@8koi139 3 жыл бұрын
I was pretty scared of touching this topic, but you made it so easily to understand!
@neolithian
@neolithian 5 жыл бұрын
We need at least a dozen of Corey Schafer in our programming learning life. I wish everyone would follow your method. You are simply the best! God bless.
@aadityalolariya2751
@aadityalolariya2751 2 жыл бұрын
Absolutely amazing video. Well if anybody is getting any error for the same code as in the video then use the main function in your code. I had encountered it (don't know why)...
@SHA256HASH
@SHA256HASH 5 жыл бұрын
This is the second video I've watched from your channel and was blown away at the quality of the content and the clarity of the concepts explained. Once I get into a data analyst role, I will support you on Patreon. Until then, I will watch every other video! 😁Thanks a million!
@coreyms
@coreyms 5 жыл бұрын
Thanks! That helps more than you know. I really appreciate it!
@vincent.borreux
@vincent.borreux 2 жыл бұрын
I struggle a bit to run this trivial pgm on my venv python 3.11.1. Ultimately, this one has run properly ... Thanks a lot anyway for this tutorial, which I definitely like a lot ! Hope this helps: import time from multiprocessing import Process start = time.perf_counter() def do_something(seconds=1): print(f'Sleeping {seconds} second(s)...') time.sleep(seconds) return f'Done Sleeping...{seconds}' if __name__ == '__main__': p1 = Process(target=do_something, args=[2]) p2 = Process(target=do_something, args=[3]) p1.start() print(f"Process 1 pid {p1.pid}") p2.start() print(f"Process 2 pid {p2.pid}") p1.join() p2.join() finish = time.perf_counter() print(f'Finished in {round(finish-start, 2)} second(s)')
@wylde780
@wylde780 5 жыл бұрын
Nice work with the real world example not only does it help the learning process after the foundation has been laid. It also encourages thought on personal projects that will help you apply this new-found knowledge.
@drforest
@drforest 10 ай бұрын
That last piece showing how to switch from processes to threads for testing was super cool. Thanks for this great video.
@kylebaker8502
@kylebaker8502 3 жыл бұрын
Hi Corey, great video covered a lot, one thing I was hoping to see here which I did not was using Pool from Multiprocessing to pass multiple parameters to a class or function. I ended up finding it after some digging using starmap with zip instead of map. To pass constants with this method import itertools and use the repeat inside the zip. Hope this helps anyone looking for this.
@cetilly
@cetilly 3 жыл бұрын
Wow! The use of map() is brilliant way to automatically batch the jobs for each process. Thanks for showing this!!
@edwardwong654
@edwardwong654 2 жыл бұрын
Corey is great. When you help people, and I mean really HELP people, he is just a saint. I am on Python 3.9.12 and my code does not execute properly, even though I am copying verbatim and I a stuck at the 10:20 mark. After I put in the p1/p2.start() statements, the script runs without printing the "sleep" statements. The final print statement does execute. But after I put in the p1/p2.join() statements, I get a runtime error: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ...
@sayhellotoroy
@sayhellotoroy 2 жыл бұрын
same here
@musicman_99
@musicman_99 Жыл бұрын
another one with same error
@deepgrewal9862
@deepgrewal9862 4 жыл бұрын
You are the best teacher I have ever had. I wish I could give this video a million likes.
@awndlr
@awndlr 5 жыл бұрын
thx so much, already implemented that into my webscraping scripts and the speed gain is tremendous. you’re explaining this so well an clear. thx a lot.
@sellosejake
@sellosejake 4 жыл бұрын
Hear trying to achieve the same
@user-he4ef9br7z
@user-he4ef9br7z 5 жыл бұрын
thanks man , you are a savior , after watching your videos on multiprocessing and threading i could finally make a project i wanted to make for a long time - python which counts the number of times i press each key and store them in an excel file for each day
@bigbangdata
@bigbangdata 5 жыл бұрын
Thanks for the smooth and didactic intro to multiprocessing, I've done this a year ago during a fast and furious degree and needed a refresher. I have a case in which this will come in handy although it's hundreds of lines of code, but I bet I can wrap them all into a function to be the executor's target.
@drygordspellweaver8761
@drygordspellweaver8761 2 жыл бұрын
A function with hundreds of lines might not be such a good idea lol. Try learning Classes and Methods and I’m sure you can incorporate multiprocessing in a much more encapsulated way
@bigbangdata
@bigbangdata 2 жыл бұрын
@@drygordspellweaver8761 right you are, god knows what insanity I was up to 2 years ago.
@drygordspellweaver8761
@drygordspellweaver8761 2 жыл бұрын
@@bigbangdata haha must have been a blast
@churchillagutu7287
@churchillagutu7287 2 жыл бұрын
This is A GREAT VIDEO!!!Thanks for explaining each step of the code
@SebaSalinas91
@SebaSalinas91 5 жыл бұрын
Thank you so much for this quality of videos on Python!... Cant tell you how much i`ve learned with them. Keep it coming! :D PS. Any GUI tutorials coming? PS2. Greetings from Chile!
@mikenesbitt8052
@mikenesbitt8052 3 жыл бұрын
Wow!! I kind of understood multi processing, but this polished up everything I know into a very sparkly vision.... Thank you!!!
@attilagyen1446
@attilagyen1446 4 жыл бұрын
In newer versions of Python instead of using a list in args you should use tuple. So instead of writing this args=[1.5] write this args=(1.5,). It's okay to use the list version, but you will get a warning. Good tutorial btw.
@salfredogonzal_
@salfredogonzal_ 2 жыл бұрын
Not anymore ... self = reduction.pickle.load(from_parent) AttributeError: Can't get attribute ...
@mrthwibble
@mrthwibble 2 жыл бұрын
By 22:48 this video had saved a project I've been working on for close to three years when the volume of data started taking to long to process and the memory load cause it to swap badly. Thank you. :)
@robertlee6529
@robertlee6529 5 жыл бұрын
Just a reminder, on Windows you need to add "if __name__ == '__main__'"
@coreyms
@coreyms 5 жыл бұрын
Thanks! I usually run my videos through my Windows laptop before posting just to make sure it's the same for both operating systems, but I didn't do that this time. I appreciate the comment!
@ThePasupathi
@ThePasupathi 5 жыл бұрын
@@coreyms Somehow, me too facing errors without using main function. Even after using the if __name__ == '__main__', i got my output in different way . done sleeping 5 second(s)..... done sleeping 4 second(s)..... done sleeping 3 second(s)..... done sleeping 2 second(s)..... done sleeping 1 second(s)..... sleeping 3 second(s).... sleeping 2 second(s).... sleeping 1 second(s).... sleeping 4 second(s).... sleeping 5 second(s).... Finished in 5.5 second(s)
@whoa_that_ruled
@whoa_that_ruled 5 жыл бұрын
Just curious (python beginner), where would that go in the script?
@veganath
@veganath 5 жыл бұрын
@@whoa_that_ruled import multiprocessing import time start = time.perf_counter() def do_something(): print('Sleeping for 1 second') time.sleep(1) print('Done sleeping...') if ___name___ == '__main__': p1 = multiprocessing.Process(target=do_something) p2 = multiprocessing.Process(target=do_something) p1.start() p2.start() p1.join() p2.join() finish = time.perf_counter() print(f'Finished in {round(finish-start,4)} seconds')
@orlkeww
@orlkeww 5 жыл бұрын
Add a shebang directive and you wouldn’t need one.
@shivstn98
@shivstn98 5 жыл бұрын
The best instructor in the world... you explain tough things very easily... better than all others(including paid) Instructor 100%👌 Content 100%👌 Quality 100%👌 Explanation 100%👌 Thankyou so much ❤ keep inspiring us...
@coreyms
@coreyms 5 жыл бұрын
Thanks!
@shivstn98
@shivstn98 5 жыл бұрын
That's so humble you replied sir... sir can you please upload numpy and pandas tutorial
@manojk1494
@manojk1494 Жыл бұрын
@@shivstn98 I hope you already found that from his playlist, there are 11 videos for Pandas.
@johntamplin
@johntamplin 4 жыл бұрын
When faced with a python challenge, I ask myself "What would Corey Schafer do?" I then, somewhat ironically, ask myself "Why am I wasting my time asking myself rhetorical questions, when I could be watching one of his outstanding videos."
@SaiSai-gs3pd
@SaiSai-gs3pd 3 жыл бұрын
Have never understood multithreading and multiprocessing this clearly ever before in my life. Would greatly appreciate if you can post a video on locks. Deadlock and livelock and how to resolve them. Thanks for the service you are doing to so many people.
@rakeshkumarkuwar6053
@rakeshkumarkuwar6053 4 жыл бұрын
Thank you, Corey, for such a wonderful explanation. The best part of this video is you have used the real-time problem, which helps to understand the concept in more depth.
@siddiksalim5037
@siddiksalim5037 3 жыл бұрын
I came to this video with a working knowledge of python threads, processes and asynchronous coroutines. I must say, this video was very well-executed. It helped me arrange my own thoughts on the matter, and I think anyone new to the concept of multiprocessing would find Corey's explanation easy to understand. I also commend his insistence on showing a practical example, as it helps a lot of people bring new concepts into perspective. Congratulations on a job well done!
@drygordspellweaver8761
@drygordspellweaver8761 2 жыл бұрын
Yeah sure you are a thread expert that’s why you watched a tutorial video. People and their cringey little egos... lmao 🤦‍♂️
@MrRobbi373
@MrRobbi373 5 жыл бұрын
That's the module I used to parallelize and optimize Mosaik smart grid co-simulations in my bachelor's thesis, which I finished this week! :D
@ecevit2804
@ecevit2804 3 жыл бұрын
Waoovv. It's a great video. Even the person who know nothing about coding will understand what you're talking about :) Thanks for description...
@9SMTM6
@9SMTM6 5 жыл бұрын
The reason it finished the 1s future in the Multiprocessing part after the 2 and (just after) the 3s future is that the PoolExecutor limited the threads to 4 (you said yourself that it chooses the number of concurrent threads based i.e. your hardware). And your list of times to wait is reversed in duration. So it only started the future for the 1s duration AFTER the first future finished, the one with a duration of 2s. So the 1s future finished after 2+1= 3s plus overhead, so slightly after the 3s future. Edit: Aaand you realized that just shortly after. Was wondering why noone else wrote this before.
@drygordspellweaver8761
@drygordspellweaver8761 2 жыл бұрын
No one mentioned it because it’s literally spelled out in the video 🤦‍♂️
@9SMTM6
@9SMTM6 2 жыл бұрын
@@drygordspellweaver8761 I did wait some time to write that comment. You meanwhile were too lazy to read my whole comment.
@drygordspellweaver8761
@drygordspellweaver8761 2 жыл бұрын
@@9SMTM6 Literally everyone who watched the video understood what was going on. Only you saw fit to go out of your way to write a comment to "explain" what was literally explained in the video like 5 seconds after it happened. You are insufferably arrogant and dense.
@TonyStark-cp3tj
@TonyStark-cp3tj 2 жыл бұрын
That was Brilliant! You are the best!
@anuprshetty6391
@anuprshetty6391 2 жыл бұрын
Hi Sir, As you mentioned, I am interested to learn advanced threading concepts like race condition, mocks, etc. Please make videos on that too. I will be waiting... Between really nice video. Thanks for the effort.
@mahathirmohammad8635
@mahathirmohammad8635 4 жыл бұрын
I watch both of threading and multiprocessing video together without skipping a moment. I enjoy your video more than movie.
@khmsalhuas1
@khmsalhuas1 5 жыл бұрын
Dude you are amazing! Excellent excellent stuff! Truly great educator and we own you a lot. I have a question. What if my function takes more than one argument. Will the ProcessPoolExecutor instance (executor) be able to map the function to however number of arguments for every process?
@drygordspellweaver8761
@drygordspellweaver8761 2 жыл бұрын
It’s the map method that would handle that. The function then would naturally need to accept multiple arguments
@rembautimes8808
@rembautimes8808 4 жыл бұрын
I joined the C# start to finish course last year and am so glad that you did this video. Hope you are keeping well. Thanks for this well explained tutorial
@sebastianbarriap.6871
@sebastianbarriap.6871 5 жыл бұрын
Excelent tutorial!, just for completeness, i'd love to see some tutorial on python coroutines, thanks a lot.
@amitdwivedi9951
@amitdwivedi9951 5 жыл бұрын
What is that
@aparnamoorthi1117
@aparnamoorthi1117 4 жыл бұрын
Me too
@Jonix-redhat
@Jonix-redhat 3 жыл бұрын
thx Corey for this great video! I'm a biomedicine master student from Sweden. This tutorial enabled me to use multiprocessing in a program I write for dna analysis during my master thesis!:)
@hacker2ish
@hacker2ish 5 жыл бұрын
You could have put the image processing and the reading and writing from disk in separate functions you could have used threads for the IO and procesess for the processing
@javedalam7383
@javedalam7383 3 жыл бұрын
One of the best video of Multiprocessing I have ever seen. Thanks for all your effort and teaching us.
@gerhardboenisch5618
@gerhardboenisch5618 4 жыл бұрын
I had to add if __name__ == '__main__': for it to work in Python 3.7.6 I did not get the print output, but the images were processed
@XinhLe
@XinhLe 3 жыл бұрын
The same for me! def main(): if __name__ == "__main__": main() and it worked, anyone can help to explain why?
@macieks3604
@macieks3604 2 жыл бұрын
Great Video!!! As always, the best, most logical explanation on KZbin.
@vincenzo3574
@vincenzo3574 5 жыл бұрын
What's the difference between time.perf_counter() and time.time()? Why should I use one over the other?
@TheOiish
@TheOiish 5 жыл бұрын
The resolution of perf_counter() is higher than that of time().
@Mayur7Garg
@Mayur7Garg 5 жыл бұрын
But he is eventually rounding off😅
@drmezo7
@drmezo7 3 жыл бұрын
This is the best explanation of that topic I have watched so far! Many thanks, Corey!
@vipuljoshi7367
@vipuljoshi7367 4 жыл бұрын
31 people didn't find a piece of code they could copy and paste for their work.
@BeattapeFactory
@BeattapeFactory 4 жыл бұрын
#indians sorry
@dylanbruner8348
@dylanbruner8348 4 жыл бұрын
your sopposed to follow along
@aeroscience9834
@aeroscience9834 4 жыл бұрын
NΛVΛZ don’t be jealous
@tarunreddy7
@tarunreddy7 4 жыл бұрын
57 now
@rzufvideo
@rzufvideo 4 жыл бұрын
incredible video! you've covered more information than all articles all together that i've read so far - thank you for that!
@piiumlkj6497
@piiumlkj6497 5 жыл бұрын
Any computer vision tutorial coming ? Would be awesome !
@agsantiago22
@agsantiago22 3 жыл бұрын
WOW! What a great KZbin channel! Congratulations and THANK YOU! I am recommending it to all my students!
@yandere_lover-tl9zs
@yandere_lover-tl9zs 4 жыл бұрын
Bro, can you tell me why this error pops up?? I'm trying to print the return value of the function. concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
@concert_music
@concert_music 4 жыл бұрын
Very good video, though yes, this is really frustrating me too!
@TheEpicPineapple56
@TheEpicPineapple56 4 жыл бұрын
@@concert_music You have to encase your code in that > if ___name___ == "__main__": thing. This is done to avoid unwanted side effects, and multiprocessing and concurrent.futures libraries both force programmers (at least on Windows; Corey is on Mac) to do that. Hope that helps! :D
@lethalfang
@lethalfang 4 жыл бұрын
Took me a while to find the issue: stackoverflow.com/questions/61860800/running-a-processpoolexecutor-in-ipython Basically I added those lines on top when I'm using MacOS on jupyter-lab: import multiprocessing as mp mp.set_start_method('fork')
@scholar_sukhiya
@scholar_sukhiya 4 жыл бұрын
thanks a lot Mr Schafer!! your videos helps me a lot, plus the tips u give to keep the code clean are making my projects look awesome
@unperrier
@unperrier 2 жыл бұрын
At 40:40 you're omitting one important thing: GaussianBlur() is an extension using a library written in C. So it can processing the image in a separately managed thread in C and relinquish execution to the python interpreter to process python bytecode on other python threads. So it's essentially a bad example, because at the low-level it behaves like multiprocessing in the sense that it can run things outside the python interpreter (multiprocessing does that in a separate process, PIL does that in a private thread, in C). So your assessment that the workload is I/O bound is incorrect. It is mostly CPU bound, but the processing you chose released the GIL, that's an unfortunate choice.
@punk0012
@punk0012 Жыл бұрын
i love the sound you made at 5:51
@guardrepresenter5099
@guardrepresenter5099 5 жыл бұрын
for i inrange(1,1000000): print ("Thank you")
@jimmy2643
@jimmy2643 5 жыл бұрын
print("Thank you "*1000000) # :)
@sebastianjost
@sebastianjost 5 жыл бұрын
@@jimmy2643 wow. I didn't know this was possible... thank you!
@joseramoncruztorres2368
@joseramoncruztorres2368 4 ай бұрын
Beautiful sir, I really love the manual way for when you are going to work with different functionalities, like in a producer consumer architecture. But use pools when you are doing a reusable module.. Thanks for this tutorial! My respects! 😎
@hongbingqin6876
@hongbingqin6876 5 жыл бұрын
It's great video. However, I receive the following error when run your code to print results in my windows. But the code works when I run it in aws environment. BrokenProcessPool Traceback (most recent call last) in 3 results = executor.map(do_something, secs) 4 ----> 5 for result in results: 6 print(result) 7 ~\AppData\Local\Continuum\anaconda3\lib\concurrent\futures\process.py in _chain_from_iterable_of_lists(iterable) 474 careful not to keep references to yielded objects. 475 """ --> 476 for element in iterable: 477 element.reverse() 478 while element: ~\AppData\Local\Continuum\anaconda3\lib\concurrent\futures\_base.py in result_iterator() 584 # Careful not to keep a reference to the popped future 585 if timeout is None: --> 586 yield fs.pop().result() 587 else: 588 yield fs.pop().result(end_time - time.monotonic()) ~\AppData\Local\Continuum\anaconda3\lib\concurrent\futures\_base.py in result(self, timeout) 430 raise CancelledError() 431 elif self._state == FINISHED: --> 432 return self.__get_result() 433 else: 434 raise TimeoutError() ~\AppData\Local\Continuum\anaconda3\lib\concurrent\futures\_base.py in __get_result(self) 382 def __get_result(self): 383 if self._exception: --> 384 raise self._exception 385 else: 386 return self._result BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
@surajitchakraborty1903
@surajitchakraborty1903 5 жыл бұрын
@Hongbing , even I am getting the same error in windows. Did you manage to get around it - if so then how ?
@darkfantasy437
@darkfantasy437 4 жыл бұрын
@@surajitchakraborty1903 I'm getting similar errors in pycharm - not exactly the same, but similar (also on windows)
@vadvenurt
@vadvenurt Ай бұрын
29:00 That's exactly what I needed. Processing lists with organized results.
@KurtSchwind
@KurtSchwind 3 жыл бұрын
Seriously learning more from your vids on Python than the Python 'class' I took one time. Clear and helpful. Everytime I watch I learn something new.
@j.caldeira
@j.caldeira 4 жыл бұрын
I do have to thank you @Corey Schafer your work in these videos. I learn a lot from them and in the end I use them on my work to automate some tasks. Some love from Portugal
@sanjeevKumar-eg6hp
@sanjeevKumar-eg6hp Жыл бұрын
I have no words to appreciate your efforts for the method of explanation and content of your videos you even took the trouble of repeating yourself even though you have covered the similar concept in threading video(others would just tell to go watch the threading video) thank you so much for sharing your knowledge.
@sandeepnagendra1531
@sandeepnagendra1531 5 жыл бұрын
Thank you very much. I had written a script to download all the rpms from remote server. It used to take approximately three minutes. With ThreadPoolExecutor, Now it is taking only 20secs. Very happy to implement the concepts on real day to day work scenarios. You are doing a great Job. Please do create videos on advance topics which you mentioned at the last section of the video. (race condition etc). Happy New Year :)
@c.c.1297
@c.c.1297 3 жыл бұрын
Thank You Very Much!!! I now understood how this works. Your way of teaching and explaining are very understandable unlike the other videos that I have watched.
@ishan1701
@ishan1701 3 ай бұрын
Both threading and multiprocessing were very nicely explained. Thanks
@Sparshchokra
@Sparshchokra Жыл бұрын
I am thankful to you for this video. I have come to know that yes conceptually threads and multiprocessing are different but it is exactly same in terms of implementation. Only methods and internal concept is different.
@FreihEitner
@FreihEitner 2 жыл бұрын
I am about to get into using Pandas to gather data from multiple spreadsheets and output cleaned up versions of those sheets as well as generate some simple reporting from the data, and I am so glad that I came across your multiprocessing video! While my use case would not take more than maybe a minute or two to run, I'm big on efficiency and if it can be run in under 15 seconds instead that is a win for me.
@Wallaby1010101
@Wallaby1010101 3 жыл бұрын
This video was a great introduction for a Python newbie like myself, with previous experience using a Future API in R.
@ItzSenaCrazy
@ItzSenaCrazy 5 жыл бұрын
This video came on time. I work with AI with python and, at university, I'm seeing operating system. Now I can join OS and python for do faster algorithms. Thanks Corey. I really like your videos!
@ShubhamKumar-br1ft
@ShubhamKumar-br1ft 4 жыл бұрын
very nice video sir, at 26:32 the intrepretation was superb, i am your fan now
@AnnWong1
@AnnWong1 11 ай бұрын
Thank you Corey, it is brilliant to start from the basics. Really helpful approach and advice !
@fayfeld5900
@fayfeld5900 4 жыл бұрын
Outstanding video! Never learned so much about multiprocessing in python - in just 45 Minutes. Thank you, Corey!
@MrNitKap
@MrNitKap 4 жыл бұрын
Awesome video! I have gone through 20min and will go through rest again later... people should learn from This how to create a teaching video with high amount of content per minute and not waste time of unnecessary sentences. I got enough in 20 minutes to create my stock scraping process in my program.
@romabilibov7612
@romabilibov7612 Жыл бұрын
Awesome tutorial! Thanks showing both methods of handling multiprocessing.
@varunn104
@varunn104 2 жыл бұрын
i cannot thank you enough for all the videos you've put out
@viswanathbarenkala1023
@viswanathbarenkala1023 3 жыл бұрын
Thanks for all the info Schafer, its a great work for community. Was checking this code if the code is not put in main this is giving error. Writing here so that you can edit for any future users. Once again thank you for all the great work and sharing it for free. Also you narration is great and I am tuned to it.
@yuzhu2547
@yuzhu2547 4 жыл бұрын
You're a grandmaster. I really appreciate you put all stuff in ONE video. So easy to navigate. Thank you!
@andrewdok3595
@andrewdok3595 3 жыл бұрын
Yep, this video was 100% helpful. Even the first 10 minutes was helpful, to start with Multiprocessing. Do you have video or lessons for TCP IP connections or\and Bluetooth connections? I teach children's in robotic-Club, we make games on python and robots on Arduino and Lego Mindstorm. So main idea connect to the robot by bluetooth on server PC, and using TCP send information to all client PC's and draw GUI using Real Data. Many years ago I made programm in that way, but not in Python. Your video was like quick start - simple, clear and easy to make, and easy to understand
@patriciodiaz2377
@patriciodiaz2377 2 ай бұрын
Dude this video is pure gold! Thank you a lot for sharing you knowledge. I already subscribed to your channel. Keep up the good work! Greetings from mexico 🫡
Caleb Pressley Shows TSA How It’s Done
0:28
Barstool Sports
Рет қаралды 60 МЛН
Andro, ELMAN, TONI, MONA - Зари (Official Music Video)
2:50
RAAVA MUSIC
Рет қаралды 2 МЛН
УНО Реверс в Амонг Ас : игра на выбывание
0:19
Фани Хани
Рет қаралды 1,3 МЛН
Multiprocessing: используем все ядра
27:50
Python Russian
Рет қаралды 4,4 М.
Python Tutorial: Calling External Commands Using the Subprocess Module
19:01
Next-Level Concurrent Programming In Python With Asyncio
19:19
ArjanCodes
Рет қаралды 186 М.
Unlocking your CPU cores in Python (multiprocessing)
12:16
mCoding
Рет қаралды 314 М.
Reacting to Controversial Opinions of Software Engineers
9:18
Fireship
Рет қаралды 2,2 МЛН
Параллельное программирование на Python
2:03:29
Тимофей Хирьянов
Рет қаралды 69 М.
Python Asynchronous Programming - AsyncIO & Async/Await
25:57
Tech With Tim
Рет қаралды 439 М.
threading vs multiprocessing in python
22:31
Dave's Space
Рет қаралды 605 М.
5 Python Libraries You Should Know in 2025!
22:30
Keith Galli
Рет қаралды 89 М.
5 Good Python Habits
17:35
Indently
Рет қаралды 704 М.
Caleb Pressley Shows TSA How It’s Done
0:28
Barstool Sports
Рет қаралды 60 МЛН