Parallelize Python Tasks with Joblib

Today we learn how to parallelize Python tasks using joblib.
◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾
📚 Programming Books & Merch 📚
🐍 The Python Bible Book: www.neuralnine.com/books/
💻 The Algorithm Bible Book: www.neuralnine.com/books/
👕 Programming Merch: www.neuralnine.com/shop
🌐 Social Media & Contact 🌐
📱 Website: www.neuralnine.com/
📷 Instagram: / neuralnine
🐦 Twitter: / neuralnine
🤵 LinkedIn: / neuralnine
📁 GitHub: github.com/NeuralNine
🎙 Discord: / discord
🎵 Outro Music From: www.bensound.com/

[@tomjohnson8986]

Straight to the point and clear. Well done. No fluff.

[@gustavobertocco554]

Awesome, this is exactly the type of pipeline I was looking for in order to integrate joblib :)

[@rahulkumarsingh1716]

With every videi , i learn something new! Good Job.

[@mohammedel2035]

Great value as always… Thanks a lot!

[@viniciusfriasaleite8016]

I want to apply parallel computing to the minimax algorithm I implemented on a connect 4 game. It looks like a really good improvement. Thanks for the content

[@west]

Very useful overview, thank you!

[@hopes3211]

Excellent content and easily explained. Awesome 👍

[@marcosoliveira8731]

I´ve learned a great deal here. Thank you.

[@thomasgoodwin2648]

Careful of your benchmarking. The 1st time you run it single threaded, then after that all multi threaded. However, the 1st time you run it, it actually has to go out to the internet and actually download the images. Running after that is a bit of a cheat since it's likely that from then on, the images will be called up from cache rather than re-DLd. Just trying to make the point that when actually benchmarking, you need to average many runs, as initial conditions runs may not be indicative of 'normal' run times. It also illustrates a bit of the difficulty in benchmarking network programs. Many thanks as always. As usual you take "That's way above my brain limits' and transform it into 'Huh, So it's really just that easy.'

[@shinrafahell]

Excellent video!

[@Y.Mahran]

Great video.

[@bw2]

well-timed video, multiprocessing module was giving me a hard time, I had to selectively scrape files (around 4K files each execution) I think this will be fantastic with HTTPX (drop-in replacement of requests module but with async support)

[@atom1833]

Intro is amazing

[@user-wr4yl7tx3w]

Great video

[@mdabdullahalhasib2920]

Awesome

[@shuxiaokai]

So cool 🥰🥰🥰

[@alexanderrubioanasco5352]

Thanks for share you aknowledge, i have a question,. how can i use parallelize if i am training a machine learning model? i use pyspark but, is parallelize better?

[@JohnWalz97]

You should always use 'time.perf_counter' instead of 'time.time()' when you're trying to benchmark code. It's way more accurate.

[@user-wr4yl7tx3w]

I use %timeit. Is it better as well?

[@JohnWalz97]

@@user-wr4yl7tx3w I think so but not completely sure. Since it's designed for timing code I would assume so but don't quote me on that 😅

[@giovannigaiardo]

I am enjoying this channel very much. Would anyone recommend a similar one but focused on JavaScript?

[@splendorman7922]

any advantages over 'unsync' library?

[@alejandrobravo1221]

What font family are you using in pycharm?

[@hamzarashid7579]

You could use httpx instead of requests!

[@alliedeena1141]

is it better than requests module?

[@Blendershick]

Amazing thank you!

[@joshuabardwell2294]

Why did your runtime change from n_jobs = 8 to -1?

[@mallikarjunrterdal]

Multi processing with shared memory possible?

[@NimaqAlizadeh]

👌👌🌹

[@user-wr4yl7tx3w]

Is it faster than JAX pmap?

[@PatrickSteil]

That stupid easy. So does colors2 get returned with all the results from each run appended to a list?

[@RidingWithGerdas]

I am currently multithreading my web scraping project, opening multi browsers and clicking many things with Selenium. Would this improve anything in my case?

[@HalfEatenMushroom]

Hey, I know it's a year later but Joblib is pretty much just a wrapper for either multiprocessing or mumtithreading depending on your specified preference

[@sarimbinwaseem]

It is separating the input list and making function parallel.. I want to speed up a function which is opening a xlsx file which take 22 seconds to open in openpyxl library.. Is thqt possible to speed up?

[@SageBetko]

Try the read- or write-only optimized modes in openpyxl.

[@sarimbinwaseem]

@@SageBetko Thanks.. I'll try.

[@unknown-cj8gv]

You should be show only work and face except others things, this is thing reduce your efforts

[@joshdheda8776]

why use a stupid example?