Easy Python Progress Bars with tqdm

In this python progress bar tutorial we go in depth about TQDM in this complete guide. The progress bar package tqdm is extremely helpful for any python programmer. Rob Mulla discusses why this package is so helpful and how you can customize it to do much more than just track the progress of a loop. After watching the video you will have learned something new to improve your python and data science coding and efficiency when evaluating your code's progress. Tqdm can be used in a jupyter notebook, command line and disabled easily.
Timeline
00:00 Intro
01:04 Setup
02:09 Tqdm Basics
06:01 Advanced Configs
09:00 Custom Interval
09:48 Disabling tqdm
10:35 Tqdm and Pandas
11:59 tqdm notebook
13:50 Read the docs
Follow me on twitch for live coding streams: / medallionstallion_
My other videos:
Speed Up Your Pandas Code: • Make Your Pandas Code ...
Speed up Pandas Code: • Make Your Pandas Code ...
Intro to Pandas video: • A Gentle Introduction ...
Exploritory Data Analysis Video: • Exploratory Data Analy...
Working with Audio data in Python: • Audio Data Processing ...
Efficient Pandas Dataframes: • Speed Up Your Pandas D...
* KZbin: youtube.com/@robmulla?sub_con...
* Discord: / discord
* Twitch: / medallionstallion_
* Twitter: / rob_mulla
* Kaggle: www.kaggle.com/robikscube
#python #datascience #tqdm

[@OmegaRxNeo]

The best explanation about tqdm in the internet. Direct to the point and with easy explanations! Please continue sharing your knowledge. I found your channel looking for data analysis in Python, and really enjoy your content. Cheer's from Brazil!!!!

[@robmulla]

Thanks so much for the feedback. That really means a lot to me.😊

[@aleks_shcherbakov]

Rob, thank you so much for your hard work and contribution to the development of the data analyst community! From Russia with love!

[@daytodatainc]

Great explanation with practical examples. Thanks!!

[@ritvikrastogi4912]

Hi Rob, warm greetings from India I got to know about your channel when you did a live stream on Abhishek Thakur's channel I've been following you since then and I really like your content, especially how you make it so easy to understand and yet it's still in-depth and cover most of the concepts about that topic I just have one little request please make some tutorials on advanced topics as well like Transformers, Auto encoders etc

[@ramonbr66]

Excelent video! Thank you, Rob.

[@robmulla]

Appreciate the kind words. Thanks for watching.

[@mateusbalotin7247]

Thanks for the video!! Hope you have a great week.

[@robmulla]

Thanks for watching!

[@ignatiusezeani6816]

Thanks so much for this insightful video.🙏

[@IntelliTradeBot]

Thank you, sir.. learning from a Pro is a much better option than reading through the help manual.

[@irfanshaikh262]

@Rob Thanks a tonn for this video. Its really helpful.

[@robmulla]

Glad it was helpful for you. Like & share with all your friends!

[@prasannakukade300]

Very smooth explanation...

[@robmulla]

Thanks! Glad you found it helpful.

[@minhnguyenquang6248]

It's great, thanks bro!

[@robmulla]

Thanks for watching!

[@robmulla]

Here is a gist with the code from the tutorial if you would like to try it yourself: gist.github.com/RobMulla/0fe4acf99001d4657df4e73c93cf3218

[@1811artur]

very usefull

[@vydo4210]

This tutorial is very good.

[@robmulla]

Thanks Vy!

[@tutosolve]

Nice video . From Vietnam

[@robmulla]

Thanks for watching!

[@7686Jin]

Great tutorial! Btw how do I update the progress bar if im using multiprocessing?

[@aaronjenkins223]

Is there a way i can make a progress bar that tracks sub loops? I have a function that loops through Members, and for each member loops through assignments and then writes to a file. Is there a way to have it track every time it writes to the file without having to be in the bottom loop? As it is if i put it on the members it just updates x% after each member and the time remaining is way off because some have a lot more assignments than others.

[@89Sawik]

It would be great to hear some of your thoughts on progress bar for multiprocessing stuff. For me it was a real pain in the back to make it work xD

[@robmulla]

Yea, nested tqdm bars can be annoying. Sometimes it's best to test on a subset of the data and then disable them for the actual run.

[@89Sawik]

@@robmulla i meant traking progress of calculations spread between multiple processes by means id multiprocessing package. I ended up manually updating progress bar after checking number of items in shared queue.

[@vivekjoshi3769]

Hi First of all thank you for sharing this. I have some doubt. When I use sleep(0.00001) like you under for dogs loop with 50000 iterations, it's taking more than 5 minutes to run but in your case I see it ran under 2 minutes. May I know why is it faster in your case but slower in mine? I am using VS code for running this. I am talking about the part at 02:50 Please clarify.

[@mtamjidhossain]

Nice video, do you mind sharing what theme are you using for jupyter notebook?

[@robmulla]

Thanks for the feedback. I'm using the Nord theme which can be installed from here: github.com/arbennett/jupyterlab-themes there are a lot of other good ones too.

[@binotabao7278]

Love your helpful videos. Could you show me the way to set up the underline Separator (_) for a number when typing in code cell, for example 50_000. Thank you.

[@robmulla]

Basically in the newer versions of python you can add _ to any integer and not have an issue. I like to use them to make large numbers easier to read. 1000000 becomes 1_000_000

[@GeoffryGifari]

Can tqdm be initialized _inside_ a loop? so without specifying the iterable's length first

[@nerassurdo]

Can you please show an example of how you would implement this for scikit-learn when doing a RandomForest or XGBoost model and you're hyper-parameter tuning?

[@robmulla]

If you are using something like random grid search or grid search in sklearn - it essentially provides a generator object which you iterate over. You simply need to wrap that generator with tqdm and have a total= the number of iterations you will run. Check out this walkthrough looks to have a good example: www.jpytr.com/post/lazy_tuning/

[@nerassurdo]

@@robmulla Thank you... that looks surprisingly simple, though they did not have the "total=" portion that you mention. By the way, what is your favorite way to tune hyper-parameters since you linked to this "lazy tuning" article? I read about Halving Grid Search recently. Wondering if there's better ways you came across in your experience.

[@telmomachado1490]

Could you make a video teaching how to generate combination number and saving in Excel? and at each end of the 1048576 line it generates a new excel. I have an excel generator but it creates in folders on the side, inside the same file so it is very heavy, but if you make a file in each folder, it would be ideal. like the mega sena, and instead of generating all the possibilities, leave a field to generate combinations only with the numbers that we would prefer. here's the tip. it would be great, thanks.

[@robmulla]

Hey Telmo. Thanks for writing this comment. This sounds like a very specific case and I’m not sure I follow exactly what you are looking to do. It might be something you could have better luck asking on my discord? Hope you figure out a good solution.

[@dingaroo2003]

Hi Rob, greetings from Singapore! I am coding following along with the video, but the section on TQDM and Pandas, doesn't seem to work for me. Both scenarios are a no go for me. Please enlighten. Thank you very much in advance. PS: I find your shorts and videos very helpful, helping me fall asleep, or rather, keeping me away, thinking how I can integrate into my code portfolio!

[@robmulla]

Glad you've found my videos helpful. I'm not sure what exactly your issue is. Sometimes you need to provide the total=len(df) to get tqdm to work correctly on dataframes. Hope that helps.

[@RafaelCataldo]

Thanks for the lesson, sir. I have one question. I defined a function and used the tqdm in it. When I call this function from a .py file, the tqdm does not get recognized, even importing as you shown it here. Do you have any clue how to solve this issue? When I define the function in a cell and then I call it, it works! But the functions are too big and, thus, I wanted it to keep on the .py file I'm using to call some functions when need it. Thanks for your time.

[@robmulla]

Hi. Thanks for the comment. Have you made sure to import tqdm at the top of your .py file? It needs to be imported in the file where it is called. Hope that helps.

[@RafaelCataldo]

@@robmulla That was it! Thanks! Now I have a new problem to solve, but little by little I'll get there lol! Thanks again!

[@robmulla]

@@RafaelCataldo Glad you figured it out!

[@escolaparapobres]

Hey, I'm Brazil, How to implement for current tasks?

[@robmulla]

Hey, thanks for watching. Not sure what you mean by current tasks. If you wrap your current loop in tqdm it will display a progress bar. Hope that helps.

[@resresres1]

I would be impressed if tqdm could give a progress bar when reading a excel or csv file as a dataframe.

[@robmulla]

I think it can if you use chunking when reading. Pretty sure I’ve seen it before.

[@resresres1]

@@robmulla hmm thanks for that comment about chunking. Found some interesting ideas regarding it. I assume it possible to use chunking with other types of libraries that load files? I have one program that uses pythonOCC library to load model .STL files, would love to get a progress bar for reading those files into the program.

[@jessemutiga6055]

Please sir, do a KeyWord Extraction video

[@robmulla]

Thanks for the feedback. I’ll see what I can do.

[@jessemutiga6055]

@@robmulla my humble request to you would be to expand the bert sentiment analysis video to include bert keyword extraction from the Amazon dataset - this would be phenomenal.

[@tradertrader2649]

how about `df.isnull().sum()` how can i use tqdm here

[@robmulla]

Not sure if you can because it's vectorized already.

[@jasonsykes4199]

I am trying to use "with torch.no_grad(): for idx, data in enumerate(tbar):" on my own code, but keep getting a " 0%| | 0/1 [00:00

[@jasonsykes4199]

tbar = tqdm(val_loader, file=sys.stdout)

[@robmulla]

Hey Jason. I’m not aware of any reason why it would work on Kaggle but not vscode. Are you using tqdm.auto? Maybe try forcing it to use the base tqdm?

[@dingaroo2003]

@@robmulla how do you do that?

[@robmulla]

@@dingaroo2003 from tqdm.auto import tqdm

[@AnastasiaZvychaynaya]

Hello! I don't know why but I have an error: out = fake_df.groupby('dog').progress_apply(lambda row: int(row['smell'])**2) my bar: 67%|██████▋ | 2/3 [00:00

[@robmulla]

Sorry I don’t think I can fully debug this for you. But maybe try setting axis=1 in the apply?

[@AnastasiaZvychaynaya]

@@robmulla Setting axis=1 doesn't help, the problem was that numpy.random.randint returns a string in my case (I really don't know why), whereas from documentation it has to be int. Interesting, but this is how a local program is executed (I used to do it in Colab). I simply converted 'smell' data to int. Anyway, thanks for you quick reply!