Progress Bars for Tracking the Progress of Data Science Workflow (Tqdm Python library)

Frustrated when the machine learning model is training? Ever wanted to track the progress of your machine learning models as it trains? In this video, I will be showing you how to use the Tqdm library in Python to monitor the progress of your machine learning model as it trains or also monitor the progress of any steps in the data science workflow. Particularly, we will creating a progress bar in a Jupyter notebook that will automatically update when your model is nearing completion.
👉 Code github.com/dataprofessor/code...
👉 Install tqdm in Python: pip install tqdm
🌟 Subscribe to this KZbin channel kzbin.info...
🌟 Join the Newsletter of Data Professor newsletter.dataprofessor.org
🌟 Buy me a coffee www.buymeacoffee.com/dataprof...
🌟 Download Kite for FREE www.kite.com/get-kite/?...
⭕ Playlist:
Check out our other videos in the following playlists.
✅ Data Science 101: bit.ly/dataprofessor-ds101
✅ Data Science KZbinr Podcast: bit.ly/datascience-youtuber-p...
✅ Data Science Virtual Internship: bit.ly/dataprofessor-internship
✅ Bioinformatics: bit.ly/dataprofessor-bioinform...
✅ Data Science Toolbox: bit.ly/dataprofessor-datascie...
✅ Streamlit (Web App in Python): bit.ly/dataprofessor-streamlit
✅ Shiny (Web App in R): bit.ly/dataprofessor-shiny
✅ Google Colab Tips and Tricks: bit.ly/dataprofessor-google-c...
✅ Pandas Tips and Tricks: bit.ly/dataprofessor-pandas
✅ Python Data Science Project: bit.ly/dataprofessor-python-ds
✅ R Data Science Project: bit.ly/dataprofessor-r-ds
✅ Weka (No Code Machine Learning): bit.ly/dp-weka
⭕ Recommended Books:
🌟kit.co/dataprofessor
✅ Hands-On Machine Learning with Scikit-Learn : amzn.to/3hTKuTt
✅ Data Science from Scratch : amzn.to/3fO0JiZ
✅ Python Data Science Handbook : amzn.to/37Tvf8n
✅ R for Data Science : amzn.to/2YCPcgW
✅ Artificial Intelligence: The Insights You Need from Harvard Business Review: amzn.to/33jTdcv
✅ AI Superpowers: China, Silicon Valley, and the New World Order: amzn.to/3nghGrd
⭕ Stock photos, graphics and videos used on this channel:
✅ 1.envato.market/c/2346717/628...
⭕ Follow us:
✅ Medium: bit.ly/chanin-medium
✅ FaceBook: / dataprofessor
✅ Website: dataprofessor.org/ (Under construction)
✅ Twitter: / thedataprof
✅ Instagram: / data.professor
✅ LinkedIn: / chanin-nantasenamat
✅ GitHub 1: github.com/dataprofessor/
✅ GitHub 2: github.com/chaninlab/
⭕ Disclaimer:
Recommended books and tools are affiliate links that gives me a portion of sales at no cost to you, which will contribute to the improvement of this channel's contents.
#tqdm #jupyter #scikitlearn #jupyternotebook #artificialintelligence #ai #machinelearning #bigdata #datascienceproject #randomforest #decisiontree #svm #neuralnet #neuralnetwork #supportvectormachine #python #learnpython #pythonprogramming #datascience #datamining #bigdata #datascienceworkshop #dataminingworkshop #dataminingtutorial #datasciencetutorial #ai #artificialintelligence #tutorial #dataanalytics #dataanalysis #machinelearningmodel #dataprofessor

[@DataProfessor]

👉Watch this video next (How to learn data science in 2021) kzbin.info/www/bejne/pYOZaGOKrdybbpo Support this Channel 👇👇👇 🌟 Buy me a coffee www.buymeacoffee.com/dataprofessor 🌟 Download Kite for FREE www.kite.com/get-kite/? 👉 Subscribe to this KZbin channel kzbin.info 👉 Join the Newsletter of Data Professor newsletter.dataprofessor.org

[@ayarikhawla5725]

Really Really amazing. Never know this before. Thanks for sharing 👍 😊

[@DataProfessor]

Glad to hear that it's helpful 😊

[@shrutijain1628]

This is actually really helpful 💯

[@DataProfessor]

Glad to hear that! Thanks for watching Shruti!

[@arielleung3917]

I like this kind of "Wow I can do that?!?!" surprise. Thank you!

[@DataProfessor]

A pleasure!

[@divyakarade4378]

Thanks for this useful information 👍😊

[@DataProfessor]

Glad to hear that!

[@absar66]

Great 👍👍👍 many thanks

[@DataProfessor]

A pleasure!

[@jeffwong1310]

I wonder if I can see the progress when I run the sklearn GridSearchCV, I tried it with verbose =3 but the progress status is minimal

[@DataProfessor]

Nope, the verbose is minimal as you mentioned and we will have to implement this manually using "for loops" to iterate through the elements of a list of hyperparameters in your hyperparameter search experiment. If you want to optimize 2 hyperparameters then you can look into doing a nested for loop e.g. for i in j fo j in i:

[@emanuelvaldes1216]

Professor, is there any way to use the progress bar for tracking RandomizedSearchC or GridSearchCV?

[@TinaHuang1]

This is so cool 😎

[@DataProfessor]

Thanks for watching Tina!

[@sevindaherath]

Thank you

[@DataProfessor]

Thanks for watching!

[@dr.merlot1532]

lol, I didn't know about this. I usually just code a print percentage: print (100*I/N) where I is ith ith iterate out of N total. But since I don't want it to print for each I, I usually put an if statement with the modular arithmetic function so that it only prints after every 10% complete. I wonder how much resources my self made progress bar eats up. Its not the most computational intense part of my simulations but, I assume the simulation time for the progress bar can make a bit of difference in finishing time. Edit: I also use the round() function. Usually progress bars are the least of my worries for my research.

[@DataProfessor]

Awesome implementation details! The progress bar is a cool eye candy and I definitely agree that it would consume some resource though not a whole lot, a text based version as you had described would be a better option for more consuming simulations.

[@dineshrajnkl]

@Dr. Merlot - Is it possible to provide an example code of how you are using it? Thanks

[@zeciljain7189]

Hi Sir, loved your simple explanation on this much needed feature. I'm the same issue while 'model.predict' stage where the cell block is running for too long a time and I don't have any means to track its progress or even determine if its functioning properly or not. Please suggest a way by which we can use tqdm library for model.predict stage as well!!

[@arianrahman4840]

Never knew i needed this , now i'm gonna use it every instance i get lol , thanks

[@DataProfessor]

Awesome!

[@UrbanxWishes]

Thanks so much for this valuable information! I would like to use the progress bar when performing grid search. In that case, how would I set up the parameter inside tqdm()?

[@DataProfessor]

we will have to implement this manually using "for loops" to iterate through the elements of a list of hyperparameters in your hyperparameter search experiment. If you want to optimize 2 hyperparameters then you can look into doing a nested for loop e.g. for i in j fo j in i:

[@zadilkhwaja]

Hi sir I had made a Hugo website using your course from Ken's channel but now I'm facing an issue I want to edit the site how do I do it cause I've already launched it... Should I edit the MD file and then replace it with the older one in GitHub? Plz guide

[@DataProfessor]

Yes you would need to edit the MD files and then re-compile the static HTML files again, then replace the newly generated files with the existing ones on the GitHub so that the github.io website would be updated as well.

[@var5723]

This is great... But is it possible to work with the progress bar when not working with iterables. For example when simply training a RandomForest model, know how much time is left until the model is completely trained. That's be helpful.

[@alexperrine3497]

How would you implement this using GridSearch?

[@DataProfessor]

Hi Alex, although scikit-learn has a function for performing grid based parameter optimization, it would not work with tqdm. To overcome this we would need to manually set the hyperparameters by iterating through a list of hyperparameter values using a for loop. I have shown how to do this for optimizing the n_estimator hyperparameter of random forest. To optimize more you will need to nest another for loop in the first for loop.

[@alexperrine3497]

@@DataProfessor Thank you for the video and your response! Nested for loops, time to get learning!

[@diprachakraborty3776]

This is excellent. I want to do real time data science projects. Can u help and guide pls.

[@DataProfessor]

Hi, I would recommend to look into computer vision, which would allow to use real time image data

[@StatusixinKirubhakaran]

Sir, I'm studying B.tech in Biotechnology and I have programming knowledge in Python..Can I get a job as a Bioinformatician?

[@arielleung3917]

I think in US any remotely STEM-related student has the capability of doing an entry-level data analyst job. But bioinformatician is a totally different story.

[@StatusixinKirubhakaran]

@@arielleung3917 I'm from India

[@DataProfessor]

Yes I agree with Ariel that if you have a STEM related background the transition shouldn’t be too difficult even those outside of STEM with the right mindset can also dive into the data domain, programming or ML are technical knowhows while there are several other soft skills that is also important such as business acumen, communication, storytelling, etc. As for becoming a Bioinformatician with your background in biotech that should also be easier.

[@StatusixinKirubhakaran]

@@DataProfessor Thank you for your reply sir, I want to be a Bioinformatician