End to end Deep Learning Project Implementation using MLOps Tool MLflow & DVC with CICD Deployment 🚀

Code: github.com/entbappy/End-to-En...
"🚀 Dive into the world of MLOps with my comprehensive tutorial on implementing an end-to-end deep learning project! In this video, we explore the power of MLflow & DVC to streamline the machine learning lifecycle, from data versioning to model deployment.
🔍 Discover how to effectively manage and version your pipeline using DVC (Data Version Control), ensuring reproducibility and easy collaboration among team members. Learn the ropes of MLflow for tracking experiments, packaging code, and sharing models seamlessly.
🧠 Master the art of continuous integration and deployment (CI/CD) specific to machine learning models, empowering you to automate testing, training, and deployment pipelines effortlessly.
🔧 Whether you're a beginner or an experienced data scientist, this tutorial provides valuable insights and hands-on guidance for implementing MLOps practices into your deep learning projects.
▬▬▬▬▬ Contents of this video ▬▬▬▬▬
00:00:00 - Introduction
00:10:50 - Project Demo
00:15:34 - MLflow Demo
01:03:09 - MLflow with Dagshub
01:20:04 - MLflow with AWS
01:44:34 - Github Repository Setup
01:47:08 - Project Template Creation
01:59:52 - Requirements Installation & Project Setup
02:06:31 - Logging, Exception & Utils Modules
02:26:58 - Project Workflows
02:30:02 - Data Ingestion Component
03:08:16 - Prepare Base Model Component
03:28:35 - Model Trainer Component
03:41:37 - Model Evaluation Component & MLflow Integration
04:23:09 - DVC Pipeline
04:48:03 - Prediction Pipeline & User App
04:57:09 - Dockerization & AWS CICD Deployment
Check out my other playlists:
► Complete Python Programming: • Complete Python Progra...
► 100 Days of Machine Learning playlist: • 100 Days Of Machine Le...
► Statistics For Machine Learning: • Statistics For Machine...
► Object Detection Using YOLO v6: • YOLO v6 | Object Detec...
► Object Detection Using YOLO v7: • YOLO v7 | Object Detec...
► Sign Language Detection Using YOLO v5: • Sign Language Detectio...
►ONNX (Open Neural Network Exchange): • ONNX (Open Neural Netw...
😀Please donate if you want to support the channel through Buy me a coffee: www.buymeacoffee.com/dswithbappy
This channel focuses on providing content on Data Science, Artificial Intelligence, Machine Learning, Deep Learning, Computer Vision, Natural language processing, Python programming, etc. in Bangla and English.
My mission is to provide inspiration, motivation & good quality education to students for learning and human development, and to become an expert in Artificial Intelligence, Machine Learning, Deep Learning, Computer Vision, Natural language processing, Python programming, and so on.
#dswithbappy aims to change this education system of Bangladesh.
I believe that high-quality education is not just for the privileged few. It is the right of everyone who seeks it. My aim is to bring quality education to every single student. All I need from you is intent, a ray of passion to learn.
Thanks!
#dswithbappy
Connect with me here:
Twitter: / bappy913873
Facebook: / dswithbappy
instagram: / entbappy
linkedin : / boktiarahmed73
Github: github.com/entbappy
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY KZbin CHANNEL

[@oyebamijimicheal8372]

This guy is so so underrated. An eloquent Data Scientist ❤️

[@tuannguyentranle7151]

You deserve my subscription, please make videos with details about MLops like that. Happy New Year!

[@parthibrathinam9050]

Amazing, one of the very first , easy to understand , e2e DL model deployment

[@mdabdullahalhasib2920]

Love it. Stay tuned my brother.

[@randomshite998]

Wow, there is not even a single end to end video on the internet. You made a complete package Thank you very much man. You earned my subscription, like and a share.

[@ZaheerKZ]

I deeply appreciate the well-structured project and video you've provided. Thank you from the bottom of my heart💖

[@dswithbappy]

You are so welcome!

[@jitheshgopinathan]

Amazing playlist. Thanks so much.

[@mojtabaaajami1112]

You have done a great job. I really admire your knowledge and how proficient you are in all and every details of an ML project.

[@arri5812]

thank you , till 3 hours 41 mins.....looking forward to complete and learn more

[@sawfhsawfh00]

Thank you so much Bappy 😀

[@nareshroyal001]

learnt everything at one go

[@DataPulse]

Great work. Thanks Buppy. Keep going. Full support

[@dswithbappy]

Thanks :)

[@donkeyally]

Thank you sir!! I really appreciate your effort to make this video for us!!!

[@dswithbappy]

It's my pleasure

[@Infinity-zc2ms]

Very Helpful, Thanks a lot ❤

[@vudumulanagasairahul1298]

your hardwork is appericated❤❤

[@dswithbappy]

Thank you!

[@MDAsadullahShibli]

Well Defined , well-structured project 🥰

[@dswithbappy]

Thank you!

[@c52obsagp50]

Very nice explanation❤

[@advaithirlekar9433]

Very nice approach! Keep up the good work!

[@olabintanibraheem8111]

Thanks bappy.

[@subashlingden9883]

Learned a lot thank you!

[@mayur.chavan]

very well explained.

[@kdAbuj]

Thank you Bappy bhaiii

[@PRIYASINGH-kq9fc]

Thanks a ton❤

[@thineshkumar5068]

Please teach us GitHub actions from basics. I'm expecting you would do that soon for us😅.

[@shivrajshinde1738]

this is really good one. however, if you want to bring the new model which is in stage how do i do it?

[@map-creator]

Thank You.

[@gauravmalik3911]

Hi, thank you for the video, although I'm on early part of the video, but I'm curious about two things, are we using fastapi in this project, and second are we using something called feature pipeline in it, I'm new in mlops, so i want to understand what is feature pipeline, ingestion pipeline. Also, I guess this video considers all those things that I wanted to learn such as DVC, MLFLOW, DOCKERISATION , BIG THANKS

[@InsightfulAIAvenue]

This is super. Thanks

[@dswithbappy]

Welcome

[@soumeshnayak4546]

Thank you sir for this project

[@dswithbappy]

Thank you!

[@haz5248]

just copy the following to copy the model from the artifacts folder to the model folder, destination_directory="model" file_path=r"artifacts/training/model.h5" if not os.path.exists(destination_directory): os.makedirs(destination_directory) shutil.copy(file_path, destination_directory) print("Model copied to Model directory")

[@specreiny775]

Thanks for your efforts in this video, it helps me a lot with my capstone project. In the future, could you please teach us an end to end project using AWS sagemaker?

[@faisalhameed7534]

Hey You are doing some amazing work here , one quick sugesstion please be a little slow while explaing the code .I have subscribed your channel after seeing your vedios on the other channel .

[@mr.strange498]

thank you soo much

[@dharmaraj7411]

If you could do end to end project like focusing on one cloud like AWS and the best practices of mlops with aws in the same way best practices of mlops with azure will really help everyone in their realtime projects.

[@maximiliano2106]

hello! Stages have been deprecated in the new Model Registry UI. It seems they are changin the way how to handle this in the new mlflow version. do you know how to work with aliases and tags?

[@SAGAR-ox6ks]

WHAT ALL EXTENSION YOU USE IN THE VSCODE? PLEASE SHARE.

[@mitijraul7126]

Very good work bro can you give the dataset you used?

[@UdiWanKenobi]

Dear Bappy sir after everything when using dvc repro I am getting ModuleNotFoundError showing I dont have module named 'src', please help me to rectify the error

[@gayathrik1517]

Your great😎

[@dswithbappy]

😀

[@saitarak1824]

I'm getting the following error. Error when deserializing class 'InputLayer' using config={'batch_shape': [None, 224, 224, 3], 'dtype': 'float32', 'sparse': False, 'name': 'input_layer'}. Exception encountered: Unrecognized keyword arguments: ['batch_shape']. Please help me in sorting this when loading the odel in model training

[@curdyco]

what if i train the model on google colab (because i need gpu) and then i save everything locally but this way i don't get an conda.yaml file in my mlruns, is that a problem? also please launch a paid course where you deploy heavy tensorflow projects (cv or nlp, large datasets) using mlflow, dvc and dagshub and github-actions

[@out-of-sight]

Hi thank you for the project, I have a question. You are making a binary classification but what should we do to make it multiclass classification?

[@dswithbappy]

just change the output layers , select the neuron with respect to the classes you have

[@vishwarajkolekar1800]

use softmax as output layer

[@kashifsadiq4145]

make NLP projects with mlops practices

[@SaurabhSingh-kz7mk]

Please help I'm getting module not found for src/cnnClassifier when running dvc repro dont know why the code was working fine with dvc but when running dvc repro it says no module found src/cnnClassifier...

[@SigmaScorpion]

Yes same thing happens with me too....

[@arri5812]

thank you till 1 hour 3 min

[@arri5812]

thank you till 2 hours 42 min

[@nijalshakya6770]

Wow

[@_YuvaKumarIrigi]

are there any prerequisites for this project bro. can you please suggest if any required.?

[@dswithbappy]

Python & basic DL

[@rezamahmoudi163]

please deploy for edge device

[@prithwishghosh1449]

During running dvc repro. । got an error that output : artifacts/prepare_base_model could not exist

[@prithwishghosh1449]

@dswithbappy Please solve my problem

[@haz5248]

mate can we not use something like react for the frontend bits , it'l be much more pretty with it .

[@dswithbappy]

yes we can

[@kashifsadiq4145]

upload mlops more projects

[@md.musfikurrahmansifar5302]

ভাইয়া, আমি basic numpy, pandas, matplotlib, seaborn, Machine learning supervization... এখনো পর্যন্ত করছি। এখন কি করব বুঝতেছিনা,,,industry based learning hcche na mne hy...but kun path e jabo?.... Btw apnar shondhan pelam...i was like emn gem Bangladesh e ase...you need to get highlighted... ❤❤❤❤

[@Rider-jn6zh]

Man how someone can go such indepth to explain and make understand the things easily with baisc things.. Really appreciated🎉

[@dswithbappy]

thanks :)

[@Rider-jn6zh]

@@dswithbappy can you please upload videos on NLP LLM projects with deployment using open AI API.. projects like text summarization( summerizing conversation of customer with service resolver), and text classification,

[@deepsuchak.09]

sir in the last after completing the notebook 4 when we execute after completing the modular code after model evaluation, in my pc it is saving a file called mlruns instead of getting connected to mlflow. pls help @4:11:00 onwards

[@sagarpadhiyar3666]

Bro did you find any solution for this issue? I am facing the same issue

[@jiyabyju]

same

[@deepsuchak.09]

Nope @sagar

[@atharvachoudhari3968]

Hi Bappy, When I run my 04_model_evaluation.ipynb file (from pipeline module) the version run is getting logged and it is visible on mlfllow ui. But when I run the pipeline from main.py, the new model version is getting but it is not showing me the new version run on mlflow ui. What could be the problem?

[@ajee1801]

having the same issue...pls help to fix this @DSwithBappy

[@Shanky1919]

@@ajee1801 remove the commented part and run #evaluation.log_into_mlflow()

[@pr0t0n86]

Uncomment the line number 20 in `cnnClassifier/pipeline /stage_04_model_evaluation.py`

[@muhammadzakiahmad8069]

Same

[@muhammadzakiahmad8069]

@@pr0t0n86 Still not working

[@Codejourneywithme]

the model is predicting all images as normal. Why it might be happening?

[@Shanky1919]

same issue with me

[@user-es2re7jo1z]

anybody having issue with this: 'export' is not recognized as an internal or external command, operable program or batch file. if yes,how you resolved it.

[@haz5248]

ello mate , perhpas ur using windows. In windows u have to use 'set' instead of export

[@user-es2re7jo1z]

@@haz5248 thank you so much.

[@shortspeeches1455]

if you are using terminal within the VScode export command will not work, use ''set" instead of expoprt

[@sohamnimbalkar66]

i am getting error for the library from box import configbox please any one can help me to resolve

[@user-es2re7jo1z]

downgrade your python to 3.8 from your conda environment.

[@stepiprincip8695]

please share your dataset images

[@muhammedjunaisk4835]

Hi, I have one doubt , here is it layer.trainable = False ? @staticmethod def _prepare_full_model(model, classes, freeze_all, freeze_till, learning_rate): if freeze_all: for layer in model.layers: model.trainable = False elif (freeze_till is not None) and (freeze_till > 0): for layer in model.layers[:-freeze_till]: model.trainable = False

[@aiproduction143]

that's transfer learning process

[@ZaheerKZ]

Could you kindly share the link to the dataset, please?

[@dswithbappy]

link is there in config.yaml file

[@ZaheerKZ]

Got it thank you@@dswithbappy

[@anishantony8]

gdown has issues,anyone experienced this issue

[@atharvachoudhari3968]

(UPDATE) Hi Bappy, I got an error towards the end of the MLflow with Dagshub part (of the basic Linear Regression model). when I executed python example.py, I got this Error: mlflow.exceptions.MlflowException: API request to endpoint /api/2.0/mlflow/runs/update failed with error code 403 != 200. Response body: '' (Update): Use git bash terminal instead of of windows powershell (from VS code). Bappy's pc has his default terminal as git bash so he didn't have to do it.

[@priyanshukumarsingh235]

did you find the solution?

[@atharvachoudhari3968]

@@priyanshukumarsingh235 use git bash terminal instead of windows powershell while running the python example.py command . This was told in one of Krish Naik's videos and it worked for me.

[@classicemmaeasy2292]

AWS account is not free😢

[@alhasanmohammedbinquraish960]

Hindi mlops course ❤

[@Shanky1919]

all images are predicted as normal image 😪

[@prithwishghosh1449]

Same thing happened here

[@prithwishghosh1449]

Same thing happened here

[@Shanky1919]

@@prithwishghosh1449 try checking with many random images ,only a few gets predicted correctly

[@samurock100]

OK?

[@gichurumuthomi8578]

You keep repeating the same end to end projects structure; create videos with other tools like kubeflow involved; using apis for real time data, other tools like kafka e.t.c,

[@dswithbappy]

its in pipeline ,, I have plan to do

[@gichurumuthomi8578]

@@dswithbappy Cool,, you're channel will blow up, If you do something different from other KZbinrs, just add those spice,, All the best,and great job

[@nagrajkaranth123]

Bappy bro he will post all unwanted comments take your time to use kubeflow Kafka, ansible, Prometheus you are excellent bro none of the youtubers have implemented modular coding except you which is highly appreciated

[@shubhamkamble9246]

hi @DSwithBappy I am getting error regarding parameter from mlflow raise RestException(json.loads(response.text)) mlflow.exceptions.RestException: INVALID_PARAMETER_VALUE: Response: {'error_code': 'INVALID_PARAMETER_VALUE'} though it was running properly first and tracking experiment but know it is not tracking experiment and showing this error. though i have not done any changes in params files. can anybody get this issue let me know.

[@avinavkashyap8802]

dear bappi sir ,please provide link to your dataset in this example🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏☝☝☝☝🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏🙏

[@dswithbappy]

available in config.yaml

[@arifkhan3262]

bondho

[@Sachdev23]

How much did it cost you to deploy it on AWS?

[@md.musfikurrahmansifar5302]

ভাইয়া, আমি basic numpy, pandas, matplotlib, seaborn, Machine learning supervization... এখনো পর্যন্ত করছি। এখন কি করব বুঝতেছিনা,,,industry based learning hcche na mne hy...but kun path e jabo?.... Btw apnar shondhan pelam...i was like emn gem Bangladesh e ase...you need to get highlighted... ❤❤❤❤

[@md.musfikurrahmansifar5302]

Apnar learning path ta blle valk hy....like kisher pr ki shikhsen..,