Model Monitoring Deep Dive

#machinelearning #datascience #modelmonitoring
This is final part of model monitoring videos in which i do a deep dive on what is model monitoring, techniques available to monitor model and take through one of the technique
In case if you have not watched my previous part they are below
Part 1 - • Model Monitoring Overv...
Part 2 - • Model Monitoring - Con...

[@optimizedml]

Hi Srivatsan, Great videos. The near-practical approach you take to explain concepts in your videos is impeccable. Most of the ML tutorial playlist tend not to delve into monitoring at all. But you have given an awesome, holistic view of the same.

[@muraki99]

Nice explanation of Model Performance Monitoring

[@rohitsuryawanshi7352]

Thanks a lot for this video. As you mentioned in this video that you will come up with a separate video on various techniques to deal with Data/Model Score Distribution. Please may I request you to come up with this topic when you get time. The content on MLOps is very minimal over the net as of now. But I feel it would be a hot topic after 5-6 months, so let's prepare beforehand.

[@jagan4269]

Hi Srivatsan, this presentation is a GEM. Kudos to you 👏for giving it to everyone.

[@swaroopsss2008]

Hi Srivatsan, great video.

[@RaviTeja-zk4lb]

In psi, when we doing service distribution, do we need to have ****service data length*** to be the *same* as ***training data length***. because if we had less data in service data or more then when binning we may not get good points in each bins. Am I right? or wrong?

[@AIEngineeringLife]

Ravi.. Not necessary in all cases. It depends on your data pattern though.. You need to check what is the best granularity to calculate psi that looks similar to training data. This is very data specific. It can be 15 min or hourly or daily based on business

[@rohitsuryawanshi7352]

Thanks a lot for video on Model Monitoring. There are very less resources available on this and you have created a video as well. May I please request you create a detailed video on Model Monitoring(Comparing Distribution of Validation vs Prod data, or any other technique) then Model Retraining, Online Learning, Active Learning, Incremental/Lifelong Learning from theory as well as practical implementation way. Many thanks for what you are doing for the Data Science Community. The entire Community is learning together to make everyone's life easy. :)

[@AIEngineeringLife]

Sure Rohit.. Will follow up with a detailed video next year

[@rohitsuryawanshi7352]

@@AIEngineeringLife Will wait for it. There are very less resources/material/videos over the net on those topics.

[@Harreesh555]

Log 0 will become undefined right if there are no scoring values in a particular decile percent range? What is the solution for this?

[@markj701]

Very well made ... nice explanation of the PSI

[@skvali3810]

Great video

[@Riyaj-Shaikh]

Hello Shrivatsan, Thanks for amazing courses. You have provided a lot of value to the community and helped data science going forward.. Can you please tell me what tools you are using to create your course/video? I want to start my own in different domain of AI which can be helpful for some niche industry.

[@AIEngineeringLife]

Riyaj.. Check my linkedin post on this where I have stated tools I am using www.linkedin.com/posts/srivatsan-srinivasan-b8131b_few-reached-out-to-me-on-how-i-created-videos-activity-6579198397696348160-2M5S .. Only thing changes is I have moved from OBS studio to stream OBS

[@chetanmundhe8619]

Very nice, keep posting

[@sukantoghosh8230]

Amazing Explaination . Can we have a liitle hands-on Demo with the same approach and ideas of how to maintain the monitor dashboard. It would be really helpful .Thank You Sir

[@AIEngineeringLife]

Sukanto.. I have it on my list to do but it is going to be towards end of year. Have lot more to complete before I get there

[@chetanmundhe8619]

@@AIEngineeringLife is it available now in any of the playlist, curious to watch✌✌

[@AIEngineeringLife]

@@chetanmundhe8619 not yet. Maybe next year. Did not get time to go there

[@chetanmundhe8619]

@@AIEngineeringLife ok, great, keep posting, and thanks a lot for sharing ur knowledge

[@tanumoyhazra6055]

very informative video , can i get the PPT ?

[@thomasmorrissey8792]

When calculating la predictive model's population stability index, why does one use natural log (ln) and not say log base 2 or log base 10?? Can't find this answer anywhere!

[@junqichen6241]

Thank you for another great video! Do you have any video that talks about KS?

[@AIEngineeringLife]

Not yet but have it on my list to do

[@eshaankirpal]

Hi Sir, Thank you for making these videos, they are invaluable. I appreciate your efforts. I am working towards deploying forecasting models at the edge and need to learn more on model monitoring. Your videos have been a great starting point. Though I need to do more research, Are there any resources that you would recommend for model deployment and monitoring?

[@AIEngineeringLife]

For model deployment you can check below playlist. It has theory initially, followed by hands on sessions. You will also find deployment on edge kzbin.info/aero/PL3N9eeOlCrP5PlN1jwOB3jVZE6nYTVswk Model monitoring you might not find much detailed info online but if you have any specific ask for your case let me know

[@valerysalov8208]

Can you use the corona example and show model monitoring in real time with a live example. Thanks for this video !

[@AIEngineeringLife]

I will stay away from creating models on corona virus as the data is linear in most places as of today and we just need exponential functions to it. But let me take another model and explain it

[@valerysalov8208]

@@AIEngineeringLife we just want to see for live data that's it doesn't matter if it's not linear.

[@AIEngineeringLife]

Yes I get that but my point was i need a model which corona is not ideal. Monitoring needs model distribution output. Let me get something to demo

[@anirbansen4059]

Sorry for a dumb question but does PSI help in tracking concept drift or data drift?

[@AIEngineeringLife]

You can use for both.

[@SaurabhSharma-im3lv]

Hi Srivatsan, I would like to know your opinion on Machine Learning model monitoring tools available in the market to track model performance over the time after deployment. Is there any readily available tool/service which is really handy for such stuff?

[@chaitanyaagrawal8481]

Hello Sir, @AIEngineering - My model is deployed on Azure platform and I want to setup model performance monitoring. Can you please point to the direction where can i find suitable information?

[@thallamkarthik7020]

is there anything else to know apart from these 24 videos on end-end machine learning project we do in companies? if yes, could u pls mention those steps atleast so that we can search and know about them before u make those videos to post here. It is imp as we are attending interviews and we have to know about this complete cycle of a ML project in companies

[@AIEngineeringLife]

Thallam Karthik.. I have covered most of it here. Few items you might want to focus on is model evaluation and recalibration which I am yet to cover. But 98% is covered here to my knowledge

[@thallamkarthik7020]

That's great Srinivasan thank u so much 👍

[@FranckePeixoto]

nice video! can you help me? do you make a video on how to create a geometric shape identifier?

[@AIEngineeringLife]

Francke.. You want to identify Geometric shapes?. Is there any dataset for this that has labelled geometric shaped, any idea?

[@FranckePeixoto]

@@AIEngineeringLife Hi, my idea is to identify geometric shapes within images (photos), I believe it would be a combination of CNN and RNN.

[@FranckePeixoto]

@@AIEngineeringLife given a imagem, i want classify all geometry shapes in it

[@FranckePeixoto]

rectangles, stars, squares, circles and until a heart! :)

[@AIEngineeringLife]

@Francke.. I feel it is possible with CNN or even with openCV provided we have some good datasets to try.. I saw this data in kaggle but it has only 3 shapes. Let me try and check it out if there is few other datasets but model building should not take time www.kaggle.com/cactus3/basicshapes

[@sarthak7413]

How to implement this in Python?

[@designwithpicmaker2785]

I am the first view and first like👍🏼😀

[@AIEngineeringLife]

🎉