Production Machine Learning Monitoring: Principles, Patterns and Techniques (Alejandro Saucedo)

[@stackinglittlesats]

Very nice explanation. Thank you.

[@8eck]

Just wow! Will dive into all of that. Thank you!

[@harshraj22_]

This video was quite insightful. Thanks

[@Yzyou11]

Which tool are you using real time statistical monitoring

[@8eck]

We don't wana our business finding out that our models are performing bad. :D

[@ajay240392]

This was an very interesting video. I liked it. Thank you for sharing this. I have few question regarding this: - Is it possible to get monitoring dashboard without feedback correct label? because this could not be possible for all of this use cases. If it is possible could you please help me to understand it? - Do we have to deploy separate microservices/API end point for drift and outlier as well? Thank you in advance.

[@axsaucedo]

Some metrics do not require correct / feedback label. For example the microservice performance metrics like requests per second or latency don't require the feedback. Similarly some of the advanced monitoring components like outliers, drift and explainers don't require the feedback label. It is the statistical performance metrics that do require the feedback label. In regards to your second question, yes it would be separate microservices for drift and outliers.

[@ChuanChihChou]

13:38 Why do we need to send the correct label to the model? Shouldn't we be able to compute performance metrics of the model by just using the stored inference data + the correct labels?

[@axsaucedo]

Good question. This is necessary as once you deploy your model, you are no longer dealing with training data with provided labels. This is production unseen inference data for which you do not have the labels. Because of this, the labels would have to be provided asynchronously through most likely a manual process where some person/system would at some point provide the correct prediction. Once the correct prediction is provided, it will then be stored with the production inference data (which would be different to the training inference data). For completeness if your question is why we actually keep track of the accuracy/precision/recall in prometheus as opposed to calculating it offline using the data in elasticsearch, in that case you actually can do both, but prometheus would be more likely used for real time insights, which can trigger alerts, and elasticsearch would be used for offline analytics which can allow for drill-downs - this how we currently approach it in Seldon Core.