LLM Evaluation With MLFLOW And Dagshub For Generative AI Application

With the emerging of ChatGPT, LLMs have shown its power of text generation in various fields, such as question answering, translating and text summarization. Evaluating LLMs’ performance is slightly different from traditional ML models, as very often there is no single ground truth to compare against. MLflow provides an API mlflow.evaluate() to help evaluate your LLMs.
mlflow.org/doc...
Code:github.com/kri...
---------------------------------------------------------------------------------------------
Support me by joining membership so that I can upload these kind of videos
/ @krishnaik06
-----------------------------------------------------------------------------------
►GenAI on AWS Cloud Playlist: • Generative AI In AWS-A...
►Llamindex Playlist: • Announcing LlamaIndex ...
►Google Gemini Playlist: • Google Is On Another L...
►Langchain Playlist: • Amazing Langchain Seri...
►Data Science Projects:
• Now you Can Crack Any ...
►Learn In One Tutorials
Statistics in 6 hours: • Complete Statistics Fo...
End To End RAG LLM APP Using LlamaIndex And OpenAI- Indexing And Querying Multiple Pdf's
Machine Learning In 6 Hours: • Complete Machine Learn...
Deep Learning 5 hours : • Deep Learning Indepth ...
►Learn In a Week Playlist
Statistics: • Live Day 1- Introducti...
Machine Learning : • Announcing 7 Days Live...
Deep Learning: • 5 Days Live Deep Learn...
NLP : • Announcing NLP Live co...
---------------------------------------------------------------------------------------------------
My Recording Gear
Laptop: amzn.to/4886inY
Office Desk : amzn.to/48nAWcO
Camera: amzn.to/3vcEIHS
Writing Pad:amzn.to/3OuXq41
Monitor: amzn.to/3vcEIHS
Audio Accessories: amzn.to/48nbgxD
Audio Mic: amzn.to/48nbgxD

[@krishnaik06]

Subscribe if you want to become a Data Scientist :)

[@jaisingh1292]

Hi Krish can you please take a look at dvc and mlflow and can they be combined and used for gen ai , the demand for LLMops is increasing. For cloud its fine but industries want on prem solution as well it would be great if you can make any video on the same

[@DarkShadow-bq5yb]

can you create video for Data engineering using LLMs

[@HimanshuGupta-ps3ib]

Great video Krish! Watching your content from USA.

[@anubhavsarkar5745]

Very Helpful Video Krish. Can you please show us the same LLM Model Evaluation using MLFlow, but using HuggingFace LLMs instead of OpenAI ? I am facing a lot of issues in adding payment method to the OpenAI platform.

[@skyplanet9858]

Dear Kirish, thank you very much for the great video. Dagshub is an Israeli company that's complicit in genocide. I wish you could explore another tool.

[@adityamaurya9521]

Can we do this with Mistral API keys ?

[@2dapoint424]

Krish, @10:15 in VS code what is that cat like icon on top right next to the run button?

[@adityarajshukla9422]

copilot

[@shriharinair1999]

can you from now on focus more on non paid apis?

[@prayagbrahmbhatt6375]

Need a video explanation on promptfoo, an opensource LLM evaluation library.

[@SantoshYadav-f4r4s]

Hi Krish ! This is excellent video, will you be able to make a video on Azure AI Prompt flow, please?

[@sourabhsomdeve1373]

Nice

[@pratiksitapara8962]

Great video on Evals! Thanks! But It would be great to see or if you could make a video on Evaluating RAG on Traditional metrics? What insights we can get by using traditional evals or automatic evals? What is the current SOTA methods for RAG Evals!?

[@YorkYongYeo]

thanks for sharing this! just what i needed for reference. Liked and looking forward to the one for RAG evaluation

[@GiovanneAfonso]

Incredible content, thank you +1sub

[@pavanpraneeth4659]

Awesome please in future show how to integrate this with aws bedrock please

[@arpitqw1]

what is the use of daghub ?, same once can see in local tracking server.

[@surbhirohilla5139]

You deserve more appreciation for this

[@ashishdayal172]

sounds like tensorflow hub

[@ridj41]

Krish honestly the content has become a bit uninteresting tbh from the past few videos. Would love if you try out some new kind of projects, some advanced ones to use in our resumes using GenAi

[@krishnaik06]

brother this is really important. I usually make all the videos based on the job market :)

[@RishiRajxtrim]

Good evening