Testing Framework Giskard for LLM and RAG Evaluation (Bias, Hallucination, and More)

  Рет қаралды 6,305

AI Anytime

AI Anytime

9 ай бұрын

Join me on a deep dive into Giskard, the cutting-edge Python library designed to revolutionize the way we test and evaluate AI models. In this tutorial, I uncover how Giskard is not just another tool, but an essential ally in detecting a wide range of vulnerabilities, from performance biases and data leakage to more nuanced issues like spurious correlations, hallucination, and even toxicity.
Learn how to harness the power of Giskard to scrutinize large language models (LLM) and retrieval-augmented generation (RAG), ensuring your models are not just high-performing but also ethical and secure. With practical examples and step-by-step guidance, I'll show you how Giskard can help you save valuable time, significantly reduce the manual effort in problem identification, and push the boundaries of what's possible in AI reliability and trustworthiness.
🔔 Subscribe for more insights into Gen AI model evaluation and development.
👍 Like this video if you find it helpful-it supports the channel and helps me create more content.
💬 Comment below if you have any questions or share your experiences with using Giskard and other Gen AI testing tools.
📢 Share this video with peers who could benefit from a robust testing framework for their Gen AI projects.
GitHub Repo: github.com/AIAnytime/Giskard-...
Find Giskard Here: github.com/Giskard-AI/giskard

Пікірлер: 21
@giridharreddy7011
@giridharreddy7011 9 ай бұрын
This channel never compromises in its quality ❤
@AIAnytime
@AIAnytime 9 ай бұрын
Glad you enjoy it!
@doge1931
@doge1931 9 ай бұрын
straight to the code ...love it
@AIAnytime
@AIAnytime 9 ай бұрын
Thanks
@SnehaRoy-pf9cw
@SnehaRoy-pf9cw 8 ай бұрын
Wow this is fantastic❤
@AIAnytime
@AIAnytime 8 ай бұрын
Thanks
@pearlmarysamuel4809
@pearlmarysamuel4809 8 ай бұрын
Will it support code llama where input is question, output is code? Or only RAG and chatbot
@milesonme
@milesonme 9 ай бұрын
Hi @AI Anytime, if i remember correctly I think you have a project where you built llms with custom data and then a user interface using I think TypeScript. I am trying to learn how to integrate a custom llm with a javascript front-end say react, I have been looking for that video for days, if you can please help me find it.
@SriNagabhirava
@SriNagabhirava 7 ай бұрын
Thanks!
@AIAnytime
@AIAnytime 7 ай бұрын
Thank you so much for the support 🙏
@mcmarvin7843
@mcmarvin7843 8 ай бұрын
Quality
@TesterOps09
@TesterOps09 2 ай бұрын
climate_qa_chain = RetrievalQA.from_llm( llm=llm, retriever=get_context_storage().as_retriever(), prompt=prompt ) This code keeps giving me 429 error every time I try to use it with OpenAI. Can you let me know why this happens? I don't see it happening in the example video. Is there anything wrong that I am doing
@RameshBaburbabu
@RameshBaburbabu 8 ай бұрын
🎯 Key Takeaways for my quick navigation: 00:15 🤖 *Giskard is a testing framework for evaluating machine learning models, especially large language models (LLMs), detecting biases, hallucination rates, and other metrics. It offers high-level APIs for easy implementation.* 01:13 🛡️ *Qualitative metrics like RAG and GPT eval might not suffice for enterprise or regulatory needs; quantitative measures like accuracy, F1 score, recall, and precision are crucial for evaluating LLMs.* 02:46 💼 *Giskard classifies issues into four major categories: performance, robustness, overconfidence, and spurious correlation. It provides metrics like accuracy, recall, precision, and more to address these issues.* 04:27 🧰 *The video demonstrates the installation process for Giskard in a Colab notebook, including the necessary dependencies such as Langchain, PiPDF, Fast, OpenAI, and others.* 21:30 📊 *The video showcases the creation of a custom Giskard wrapper class for serialization, including methods for defining, saving, and loading the Giskard model.* 23:56 📄 *The class methods include 'Define', 'Save', and 'Load' for the Giskard model, allowing for easy serialization and deserialization, enabling the saving of models, fast retrievers, and OpenAI embeddings.* 26:28 🤖 *Giskard is used to create a climate change question-answering model. Fast RAG model is employed for text generation tasks, specifying model type, name, description, and features.* 30:47 🧠 *Giskard model is validated with a dataset containing questions related to climate change based on IPCC reports.* 32:49 🕵️ *Giskard is used to scan for hallucination in the model's outputs, identifying potential issues in generating non-factual or incoherent responses.* 34:40 🧪 *A test suite is generated by Giskard to assess the model's performance, including metrics for hallucination, security, and other categories.* 37:02 🚨 *Giskard detects an issue in model responses, where the model contradicts itself in different outputs, emphasizing the importance of handling varied user queries effectively.* 39:34 📊 *Giskard provides comprehensive results, including accuracy, F1 score, precision, recall, robustness, and overconfidence measures, making it a versatile testing framework for language models and retrievers.*
@b15ganeshgulhane90
@b15ganeshgulhane90 Ай бұрын
how can i use opensource model in giskard instead of openai?
@arpsami7797
@arpsami7797 7 ай бұрын
Very informative as always, Thank you. And one question, Is this applicable for a non English document? I am specifically looking an evolution for Japanese
@AIAnytime
@AIAnytime 7 ай бұрын
Yes, it is
@chuanjiang6931
@chuanjiang6931 8 ай бұрын
How do we know if a llm is supported by Giskard?
@pearlmarysamuel4809
@pearlmarysamuel4809 8 ай бұрын
Also can we replace openAI with any other opensource llm ro act as judge? Any thoughts
@AIAnytime
@AIAnytime 8 ай бұрын
Not possible at the moment. But Giskard creators are working on it.
@deepaksingh9318
@deepaksingh9318 4 ай бұрын
Could you please also make a video Specially on Embeddings(What are different Embedding available, What are their Pros and Cons, Which one is ideal in which scenarios , What are few popular ones etc. with code)
@FalahgsGate
@FalahgsGate 3 ай бұрын
Testing Framework Giskard does not support the Gemini Pro model...I'm working with the Gemini Pro model ...the Giskard it is not working with it
I'm Excited To see If Kelly Can Meet This Challenge!
00:16
Mini Katana
Рет қаралды 29 МЛН
Slow motion boy #shorts by Tsuriki Show
00:14
Tsuriki Show
Рет қаралды 10 МЛН
CHOCKY MILK.. 🤣 #shorts
00:20
Savage Vlogs
Рет қаралды 14 МЛН
A teacher captured the cutest moment at the nursery #shorts
00:33
Fabiosa Stories
Рет қаралды 55 МЛН
How to evaluate an LLM-powered RAG application automatically.
50:42
What are AI Agents?
12:29
IBM Technology
Рет қаралды 119 М.
Multi-modal RAG: Chat with Docs containing Images
17:40
Prompt Engineering
Рет қаралды 16 М.
Why Agent Frameworks Will Fail (and what to use instead)
19:21
Dave Ebbelaar
Рет қаралды 35 М.
Intro to RAG for AI (Retrieval Augmented Generation)
14:31
Matthew Berman
Рет қаралды 48 М.
Graph RAG: Improving RAG with Knowledge Graphs
15:58
Prompt Engineering
Рет қаралды 43 М.
LLM Function Calling - AI Tools Deep Dive
31:14
Adam Lucek
Рет қаралды 4,6 М.
Rate This Smartphone Cooler Set-up ⭐
0:10
Shakeuptech
Рет қаралды 6 МЛН
$1 vs $100,000 Slow Motion Camera!
0:44
Hafu Go
Рет қаралды 29 МЛН
low battery 🪫
0:10
dednahype
Рет қаралды 1,7 МЛН
Xiaomi SU-7 Max 2024 - Самый быстрый мобильник
32:11
Клубный сервис
Рет қаралды 551 М.
Tag him😳💕 #miniphone #iphone #samsung #smartphone #fy
0:11
Pockify™
Рет қаралды 4,5 МЛН