Why you should build an LLM benchmark [English]

📊 Dive Deep into the World of LLM Benchmarks! 📊
Objective: By the end of this session, you should have a good understanding of how to select and maintain your own LLM benchmark.
Agenda:
🔬 Demo!
🔍Discover what ARC, HellSwag, and MMLU are exactly
🧫 Learn how to select the right benchmark
🧪 Methods to test LLMs tailored to your unique use case
🧱 Q&A
Speaker: J. Yarkoni ex-Google AI/ML Specialist (Shujin.ai)
Jonathan comes from a background of leading R&D teams. Previously he co-founded NAM, an advertising startup, and AA-TLV meetup, which at its peak had 3,500 members. Over the last six years, he spearheaded AI/ML initiatives at Google Cloud Israel. More recently, he established Shujin.AI, a consultancy specializing in ML projects with an emphasis on Generative AI.
big-data-demystified.ninja/20...

[@jazzvids]

Thank you for this valuable talk! I am currently writing my masters' thesis in nlp and this is very helpful