MedAI #72: Large Language Models Encode Clinical Knowledge | Karan Singhal

Title: Large Language Models Encode Clinical Knowledge
Speaker: Karan Singhal
Abstract:
Large language models (LLMs) have demonstrated impressive capabilities in natural language understanding and generation, but the quality bar for medical and clinical applications is high. Today, attempts to assess models' clinical knowledge typically rely on automated evaluations on limited benchmarks. There is no standard to evaluate model predictions and reasoning across a breadth of tasks. To address this, we present MultiMedQA, a benchmark combining six existing open question answering datasets spanning professional medical exams, research, and consumer queries; and HealthSearchQA, a new free-response dataset of medical questions searched online. We propose a framework for human evaluation of model answers along multiple axes including factuality, precision, possible harm, and bias. In addition, we evaluate PaLM (a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM, on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA, MedMCQA, PubMedQA, MMLU clinical topics), including 67.6% accuracy on MedQA (US Medical License Exam questions), surpassing prior state-of-the-art by over 17%. However, human evaluation reveals key gaps in Flan-PaLM responses. To resolve this we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, recall of knowledge, and medical reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal important limitations of today's models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLM models for clinical applications.
Speaker Bio:
Karan co-leads teams at Google Research working on medical AI, foundation models, representation learning, and federated learning. He is broadly interested in developing and validating techniques that lead to wider adoption of safe, beneficial AI. Prior to joining Google, he received an MS and BS in Computer Science from Stanford University.
------
The MedAI Group Exchange Sessions are a platform where we can critically examine key topics in AI and medicine, generate fresh ideas and discussion around their intersection and most importantly, learn from each other.
We will be having weekly sessions where invited speakers will give a talk presenting their work followed by an interactive discussion and Q&A. Our sessions are held every Thursday from 1pm-2pm PST.
To get notifications about upcoming sessions, please join our mailing list: mailman.stanfo....
For more details about MedAI, check out our website: medai.stanford.... You can follow us on Twitter @MedaiStanford
Organized by members of the Rubin Lab (rubinlab.stanfo...)
Nandita Bhaskhar (www.stanford.e...)
Siyi Tang (siyitang.me)

[@zooKcx]

Great talk Karan and team, straight talk, zero hype and most useful. Exciting times ahead

[@mrghulamrasool]

Thanks, Karan and organizers!!!

[@ChâuThanhNgọc]

Hi @StanfordMedAI, could you please share the slides with me? Thank you very much!!!

[@pranavnair7714]

Hi! Any idea if Med-PaLM would be available to the general public?