Thread by @alan_karthi

So happy to share #MedPaLM2 - our team's evolution of Med-PaLM. A new state of art for medical question-answering!

Med-PaLM 2 scores 86.5% on MedQA-USMLE, exceeding Med-PaLM's score by >19% 🤯, & 81.8% on PubMedQA...

More here: arxiv.org/pdf/2305.09617…

We believe in rigorous, careful evaluation. Physicians even preferred #MedPaLM2's long-form answers to answers from other real 🇮🇳🇺🇸🇬🇧 physicians along 8/9 axes of quality including medical accuracy (consensus w/medical opinion) and reasoning, with less likelihood of harm

MedPaLM-2's performance was superior to Med-PaLM far beyond exam performance. To highlight the real-world importance of nuanced evaluation we introduce a new dataset of "adversarial" questions designed specifically to probe LLM weaknesses including #HealthEquity

Lay raters also consistently find MedPaLM-2 more helpful, and that it directly addresses the intent behind a medical question:

I can't believe I get to say this but you can see a summary by @sundarpichai with a sneak peek at where our research is heading next! Also co-senior authors' feeds at @AziziShekoofeh @vivnat and first authors @taotu831 @thekaransinghal @Mysiak
👀 youtube.com/clip/Ugkxb7W_k…

Grateful for our amazing team @GoogleAI @GoogleHealth @DeepMind @AziziShekoofeh
@thekaransinghal
@vivnat
@Mysiak
@weballergy
@clark_kev
@stephenpfohl
@hcolelewis
@ymatias
@greg_corrado