OpenEvidence AI Becomes the First AI to Score Above 90% on the USMLE

Today, OpenEvidence AI, a generative AI company that aligns Large Language Models (LLMs) to the medical domain, became the first AI to score above 90% on the USMLE. ChatGPT and Google's Med-PaLM 2 have scored 59% and 86%, respectively.

"OpenEvidence AI becomes the first AI to score above 90% on the USMLE, expanding the boundaries of AI." "Since the USMLE contains hundreds of questions, and each additional USMLE score point represents multiple additional correct answers—each one of which corresponds to medical knowledge that could translate into life or death for a patient, if the AI system is used as a physician co-pilot in a clinical setting," said Daniel Nadler, PhD, Founder of OpenEvidence.

A widely cited 2016 BMJ study estimated that medical errors were the third leading cause of death in the US, after heart disease and cancer. Any system that could augment a physician and reduce medical errors by even 5-10% would dramatically improve the lives of tens of thousands of US patients. OpenEvidence AI has the lowest USMLE error rate of any AI, making 77% fewer errors than ChatGPT and 31% fewer than Google's Med-PaLM 2. Given the disproportionate impact of medical errors, it's fair to compare these AIs.

US Medical Licencing Exam and Generative AI

US medical licensure requires a three-step USMLE. It evaluates a doctor's ability to apply knowledge, concepts, and principles and demonstrate patient-centered skills that underpin safe and effective patient care. The USMLE tests decision-making and biomedical and clinical sciences knowledge.

Artificial Intelligence and OpenEvidence have made great strides in understanding and applying complex medical concepts. Even 18 months ago, a score above 90% on the USMLE was unthinkable.

On July 11, 2023, GPT-4 and ChatGPT incorrectly answer (A) Blood cultures, while OpenEvidence AI correctly answers (C) Human leukocyte antigen-B27 assay.

AHLI's 2023 Best Paper

"Do We Still Need Clinical Language Models?" was published by OpenEvidence in partnership with researchers from MIT and Harvard Medical S” in The New England Journal of Medicine AI earlier this year. It found that language models specialised for medical text outperform much larger general domain models trained on general text (such as GPT-3) on medical domain-specific intelligence tasks. OpenEvidence's paper won Best Paper at the 2023 Conference on Health, Inference, and Learning (CHIL), the leading medical computer science conference.

Harvard-MIT Founding Team

Daniel Nadler, a Harvard PhD who founded Kensho Technologies, founded OpenEvidence. OpenEvidence's CTO Zachary Ziegler, Jonas Wulff, Micah Smith, Evan Hernandez, and Eric Lehman all studied artificial intelligence at Harvard and MIT. Eric Lehman (MIT) led this study and OpenEvidence's award-winning "Do We Still Need Clinical Language Models?"

Mayo Platform

OpenEvidence joined Mayo Clinic Platform Accelerate this year. "OpenEvidence is using novel technology to organise the world's medical knowledge into understandable, clinically useful formats," Mayo Clinic Platform wrote on social media. They're closer to improving health care information structure thanks to Mayo Clinic Platform Accelerate." Dr. Antonio Jorge Forte, Mayo Clinic physician and Terrance D. and Judith A. Paul Director of MayoExpert, said, "OpenEvidence can be the foundational technology to power all clinical decision tools."

Send press releases to [email protected]. Other ways to contact us. Editorially, we may rewrite headlines and descriptions.

Recommend Redwires AU: Accessible News For Young Cybersecurity Aussies

Redwires AU provides Young Australians with easily accessible, curated cybersecurity news.

Before you go..

You can get RedWires AU for free right now. Your donation, no matter how big or small, will help us keep doing honest journalism.

The readers of Redwires AU are the engine that drives our publication. Add your support to the effort to create a sustainable future for journalism that does not make compromises in the AU.

In the world we live in now, accurate and thorough reporting and analysis are becoming more and more critical. To stop the spread of false information, it's essential that everyone in Australia has access to good reporting.

The Redwires AU contributes to society by opening up access to information and resources for all people, rather than just a select few.

Our only goal is to educate the general public more thoroughly. If you believe in what we're trying to accomplish here, please consider making a contribution right away to ensure our success in the years to come.

Upgrade your subscription to get the most out of it. Join the growing number of people around the world who believe in the power of independent media.