An AI tool grounded in evidence-based medicine outperformed other AI tools — and most doctors — on USMLE exams

BUFFALO, N.Y. — A powerful clinical artificial intelligence tool developed by University at Buffalo biomedical informatics researchers has demonstrated remarkable accuracy on all three parts of the United States Medical Licensing Exam (Step exams), according to a paper published today (April 22) in JAMA Network Open.

Achieving higher scores on the USMLE than most physicians and all other AI tools so far, Semantic Clinical Artificial Intelligence (SCAI, pronounced “Sky”) has the potential to become a critical partner for physicians, says lead author Peter L. Elkin, MD, chair of the Department of Biomedical Informatics in the Jacobs School of Medicine and Biomedical Sciences at UB and a physician with UBMD Internal Medicine.  

Elkin says SCAI is the most accurate clinical AI tool available to date, with the most advanced version scoring 95.2% on Step 3 of the USMLE, while a GPT4 Omni tool scored 90.5% on the same test.

“As physicians, we are used to using computers as tools,” he explains, “but SCAI is different; it can add to your decision-making and thinking based on its own reasoning.”

The tool can respond to medical questions posed by clinicians or the public at https://halsted.compbio.buffalo.edu/chat/.

The researchers tested the model against the USMLE, required for licensing physicians nationwide, which assesses the physician’s ability to apply knowledge, concepts and principles, and to demonstrate fundamental patient-centered skills. Any questions with a visual component were eliminated.

Elkin explains that most AI tools function by using statistics to find associations in online data that allow them to answer a question. “We call these tools generative artificial intelligence,” he says. “Some have postulated that they are just plagiarizing what’s on the internet because the answers they give you are what others have written.” However, these AI models are now becoming partners in care rather than simple tools for clinicians to utilize in their practice, he says.

“But SCAI answers more complex questions and performs more complex semantic reasoning,” he says, “We have created knowledge sources that can reason more the way people learn to reason while doing their training in medical school.”

The team started with a natural language processing software they had previously developed. They added vast amounts of authoritative clinical information gleaned from widely disparate sources ranging from recent medical literature and clinical guidelines to genomic data, drug information, discharge recommendations, patient safety data and more. Any data that might be biased, such as clinical notes, were not included.

13 million medical facts

SCAI contains 13 million medical facts, as well as all the possible interactions between those facts. The team used basic clinical facts known as semantic triples (subject-relation-object, such as “Penicillin treats pneumococcal pneumonia”) to create semantic networks. The tool can then represent these semantic networks so that it is possible to draw logical inferences from them.

“We have taught large language models how to use semantic reasoning,” says Elkin.

Other techniques that contributed to SCAI include knowledge graphs that are designed to find new links in medical data as well as previously “hidden” patterns, as well as retrieval-augmented generation, which allows the large language model to access and incorporate information from external knowledge databases before responding to a prompt. This reduces “confabulation,” the tendency for AI tools to always respond to a prompt even when it doesn’t have enough information to go on.

Elkin adds that using formal semantics to inform the large language model provides important context necessary for SCAI to understand and respond more accurately to a particular question.

‘It can have a conversation with you’

“SCAI is different from other large language models because it can have a conversation with you and as a human-computer partnership can add to your decision-making and thinking based on its own reasoning,” Elkin says.

He concludes: “By adding semantics to large language models, we are providing them with the ability to reason similarly to the way we do when practicing evidence-based medicine.”

Because it can access such vast amounts of data, SCAI also has the potential to improve patient safety, improve access to care and “democratize specialty care,” Elkin says, by making medical information on specialties and subspecialties accessible to primary care providers and even to patients.

While the power of SCAI is impressive, Elkin stresses its role will be to augment, not replace, physicians.

“Artificial intelligence isn’t going to replace doctors,” he says, “but a doctor who uses AI may replace a doctor who does not.”

In addition to Elkin, UB co-authors from the Department of Biomedical Informatics are Guresh Mehta; Frank LeHouillier; Melissa Resnick, PhD; Crystal Tomlin, PhD; Skyler Resendez, PhD; and Jiaxing Liu. 

Sarah Mullin, PhD, of Roswell Park Comprehensive Cancer Center, and Jonathan R. Nebeker, MD, and Steven H. Brown, MD, both of the Department of Veterans Affairs, also are co-authors.

The work was funded by grants from the National Institutes of Health and the Department of Veterans Affairs.