Generative AI correctly diagnoses 39% of complex, challenging medical cases

By Nick Paul Taylor, The Science Advisory Board contributing writer

June 20, 2023 -- The generative artificial intelligence (AI) model GPT-4 correctly diagnosed 39% of challenging medical cases in a study published in JAMA.

GPT-4 is the successor to the technology behind ChatGPT, the chatbot that took discussions of generative AI into the mainstream late last year. Such generative models are trained on large datasets and provide accurate and detailed text-based responses to written prompts. Because the models can interpret and respond to language, they could theoretically mimic the deductions of doctors and diagnose disease.

To test that theory, researchers at Beth Israel Deaconess Medical Center (BIDMC) applied GPT-4 to a set of 70 clinicopathological case conferences (CPCs). CPCs are complex and challenging cases published in the New England Journal of Medicine for educational purposes. The reports include clinical and laboratory data, imaging studies, and histopathological findings.

"We wanted to know if such a generative model could 'think' like a doctor, so we asked one to solve standardized complex diagnostic cases used for educational purposes. It did really, really well," Dr. Adam Rodman, co-director of the Innovations in Media and Education Delivery Initiative at BIDMC, said in a statement. Rodman is an instructor in medicine at Harvard Medical School.

GPT-4 matched the final CPC diagnosis 39% of the time. For 64% of cases, the AI included the final CPC diagnosis in its list of possible conditions that could account for a patient's symptoms, medical history, clinical findings, and laboratory or imaging results.

The JAMA paper lacks a control arm showing how GPT-4 compares to the success rate of doctors, but the results of another study suggest the AI performed well. Last year, two people described as "medical doctors, but non-experts" reached the correct diagnosis for 28% of the 50 CPCs they reviewed.

While GPT-4 performed well in the study, the fact that the AI can solve CPCs does not mean it can execute the process physicians use to diagnose patients. Experts create CPCs for physicians to learn from, providing all the data they need to reach a diagnosis. In the real world, physicians start with limited information and need to form hypotheses and generate data to test them, working step by step toward a diagnosis.

Even so, the study suggests generative AI could play a role in supporting doctors, as first author Dr. Zahir Kanjee, a physician at BIDMC and assistant professor of medicine at Harvard Medical School, said in a statement.

"It has the potential to help physicians make sense of complex medical data and broaden or refine our diagnostic thinking. We need more research on the optimal uses, benefits, and limits of this technology, and a lot of privacy issues need sorting out, but these are exciting findings for the future of diagnosis and patient care," Kanjee said.

Nucleai, Mayo Clinic partner on digital pathology
Nucleai and Mayo Clinic BioPharma Diagnostics on Monday announced a strategic collaboration involving digital pathology solutions, technologies, and services...
Nucleai, Adlai Nortye collaborate on biomarkers for clinical trials
Artificial intelligence firm Nucleai and biopharmaceutical firm Adlai Nortye on Wednesday announced a strategic partnership aimed at identifying, validating,...
Artificial intelligence tool identifies current drugs for future pandemics
Researchers have created an artificial intelligence (AI)-based algorithmic tool that calculates how to effectively repurpose current drugs to combat both...
Atomic AI closes $35M Series A financing
Biotech firm Atomic AI recently announced it the closing of a $35 million Series A round led by Playground Global, with participation from 8VC, Factory...
AI tool makes speedy gene editing possible
Researchers have designed an artificial intelligence (AI) program that may enable the production of customizable proteins called zinc fingers to treat...

Copyright © 2023

Science Advisory Board on LinkedIn
Science Advisory Board on Facebook
Science Advisory Board on Twitter