TY  - JOUR
AU  - Hoppe, John Michael
AU  - Auer, Matthias K
AU  - Strüven, Anna
AU  - Massberg, Steffen
AU  - Stremmel, Christopher
PY  - 2024
DA  - 2024/7/8
TI  - ChatGPT With GPT-4 Outperforms Emergency Department Physicians in Diagnostic Accuracy: Retrospective Analysis
JO  - J Med Internet Res
SP  - e56110
VL  - 26
KW  - emergency department
KW  - diagnosis
KW  - accuracy
KW  - artificial intelligence
KW  - ChatGPT
KW  - internal medicine
KW  - AI
KW  - natural language processing
KW  - NLP
KW  - emergency medicine triage
KW  - triage
KW  - physicians
KW  - physician
KW  - diagnostic accuracy
KW  - OpenAI
AB  - Background: OpenAI’s ChatGPT is a pioneering artificial intelligence (AI) in the field of natural language processing, and it holds significant potential in medicine for providing treatment advice. Additionally, recent studies have demonstrated promising results using ChatGPT for emergency medicine triage. However, its diagnostic accuracy in the emergency department (ED) has not yet been evaluated. Objective: This study compares the diagnostic accuracy of ChatGPT with GPT-3.5 and GPT-4 and primary treating resident physicians in an ED setting. Methods: Among 100 adults admitted to our ED in January 2023 with internal medicine issues, the diagnostic accuracy was assessed by comparing the diagnoses made by ED resident physicians and those made by ChatGPT with GPT-3.5 or GPT-4 against the final hospital discharge diagnosis, using a point system for grading accuracy. Results: The study enrolled 100 patients with a median age of 72 (IQR 58.5-82.0) years who were admitted to our internal medicine ED primarily for cardiovascular, endocrine, gastrointestinal, or infectious diseases. GPT-4 outperformed both GPT-3.5 (P<.001) and ED resident physicians (P=.01) in diagnostic accuracy for internal medicine emergencies. Furthermore, across various disease subgroups, GPT-4 consistently outperformed GPT-3.5 and resident physicians. It demonstrated significant superiority in cardiovascular (GPT-4 vs ED physicians: P=.03) and endocrine or gastrointestinal diseases (GPT-4 vs GPT-3.5: P=.01). However, in other categories, the differences were not statistically significant. Conclusions: In this study, which compared the diagnostic accuracy of GPT-3.5, GPT-4, and ED resident physicians against a discharge diagnosis gold standard, GPT-4 outperformed both the resident physicians and its predecessor, GPT-3.5. Despite the retrospective design of the study and its limited sample size, the results underscore the potential of AI as a supportive diagnostic tool in ED settings. 
SN  - 1438-8871
UR  - https://www.jmir.org/2024/1/e56110
UR  - https://doi.org/10.2196/56110
DO  - 10.2196/56110
ID  - info:doi/10.2196/56110
ER  -