A Multiagent Large Language Model Framework for Emergency Treatment Recommendation in Acute Ischemic Stroke: Development and Validation Study

Introduction

Stroke remains a leading cause of disability and death worldwide, with the greatest burden borne by low- and middle-income countries [1,2]. While acute ischemic stroke (AIS) management requires rapid and precise decision-making within narrow therapeutic windows, stroke care resources and diagnostic expertise remain profoundly uneven globally. Driven by socioeconomic inequality, geographic barriers, and workforce shortages across both resource-limited and high-income settings, these disparities lead to critical delays and misdiagnoses, worsening patient outcomes and reinforcing inequities in access to timely, specialist-level care [3-7]. These systemic challenges make equitable access to evidence-based AIS management a pressing global priority. As underscored by the World Stroke Organization Global Declaration on Stroke, which calls for equitable, evidence-based stroke systems worldwide, efforts to expand manpower and training have so far yielded only modest and uneven progress [8,9].

Large language models (LLMs) have shown potential to support clinical information synthesis and decision-making, offering a possible strategy to improve the consistency and accessibility of AIS decision support [10]. LLMs have demonstrated strong capabilities in diagnostic reasoning, differential generation, and information synthesis [11-15]. However, whether LLMs can provide reliable, guideline-concordant, and clinically safe decision support for AIS treatment selection remains insufficiently established. Most existing studies are benchmark- or theory-driven, focusing on tasks such as MedQA (United States Medical Licensing Examination) that do not adequately reflect real-world, guideline-based clinical decision-making [16,17]. This gap underscores the need for systematic case-based evaluation using real-world AIS data and physician decision-making experiments.

Our framework augments LLMs with structured clinical reasoning to support guideline-concordant therapy selection and more reliable AIS decision-making. This study had 2 objectives: to determine whether a multiagent design improves LLM performance on real-world AIS decision tasks in a multicenter, multisource evaluation, and to assess its effect on physician decision accuracy in a human-AI interaction experiment involving physicians with varying seniority and specialty backgrounds. We further examined whether LLM assistance preferentially benefits less-experienced clinicians and narrows expertise-related decision gaps, thereby supporting more consistent, safe, and accessible stroke care decision support across diverse care settings.

Methods

Ethical Considerations

Approval was obtained from the institutional review board of the Medical Faculty Ethics Committee of Shanghai Sixth People's Hospital affiliated to Shanghai Jiao Tong University School of Medicine (approval 2024-KY-203), and the study was registered in the Chinese Clinical Trial Registry (ChiCTR2400092800) on November 22, 2024. Informed consent was obtained from all participants. This study was conducted in accordance with the Declaration of Helsinki and relevant ethical guidelines and regulations.

Study Design

The overall study design and data sources are summarized in Figure 1. This schematic highlights the integration of multicenter retrospective and prospective real-world clinical cases with PubMed case reports, the application of the multiagent framework, and the evaluation of both model performance and human-AI interaction.

‎

Data Collection

We retrospectively screened clinical cases from 2 tertiary care centers: 1228 cases from center A (group A, January 2018-January 2025, tertiary grade A hospital) and 938 cases from center B (group B, May 2018-March 2025, tertiary grade B hospital). All cases were deidentified encounters of patients diagnosed with acute cerebrovascular disease. In addition, 327 stroke case reports were retrieved from PubMed between January 2024 and January 2025 (group C), which predominantly represented diagnostically challenging and clinically complex presentations. For prospective validation, 213 patients were consecutively enrolled at center A between February and May 2025 (group D). Detailed inclusion and exclusion criteria for each group are provided in the Supplementary Material Methods section and Figure S1 in Multimedia Appendix 1.

Patient Cases

To ensure patient privacy, all personally identifiable information was removed. Each case was formatted as a single paragraph containing all or a subset of the following elements: patient age and sex, chief complaint, current symptoms, medical history (including illnesses and medications), relevant family history, physical examination findings, laboratory test results, and imaging reports.

Evaluation of Standalone LLMs

To evaluate LLM performance in generating AIS treatment recommendations and assigning stroke subtypes according to the Trial of ORG 10172 in Acute Stroke Treatment (TOAST) classification, each model was tasked with generating a treatment recommendation and the corresponding TOAST classification conclusion. Seven models were tested: Baichuan-M1-14B [18], GPT-OSS-20B [19], Qwen2.5-32B [20], DeepSeek-R1-Distill-Qwen2.5-32B [21], GPT-OSS-120B [19], DeepSeek-R1-671B [21], and GPT-4o (OpenAI; only in group C dataset) [22] (Table S1 in Multimedia Appendix 1). Real-world clinical cases from groups A to D were used for this assessment. Outputs were produced in free-text format without predefined options, reflecting the probabilistic nature of clinical reasoning. To ensure consistent and reproducible evaluation, an automated grader agent was used to quantify accuracy across all LLMs (Supplementary Material Methods section in Multimedia Appendix 1).

All LLMs were evaluated in single-turn interactions using their default parameter configurations, without additional manual tuning. Model inference was performed with the official vLLM framework, providing an optimized environment for efficient large-scale deployment. All models were executed on a cluster of 8 NVIDIA H20-141GB graphics processing units (GPUs) using the official Docker release. For version control, vLLM v0.8.4 was applied to all models except GPT-OSS-20B and GPT-OSS-120B, which were deployed under v0.10.1, thereby ensuring reproducibility and transparency across experiments.

The Multiagent Framework

Overview

In addition to standalone evaluations, we implemented a multiagent framework to examine whether structured reasoning and constrained outputs could enhance model performance in AIS-specific tasks. The framework integrates three components: (1) a workflow-oriented summary agent that extracts disease-relevant evidence from lengthy clinical narratives, (2) a guideline-concordant reasoning-path chain of thought (Short-CoT) Agent that enforces structured diagnostic steps, and (3) a clinically inspired multiple-choice constraint agent that standardizes outputs within evidence-based decision boundaries.

Summarization Agent

To mitigate performance degradation caused by lengthy case tokens and to ensure the extraction of salient details, a workflow-oriented summary agent was implemented. Using DeepSeek-R1 with few-shot prompting, this agent generated structured case summaries for downstream reasoning.

Short-CoT Agent

The Short-CoT module was designed as a concise sequence of 4 critical diagnostic steps, mirroring clinical decision trees. It was developed collaboratively with neurologists, interventional radiologists, and emergency physicians by restructuring existing clinical guidelines (refer to the prompt provided in Supplementary Material Methods in Multimedia Appendix 1).

Multiple-Choice Constraint Agent

For clinically inspired final decision-making, models augmented by the multiagent framework were evaluated using 4 predefined categories as answer options: thrombolysis, mechanical thrombectomy, standard medical therapy, and non-AIS or nonstroke conditions. This constraint not only improved consistency but also aligned model outputs with clinically interpretable categories.

Alternative Framework Compositions

We also evaluated 3 alternative framework compositions: F1 included the Short-CoT agent and the multiple-choice constraint agent but omitted the summary agent; F2 included the guideline-derived long CoT (Long-CoT) agent and the multiple-choice constraint agent but omitted the summary agent; F3 included the summary agent and the Long-CoT agent combined with the multiple-choice constraint agent. This part used a Long-CoT agent to augment the selected LLM decision-making. The Long-CoT involved filtering and restructuring clinical guidelines into a structured decision-making pathway for AIS (refer to the prompt provided in Supplementary Material Methods in Multimedia Appendix 1) [23].

Model-Level Performance and Output Safety Assessment

Ground truth for model-level evaluation was defined as the actual treatment decision and corresponding TOAST classification documented in the clinical record, with independent verification for guideline concordance performed by 2 senior stroke clinicians: a neurologist with >10 years of experience and a neurointerventional physician with >15 years of experience. Cases with disagreement or ambiguity regarding guideline concordance were jointly re-evaluated, and eligibility for inclusion was determined by consensus.

The model-level primary outcome was the overall therapeutic and TOAST diagnostic accuracy of LLMs. Accuracy was the case-level proportion correct; TOAST F1 was macroaveraged one-vs-rest F1 across classes, and diagnostic F1 was computed on binary correctness (1/0). Model-level secondary outcomes encompassed safety-related assessments, including instruction adherence, structured harmfulness assessment, and other exploratory safety metrics. For each case, the presence of any omission or hallucination was counted as one event (yes or no), and event rates were computed as events per case (events/total cases) [24,25].

Comprehensive Multicenter, Multisource, and Cross-Scenario Evaluation

Validation Using PubMed Case Reports and External Cohorts

To clinically validate the framework, we analyzed real-world cases from group B using 6 open-source LLMs. To further test generalizability across model families, we also evaluated 6 open-source LLMs and an additional closed-source LLM in group C (PubMed case reports). The PubMed case report dataset is publicly accessible and predominantly comprises diagnostically challenging cases. This dataset therefore provided a more stringent test of the generalizability of the multiagent framework across patients with varying levels of difficulty and complexity.

Prospective Clinical Validation With Multilevel and Multispecialty Physicians

To evaluate the clinical impact of LLM assistance, we conducted a prospective, blinded physician study at center A (approval 2024-KY-203) between February and May 2025. Twelve physicians with heterogeneous AIS expertise were recruited across 4 Chinese cities or provinces (Hunan, Guangdong, Jilin, and Shanghai), comprising junior (n=5), senior (n=5), and expert strata (n=2) and including stroke specialists and nonspecialists. Consecutive eligible patients were enrolled and randomized at enrollment to an AI-assisted arm (with LLM support) or a standard review arm (without LLM support). Each case was evaluated only once by a single assigned physician.

In the AI-assisted arm, the best-performing model from the retrospective evaluation (DeepSeek-R1) generated treatment recommendations, TOAST classifications, and structured reasoning. These model outputs were provided to physicians together with the routinely available clinical materials for each assigned case. Physicians additionally provided a 5-point Likert rating of the perceived effectiveness of the AI-assisted reasoning (with higher scores indicating greater perceived usefulness). In the standard review arm, physicians reviewed identical case materials without any model output or additional computer-based decision support beyond routine hospital systems.

The reference standard was the final multidisciplinary team decision. Additional design details, including sample-size estimation and allocation procedures, are reported in the Supplementary Material Methods in Multimedia Appendix 1.

Statistical Analysis

All analyses were performed in Python (version 3.10; Python Software Foundation). Statistical significance was set at a 2-sided P<.05. Binary and continuous variables were summarized descriptively. Accuracy differences between standalone and LLMs augmented by the multiagent framework were tested with the McNemar test, and F₁-score differences were estimated by bootstrap resampling (1000 iterations) with 95% CIs. Paired ordinal outcomes were compared using the Wilcoxon signed-rank test. The ablation study compared the full multiagent framework with predefined variants that removed or substituted key modules (summary agent, guideline-derived Long-CoT vs Short-CoT, and free-form vs constrained multiple-choice outputs) [26]. We fitted physician-case–level binomial-logit generalized linear mixed-effects models (GLMMs), with decision correctness as the binary outcome, and LLM support as the main fixed effect for treatment decision and TOAST classification [27]. Event rates—including omission and hallucination frequencies—and differences between AI-assisted and nonassisted arms were analyzed using chi-square or Fisher exact tests, as appropriate. Outcome metrics and definitions are summarized in Table S2 in Multimedia Appendix 1.

Results

Patient Characteristics

At center A, 1228 cases were screened retrospectively, and 1055 (85.91%) were included in group A. At center B, 938 were screened, and 721 (76.87%) were included in group B. In addition, 213 prospective cases were screened at center A between February and May 2025, of which 161 (75.59%) were included in group D. Of 327 PubMed case reports retrieved, 144 (44.04%) met the inclusion criteria and were included in group C. We finally analyzed 1937 clinical cases (mean age 68.0, SD 13.6 years; n=761, 39.29% women) between January 2018 and May 2025, and 144 PubMed case reports identified from January 2024 to January 2025 (Table 1).

Baseline Performance Disparities Among Standalone LLMs

Figure 2A and 2B and Figure S2 in Multimedia Appendix 1 illustrate the simplified system architecture for the standalone LLM and the LLMs augmented by the multiagent framework. Performance varied markedly by model size. Standalone larger-scale LLMs consistently outperformed smaller ones in both treatment recommendation and TOAST classification (Table 2). Additionally, the multiagent framework consistently improved macroaveraged sensitivity and specificity across all evaluated models (Table S3 in Multimedia Appendix 1). For example, across the pooled cohort, GPT-OSS-120B achieved 0.724 accuracy for treatment recommendation, compared with 0.570 for Baichuan-M1-14B and 0.464 for GPT-OSS-20B (all adjusted P<.0001; Table 2). Performance trends were consistent across individual subgroups (Table S4-S7 in Multimedia Appendix 1). Similar gaps were observed for TOAST classification, with DeepSeek-R1 surpassing all smaller models. These results establish model size as a key determinant of baseline performance.

‎

Multiagent Framework-Induced Improvements Across LLMs

Augmentation with the multiagent framework substantially improved outcomes across all models (Figure 2C-2H and Figure 3A; Figure S3 in Multimedia Appendix 1), with an average improvement of 18.9% compared with standalone LLMs. In group A, Baichuan-M1-14B accuracy increased by 25.8% (F₁-score +15.1%), while DeepSeek-R1-671B improved more modestly (+23.3% accuracy and F₁-score +12.7%). For TOAST classification, GPT-OSS-20B accuracy rose by 39.0% (F₁-score +4.9%), compared with only 2.5% (F₁-score −2.1%) for GPT-OSS-120B. Notably, GPT-OSS-20B exhibited unstable outputs with fluctuating gains. Collectively, these findings show that while model size remains critical for baseline accuracy, the multiagent framework reduces size-related disparities, enabling smaller models to approach the clinical utility of their larger counterparts.

‎

Cross-Group and Multisource Validation

Validation in groups B and C demonstrated consistent performance gains across all groups (Figure 2C-2H and Figure 3A; Tables S4-S7 in Multimedia Appendix 1). The framework-augmented DeepSeek-R1-671B achieved the highest accuracy in both treatment recommendation and TOAST classification. In group C, GPT-4o gained accuracy in treatment recommendation (+14.9%, 0.750 vs 0.653) but showed a decline in TOAST classification (−6.7%, 0.486 vs 0.521), reflecting the predominance of “other causes” and cryptogenic subtypes. The results for alternative framework compositions are shown in Figure 4. Collectively, these findings confirm that the framework provides reliable gains across heterogeneous cohorts.

‎

Contribution of Framework Components

Generalized linear mixed model analysis in group A confirmed that the multiagent framework was independently associated with higher accuracy in AIS treatment recommendations (odds ratio [OR] 1.72, 95% CI 1.63-1.81; P<.001) and increased selection of appropriate reperfusion strategies (Figure 3B).

In the ablation study (average-only aggregation), treatment recommendation performance improved when using a multiple-choice constraint compared with free-form answer formats (constrained vs free-form: +0.038), and CoT prompting outperformed direct prompting (long vs direct: +0.052; short vs direct: +0.063), suggesting that treatment recommendations benefit from structured answer formats and explicit reasoning scaffolds. Summarization yielded only a modest average gain for treatment recommendation (summary vs no summary: +0.006). In contrast, diagnosis benefited more from summarization (+0.041) and performed better with free-form answer formats (constrained vs free-form: −0.051), motivating task-specific choices of prompting strategy and answer-format control.

Qualifying LLM Reasoning for Safe Clinical Deployment

Beyond accuracy, we visualized step-by-step reasoning traces for representative success and failure cases to characterize failure modes and deployment-relevant clinical safety risks (Figure 5A and 5B). Because accuracy alone provides an incomplete view of clinical applicability, we further assessed output safety. Compared with the standalone LLM, the multiagent framework achieved higher mean clinical safety scores (4.01, SD 0.35 vs 3.70, SD 0.43) and lower mean hallucination (20.6% vs 33.6%) and omission rates (24.5% vs 38.5%; Figure 5C-5E; Table S8 in Multimedia Appendix 1). Other exploratory safety metrics are shown in Table S9 in Multimedia Appendix 1. These findings demonstrate that structured reasoning may improve reliability and mitigate the risk of unsafe content entering clinical workflows.

‎

LLM Support Benefits Less-Experienced Physicians

Integration of LLM support substantially enhanced physician performance, with the most pronounced gains observed for less-experienced physicians in treatment decisions (0.667 to 0.833 in junior specialists; 0.600 to 0.846 in nonspecialists). On the basis of physician-case decision-level observations (1 observation per physician per case), we fitted a binomial-logit GLMM with decision correctness as the binary outcome, LLM support, physician experience, and specialty as fixed effects, and a random intercept for case and physician. After adjustment, LLM support was associated with higher odds of correct treatment decisions (OR 2.86, 95% CI 2.27‐3.60; P<.0001) and correct TOAST classification (OR 3.63, 95% CI 2.85‐4.64; P<.0001; Figure S4 in Multimedia Appendix 1). The estimated association was strongest among junior and nonspecialist physicians. For treatment decisions, the ORs were 3.96 (95% CI 2.64-5.96) for junior nonspecialists and 3.22 (95% CI 2.13-4.87) for senior nonspecialists; for TOAST classification, the corresponding ORs were 4.66 (95% CI 3.01-7.22) and 4.27 (95% CI 2.74-6.66). Expert specialists showed positive but nonsignificant estimates, consistent with a ceiling effect at higher baseline performance levels (P<.001). Physicians further rated the assistance positively, with a mean score of 3.849 out of 5. Improvements were more modest in senior and specialist physicians (Figure 6).

‎

Discussion

This study advances the clinical translation of LLMs for AIS decision support by moving beyond benchmark-style evaluation to a multicenter, multisource assessment across retrospective, prospective, and literature-derived cases. First, the structured multiagent framework improved guideline-concordant therapy selection while producing more reliable and clinically accountable outputs, with lower hallucination and omission rates. Second, in a human-AI interaction experiment involving physicians of varying seniority and specialty backgrounds, the framework improved physician decision accuracy, with the largest gains observed among junior and nonspecialist physicians. Collectively, these findings suggest that LLM-integrated systems can enhance both intrinsic decision reliability and downstream clinical performance, offering a practical means to improve consistency, safety, and equity in AIS care when access to stroke expertise is uneven.

Global inequities in stroke care remain profound, with low- and middle-income countries constrained by limited resources and specialist expertise, while high-income countries face overcrowded emergency services and workforce pressures. Existing evaluations of LLMs have been largely confined to simplified benchmark tasks that fail to capture the real-world complexity of disease management [16,20,21]. To address this gap, we applied LLMs to real-world clinical and literature-derived cases, thereby simulating authentic clinical scenarios and reflecting heterogeneous contexts. Performance declined in complex settings, partly due to limitations in handling long token inputs [28,29]. To address this challenge, our multiagent framework incorporated a custom-designed summarization agent that extracted salient features from clinical narratives, improving accuracy across diverse scenarios. Augmented LLMs achieved accuracies ranging from 0.541 to 0.830 across model sizes—a level comparable to question-and-answer–style or simulated patient scenarios [13,30], and consistently higher than standalone LLMs.

This study represents an initial step toward translating LLM-assisted decision support for AIS into clinical practice, with a focus on reducing variability in care that arises from uneven clinical expertise. Prior work suggests that CoT prompting and fixed-answer formats can partially address these limitations [31,32]. Our multiagent framework introduces a workflow-oriented, guideline-concordant structure that improves decision accuracy while maintaining clinically consistent recommendations, supporting feasibility for future clinical integration. Unlike approaches that rely on ever-larger proprietary models [33], our results show that workflow-inspired structured design improvements can yield substantial gains without increasing model scale, potentially lowering adoption barriers and supporting broader access to specialist-level decision support.

Our evaluation offers one of the most comprehensive simulations of clinical practice to date, establishing a foundation for the deployment of LLMs in AIS workflows. While the promise is substantial, deployment at scale carries risks of unintended harmful consequences [34,35]. By integrating data from multicenter, multitier hospitals, retrospective and prospective real-world cases, literature-derived high-difficulty cases, and human-AI interactions across physicians of different levels and specialties, our evaluation captured the heterogeneity and complexity of AIS care. Notably, LLMs provided the most benefit to less-experienced physicians, narrowing expertise gaps across experience, specialty, and geography, and aligning with prior reports of near expert-level performance [36,37]. In alignment with the World Stroke Organization’s Global Stroke Declaration, which underscores that quality stroke care should be universal [2,38], our results advance the case for AI-enabled strategies to promote equity in global stroke systems.

This study systematically assessed safety, a critical prerequisite for clinical deployment. Because LLMs predict the next token without verifying evidence, they remain prone to hallucinations that can erode trust and generate harmful or misleading recommendations [39,40]. In our evaluation, hallucinations and omissions were not eliminated but occurred at relatively low frequencies, with the multiagent DeepSeek-R1-671B achieving a hallucination rate of 10.9%, an omission rate of 14.7%, and an overall clinical safety score of 4.36 out of 5. These findings indicate that while safety concerns remain, the error rates are within a range that may be acceptable for decision support use, supporting the feasibility of cautious clinical integration [41]. The multiagent-augmented LLMs further demonstrated stronger instruction adherence, lower hallucination and omission rates, and more reliable structured outputs, thereby addressing a key barrier to real-world deployment.

Our results suggest that structured multiagent LLM frameworks may improve guideline-concordant AIS decision support, output safety, and auditability, but the remaining safety-critical errors highlight the need for caution before clinical deployment. Such systems should be used only as human-in-the-loop decision-support tools, with final treatment decisions remaining under physician responsibility. Future workflow-based studies should prospectively test safeguards, including uncertainty signaling, guideline and local-protocol alignment, contraindication review, deferral under missing or ambiguous information, high-risk warnings, and complete logging of model inputs, outputs, and reasoning traces.

This study has several limitations. First, the framework was evaluated in case-based rather than real-time emergency room settings, precluding assessment of usability, clinician trust, latency, time to decision, bedside integration, patient outcomes, and uncertainty escalation. Second, the exclusion criteria may have produced a more standardized dataset than real-world emergency stroke care. By excluding cases with incomplete information, chronic-phase presentations, competing urgent conditions, or treatment refusal or discontinuation, the study may underrepresent information-limited and operationally constrained scenarios. Third, although the framework reduced omission and hallucination events, the proposed deployment guardrails were not prospectively tested in real clinical workflows, limiting conclusions about operational safety. Fourth, the automated grader used to categorize free-text standalone LLM outputs may have introduced evaluation error, particularly for ambiguous or internally inconsistent responses. Finally, rapid iteration of commercial LLMs and restricted access to proprietary systems may affect reproducibility and long-term stability. Future prospective multicenter workflow studies should evaluate the framework under more complex real-world conditions, with predefined safety guardrails, human adjudication of uncertain outputs, and patient-level outcome assessment.

In conclusion, our multicenter evaluation demonstrates that a structured, multiagent LLM framework significantly enhances guideline-concordant AIS decision-making (average accuracy gain: 18.9%) and output safety. Crucially, LLM support substantially increased the odds of correct physician decisions for both treatment (OR 2.86, 95% CI 2.27‐3.60) and TOAST classification (OR 3.63, 95% CI 2.85‐4.64), with the most pronounced benefits among less-experienced and nonspecialist clinicians. These findings highlight the framework’s potential to narrow expertise gaps and support equitable stroke care, warranting further prospective evaluations of real-world workflows and patient outcomes before clinical deployment.