Benchmarking State-of-the-Art Large Language Models for Migraine Patient Education: Performance Comparison of Responses to Common Queries

Introduction

Migraine is a highly debilitating primary headache disorder. Long-term education for migraine is essential to help patients, identify triggers, use medications appropriately, and adopt lifestyle changes that can reduce the frequency and severity of attacks [1-3]. Large language models (LLMs), complex artificial intelligence systems trained on extensive data sets to produce human-like text responses, are promising tools for patient education. However, LLMs could lead to inaccurate responses known as “hallucinations” [4]. Therefore, rigorous evaluations within specific medical domains are essential. Nevertheless, there is a lack of benchmarking for popular online LLMs in patient education for migraines. We performed a comparative analysis of responses to migraine-related queries from five leading LLMs: OpenAI’s ChatGPT 3.5 (December 2022) and 4.0 (March 2023), Google Bard (February 2023), Meta Llama2 (July 2023), and Anthropic Claude2 (July 2023).

Methods

Study Process

This study was conducted from October 1 to October 28, 2023. With reference to guidelines and their clinical expertise, neurologists meticulously crafted 30 migraine-related queries spanning a wide array of topics, including evaluation and definition, testing and diagnosis, treatment, follow-up, and prognosis, as well as special population considerations [5-7]. Each query was individually presented to the 5 leading LLMs through an independent conversation web-based interface to avert chained prompting. Responses from all LLM chatbots were converted into plain text, obscuring specific chatbot features to ensure blinding. The generated answers were then randomly ordered in each set of queries and assessed by blinded reviewers, with a 24-hour interval between each assessment to mitigate memory bias (Figure 1).

‎

The rating panel comprised three seasoned neurologists, each with at least 15 years of experience in the field. Their primary task was to independently evaluate the accuracy of the LLM chatbots’ responses using a 3-point scale (“poor,” “borderline,” and “good”).

Ethical Considerations

Approval from an ethics committee was not required for this study as no patients were involved [8].

Results

ChatGPT-3.5, ChatGPT-4.0, Google Bard, Meta Llama2, and Anthropic Claude2 achieved appropriate response rates (Pearson χ² test P=.48; Table 1); Multimedia Appendix 1 provides examples of the prompts and responses from each LLM. Google Bard had a relatively higher “poor” rating proportion than that of the other LLMs (Pearson χ² test P=.96; Table 1). None of the LLMs provided an appropriate response to the query “In severe cases, are there surgical options available for treating migraine?” This deficiency potentially stems from their limited ability to distinguish between migraine and secondary headaches, a crucial distinction in medical diagnosis. The complexity and ongoing debate surrounding surgical interventions for migraine treatment likely contribute to this gap in appropriate advice. Specifically, ChatGPT-3.5 erroneously proposed hemicraniectomy for persistent and severe migraines.

Discussion

Managing migraines over an extended period significantly strains both health care systems and patients [2]. In this study, most LLM responses were accurate and practical, with the majority of responses graded as “good” or “borderline” rather than “poor.” Among the five LLMs tested, ChatGPT-4.0 had the highest accuracy, although the difference in performance was not statistically significant. These observations suggest that LLMs may function as assistive tools in providing advice, enhancing information acquisition, and offering personalized responses, thereby supporting both patients and physicians.

This study represents the first effort to evaluate the capability of state-of-the-art LLMs in educating patients with migraine. Although prior studies have primarily focused on migraine diagnostic tests [9], there was no thorough head-to-head comparison of emerging LLMs for patient education about migraines. The observed variation in performance may be influenced by differences in LLM settings or the limited range of queries tested. Our study benefits from a rigorous study design, incorporating blinding, proper randomization, and an expert review by three neurology specialists. However, it also has limitations. First, the potential generation of false information such as hallucinations could lead to patient confusion and even delays in seeking proper medical care. This highlights the need for strategies to mitigate risks, such as improving model bias monitoring and implementing more stringent testing phases before real-world deployment. Second, this study is potentially underpowered due to the limited sample size; therefore, statistical insignificance should not be viewed as proof of equivalent LLM performance. Given the continually evolving nature of LLMs, ongoing evaluation with broader data sets is crucial for maintaining the validity of the models and clarifying performance differences between LLMs.

In conclusion, these state-of-the-art LLMs have the potential to accurately respond to common migraine-related queries, which may have implications for generative artificial intelligence–assisted migraine education.