Performance of Retrieval-Augmented Large Language Models to Recommend Head and Neck Cancer Clinical Trials

Introduction

Chatbots based on large language models (LLMs) have demonstrated the ability to answer oncology examination questions with impressive accuracy without specialized training or reinforcement [1,2]; however, leveraging LLMs in oncology decision support has not yet demonstrated suitable performance, as LLMs would produce responses that deviate from cancer expert recommendations and guidelines [3-5]. Furthermore, the rapidly changing oncology landscape, including knowledge of cancer clinical trials, limits the meaningful use of LLMs in practice given delays in training dataset updates. To enhance LLM utility in oncology practice, we developed a retrieval-augmented LLM, powered by GPT-4, and evaluated its performance to provide appropriate clinical trial recommendations for a head and neck (HN) cancer population.

Methods

On February 1, 2022, we piloted a clinical trial knowledge management application, LookUpTrials, at the Memorial Sloan Kettering Cancer Center (MSK) [6]. Using LookUpTrials’ real-time database, we applied retrieval-augmented generation architecture and direct preference optimization to fine-tune GPT-4 as a clinical trial decision assistant [7]. Specifically, we enabled retrieval-augmented GPT-4 to respond with up-to-date information—such as trial availability—developed initial prompts, and validated GPT-4 responses from 1120 preference pairs across 56 MSK HN clinical trials. Preference pairs were constructed in [trial : attributes] format, including 20 organizational, investigator, and study attribute types (Multimedia Appendix 1). Data labels were annotated by author TKWH and cross-verified by 2 trial managers. From November 7, 2023, to January 30, 2024, we collected all consecutive new patient cases and their respective clinical trial recommendations, which were made by consensus during a weekly HN conference attended by 5-8 oncologists with 2 to more than 25 years of practice experience. Cases were categorized by diagnosis, biomarkers, cancer stage, treatment setting, and physician recommendations on clinical trials. Using these cases as test datasets, we prompted retrieval-augmented GPT-4 using a semistructured template, as follows: “Given patient with a <biomarkers>, <diagnosis>, <cancer stage>, <treatment setting>, what are possible clinical trials?” (eg, given a patient with human papillomavirus–associated HN cancer, metastatic stage, in a first-line treatment setting, what are the possible clinical trials?). GPT-4 responses were compared with physician recommendations, with concordance defined a priori: a GPT-4 response was a true positive if it included the recommended clinical trial(s); a true negative if neither the GPT-4 response nor the physicians recommended any clinical trial(s); a false positive if the GPT-4 response recommended clinical trial(s) but physicians did not; and a false negative if the GPT-4 response did not recommend clinical trial(s) but the physicians did. We analyzed the performance of GPT-4 based on its response precision (positive predictive value), recall (sensitivity), and F₁-score (harmonic mean of precision and recall). We further analyzed subgroup performance by cancer types and the presence of biomarkers. Statistical analyses were performed using JMP-17.2.0.

Ethical Considerations

MSK institutional review board approved the study (application number: 24-120).

Results

We analyzed 178 patient cases (mean age 66, SD 13.9 years), primarily male (n=134, 75.3%), with local/locally advanced cancers (n=121, 68.0%), including HN (n=109, 61.2%), thyroid (n=29, 16.3%), skin (n=16, 9.0%), or salivary gland (n=14, 7.9%) cancers (Table 1). Over one-third of cases had biomarkers (n=66, 37.1%). The majority were treated in the definitive setting with combined modality therapy (n=75, 42.1%), and a modest proportion were treated under clinical trials (n=18, 10.1%). Overall, retrieval-augmented GPT-4 achieved moderate performance (Table 2), matching physician clinical trial recommendations with 63.0% precision and 100.0% recall (F₁-score 0.77), narrowing a total of 56 HN clinical trials to a range of 0-4 relevant trials per patient case (mean 1, SD 1.2 trials). In comparison, baseline non–retrieval-augmented GPT-4 demonstrated 0.0% precision, recall, and F₁-score—given the lack of response specificity to MSK clinical trials. Subgroup precision varied by cancer types (HN cancers: 72.7%, skin cancers: 50.0%, salivary gland cancers: 36.4%, and thyroid cancers: 33.3%) and the presence of biomarkers (presence 72.7%, absent 62.1%).

Discussion

Our study demonstrated that retrieval-augmented GPT-4 achieved moderate performance in matching physician clinical trial recommendations in HN oncology. Comparatively, our retrieval-augmented LLM outperformed its pre–fine-tuned baseline and exceeded the historical performance of pretrained LLMs for providing oncology treatment recommendations by 4-20 folds (F₁-score 0.04-0.19) [4]. Prior studies have evaluated LLM performance in matching patients to clinical trials, achieving high accuracy [8-10]; however, to our knowledge, our study is the first to evaluate an oncology-specific, retrieval-augmented LLM as a point-of-care, clinical trial decision support application. As our subgroup analyses demonstrated, LLM performance varies based on the specificity of the prompt and dataset, with enhanced precision achieved through reduced search ambiguity for biomarker-specific trials and cancer types with more well-defined datasets. Study limitations included small sample size, short-term assessment, cross-sectional design, disease-specific focus, and being conducted in a single institution, which limits generalizability and subgroup analyses; however, our study provides insights into the rarely measured performance of retrieval-augmented LLMs using real-world patient cases. Future research is needed to optimize LLMs’ precision and stability and to assess their implementation and effectiveness as a scalable solution for enhancing clinical trial participation.