TY  - JOUR
AU  - Hao, Jie
AU  - Chen, Zhenli
AU  - Peng, Qinglong
AU  - Zhao, Liang
AU  - Zhao, Wanqing
AU  - Cong, Shan
AU  - Li, Junlian
AU  - Li, Jiao
AU  - Qian, Qing
AU  - Sun, Haixia
PY  - 2025
DA  - 2025/3/18
TI  - Prompt Framework for Extracting Scale-Related Knowledge Entities from Chinese Medical Literature: Development and Evaluation Study
JO  - J Med Internet Res
SP  - e67033
VL  - 27
KW  - prompt engineering
KW  - named entity recognition
KW  - in-context learning
KW  - large language model
KW  - Chinese medical literature
KW  - measurement-based care
KW  - framework
KW  - prompt
KW  - prompt framework
KW  - scale
KW  - China
KW  - medical literature
KW  - MBC
KW  - LLM
KW  - MedScaleNER
KW  - retrieval
KW  - information retrieval
KW  - dataset
KW  - artificial intelligence
KW  - AI
AB  - Background: Measurement-based care improves patient outcomes by using standardized scales, but its widespread adoption is hindered by the lack of accessible and structured knowledge, particularly in unstructured Chinese medical literature. Extracting scale-related knowledge entities from these texts is challenging due to limited annotated data. While large language models (LLMs) show promise in named entity recognition (NER), specialized prompting strategies are needed to accurately recognize medical scale-related entities, especially in low-resource settings. Objective: This study aims to develop and evaluate MedScaleNER, a task-oriented prompt framework designed to optimize LLM performance in recognizing medical scale-related entities from Chinese medical literature. Methods: MedScaleNER incorporates demonstration retrieval within in-context learning, chain-of-thought prompting, and self-verification strategies to improve performance. The framework dynamically retrieves optimal examples using a k-nearest neighbors approach and decomposes the NER task into two subtasks: entity type identification and entity labeling. Self-verification ensures the reliability of the final output. A dataset of manually annotated Chinese medical journal papers was constructed, focusing on three key entity types: scale names, measurement concepts, and measurement items. Experiments were conducted by varying the number of examples and the proportion of training data to evaluate performance in low-resource settings. Additionally, MedScaleNER’s performance was compared with locally fine-tuned models. Results: The CMedS-NER (Chinese Medical Scale Corpus for Named Entity Recognition) dataset, containing 720 papers with 27,499 manually annotated scale-related knowledge entities, was used for evaluation. Initial experiments identified GLM-4-0520 as the best-performing LLM among six tested models. When applied with GLM-4-0520, MedScaleNER significantly improved NER performance for scale-related entities, achieving a macro F1-score of 59.64% in an exact string match with the full training dataset. The highest performance was achieved with 20-shot demonstrations. Under low-resource scenarios (eg, 1% of the training data), MedScaleNER outperformed all tested locally fine-tuned models. Ablation studies highlighted the importance of demonstration retrieval and self-verification in improving model reliability. Error analysis revealed four main types of mistakes: identification errors, type errors, boundary errors, and missing entities, indicating areas for further improvement. Conclusions: MedScaleNER advances the application of LLMs and prompts engineering for specialized NER tasks in Chinese medical literature. By addressing the challenges of unstructured texts and limited annotated data, MedScaleNER’s adaptability to various biomedical contexts supports more efficient and reliable knowledge extraction, contributing to broader measurement-based care implementation and improved clinical and research outcomes. 
SN  - 1438-8871
UR  - https://www.jmir.org/2025/1/e67033
UR  - https://doi.org/10.2196/67033
DO  - 10.2196/67033
ID  - info:doi/10.2196/67033
ER  -