Identifying Functional Status Impairment in People Living With Dementia Through Natural Language Processing of Clinical Documents: Cross-Sectional Study

Introduction

In the United States, over 6 million people are living with Alzheimer disease or related dementia, and this number is projected to increase to 13 million by 2050 [1]. As dementia progresses, the affected individuals lose the ability to carry out everyday activities, including basic activities of daily living (ADLs) and instrumental ADLs (iADLs), which are fundamental skills required to independently care for oneself and serve as an indicator of a person’s functional status [2]. Assessment of ADLs and iADLs is also essential in determining the severity of dementia and the care needs of older adults [3]. This information is important in predicting a patient’s risk of mortality, long-term nursing home admission, and health care use.

Despite the significance of assessing disability in performing ADLs and iADLs in people living with dementia, the assessment is not routinely done in clinical practice; if performed, it is often documented within the unstructured clinical notes of a patient’s electronic health record (EHR), making it difficult to readily locate. This process could be improved with natural language processing (NLP) and machine learning. NLP has been applied to health care research in a variety of ways, including quantifying changes in social media posts, better understanding the mental health impacts of COVID-19 [4], and extracting cancer phenotypes from clinical note text [5]. An NLP approach can convert free-text information on ADLs and iADLs from an EHR into structured data. The structured data of ADL and iADL can then be readily used in clinical care and research to perform statistical modeling for outcome prediction, patient phenotyping, and confounding or risk adjustment. In this study, we seek to develop and validate machine learning models that can identify clinical note text containing information on ADL and iADL impairment in people living with dementia.

Methods

Setting and Data Sources

We used data from the Research Patient Data Repository (RPDR) [6] linked to Medicare fee-for-service parts A (inpatient coverage), B (outpatient coverage), and D (prescription benefits) claims data for over 700,000 individuals from 2007 to 2017. The RPDR includes longitudinal EHR data from 2 tertiary hospitals, 3 community hospitals, and more than 35 primary care centers in Greater Boston, Massachusetts. This data set includes demographic information, inpatient and outpatient diagnoses and procedures, medical orders and drug prescriptions, vital signs, laboratory and radiology test results, and free-text notes and reports from inpatient and ambulatory encounters. We linked the EHR with Medicare claims data to reduce information leakage of the EHR due to care provided outside of our EHR [7,8]. The linkage was done deterministically by the unique Medicare beneficiary number, date of birth, and sex, with a success linkage rate of 98.7% [9]. Medicare is a US federal health insurance program that currently covers approximately 50 million Americans by providing medical and prescription drug coverage to individuals aged 65 years or older and to younger individuals with disabilities. The Medicare claims database contains longitudinal, individual-level data on health care use, diagnoses, diagnostic tests, procedures, and pharmacy-filled prescriptions.

Study Population

From the linked RPDR-claims data from 2007 to 2017, we identified older adults aged 65 years or older with at least 1 diagnosis of dementia using validated algorithms with positive predictive values of 78%-92% to define dementia [10]. The first date of dementia diagnosis during our study period was the cohort entry (index) date. We further required the study cohort to have at least 365 days of continuous enrollment in Medicare parts A and B and at least 1 admission, progress, or discharge note from an inpatient encounter or ambulatory visit within 365 days before the index date.

Labeled Data Set Development

Patient clinical notes were split into sentences using the Medical Text Extraction, Reasoning, and Mapping System NLP system [11]. In our preliminary exploration, the information relevant for ADL and iADL was noted to be sparse, resulting in a highly imbalanced data set with little input data for model development. To increase the prevalence of data containing information relevant to ADL and iADL impairment in our model training data set, we created a lexicon of ADL- and iADL-related key terms curated with physician expert guidance. A total of 3 medical doctors (DHK, MM, and KJL) came up with the initial list of terms with automatically generated synonyms, followed by list refinement based on clinical knowledge (see Table S1 in Multimedia Appendix 1 for a final list of the terms). We filtered and kept sentences that included at least 1 key term from the final lexicon using a filtering tool coded in Python. Notes were split into sentences but retained a context window of 250 characters before and after key terms. Each medical doctor was paired with a research assistant to form a review team. The 3 review teams first conducted preliminary reviews on the same set of notes, filtered by our key term lexicon. They discussed sentences with different classifications regarding ADL and iADL impairment and assessed interrater agreement (κ was 85.2%, 89.8%, and 85.4% between the 3 teams after 5 rounds of training sessions). Then the review teams manually labeled 10,000 randomly selected filtered sentences in 2743 clinical notes from 441 patients for evidence of ADL or iADL impairment. The data were randomly split into a 70% subset (filtered training cohort) and a 30% subset (filtered validation cohort). We used the training cohort to train and tune model parameters by 5-fold cross-validation and used the validation cohort to test the performance. Models were further evaluated using 1000 randomly selected, unfiltered sentences from an independent set of 80 patients for assessing generalizability.

Classifier Development

We implemented 5 commonly used statistical models in machine learning literature: logistic regression, support vector machine (SVM), least absolute shrinkage and selection operator (LASSO) regression [12], random forest [13], implemented using the Python Scikit-learn module [14], and gradient boosting, implemented using the Python XGBoost module [15]. Training data for these 5 models were represented as unigrams transformed using term frequency—inverse document frequency [16]. We also implemented a hierarchical attention-based deep learning model consisting of a convolutional neural network and long short-term memory network, developed in a previous study [17]. Additionally, we implemented a model derived from Bio+Clinical bidirectional encoder representations from transformers (BERT), a contextualized word representation model based on BioBERT, and trained further on Medical Information Mart for Intensive Care (MIMIC) data [18-21]. We performed 5-fold cross-validation on the training cohort to tune the parameters for each model based on area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC) metrics. Tuned model performance was then evaluated on the filtered validation cohort, and generalizability was tested on the unfiltered validation cohort (ie, the 1000 unfiltered sentence set). See Tables S2 and S3 in Multimedia Appendix 1 for the final parameters of the ADL and iADL impairment classifiers, respectively.

Ethical Considerations

The study was approved by the institutional review board of Brigham and Women’s Hospital, Boston, Massachusetts (2018P002462). Personal health information was used as minimally as possible within the needs of the study. Data were not shared with any individuals not directly involved in the study.

Results

Study Sample

The filtered cohort was used to extract our data set of 10,000 filtered sentences; it comprises 441 people with a mean age of 82.7 (SD 7.9) years. A total of 64% (n=283) of the people were female, 88% (n=389) were White, and 4% (n=19) were Black. During the 365-day baseline period, the mean frailty score in our cohort was 0.26 (SD 0.08) with the most commonly observed comorbidities being urinary tract infections (n=192, 43.5%), history of falls (n=172, 39%), failure to thrive (n=89, 20.2%), incontinence (n=73, 16.5%), pressure ulcers (n=57, 12.9%), and dysphagia (n=50, 11.3%). As far as health care use is concerned, people on average had 13 (SD 7.8) unique medications along with a mean of 11.5 (SD 8.8) outpatient visits in the baseline period. Furthermore, the mean number of baseline hospitalizations and emergency room visits were 1.3 (SD 1.7) and 2.4 (SD 2.7), respectively. Our key-term filtered data set contained 1628 (16.3%) sentences annotated as positive (ie, containing relevant information) for ADL impairment (1128 in the training subset and 500 in the internal evaluation subset) and 323 (3.2%) sentences annotated as positive for iADL impairment (234 in the training subset and 89 in the internal evaluation subset). In contrast, the unfiltered data set was used to extract the external validation cohort of 1000 unfiltered sentences from 80 patients that contained 7 (0.7%) sentences labeled positive for ADL impairment and 4 (0.4%) sentences labeled positive for iADL impairment. Compared to the filtered data set, the unfiltered data set has a comparable mean age and race composition but a slightly higher female percentage (56/80, 70% vs 283/441, 64.2%). The baseline comorbidity profile of the 2 study cohorts was largely comparable, except that the prevalence of aspiration pneumonia was higher in the filtered than unfiltered set. The health care use was also noted to be slightly higher in the filtered than in the unfiltered set (Table 1).

Performance of the ADL Model

AUROC and AUPRC performance across all models for ADL impairment detection in the training set and both evaluation sets are shown in Table 2 with receiver operating characteristic (ROC) and precision-recall curves for the filtered validation cohort evaluation in Figures 1 and 2, respectively. While most models scored high AUROC across data sets, there was more notable variation in AUPRC scores, particularly in the unfiltered validation set. LASSO performed best in training set cross-validation, with an AUROC of 0.958 and AUPRC of 0.865. Top predictors of the LASSO model include “incontinence,” “tube,” “incontinent,” “PEG (percutaneous endoscopic gastrostomy),” “bathing,” “dressing,” “feeding,” “total parenteral nutrition (TPN),” “assistance,” and “toileting.” These features tended to have high importance for the remaining models, as well as “gastrostomy” and “body.” As shown in Table 2, most ADL models performed similarly on the filtered validation cohort, with the random forest model achieving slightly better AUROC and AUPRC measures (0.971 and 0.887, respectively) than the others. All models’ AUROC scores improved for the unfiltered validation cohort, explained by the notable imbalance of the data set—only 0.7% (7/1000) cases were positive for ADL impairment in the unfiltered validation cohort versus 16.7% (500/3000) in the filtered validation cohort. Unfiltered validation AUROC was highest for the deep learning model (0.991). The AUPRC scores of all models decreased for the unfiltered validation cohort prediction, with the SVM model’s score the highest (0.822) and dropping the least. Though trained on more data than the deep learning model, the data that the Bio+Clinical BERT model is pretrained on is not specific to the Mass General Brigham (MGB) EHR. This is likely why Bio+Clinical BERT exhibited lower performance than the deep learning model, which was trained entirely on our annotated MGB data set.

‎

‎

Performance of the iADL Model

Evaluation data sets were more imbalanced for the iADL impairment classification task—the filtered validation cohort had a 3.0% positive rate for iADL impairment and just 0.4% in the unfiltered validation cohort. This resulted in wide CIs for reported performance. Across data sets, AUROC scores remained high for all models except the deep learning and Bio+Clinical BERT models, which may have been hindered due to the low prevalence of positive training instances (n=234, 3.3% of the training set). Table 3 presents model performance results for iADL impairment detection in the training and 2 evaluation sets. Logistic regression and XGBoost performed best in training set cross-validation AUROC (0.97), while SVM produced the highest AUPRC (0.735). Top predictors of the LASSO model include iADL-related terms such as “cooking,” “shopping,” “management,” “laundry,” “finances,” “meals,” “cleaning,” “food,” and “medication.” These features also tend to have high importance for the remaining models, along with “husband,” “drives,” and “bills.” The XGBoost model’s AUROC scores were best for both data sets (0.995 for filtered validation and 0.991 for unfiltered validation), while the Bio+Clinical BERT model had the highest AUPRC scores for each validation data set (0.551 filtered and 0.568 unfiltered). ROC curves for the filtered validation cohort iADL classification appear in Figure 3, and precision-recall curves are provided in Figure 4.

‎

‎

Clinical Implications

Table S4 in Multimedia Appendix 1 provides examples identified by our NLP classifier of various ways that ADL and iADL impairment can present in patient note text. The most prevalent key term categories (Table S1 in Multimedia Appendix 1) across the unfiltered cohort data set appear in Tables S5 and S6 in Multimedia Appendix 1 for ADL and iADL impairment, respectively. As shown by these tables, common ADL- and iADL-related terms appear in just a small fraction of patient note sentences. Using NLP can demonstrably improve the ability to locate evidence of diverse ADL and iADL impairments within patient notes. We performed a key term search on the unfiltered cohort data set using our ADL and iADL lexicons to see how well such an approach identified patient note sentences with evidence of ADL and iADL impairment. Key term search proved to be a sensitive approach—no false negatives occurred for ADL impairment identification and 1 (0.1%) occurred for iADL impairment identification. The false positive rate, however, was greater (45/1000, 4.5% for ADL vs 18/1000, 1.8% for iADL) when compared with the highest performing machine learning models (2/1000, 0.2% for ADL vs 1/1000, 0.1% for iADL).

Discussion

Principal Findings

Among people living with dementia in a cohort using US multicenter EHRs linked with Medicare claims data, we developed and validated NLP models to determine evidence of ADL and iADL impairment. Although the proportion of sentences in clinical notes that contained ADL and iADL information was low, our best-performing models effectively identified relevant sentences, with an AUROC of 0.990 (95% CI 0.984-0.995) for ADL (random forest) and 0.991 (95% CI 0.972-1.00) for iADL (XGBoost).

Identifying people living with dementia who have difficulty with basic ADLs and iADLs is important for clinical care and population health management. The degree of ADL and iADL impairment is associated with dementia severity and progression. The iADLs begin to decline at the mild cognitive impairment stage [22]. Widely used dementia severity scales, such as the Clinical Dementia Rating and Functional Assessment Staging Tool, require assessment of iADLs and ADLs. As the ability to perform iADLs and ADLs declines with the progression of dementia [23], early detection of iADL and ADL impairment can lead to early rehabilitation to preserve their daily function. In acute hospital care settings, assessment of iADL and ADL function could help identify those at risk of loss of independence and arrange care transition interventions [24]. Moreover, ADL dependence is a risk factor for falls in community-dwelling adults with dementia [25]. Similarly, iADL impairment is predictive of 30-day readmission and can be helpful in identifying high-risk patients for early interventions [26].

Despite the importance of ADL and iADL assessment, documentation of this information is neither standardized nor available in most EHR and claim data. As a result, measures of ADL and iADL impairment are not included in prediction models of readmission. Recently, an effort to use machine learning methods to extract ADLs and iADLs information from EHR free-text notes or reports showed a potential to improve risk prediction or clinical decision support systems. The iADL impairment identified using machine learning was predictive of 30-day readmission [26]. Similarly, geriatric syndromes that are not documented in structured EHR data can be further identified in unstructured clinical notes in the EHR using NLP algorithms [27]. It has also been shown that frailty described in clinical notes was associated with greater health care use [28].

Comparison to Previous Work

Our work adds to previous research by showing the utility of NLP and machine learning algorithms to identify ADL and iADL information in unstructured EHR data with high accuracy for older adults with dementia. ADL and iADL impairment information from clinical notes of people living with dementia can help researchers identify medically stable and ambulatory older adults with dementia and specific functional levels who can be enrolled in clinical trials. In addition, information on ADL and iADL function is an important confounder in administrative claims-based studies of medical interventions in the fields of geriatrics, neurology, rehabilitation medicine, and family medicine. Combining our NLP approach with other data from the EHR could further improve the validity of EHR-based analysis. The extent of confounding and further adjustment has become possible through EHR-claims linkage in clinical research networks [8,29].

Study Limitations

Our study has several limitations. Our model is based on a US metropolitan academic care delivery network. Because of the subjectivity of self-reported information, variations in documentation conventions, and different demographic or cultural backgrounds of the study population, it is unclear if our findings can be generalizable to other health care systems. Additionally, our findings were based on a relatively small cohort with 441 patients in the filtered set and 80 in the unfiltered set, so validation in a larger cohort, preferably with a different demographic profile, is needed to confirm generalizability. Additionally, the measure and conceptualization of iADLs can be complex due to the differences between cultural norms and gender roles. For example, women have greater health-related iADL limitations than men [30]. Cross-national variations in ADL and iADL impairment may reflect item-response bias due to culture-based gender norms rather than actual differences in disability levels [31].

Conclusion

In conclusion, we have developed models to determine ADL and iADL impairment among US Medicare beneficiaries using NLP and EHR unstructured data. Because ADL and iADL are typically not available as structured EHR data, our models can enhance researchers’ ability to identify subgroups among people living with dementia according to their ADL and iADL dependency. Our models can improve patient phenotyping and confounding adjustment in EHR data that are used in comparative effectiveness and safety research.