This paper is in the following e-collection/theme issue:

Digital Mental Health Interventions, e-Mental Health and Cyberpsychology (1514) Machine Learning (1465) Depression and Mood Disorders; Suicide Prevention (1363) Diagnostic Tools in Mental Health (242)

Published on in Vol 21, No 7 (2019): July

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/13809, first published .
Identification of Patients in Need of Advanced Care for Depression Using Data Extracted From a Statewide Health Information Exchange: A Machine Learning Approach

Identification of Patients in Need of Advanced Care for Depression Using Data Extracted From a Statewide Health Information Exchange: A Machine Learning Approach

Identification of Patients in Need of Advanced Care for Depression Using Data Extracted From a Statewide Health Information Exchange: A Machine Learning Approach

Authors of this article:

Suranga N Kasthurirathne1, 2 Author Orcid Image ;   Paul G Biondich1, 3 Author Orcid Image ;   Shaun J Grannis1, 3 Author Orcid Image ;   Saptarshi Purkayastha4 Author Orcid Image ;   Joshua R Vest1, 2 Author Orcid Image ;   Josette F Jones4 Author Orcid Image
Article Authors Cited by (19) Tweetations (26) Metrics

    Original Paper

    1Center for Biomedical Informatics, Regenstrief Institute, Indianapolis, IN, United States

    2Indiana University Fairbanks School of Public Health, Indianapolis, IN, United States

    3Indiana University School of Medicine, Indianapolis, IN, United States

    4Indiana University School of Informatics and Computing, Indianapolis, IN, United States

    Corresponding Author:

    Suranga N Kasthurirathne, PhD

    Center for Biomedical Informatics

    Regenstrief Institute

    1101 W 10th St

    Indianapolis, IN, 46202

    United States

    Phone: 1 3173323480

    Email: snkasthu@iupui.edu


    Background: As the most commonly occurring form of mental illness worldwide, depression poses significant health and economic burdens to both the individual and community. Different types of depression pose different levels of risk. Individuals who suffer from mild forms of depression may recover without any assistance or be effectively managed by primary care or family practitioners. However, other forms of depression are far more severe and require advanced care by certified mental health providers. However, identifying cases of depression that require advanced care may be challenging to primary care providers and health care team members whose skill sets run broad rather than deep.

    Objective: This study aimed to leverage a comprehensive range of patient-level diagnostic, behavioral, and demographic data, as well as past visit history data from a statewide health information exchange to build decision models capable of predicting the need of advanced care for depression across patients presenting at Eskenazi Health, the public safety net health system for Marion County, Indianapolis, Indiana.

    Methods: Patient-level diagnostic, behavioral, demographic, and past visit history data extracted from structured datasets were merged with outcome variables extracted from unstructured free-text datasets and were used to train random forest decision models that predicted the need of advanced care for depression across (1) the overall patient population and (2) various subsets of patients at higher risk for depression-related adverse events; patients with a past diagnosis of depression; patients with a Charlson comorbidity index of ≥1; patients with a Charlson comorbidity index of ≥2; and all unique patients identified across the 3 above-mentioned high-risk groups.

    Results: The overall patient population consisted of 84,317 adult (aged ≥18 years) patients. A total of 6992 (8.29%) of these patients were in need of advanced care for depression. Decision models for high-risk patient groups yielded area under the curve (AUC) scores between 86.31% and 94.43%. The decision model for the overall patient population yielded a comparatively lower AUC score of 78.87%. The variance of optimal sensitivity and specificity for all decision models, as identified using Youden J Index, is as follows: sensitivity=68.79% to 83.91% and specificity=76.03% to 92.18%.

    Conclusions: This study demonstrates the ability to automate screening for patients in need of advanced care for depression across (1) an overall patient population or (2) various high-risk patient groups using structured datasets covering acute and chronic conditions, patient demographics, behaviors, and past visit history. Furthermore, these results show considerable potential to enable preventative care and can be easily integrated into existing clinical workflows to improve access to wraparound health care services.

    J Med Internet Res 2019;21(7):e13809

    doi:10.2196/13809

    Keywords

    depression; supervised machine learning; delivery of health care 


    Background

    Depression is the most commonly occurring mental illness worldwide [1]. It negatively affects how up to 350 million persons worldwide think, feel, and interact [2]. Depression poses significant health and economic burdens to both the individual and community [3]. Previous studies have presented a strong comorbidity between mental health and medical conditions [4]. Depression is highly prevalent among patients suffering from various chronic conditions [5,6]. Such patients may suffer up to a 10-to-25-year reduction in life expectancy [7,8]. Depression is also a leading cause of disability for Americans aged between 15 and 44 years [9]. The incremental economic burden of depression covering medical, pharmaceutical, workplace, and suicide-related costs in the United States was evaluated at US $210.5 billion in 2010, a 21.5% increase from 2005 [10].

    Different types of depression pose different levels of risk. Individuals who suffer from mild forms of depression may recover without any assistance. Other less severe cases can be effectively managed by primary care or family practitioners [11-13]. However, other forms of depression are far more severe and require advanced care above and beyond that provided by primary care or family practitioners [14,15]. Identifying cases of depression that require advanced care may be challenging to primary care providers and health care team members whose skill sets run broad rather than deep. Training health care teams to successfully identify patients with severe depression would resolve the problem but is unfeasible given cost, effort, and time considerations [16,17]. Social stigma and ignorance of health issues also encourage depression sufferers to downplay their condition, further increasing difficulty in detection and assessment [18].

    Many health care systems leverage screening tools such as the Beck Depression Scale [19], the Patient Health Questionnaire-9 (PHQ-9) [20], PHQ-15 [21], the Cornell Scale for Depression in Dementia [22], and the Hamilton Rating Scale for Depression [23] to evaluate depression severity. However, such tools are not optimal as they (1) tie up significant resources [24], (2) rely heavily on potentially inaccurate patient-reported outcomes for decision making [25], and (3) utilize only a small subset of clinical and behavioral data for decision making. In addition, traditional depression screening approaches may increase risk of overdiagnosis and overtreatment of depression across community and primary care settings [26-28] without contributing to better mental health [29]. Recent studies have questioned the benefits of routine screening [30,31] as well as the US Preventive Services Task Force recommendations to screen adults for depression in primary care settings where staff-assisted depression management programs are available [29].

    Given such limitations, it is more appropriate to develop machine learning–based screening approaches capable of leveraging more comprehensive patient datasets representing a patient’s overall health status to identify individuals who cannot be treated at primary care alone and would suffer from worsening health conditions unless they are provided with specialized, high-intensity treatment for depression [14,15]. Machine learning enables us to learn from multiple primary and secondary care datasets that might be missed by a clinician because of cognitive burden, and therefore, are a suitable solution to this challenge.

    Objectives

    For purposes of this research, we have defined individuals whose quality of life and health status will degrade if they do not receive specialized treatment above and beyond primary care as patients in need of advanced care for depression. Operationally, such patients would be identified by evaluating clinical data to detect patients who had received referrals to a certified mental health provider for specialized treatment for depression, indicating that their illnesses cannot be treated at primary care alone. In this study, we leveraged data obtained from varied structured and unstructured datasets to build decision models capable of identifying patients in need of advanced care for depression.


    Patient Population

    We identified a population of 84,317 adult patients (≥18 years of age) with at least 1 primary care visit between the years 2011 and 2016 at Eskenazi Health, a leading health care provider in Indianapolis, Indiana.

    Patient Subset Selection

    We sought to predict the need for advanced care for depression across (1) the overall patient population and (2) different groups of high-risk patient populations. We selected 3 high-risk patient groups: group A: patients with a past diagnosis of depression, group B: patients with a Charlson Comorbidity Index [32] of ≥1, and group C: patients with a Charlson Comorbidity Index of ≥2. Patients with a past diagnosis of depression were flagged as a high-risk group as their illness may re-emerge or worsen based on other health conditions. Patients with Charlson indexes ≥1 and ≥2 were selected because of the high prevalence of depression among patients suffering from one or many chronic illnesses [33] and its ability to worsen health outcomes of patients. Thresholds of ≥1 and ≥2 were selected because they captured patient populations that were adequately large for machine learning processes, as well as the cost/effort of potential implementation. We also identified a fourth group (Group D) that comprised all unique patients identified in groups A to C.

    We trained models for different populations to capture as many of the overall number of patients in need of advanced care for depression and to identify which patient groups were most suitable for use in screening for need of advanced care. Furthermore, focusing on a smaller population of high-risk patients may be easier to operationalize and cost-efficient to implement across chronic care clinics. Groups A to D were identified by analyzing diagnostic data on each of the 84,317 unique patients (master patient list) for past diagnosis of depression and to calculate Charlson Comorbidity Index for each patient.

    Data Preparation

    In a previous effort, we developed a depression taxonomy [34] using knowledge-based terminology extraction of the Unified Medical Language System (UMLS) Metathesaurus [35]. The taxonomy was developed by performing a literature search on Ovid Medline to identify publications that discuss depression and its treatment and then using Metamap [36], a Natural Language Processing–based tool to map these abstracts against the UMLS Metathesaurus, a large, multipurpose, multilingual thesaurus that contains millions of biomedical and health-related concepts, synonymous names, and their relationships across 199 medical dictionaries [37]. The most frequently occurring UMLS concepts were compiled into a terminology using the Web Ontology Language, a semantic Web language that is widely used to represent ontologies. These features presented a wide variety of diagnostic, demographic, and behavioral features that impacted the onset and severity of depression [34].

    We obtained longitudinal health records on each patient from the Indiana Network for Patient Care (INPC), a statewide health information exchange [38,39]. Thus, our dataset included records on each patient, including data that may have been captured at any hospital system that participated in the INPC. The dataset included a wide array of patient data, including patient demographic, diagnostic, behavioral, and visit data reported in both structured and unstructured form. All diagnostic data were obtained in the form of structured International Classification of Diseases, ninth revision (ICD-9) and ICD-10 codes. We assessed extracted data against the depression terminology and used relationships presented within the UMLS Metathesaurus to identify ICD codes for inclusion as features. We tabulated vectors of features for each patient group under study. We predict current risk levels based on past patient data. We did not assess the impact of temporality because of our dataset representing a (1) relatively short time period and (2) an older population with high chronic conditions that do not change significantly over time. In the event that the patient under study had received a referral for depression treatment, the data vector only comprised medical data recorded up to 24 hours before the aforesaid referral order. If no past referrals for depression treatment were present, then the vector comprised all available data on the patient. A master data vector encompassing all 84,317 patients was also created using the same approach.

    Preparation of Gold Standard

    We applied regular expressions to physician referrals to certified mental health providers to identify referrals where the physician was recommending specialized treatment for depression. We determined that our use of regex patterns was 100% accurate via manual review.

    Decision Model Building

    We split each of the 5 data vectors (4 patient subgroups and 1 master data vector) into random groups of 90% training data and 10% test data. Each training dataset was used to train a decision model using the random forest classification algorithm [40]. The random forest algorithm was selected because of its track record of successful use in decision modeling for health care challenges [41,42] and its ability to develop interpretable machine learning predictions [43]. We used Python programming language (version 2.7.6) for all data preprocessing tasks and the Python scikit-learn package for decision model development and testing [44].

    Analysis

    Each decision model was evaluated using the 10% holdout test set. Results produced by each decision model were evaluated using area under the curve (AUC) values, which measure classifier accuracy. Youden J Index [45] was used to identify optimal sensitivity and specificity for each decision model.

    A flowchart representing our workflow from patient group selection to decision model evaluation can be seen in Figure 1.

    ‎
    Figure 1. A flowchart representing our workflow from patient group selection to decision model evaluation. IHIE: Indiana Health Information Exchange.
    View this figure

    Evaluation of Patient Groups

    We identified a total of 12,432 patients with a diagnosis of depression (group A), 32,249 patients with a Charlson Index of 1 or greater (group B), and 7415 patients with a Charlson Index of 2 or more (group C). Overall, these 3 groups identified a total of 37,560 unique patients (group D).

    The master patient list as well as each of the 4 high-risk patient groups represented an adult, urban population: predominantly female and with high disease burdens (Table 1). The populations identified by their Charlson Indexes were older (mean age >50 years) than the population identified with depression (46.31 mean age). In addition, populations identified based on Charlson Indexes were predominantly African American. In contrast, the population with a past diagnosis of depression was predominantly composed of non-Hispanic whites. As anticipated, the prevalence of depression across a patient population with a Charlson Index of 1 or greater (30.18%) and a patient population with a Charlson Index of 2 or greater (37.25%) was greater than across the master patient list (19%).

    Figure 2 presents a Venn diagram presenting overlap across the high-risk patient groups identified for the study.

    A total of 6992 (8.29%) of the 84,317 patients in the master patient list were in need of advanced care for depression. Group A captured 3683 (52.68%) of these patients, group B captured 4016 (57.43%), and group C captured 1026 (14.67%). Overall, all 3 patient groups were able to identify 5612 (80.26%) of all patients in need of advanced care for depression.

    Table 1. Characteristics of the master patient list/groups of high-risk patients used for decision model building.
    Characteristic of interestMaster patient
    set: all patients
    (N=84,317)    		Group A: patients
    with a past diagnosis
    of depression    		Group B: patients
    with a Charlson
    Index of ≥1    		Group C: patients
    with a Charlson
    Index of ≥2    		Group D: all unique
    patients in groups
    A-C
    Patient group size, n (%)—a12,432 (14.74)32,249 (38.25)7415 (8.8)37,560 (44.5)
    Need of advanced care for depression, n (%)6992 (8.29)3683 (30.04)4016 (12.94)1026 (21.6)5612 (80.26)
    Demographics

    		Age (years), mean (SD)43.88 (15.60)46.31 (14.74)51.94 (14.55)59.50 (12.33)50.31 (14.93)

    		Male gender (%)35.0930.2239.843.9842.03

    		Race/ethnicity (%)


    		White (non-Hispanic)25.2144.6233.3837.0235.31


    		African American (non-Hispanic)37.2332.0142.7847.2640.12


    		Hispanic or Latino19.4711.1210.604.947.38

    		Diagnoses


    		Depression (%)19.0710030.1837.2537.51


    		Charlson Index score, mean (SD)0.77 (1.21)0.22 (0.75)1.89 (1.27)3.85 (1.14)1.62 (1.35)

    		Hospitalizations, mean (SD)


    		EDb visits during current month0.21 (1.03)0.33 (1.48)0.26 (1.15)0.31 (1.14)0.27 (1.17)


    		ED visits before previous months3.73 (14.40)4.69 (18.73)8.63 (24.2)10.71 (31.36)8.03 (23.67)

    aNot applicable.

    bED: emergency department.

    ‎
    Figure 2. Overlap between the patient groups identified for the study.
    View this figure

    Feature Selection Using the Depression Terminology

    Comparison of patient data against the depression terminology resulted in the identification of 1150 unique concepts for inclusion in each decision model. A description of features included in each of the decision models is presented in Multimedia Appendix 1.

    Decision Model Performance

    The decision model predicting need of advanced care across the master population reported a moderate AUC score of 78.87% (optimal sensitivity=68.79%, optimal specificity=76.30%). However, decision models to predict need of advanced care across patients’ groups A to D performed significantly better. Group A (patients with a past diagnosis of depression) reported an AUC score of 87.29% (optimal sensitivity=77.84%, optimal specificity=82.66%). Group B (patients with a Charlson Index of ≥1) reported an AUC score of 91.78% (optimal sensitivity=81.05%, optimal specificity=89.21%). Group C (patients with a Charlson Index of ≥ 2) reported an AUC score of 94.43% (optimal sensitivity=83.91%, optimal specificity=92.18%), whereas Group D (list of unique patients from groups A-C) reported an AUC score of 86.31% (Figure 3; optimal sensitivity=75.31%, optimal specificity=76.03%). Precision-recall curves for each decision model are presented in Multimedia Appendix 2.

    The top 20 features for each decision model can be seen in Multimedia Appendix 3. Multimedia Appendix 4 presents the co-occurrence of these top 20 features across each decision model under study. In assessing the top ranked features selected for each decision model, we found significant overlap among the top features for each of the high-risk patient populations. Furthermore, essential (primary) hypertension, depression, gender, and number of outpatient visits appears in the top 20 feature lists for every patient population under test.

    To demonstrate that the models did not suffer from overtraining, we added an additional evaluation step where we compared model performance across smaller feature subset sizes. We ranked all features for each decision model using information gain aka. Kullback-Leibler divergence [46]. For each patient subgroup, we used the ranked feature lists to build multiple decision models starting with a decision model trained using only the 5 top ranking features, iteratively adding on the next most important feature, retraining the model and evaluating performance using F1 core. We continued this process until we had trained n-5 models using all n features in the feature set. As an example, for patient group A, we began by building a decision model consisting of 5 patient-centric features and assessing its performance using F1 score. Afterwards, we added in the 6th most important feature and retrained a decision model consisting of these 6 features. We continued building models and evaluating F1 scores until we had included all features from each dataset. The results of this exercise (Multimedia Appendix 5) demonstrated that model performance plateaued after the top 10 to 20 features and that inclusion of further features did not improve model performance. This demonstrates that the models were not overfit and that they reached optimal performance after a relatively small number of features.

    ‎
    Figure 3. Receiver operating characteristic curves produced by decision models predicting need of advanced care across each patient group under study. AUC: area under the curve.
    View this figure

    Principal Findings

    The decision model to predict need of advanced care for depression across the overall patient population achieved an AUC score of 78.87%. In comparison, decision models that predicted need of advanced care across 4 high-risk patient groups performed better, with AUC scores ranging from 86.31 to 94.43%. In addition, optimal sensitivity and specificity for each decision model was significantly high and demonstrated the models’ potential for practical implementation.

    We attribute the comparatively lower performance of the decision model developed using the overall population to the unbalanced nature of the gold standard [47] caused by the relatively low prevalence (8.29%) of patients in need of advanced care and the sparsity of data available for some of the patients in the overall patient population. The high performance of the decision models built using high-risk patient groups could be attributed to the higher prevalence of patients in need of advanced care. Although various publications have presented approaches to address data imbalance [48,49], we did not pursue such as approach as we wished to focus on demonstrating methods that could be replicated across other datasets that may or may not be imbalanced.

    In assessing prediction performance, group C (patients with a Charlson Index ≥2) yielded the highest AUC score (97.43%). Groups A (patients with a diagnosis of depression) and B (patients with a Charlson Index ≥1) reported lesser AUC scores. Group C captured the least number of patients in need of advanced care in comparison with groups A and B. However, it is noteworthy that none of the decision models developed using high-risk populations could capture all patients in need of advanced care. Overall, all 3 models could capture only 80.26% of all patients in need of advanced care. The remainder (19.74%) of the patients in need of advanced care did not qualify for any of the three high-risk patient populations. We hypothesize that a share of the missing 19.74% patients would have fallen into 1 of the 3 high-risk patient groups had more comprehensive data been available, and thus, been eligible for detection.

    We present a novel application of machine learning to address a question of significant clinical relevance. We demonstrated the ability to predict the need of advanced care for depression across various patient populations with considerable predictive performance. These efforts can easily be integrated into existing hospital workflows [42]. As wraparound services are not delivered by primary care providers [50], the ability to identify and refer patients in need of such services is extremely important [51]. Our efforts yield a highly accurate, automated approach for identifying patients in need of wraparound services for mental health, which is of growing importance to health care organizations and incentivized by changing reimbursement policies. By predicting the need for advanced care across various high-risk populations, we offer potential implementers the option of selecting the best screening approach that meets their needs. Our approach is also well suited to leverage increasing health information technology adoption and interoperability of health care datasets for community-wide health transformations [52,53]. In the field of population health informatics, it enables organizations to leverage widespread acceptance and use of machine learning for cross organizational collaboration and management of various datasets [53] while giving implementers the freedom to select methods best suited for each hospital system. Furthermore, such applications of predictive modeling could support organization-level population health initiatives as risk stratification is fundamental to identifying those patients who are most in need of services to improve health and well-being. In addition, implementing such solutions at primary care ensure that facilitating the entry of all patients into the health care system is more efficient than stand-alone implementations at each chronic care clinic. Thus, our approach presents the ability to effectively identify need of advanced care for depression without risk of overdiagnosis and overtreatment and without the use of manual screening mechanisms.

    There is limited knowledge on the best approach to integrate machine learning approaches into existing clinical workflows. As highlighted above, primary care facilities are the point of entry by which a majority of patients suffering from depression seek care [54,55]. However, application of machine learning solutions to screen every primary care visit may be cost-intensive and inefficient for certain clinical practices. Thus, alternate models to evaluate a subset of high-risk patients in need of advanced care for depression would be useful. The 2 potential high-risk patient populations are (1) patients already diagnosed with depression and (2) patients with chronic conditions, who are thus, are at higher risk of suffering from depression [56]. Models built using these subsets may be more practical and result in better machine learning performance than models built using all primary care patients because of variability of underlying data and higher prevalence of outcome of interest, which enables better model training.

    We identified several limitations in our study approach. We adopted a binary (present/absent) flag for each feature used to train decision models. We hypothesize that switching to tabulated counts for each feature will increase the granularity of the feature vector, thereby increasing model performance. The patient group used in our study was obtained from the Eskenazi Health system, a safety-net population with significant health burdens. Thus, our models may not generalize to other commercially insured or broader populations. Our diagnostic data were limited to ICD codes. Integrating medications, laboratory, and clinical procedure data may further improve decision model performance. Furthermore, studies present that social determinants of health such as low-educational attainment, poverty, unemployment, and social isolation may have a significant impact on depression and the need for treatment [57,58]. We propose to expand our models using social determinants of health to assess their impact on decision model performance.

    We acknowledge that incompleteness of EMR data [59] may impact model performance. Use of claims data may have enabled us to identify a greater number of patients in need of specified treatment into each of our patient subgroups [60]. Furthermore, our outcome of interest are patients in need of advanced care, as identified by primary care providers. Thus, we were unable to account for patients who received advanced care for depression without a past referral. Such patients could have been identified from claims data and used to augment our gold standard.

    We selected the random forest classification algorithm for decision-model building based on the need to develop high performance models that were easily interpretable to our clinical audience [42,43]. Other, more advanced decision-modeling approaches such as neural networks [61] have shown potential to improve machine learning performance across various health care challenges. However, neural networks are more complex, cost-intensive, and difficult to interpret [62], making it harder to gain provider acceptance of such models. In addition, it is unclear if they can contribute to our study given the significant performance measures already achieved using random forest models. We recommend that neural networks be considered in a scenario where the sequence or temporality of clinical events is being evaluated, or where the performance of random forest models is unsatisfactory.

    In conclusion, these results present considerable potential to enable preventative care and can be potentially integrated into existing clinical workflows to improve access to wraparound health care services.

    Conclusions

    Our efforts demonstrate the ability to identify patients in need of advanced care for depression across (1) an overall patient population and (2) various groups of high-risk patients using a wide range of acute and chronic conditions, patient demographics, behaviors, and past visit history. Although all models yielded significant performance accuracy, models focused on high-risk patient populations yielded comparatively better results. Furthermore, our methods present a replicable approach for implementers to adopt based on their own needs and priorities. However, decision model performance may differ based on the availability of patient data at each health care system. These results show considerable potential to enable preventative care and can be easily integrated into existing clinical workflows to improve access to wraparound health care services.

    Acknowledgments

    This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

    Authors' Contributions

    The primary author proposed and completed the study as part of his doctoral dissertation. The coauthors, who are all members of his dissertation committee, contributed significantly to the acquisition, analysis, and interpretation of data, rigor of the methodology and analysis, as well as contributing to write the manuscript.

    Conflicts of Interest

    None declared.

    ‎

    Multimedia Appendix 1

    Types of patient data used in decision-model building.

    DOCX File, 13KB
    ‎

    Multimedia Appendix 2

    Precision-recall curve for each decision model under study.

    DOCX File, 1MB
    ‎

    Multimedia Appendix 3

    List of 20 top features (ranked in order of best to worst) for each decision model, together with their least absolute shrinkage and selection operator scores.

    DOCX File, 16KB
    ‎

    Multimedia Appendix 4

    Co-occurrence of top 20 features across each of the patient populations under test (1=most important, 20=least important).

    DOCX File, 19KB
    ‎

    Multimedia Appendix 5

    F1 scores for each decision model trained using iteratively increasing feature subset sizes. For clarity, our plot includes only models trained using the top 50 features for each patient subgroup.

    PNG File, 102KB
    1. World Health Organization. 2012. Depression: A Global Crisis. World Mental Health Day, October 10 2012   URL: https://www.who.int/mental_health/management/depression/wfmh_paper_depression_wmhd_2012.pdf
    2. American Psychiatric Association. 2016. What is Depression?   URL: https://www.psychiatry.org/patients-families/depression/what-is-depression [accessed 2018-01-01]
    3. Lépine JP, Briley M. The increasing burden of depression. Neuropsychiatr Dis Treat 2011;7(Suppl 1):3-7 [FREE Full text] [CrossRef] [Medline]
    4. Goodell S, Druss BG, Walker ER. Mental disorders and medical comorbidity. Synth Proj Res Synth Rep 2011 Feb(21):1-26 [FREE Full text] [Medline]
    5. Yohannes A, Willgoss T, Baldwin R, Connolly M. Depression and anxiety in chronic heart failure and chronic obstructive pulmonary disease: prevalence, relevance, clinical implications and management principles. Int J Geriatr Psychiatry 2010 Dec;25(12):1209-1221. [CrossRef] [Medline]
    6. Nouwen A, Winkley K, Twisk J, Lloyd C, Peyrot M, Ismail K, European Depression in Diabetes (EDID) Research Consortium. Type 2 diabetes mellitus as a risk factor for the onset of depression: a systematic review and meta-analysis. Diabetologia 2010 Dec;53(12):2480-2486 [FREE Full text] [CrossRef] [Medline]
    7. World Health Organization. 2016. Premature Death Among People With Severe Mental Disorders   URL: http://www.who.int/mental_health/management/info_sheet.pdf [accessed 2018-03-01]
    8. Hawton K, Casañas IC, Haw C, Saunders K. Risk factors for suicide in individuals with depression: a systematic review. J Affect Disord 2013 May;147(1-3):17-28. [CrossRef] [Medline]
    9. Mathers C, Fat DM, Boerma JT. World Health Organization. 2008. The Global Burden of Disease: 2004 Update   URL: https://www.who.int/healthinfo/global_burden_disease/2004_report_update/en/
    10. Greenberg PE, Fournier AA, Sisitsky T, Pike CT, Kessler RC. The economic burden of adults with major depressive disorder in the United States (2005 and 2010). J Clin Psychiatry 2015 Feb;76(2):155-162 [FREE Full text] [CrossRef] [Medline]
    11. Wagner HR, Burns B, Broadhead W, Yarnall K, Sigmon A, Gaynes B. Minor depression in family practice: functional morbidity, co-morbidity, service utilization and outcomes. Psychol Med 2000 Nov;30(6):1377-1390. [CrossRef] [Medline]
    12. Wells KB, Sherbourne C, Schoenbaum M, Duan N, Meredith L, Unützer J, et al. Impact of disseminating quality improvement programs for depression in managed primary care: a randomized controlled trial. J Am Med Assoc 2000 Jan 12;283(2):212-220. [CrossRef] [Medline]
    13. Williams Jr JW, Barrett J, Oxman T, Frank E, Katon W, Sullivan M, et al. Treatment of dysthymia and minor depression in primary care: a randomized controlled trial in older adults. J Am Med Assoc 2000 Sep 27;284(12):1519-1526. [CrossRef] [Medline]
    14. Saloheimo HP, Markowitz J, Saloheimo TH, Laitinen JJ, Sundell J, Huttunen MO, et al. Psychotherapy effectiveness for major depression: a randomized trial in a Finnish community. BMC Psychiatry 2016 May 6;16:131 [FREE Full text] [CrossRef] [Medline]
    15. Cuijpers P, van Straten A, Andersson G, van Oppen P. Psychotherapy for depression in adults: a meta-analysis of comparative outcome studies. J Consult Clin Psychol 2008 Dec;76(6):909-922. [CrossRef] [Medline]
    16. Sharp LK, Lipsky MS. Screening for depression across the lifespan: a review of measures for use in primary care settings. Am Fam Physician 2002 Sep 15;66(6):1001-1008 [FREE Full text] [Medline]
    17. US Preventive Services Task Force. Screening for depression in adults: US preventive services task force recommendation statement. Ann Intern Med 2009 Dec 1;151(11):784-792. [CrossRef] [Medline]
    18. Corrigan P. How stigma interferes with mental health care. Am Psychol 2004 Oct;59(7):614-625. [CrossRef] [Medline]
    19. Beck AT, Steer RA, Brown GK. BDI-II, Beck Depression Inventory: Manual. Second Edition. San Antonio, Texas: Psychological Corporation; 1996.
    20. Gilbody S, Richards D, Brealey S, Hewitt C. Screening for depression in medical settings with the patient health questionnaire (PHQ): a diagnostic meta-analysis. J Gen Intern Med 2007 Nov;22(11):1596-1602 [FREE Full text] [CrossRef] [Medline]
    21. Kroenke K, Spitzer RL, Williams JB. The PHQ-15: validity of a new measure for evaluating the severity of somatic symptoms. Psychosom Med 2002;64(2):258-266. [CrossRef] [Medline]
    22. Alexopoulos GS, Abrams RC, Young RC, Shamoian CA. Cornell scale for depression in dementia. Biol Psychiatry 1988 Feb 1;23(3):271-284. [CrossRef] [Medline]
    23. Williams JB. Standardizing the Hamilton depression rating scale: past, present, and future. Eur Arch Psychiatry Clin Neurosci 2001;251(Suppl 2):II6-I12. [CrossRef] [Medline]
    24. Valenstein M, Vijan S, Zeber JE, Boehm K, Buttar A. The cost-utility of screening for depression in primary care. Ann Intern Med 2001 Mar 6;134(5):345-360. [CrossRef] [Medline]
    25. Kerr LK, Kerr Jr LD. Screening tools for depression in primary care: the effects of culture, gender, and somatic symptoms on the detection of depression. West J Med 2001 Nov;175(5):349-352 [FREE Full text] [CrossRef] [Medline]
    26. Mojtabai R, Olfson M. Proportion of antidepressants prescribed without a psychiatric diagnosis is growing. Health Aff (Millwood) 2011 Aug;30(8):1434-1442. [CrossRef] [Medline]
    27. Mojtabai R. Clinician-identified depression in community settings: concordance with structured-interview diagnoses. Psychother Psychosom 2013;82(3):161-169. [CrossRef] [Medline]
    28. Dowrick C, Frances A. Medicalising unhappiness: new classification of depression risks more patients being put on drug treatment from which they will not benefit. Br Med J 2013 Dec 9;347:f7140. [CrossRef] [Medline]
    29. Thombs BD, Ziegelstein RC, Roseman M, Kloda LA, Ioannidis JP. There are no randomized controlled trials that support the United States Preventive Services Task Force Guideline on screening for depression in primary care: a systematic review. BMC Med 2014 Jan 28;12:13 [FREE Full text] [CrossRef] [Medline]
    30. Thombs BD, Coyne JC, Cuijpers P, de Jonge P, Gilbody S, Ioannidis JP, et al. Rethinking recommendations for screening for depression in primary care. CMAJ 2012 Mar 6;184(4):413-418 [FREE Full text] [CrossRef] [Medline]
    31. Gilbody S, Sheldon T, Wessely S. Should we screen for depression? Br Med J 2006 Apr 29;332(7548):1027-1030 [FREE Full text] [CrossRef] [Medline]
    32. Charlson ME, Charlson RE, Peterson JC, Marinopoulos SS, Briggs WM, Hollenberg JP. The Charlson comorbidity index is adapted to predict costs of chronic disease in primary care patients. J Clin Epidemiol 2008 Dec;61(12):1234-1240. [CrossRef] [Medline]
    33. DeJean D, Giacomini M, Vanstone M, Brundisini F. Patient experiences of depression and anxiety with chronic disease: a systematic review and qualitative meta-synthesis. Ont Health Technol Assess Ser 2013;13(16):1-33 [FREE Full text] [Medline]
    34. Kasthurirathne SN. IUPUI Scholar Works Repository. 2018. The Use of Clinical, Behavioral, and Social Determinants of Health to Improve Identification of Patients in Need of Advanced Care for Depression   URL: https:/​/scholarworks.​iupui.edu/​bitstream/​handle/​1805/​17765/​Kasthurirathne_iupui_0104D_10328.​pdf?sequence=1&isAllowed=y
    35. National Library of Medicine - National Institutes of Health. 2016. Unified Medical Language System® (UMLS®): Metathesaurus   URL: https://www.nlm.nih.gov/research/umls/knowledge_sources/metathesaurus/ [accessed 2018-02-01]
    36. Aronson AR, Lang FM. An overview of MetaMap: historical perspective and recent advances. J Am Med Inform Assoc 2010;17(3):229-236 [FREE Full text] [CrossRef] [Medline]
    37. National Library of Medicine - National Institutes of Health. 2015. Fact Sheet: Unified Medical Language System   URL: https://www.nlm.nih.gov/pubs/factsheets/umls.html [accessed 2018-01-01]
    38. McDonald CJ, Overhage JM, Barnes M, Schadow G, Blevins L, Dexter PR, INPC Management Committee. The Indiana network for patient care: a working local health information infrastructure. An example of a working infrastructure collaboration that links data from five health systems and hundreds of millions of entries. Health Aff (Millwood) 2005;24(5):1214-1220. [CrossRef] [Medline]
    39. Overhage JM. The Indiana health information exchange. In: Dixon BE, editor. Health Information Exchange: Navigating and Managing a Network of Health Information Systems. Waltham, MA: Academic Press; 2016:267-279.
    40. Breiman L. Random forests. Mach Learn 2001;45(1):5-32. [CrossRef]
    41. Kasthurirathne SN, Dixon BE, Gichoya J, Xu H, Xia Y, Mamlin B, et al. Toward better public health reporting using existing off the shelf approaches: a comparison of alternative cancer detection approaches using plaintext medical data and non-dictionary based feature selection. J Biomed Inform 2016 Apr;60:145-152 [FREE Full text] [CrossRef] [Medline]
    42. Kasthurirathne SN, Vest JR, Menachemi N, Halverson PK, Grannis SJ. Assessing the capacity of social determinants of health data to augment predictive models identifying patients in need of wraparound social services. J Am Med Inform Assoc 2018 Dec 1;25(1):47-53. [CrossRef] [Medline]
    43. Qi Y. Random forest for bioinformatics. In: Zhang C, Ma Y, editors. Ensemble Machine Learning: Methods and Applications. New York: Springer; 2012:307-323.
    44. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-Learn: Machine Learning in Python. J Mach Learn Res 2011;12:2825-2830 [FREE Full text]
    45. Youden WJ. Index for rating diagnostic tests. Cancer 1950 Jan;3(1):32-35. [CrossRef] [Medline]
    46. Polani D. Kullback-leibler divergence. In: Dubitzky W, Wolkenhauer O, Cho KH, Yokota H, editors. Encyclopedia of Systems Biology. New York: Springer; 2013:1087-1088.
    47. Khalilia M, Chakraborty S, Popescu M. Predicting disease risks from highly imbalanced data using random forest. BMC Med Inform Decis Mak 2011 Jul 29;11:51 [FREE Full text] [CrossRef] [Medline]
    48. Zong W, Huang GB, Chen YJ. Weighted extreme learning machine for imbalance learning. Neurocomputing 2013 Feb;101:229-242. [CrossRef]
    49. Lemaître G, Nogueira F, Aridas CK. Imbalanced-learn: a python toolbox to tackle the curse of imbalanced datasets in machine learning. J Mach Learn Res 2017;18(1):559-563 [FREE Full text]
    50. Loeb DF, Binswanger IA, Candrian C, Bayliss EA. Primary care physician insights into a typology of the complex patient in primary care. Ann Fam Med 2015 Sep;13(5):451-455 [FREE Full text] [CrossRef] [Medline]
    51. Jackson GL, Powers BJ, Chatterjee R, Bettger JP, Kemper AR, Hasselblad V, et al. The patient centered medical home. A systematic review. Ann Intern Med 2013 Dec 5;158(3):169-178. [CrossRef] [Medline]
    52. Kharrazi H, Weiner JP. IT-enabled community health interventions: challenges, opportunities, and future directions. EGEMS (Wash DC) 2014;2(3):1117 [FREE Full text] [CrossRef] [Medline]
    53. Kharrazi H, Lasser EC, Yasnoff WA, Loonsk J, Advani A, Lehmann HP, et al. A proposed national research and development agenda for population health informatics: summary recommendations from a national expert workshop. J Am Med Inform Assoc 2017 Dec;24(1):2-12 [FREE Full text] [CrossRef] [Medline]
    54. Cape J, Whittington C, Buszewicz M, Wallace P, Underwood L. Brief psychological therapies for anxiety and depression in primary care: meta-analysis and meta-regression. BMC Med 2010 Jun 25;8:38 [FREE Full text] [CrossRef] [Medline]
    55. Gilbody S, Littlewood E, Hewitt C, Brierley G, Tharmanathan P, Araya R, et al. Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial): large scale pragmatic randomised controlled trial. Br Med J 2015;351:h5627 [FREE Full text] [CrossRef] [Medline]
    56. Katon WJ. Epidemiology and treatment of depression in patients with chronic medical illness. Dialogues Clin Neurosci 2011;13(1):7-23 [FREE Full text] [Medline]
    57. World Health Organization. 2014. Social Determinants of Mental Health   URL: https:/​/www.​who.int/​mental_health/​publications/​gulbenkian_paper_social_determinants_of_mental_health/​en/​
    58. Fryers T, Melzer D, Jenkins R, Brugha T. The distribution of the common mental disorders: social inequalities in Europe. Clin Pract Epidemiol Ment Health 2005 Sep 5;1:14 [FREE Full text] [CrossRef] [Medline]
    59. Kharrazi H, Chi W, Chang HY, Richards TM, Gallagher JM, Knudson SM, et al. Comparing population-based risk-stratification model performance using demographic, diagnosis and medication data extracted from outpatient electronic health records versus administrative claims. Med Care 2017 Dec;55(8):789-796. [CrossRef] [Medline]
    60. Kharrazi H, Anzaldi LJ, Hernandez L, Davison A, Boyd CM, Leff B, et al. The value of unstructured electronic health record data in geriatric syndrome case identification. J Am Geriatr Soc 2018 Aug;66(8):1499-1507. [CrossRef] [Medline]
    61. Haykin S. Neural Networks: A Comprehensive Foundation. New York: Pearson; 2004.
    62. Bengio Y, Courville A, Vincent P. Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell 2013 Aug;35(8):1798-1828. [CrossRef] [Medline]

    ‎
    AUC: area under the curve
    ICD: International Classification of Diseases
    INPC: Indiana Network for Patient Care
    PHQ: Patient Health Questionnaire
    UMLS: Unified Medical Language System

    Edited by G Eysenbach; submitted 25.02.19; peer-reviewed by A Flynn, H Kharrazi; comments to author 11.04.19; revised version received 01.06.19; accepted 12.06.19; published 22.07.19

    Copyright

    ©Suranga N Kasthurirathne, Paul G Biondich, Shaun J Grannis, Saptarshi Purkayastha, Joshua R Vest, Josette F Jones. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 22.07.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.