This paper is in the following e-collection/theme issue:

Cost-Effectiveness and Economics (252)

Published on in Vol 24, No 6 (2022): June

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/29420, first published .
Impactability Modeling for Reducing Medicare Accountable Care Organization Payments and Hospital Events in High-Need High-Cost Patients: Longitudinal Cohort Study

Impactability Modeling for Reducing Medicare Accountable Care Organization Payments and Hospital Events in High-Need High-Cost Patients: Longitudinal Cohort Study

Impactability Modeling for Reducing Medicare Accountable Care Organization Payments and Hospital Events in High-Need High-Cost Patients: Longitudinal Cohort Study

Authors of this article:

Maureen A Smith1, 2 Author Orcid Image ;   Menggang Yu1 Author Orcid Image ;   Jared D Huling3 Author Orcid Image ;   Xinyi Wang1 Author Orcid Image ;   Allie DeLonay1 Author Orcid Image ;   Jonathan Jaffery4, 5 Author Orcid Image
Article Authors Cited by Tweetations (3) Metrics

    Original Paper

    1Department of Population Health Sciences, University of Wisconsin–Madison, Madison, WI, United States

    2Department of Family Medicine and Community Health, University of Wisconsin–Madison, Madison, WI, United States

    3Division of Biostatistics, University of Minnesota, Minneapolis, MN, United States

    4University of Wisconsin Health Office of Population Health, Madison, WI, United States

    5Department of Medicine, University of Wisconsin–Madison, Madison, WI, United States

    Corresponding Author:

    Maureen A Smith, MD, MPH, PhD

    Department of Population Health Sciences

    University of Wisconsin–Madison

    800 University Bay Dr

    Suite 210

    Madison, WI, 53794

    United States

    Phone: 1 6082624802

    Email: maureensmith@wisc.edu


    Background: Impactability modeling promises to help solve the nationwide crisis in caring for high-need high-cost patients by matching specific case management programs with patients using a “benefit” or “impactability” score, but there are limitations in tailoring each model to a specific program and population.

    Objective: We evaluated the impact on Medicare accountable care organization savings from developing a benefit score for patients enrolled in a historic case management program, prospectively implementing the score, and evaluating the results in a new case management program.

    Methods: We conducted a longitudinal cohort study of 76,140 patients in a Medicare accountable care organization with multiple before-and-after measures of the outcome, using linked electronic health records and Medicare claims data from 2012 to 2019. There were 489 patients in the historic case management program, with 1550 matched comparison patients, and 830 patients in the new program, with 2368 matched comparison patients. The historic program targeted high-risk patients and assigned a centrally located registered nurse and social worker to each patient. The new program targeted high- and moderate-risk patients and assigned a nurse physically located in a primary care clinic. Our primary outcomes were any unplanned hospital events (admissions, observation stays, and emergency department visits), count of event-days, and Medicare payments.

    Results: In the historic program, as expected, high-benefit patients enrolled in case management had fewer events, fewer event-days, and an average US $1.15 million reduction in Medicare payments per 100 patients over the subsequent year when compared with the findings in matched comparison patients. For the new program, high-benefit high-risk patients enrolled in case management had fewer events, while high-benefit moderate-risk patients enrolled in case management did not differ from matched comparison patients.

    Conclusions: Although there was evidence that a benefit score could be extended to a new case management program for similar (ie, high-risk) patients, there was no evidence that it could be extended to a moderate-risk population. Extending a score to a new program and population should include evaluation of program outcomes within key subgroups. With increased attention on value-based care, policy makers and measure developers should consider ways to incorporate impactability modeling into program design and evaluation.

    J Med Internet Res 2022;24(6):e29420

    doi:10.2196/29420

    Keywords

    case management; high-risk patients; benefit score 


    With a national imperative to reduce costs and improve care for high-need high-cost patients [1-3], most accountable care organizations (ACOs) [4] are implementing outpatient case management programs with a nurse or social worker coordinating care. However, there is little evidence of cost savings from these resource-intensive programs [5-7], and they vary widely in design and implementation [5,8]. This widespread implementation of unproven programs has concerned policy makers [9,10] and led to recommendations to design more effective programs [11], to improve the identification of potentially high-cost patients using predictive models [12], and even to abandon care coordination as a cost-saving strategy [13]. For example, case managers often identify patients for enrollment in a case management program using a predictive risk score to find patients at high risk of poor outcomes, such as hospital admission [14].

    Rather than attempting to identify effective case management programs and standardize implementation across health systems, a fundamentally different strategy would identify patients who benefit from specific case management programs as they are implemented in practice and match patients to the most beneficial program [5]. Described by Lewis et al as “impactability modeling” [15], this pragmatic approach predicts who is likely to benefit from a particular intervention with respect to an outcome and not who is likely to have a poor outcome. Different from risk scores, these “impactability” or “benefit” scores identify patients who are likely to benefit from enrollment into case management with respect to a specific outcome (eg, preventing hospital admissions). In this way, a benefit score could allow further partitioning of a high-risk population of patients into those who are and those who are not likely to benefit from case management. For example, a high-risk patient may or may not be likely to avoid hospitalizations if enrolled into case management. This additional stratification of a high-risk population using a benefit score may help an ACO target patients for enrollment into case management. Early analyses on impactability in the Medicare population [16] (labeled “benefit score”) and in the Medicaid population [17] (labeled “impactability score”) successfully developed scores to identify individuals who were more likely to benefit from certain case management programs and suggested significant savings [16]. However, neither score was evaluated to determine if it could extend beyond the case management program and population on which it was developed.

    The promise of impactability modeling is based on substantial evidence that patients may be more or less likely to benefit from a specific intervention depending on their personal and clinical characteristics [5], although this tailoring of program enrollment may come at a cost. Because impactability models are intrinsically linked to the specific program and population used in their development and because there is wide variation in case management program design and implementation [5,8], it is unclear whether these scores could extend to new programs or populations. To address this question, we evaluated the impact on Medicare ACO savings from a case management “benefit score” developed using a historic case management program enrolling high-risk patients (published elsewhere [16]), and compared the results to prospectively implementing the score in a new case management program enrolling both high- and moderate-risk populations. Our work extends analyses conducted under a Patient-Centered Outcomes Research Institute (PCORI)-funded health systems demonstration grant (HSD-1603-35039; “Variation in case management programs and their effectiveness in managing high-risk patients for Medicare ACOs” [18]).


    Study Design and Setting

    We used a longitudinal cohort study design with multiple before-and-after measures of the outcome for each case management patient and matched comparison patient [19-21]. Linked electronic health record (EHR) and Medicare enrollment and claims data were extracted from January 1, 2012, through April 30, 2019, to characterize patients during a 1-year baseline period and up to a 1-year follow-up period, as well as census data from the 2007-2011 American Community Survey. The setting was UW Health, a large health system in Wisconsin with 30 statewide academic and community-based primary care clinics and 279 primary care providers. UW Health began participating in the Medicare ACO program in 2013 and, as part of its commitment to become a learning health system [22], began developing scores to support targeted enrollment of patients into population management programs and regularly evaluating program outcomes.

    Ethical Considerations

    This project was deemed exempt from institutional review board oversight at the University of Wisconsin–Madison as it constitutes quality improvement or program evaluation [23]. Institutional review board review was not required because, in accordance with federal regulations, the project does not constitute research.

    Case Management Patients

    For ease of comparison, we described patients from both the historic [16] and new case management programs. We included patients aged 18 years or older enrolled for at least 1 month with (1) continuous EHR and claims data available for at least 1 year prior to enrollment in case management; (2) assignment to the ACO during baseline and follow-up periods; and (3) at least 1 month of continuous EHR and claims data during the follow-up period. Patients were excluded if they were enrolled in hospice, were on dialysis, or had end-stage renal disease during baseline. Patients were recruited for the historic case management program from June 2013 to December 2018. For the new program, we also excluded any patients previously enrolled in the historic program. Patients were recruited for the new case management program from April 2018 through March 2019. The final sample size of patients in the historic program was 489 (Figure 1) and in the new program was 830 (Figure 2).

    ‎
    Figure 1. Final analysis sample for the historic case management program.
    View this figure
    ‎
    Figure 2. Final analysis sample for the new case management program.
    View this figure

    Matched Comparison Patients

    We identified all possible comparison patients receiving usual care from the Medicare ACO, who had not been enrolled in the program but who had comparable patient characteristics, had data available, and met the inclusion and exclusion criteria. For each possible comparison patient, we constructed multiple baseline 1-year time periods (63,047 potential comparison patients; 732,799 potential comparison patient-episodes). We matched each case to a maximum of four of the closest eligible comparison patient-episodes. The date at which a possible comparison patient-episode had the closest match to a case with respect to baseline characteristics was the “match date” and was treated identically to the case’s “enrollment date.” The final sample size of matched comparison patient-episodes was 1550 for the historic program (Figure 1) and 2368 for the new program (Figure 2).

    Potential Input Variables

    Our strategy leveraged the high-dimensional nature of combined EHR and claims data [24]. For the baseline year for each patient, we constructed 18,406 possible input variables that encompassed sociodemographics (eg, demographics and homelessness), chronic conditions (eg, diagnoses), utilization (eg, procedures and hospitalizations), vital signs (eg, blood pressure), behaviors (eg, tobacco), laboratory values, payments, medications, patient engagement (eg, “no show” appointments), and other information (eg, advance directives). For missing information for continuous variables, we used simple mean imputation within each decile of a hierarchical condition category (HCC) score [25], and for categorical variables, a missing category was created [26]. Continuous variables were transformed into indicators representing “high” and “low” values using the median from the cases. For our core set of descriptive characteristics presented in tables, we used claims data unless otherwise specified. Baseline sociodemographic variables included age (continuous), sex (female/male), race/ethnicity (White/non-White or Hispanic), Medicaid (yes/no), disability entitlement (yes/no), residence (urban, suburban, large town, and small town/rural; categorizing ZIP code from claims) [27], and mean percentage with a high school degree in the 2007-2011 census tract (after geocoding the address from the EHR). Other variables included the HCC score (continuous) [25] and a risk score based on both claims and EHR data that predicted the risk of hospital admission or death within the next 6 months (categorized as “high” risk if the risk was >13%; otherwise, designated as “moderate”) [28]. Chronic conditions included 17 medical conditions defined by Elixhauser et al using International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnostic codes, along with an indicator variable for ≥3 of these conditions [29]. Utilization included counts of emergency department (ED) visits, unplanned hospitalizations and hospital days [30], observation stays and observation days, and total Medicare payments. ED visits that resulted in hospitalization were not counted as ED visits but were counted as part of the hospitalization.

    Exact and Propensity Score Matching

    To conduct matching, we constructed a high-dimensional propensity score for case management enrollment by adapting the approach from Schneeweiss et al [24]. This included (1) requiring the variables to have a prevalence between 5% and 95% among the cases and a maximum correlation of 0.8 for each covariate (14,909 variables remained); (2) prioritizing covariates using a measure of confounding bias (threshold=95% significance level; 1905 variables remained); (3) selecting covariates using logistic regression with a lasso penalty, with tuning parameters selected using a variant of the traditional stepwise selection, where the final model was chosen on the basis of the best Schwarz Bayesian criterion (37 variables remained) [31]; (4) estimating the propensity score using logistic regression and the 37 predictors, including chronic conditions, HCC scores, procedures, medication counts, telephone encounter counts, etc, for each patient-episode; (5) selecting up to four of the closest eligible comparison patient-episodes using 5 rounds of exact matching (19 exact match variables) and within exact match strata; and (6) selecting final matches using global optimal propensity score matching to minimize the overall distance between propensity scores, using a matrix of distances between all cases and potential matches [32,33]. The quality of our matching process was determined by examining standardized mean differences, which describe a variance-normalized difference in the means of confounders of the control group and the group enrolled in case management. Standardized mean differences with values around 20%-25% were considered moderately imbalanced, but with a range that was amenable to further adjustment through regression [33,34]. Of 1905 baseline variables, 4 had standardized differences between cases and comparison patients above 25%, including the count of unique prescription medication, nonthrombotic nonathlerosclerotic vascular disease or hypertensive heart disease, and professional service payment, and were included in regressions to adjust for residual confounding [33].

    Outcome Measures

    Our outcome measures were (1) any unplanned hospital events (admissions, observation stays, and ED visits) during a month, (2) the count of days during the month with any unplanned hospital events, and (3) total Medicare payments during a month, excluding payments for planned hospitalizations and pharmacy payments. We created a data set with 1 observation per patient-episode per month. The first month was 12 months prior to the enrollment/match date and continued for 1 to 12 months after the enrollment/match date until death or censoring due to lack of data.

    Benefit Score

    The benefit score [16] differs from a typical risk score in that it predicts the effectiveness or “benefit” of a treatment with respect to the outcome using patient and clinical characteristics (eg, the effectiveness of case management with respect to reducing payments), rather than predicting the outcome directly (eg, payments). This modeling approach was developed under a PCORI methodology grant (ME-1409-21219; “Developing new methods for determining treatment benefits based on individual patient traits” [35,36]). The benefit score represents the estimated reduction in Medicare payments within 1 year if the patient is enrolled in case management [16]. Important variables that determined the benefit from case management included chronic conditions (liver disease, dementia, cardiac dysrhythmias, psychiatric disease, and back disease), count of medication, count of appointment “no-shows,” and use of the electronic medical record patient portal. Patients with negative savings have “no benefit” from case management, and those with positive savings have “benefit.” To provide a qualitative summary of the benefit score, we divided the score into quintiles above 0 (1 to 5) and below zero (−5 to −1). As values close to 0 were ambiguous, scores from 2 to 5 were designated “high benefit” and scores from −5 to 1 were designated “no/low benefit.”

    Statistical Analysis

    To estimate the effect of our intervention, our statistical analysis used longitudinal regression modeling of the risk-adjusted difference in outcome trajectories between the cases and comparison patients, using patient-month data. We used an intent-to-treat approach in which individuals who disenrolled from the program were treated as enrolled. We controlled for confounding using exact and propensity score matching (see “Exact and Propensity Score Matching”). After matching, our regression modeling accounted for residual confounding using inverse weighting by the propensity score and for differences in the number of matched comparison patients for each case (ranging from 1-4) by weighting using the inverse of the number of matches. We used the following link functions: logit/binomial (any events), log/zero inflated Poisson (count of event-days), and log/zero inflated gamma (payments). Models included terms for the preintervention trend, change in level, and postintervention trend in monthly events for both cases and comparison patients, and were risk-adjusted for 4 indicator variables with standardized differences above 25% (see above). We stratified our regression analyses of the new case management program by high versus moderate risk, and the final model included benefit category as an interaction term. Treatment of missing data is described in the section on input variables. Results were transformed into predicted outcomes (ie, dollar amount of Medicare payment reduction, number of event-months prevented, or number of event-days prevented) for 100 patients enrolled in case management programs for 1 year, who were similar to those included in our analyses [37]. Because the benefit score was developed on 69% of the cases in the historic program (339 patients enrolled prior to December 1, 2016) [16], intervention effect estimates for the historic program may be biased. We debiased the intervention effect estimation for the historic program using a Harrell bootstrap bias-correction procedure [38], but found no difference after correction and thus presented uncorrected estimates; this procedure is not needed for the new program. We calculated bootstrapped 95% CIs using 400 replications for all outcome models.


    Characterizing Case Management Programs

    The historic case management program used a team approach with a centrally located registered nurse and social worker assigned to each patient and enrolled mostly high-risk patients (Table 1). At program initiation in 2013, patients were identified for further screening using a risk score [28] (calculated monthly) that represented risk of hospital admission or death within 6 months (with “high risk” defined as >13%) or through referral by their primary care provider. After initial identification, patients were screened by nurses or social workers using an assessment tool [39]. Beginning in 2017, the benefit score was also used to identify patients for further screening (with high benefit defined as greater than US $1200 estimated reduction in Medicare payments) [16,40].

    The new program relied on nurses physically located in each primary care clinic. Social workers were available only through referral and, in practice, consulted infrequently. The program enrolled both high- and moderate-risk patients. At program initiation, the health system decided to identify 80% of patients through the monthly benefit and risk scoring process developed for the historic program and 20% through the primary care provider referral process [22].

    Table 1. Case management program characteristics.
    CharacteristicHistoric programNew program
    Established (year)20132018
    Enrolled patients, n5503600
    Case manager


    		ProfessionRNa+SWb dyadRN

    		TrainingCCMCcIn-house

    		Number1540
    Case finding process


    		Risk scoreYesYes

    		Referrals from providersYesYes (20% of cases)

    		Benefit scoreBeginning 2017Yes (80% of cases)
    Intake process


    		Comprehensive assessmentWithin 30 daysWithin 30 days

    		Collaborative goal-settingWithin 60 daysWithin 60 days
    Intervention intensity


    		Contacts (#)3 per month1 per month

    		Duration, mean160 days200 days

    		Caseload20-25 primary; 40-50 secondary75 primary
    Care integration


    		Physician collaborationYesYes

    		Mode of contactTelephone and in-personTelephone and in-person

    		Program locationCentralPrimary care clinic
    Quality assurance


    		Medical director attends weekly case consultsYesNo

    		Medical director chart reviewYesYes

    aRN: registered nurse.

    bSW: social worker.

    cCCMC: Commission for Case Manager Certification.

    Characterizing Case Management and Comparison Patients

    After matching and propensity score weighting, case management and comparison patients were similar with respect to a predetermined set of baseline sociodemographic, chronic condition [29], behavioral, and utilization variables, although cases had slightly more anxiety than comparison patients (Table 2). However, patients in the historic and new programs differed. Patients in the new program were older but less likely to live in an urban area, have Medicaid, or have disability entitlement. They also had less alcohol or drug abuse, less depression but more hypertension and diabetes with complications, lower HCC scores, and less baseline utilization, and were less likely to be high risk. Specifically, 70% of cases in the historic program were high risk compared with 58% of cases in the new program, and the median risk score for cases in the historic program was twice as high as that for cases in the new program (32% vs 16%; data not shown).

    Because of these differences, we stratified cases in the new program by high risk versus moderate risk (Table 3). High-risk patients in the new program had similar or slightly higher HCC scores compared with the scores of high-risk patients in the historic program, but were older, less likely to be on Medicaid, more suburban, and more likely to have 3 or more chronic conditions. Moderate-risk patients in the new program had lower HCC scores compared with the scores of high-risk patients and were less likely to have chronic conditions but were more likely to have anxiety and depression.

    Table 2. Sociodemographics, chronic conditions, and baseline utilization for historic and new case management and matched comparison patients.
    CharacteristicHistoric programaNew programa

    		Cases (N=489)Comparisons (N=1550)Cases (N=830)Comparisons (N=2368)
    Sociodemographics




    		Age (years), mean (SD)67 (20)68 (13)76 (11)76 (12)

    		Female, n (%)320 (65)987 (64)542 (65)1564 (66)

    		Non-White or Hispanic, n (%)66 (13)121 (8)42 (5)128 (5)

    		Medicaid insurance ever, n (%)219 (45)643 (42)184 (22)594 (25)

    		Disability entitlement, n (%)218 (44)611 (39)144 (17)408 (17)
    Rural/urban, n (%)




    		Urban code387 (79)1031 (67)508 (61)1609 (68)

    		Suburban56 (12)284 (18)180 (22)425 (18)

    		Large town35 (7)186 (12)128 (15)295 (12)

    		Small town/rural10 (2)45 (3)14 (2)38 (2)
    Percentage with a high school degree, mean (SD)1 (0)1 (0)1 (0)1 (0)
    HCCb score, mean (SD)3 (3)3 (2)2 (1)2 (1)
    High risk, n (%)344 (70)997 (64)483 (58)1341 (57)
    Chronic conditions, n (%)




    		3 or more chronic conditions420 (86)1322 (85)757 (91)2045 (86)

    		Congestive heart failure203 (42)650 (42)322 (39)862 (36)

    		COPDc/asthma224 (46)709 (46)375 (45)906 (38)

    		Chronic kidney disease220 (45)728 (47)346 (42)964 (41)

    		Alcohol or drug abuse159 (33)328 (21)133 (16)370 (16)

    		Anxiety274 (56)671 (43)393 (47)825 (35)

    		Depression234 (48)699 (45)304 (37)830 (35)

    		Diabetes with complications125 (26)396 (26)188 (23)479 (20)

    		Diabetes without complications40 (8)138 (9)84 (10)246 (10)

    		Hypertension296 (60)1044 (67)676 (81)1642 (69)

    		Liver disease45 (9)112 (7)59 (7)84 (4)

    		Fluid/electrolyte disorders227 (46)681 (44)339 (41)867 (37)

    		Metastatic cancer19 (4)37 (2)38 (5)77 (3)

    		Obesity154 (31)430 (28)262 (32)570 (24)

    		Psychosis143 (29)424 (27)179 (22)436 (18)

    		Peripheral vascular disease115 (23)396 (26)234 (28)667 (28)

    		Renal failure129 (26)470 (30)209 (25)613 (26)

    		Solid tumor without metastasis34 (7)127 (8)87 (10)258 (11)
    Baseline utilization




    		Number of EDd visits, mean (SD)2.2 (4.97)2.09 (2.89)1.07 (1.43)0.96 (1.44)

    		Number of hospitalizations, mean (SD)1.17 (2.46)1.11 (1.20)0.64 (0.93)0.54 (0.84)

    		Number of days in hospital, mean (SD)5.55 (14.98)5.30 (7.07)2.77 (5.11)2.33 (4.57)

    		Number of observation stays, mean (SD)0.25 (0.83)0.25 (0.49)0.17 (0.42)0.15 (0.41)

    		Number of days in observation stay, mean (SD)0.29 (1.08)0.33 (0.74)0.21 (0.57)0.18 (0.53)

    		Medicare paymente, mean (SD)29.43 (54.14)30.57 (32.50)21.05 (25.86)17.37 (22.62)

    aNumbers were adjusted for varying case control ratios.

    bHCC: hierarchical condition category.

    cCOPD: chronic obstructive pulmonary disease.

    dED: emergency department.

    ePer US $1000.

    Table 3. Sociodemographics, chronic conditions, and baseline utilization for new case management and matched comparison patients, by risk.
    CharacteristicNew program, high riskaNew program, moderate riska

    		Cases (N=483)Comparisons (N=1315)Cases (N=347)Comparisons (N=1053)
    Sociodemographics




    		Age (years), mean (SD)79 (11)78 (11)71 (11)75 (12)

    		Female, n (%)308 (64)827 (63)234 (67)727 (69)

    		Non-Hispanic White, n (%)462 (96)1247 (95)326 (94)989 (94)

    		Non-White or Hispanic, n (%)21 (4)68 (5)21 (6)64 (6)

    		Medicaid insurance ever, n (%)127 (26)350 (27)57 (16)241 (23)

    		Disability entitlement, n (%)79 (16)204 (16)65 (19)196 (19)
    Rural/urban, n (%)




    		Urban code280 (58)909 (69)228 (66)699 (66)

    		Suburban111 (23)213 (16)69 (20)211 (20)

    		Large town82 (17)170 (13)46 (13)128 (12)

    		Small town/rural10 (2)22 (2)4 (1)15 (1)
    Percentage with a high school degree, mean (SD)1 (0)1 (0)1 (0)1 (0)
    HCCb score, mean (SD)3 (1)3 (1)2 (1)2 (1)
    High risk, n (%)483 (100)991 (75)0 (0)363 (34)
    Chronic conditions




    		3 or more chronic conditions472 (98)1241 (94)285 (82)810 (77)

    		Congestive heart failure249 (52)651 (50)73 (21)219 (21)

    		COPDc/asthma242 (50)558 (42)133 (38)349 (33)

    		Chronic kidney disease268 (55)665 (51)78 (22)303 (29)

    		Alcohol or drug abuse78 (16)207 (16)55 (16)158 (15)

    		Anxiety213 (44)422 (32)180 (52)394 (37)

    		Depression165 (34)419 (32)139 (40)405 (38)

    		Diabetes with complications134 (28)337 (26)54 (16)146 (14)

    		Diabetes without complications38 (8)129 (10)46 (13)121 (11)

    		Hypertension406 (84)969 (74)270 (78)677 (64)

    		Liver disease34 (7)55 (4)25 (7)27 (3)

    		Fluid/electrolyte disorders277 (57)582 (44)62 (18)289 (27)

    		Metastatic cancer29 (6)60 (5)9 (3)17 (2)

    		Obesity142 (29)329 (25)120 (35)238 (23)

    		Psychosis85 (18)230 (17)94 (27)203 (19)

    		Peripheral vascular disease184 (38)440 (33)50 (14)235 (22)

    		Renal failure166 (34)431 (33)43 (12)188 (18)

    		Solid tumor without metastasis53 (11)160 (12)34 (10)100 (10)
    Baseline utilization




    		Number of EDd visits, mean (SD)1.31 (1.60)1.15 (1.62)0.73 (1.07)0.72 (1.11)

    		Number of hospitalizations, mean (SD)0.93 (1.06)0.82 (0.96)0.24 (0.50)0.22 (0.51)

    		Number of days in hospital, mean (SD)4.18 (6.11)3.65 (5.52)0.80 (1.99)0.81 (2.39)

    		Number of observation stays, mean (SD)0.22 (0.46)0.17 (0.44)0.10 (0.33)0.13 (0.36)

    		Number of days in observation stay, mean (SD)0.28 (0.67)0.21 (0.59)0.10 (0.35)0.15 (0.43)

    		Medicare Paymente, mean (SD)28.43 (28.39)24.27 (26.33)10.78 (17.24)9.37 (13.26)

    aNumbers were adjusted for varying case control ratios.

    bHCC: hierarchical condition category.

    cCOPD: chronic obstructive pulmonary disease.

    dED: emergency department.

    ePer US $1000.

    Characterizing Case Management Patients by Benefit Category

    Approximately one-third of the cases in the historic program were identified as high benefit, while in the new program, 43% of high-risk and 37% of moderate-risk cases were identified as high benefit (Table 4). High-benefit patients in the historic program had higher HCC scores and baseline utilization but were less likely to be high risk and had less disability entitlement when compared with the findings for no/low-benefit patients in the historic program. They were also less likely to have chronic conditions, including chronic obstructive pulmonary disease (COPD)/asthma and anxiety, but more likely to have diabetes with complications. Among high-risk patients in the new program, high-benefit patients were more likely to be female and less likely to have congestive heart failure, chronic kidney disease, alcohol or drug abuse, or valvular disease when compared with the findings for no/low-benefit patients. Among moderate-risk patients in the new program, high-benefit patients were slightly more likely to be female and were less likely to have Medicaid or disability entitlement, COPD/asthma, and alcohol or drug abuse, but more likely to have higher HCC scores and 3 or more chronic conditions, including obesity.

    Table 4. Sociodemographics, chronic conditions, and baseline utilization for historic and new case management patients, by benefit category.
    CharacteristicHistoric program (cases only)New program (cases only), high riskNew program (cases only), moderate risk

    		High benefit (N=170)No/low benefit (N=319)High benefit (N=209)No/low benefit (N=274)High benefit (N=129)No/low benefit (N=218)
    Sociodemographics






    		Age (years), mean (SD)64 (15)68 (14)79 (11)78 (11)73 (9)71 (11)

    		Female, n (%)107 (63)211 (66)147 (70)161 (59)91 (71)143 (66)

    		Non-White or Hispanic, n (%)23 (14)39 (12)10 (5)11 (4)10 (8)11 (5)

    		Medicaid insurance ever, n (%)76 (45)15 (48)54 (26)73 (27)14 (11)43 (20)

    		Disability entitlement, n (%)69 (40)18 (56)37 (18)42 (15)21 (16)44 (20)
    Rural/urban, n (%)






    		Urban code137 (80)241 (75)117 (56)163 (59)85 (66)143 (66)

    		Suburban19 (11)45 (14)47 (22)64 (23)27 (21)42 (19)

    		Large town11 (6)27 (9)43 (21)39 (14)17 (13)29 (13)

    		Small town/rural3 (2)6 (2)2 (1)8 (3)0 (0)4 (2)
    Percentage with a high school degree, mean (SD)1 (0)1 (0)1 (0)1 (0)1 (0)1 (0)
    HCCa score, mean (SD)3 (2)3 (2)3 (1)3 (2)2 (1)1 (1)
    High risk, n (%)117 (69)256 (80)209 (100)274 (100)0 (0)0 (0)
    Chronic conditions






    		3 or more chronic conditions142 (84)301 (94)204 (98)268 (98)117 (91)168 (77)

    		Congestive heart failure74 (44)141 (44)102 (49)147 (54)29 (22)44 (20)

    		COPDb/asthma73 (43)184 (58)106 (51)136 (50)40 (31)93 (43)

    		Chronic kidney disease79 (47)155 (48)103 (49)165 (60)28 (22)50 (23)

    		Alcohol or drug abuse55 (33)115 (36)27 (13)51 (19)8 (6)47 (22)

    		Anxiety88 (52)207 (65)86 (41)127 (46)70 (54)110 (50)

    		Depression79 (46)168 (53)71 (34)94 (34)50 (39)89 (41)

    		Diabetes with complications43 (25)98 (31)51 (24)83 (30)20 (16)34 (16)

    		Diabetes without complications17 (10)14 (4)16 (8)22 (8)14 (11)32 (15)

    		Hypertension104 (61)198 (62)174 (83)232 (85)106 (82)164 (75)

    		Liver disease9 (6)57 (18)20 (10)14 (5)11 (9)14 (6)

    		Fluid/electrolyte disorders74 (44)182 (57)124 (59)153 (56)24 (19)38 (17)

    		Metastatic cancer8 (5)12 (4)12 (6)17 (6)4 (3)5 (2)

    		Obesity55 (33)104 (33)64 (31)78 (28)54 (42)66 (30)

    		Psychosis46 (27)108 (34)37 (18)48 (18)29 (22)65 (30)

    		Peripheral vascular disease39 (23)88 (28)74 (35)110 (40)17 (13)33 (15)

    		Renal failure48 (28)84 (26)59 (28)107 (39)16 (12)27 (12)

    		Solid tumor without metastasis13 (8)23 (7)22 (11)31 (11)12 (9)22 (10)
    Baseline utilization






    		Number of EDc visits, mean (SD)3.04 (4.20)2.08 (3.30)1.33 (1.61)1.30 (1.60)0.73 (1.07)0.72 (1.07)

    		Number of hospitalizations, mean (SD)1.70 (2.13)1.10 (1.62)0.88 (0.90)0.97 (1.17)0.39 (0.62)0.15 (0.39)

    		Number of days in hospital, mean (SD)8.96 (14.77)4.79 (8.36)4.14 (5.44)4.21 (6.58)1.27 (2.48)0.53 (1.58)

    		Number of observation stays, mean (SD)0.33 (0.66)0.25 (0.60)0.21 (0.45)0.23 (0.47)0.15 (0.42)0.07 (0.26)

    		Number of days in observation stay, mean (SD)0.37 (0.77)0.29 (0.80)0.25 (0.62)0.30 (0.72)0.16 (0.46)0.07 (0.26)

    		Medicare paymentd, mean (SD)44.39 (52.84)27.20 (33.55)27.62 (26.12)29.05 (30.04)15.66 (24.34)7.89 (10.09)

    aHCC: hierarchical condition category.

    bCOPD: chronic obstructive pulmonary disease.

    cED: emergency department.

    dPer US $1000.

    Relationship Between Case Management and Outcomes by Benefit Category

    Across all patients, enrollment in the historic case management program was associated with 80 fewer events and 368 fewer event-days per 100 enrolled patients, although there was no difference in Medicare payments (Table 5). Among high-benefit patients, enrollment in the historic program was associated with 117 fewer events, 536 fewer event-days, and US $1,151,063 reduction in Medicare payments over the subsequent year per 100 enrolled patients when compared with the findings for comparison patients. Among no/low-benefit patients, there was no association between enrollment and outcomes.

    For the new case management program, among high-risk high-benefit patients, enrollment was associated with 65 fewer events per 100 patients, with no difference in event-days or Medicare payments. Among high-risk no/low-benefit patients, there was no association between enrollment and outcomes. Among moderate-risk patients, there was no association between enrollment and outcomes for either high-benefit or no/low-benefit patients.

    Table 5. Average adjusted predicted outcomes per 100 patients enrolled in case management for 1 year among historic and new case management patients, by benefit category.
    OutcomeOverallHigh benefitNo/low benefit

    		Value95% CIValue95% CIValue95% CI
    Historic case management program






    		Reduction in the number of unplanned events8018 to 16111751 to 2029−65 to 97

    		Reduction in the number of unplanned event-days36832 to 820536214 to 96244−357 to 522

    		Medicare savings (from the outcome model) (US$)663,742−204,900 to 1,827,6051,151,063368,423 to 2,216,791−272,124−1,449,624 to 1,179,157
    New case management program (all patients)






    		Reduction in the number of unplanned events0−43 to 4417−35 to 61−12−57 to 41

    		Reduction in the number of unplanned event-days49−136 to 245155−58 to 365−24−248 to 203

    		Medicare savings (US$)−33,840−828,888 to 740,516181,660−741,324 to 1,059,818−181,032−964,032 to 658,798
    New case management
    program (high-risk patients)






    		Reduction in the number of unplanned events34−32 to 102651 to 12811−64 to 100

    		Reduction in the number of unplanned event-days132−158 to 447288−29 to 59613−333 to 411

    		Medicare savings (US$)350,407−714,876 to 1,526,562891,572−291,096 to 1,976,962−58,524−1,193,112 to 1,263,905
    New case management program
    (moderate risk patients)






    		Reduction in the number of unplanned events−57−114 to 0.30−72−156 to 0.01−47−112 to 8

    		Reduction in the number of unplanned event-days−190−486 to 27−191−527 to 62−189−549 to 52

    		Medicare savings (US$)−288,552−1,447,932 to 860,098−449,688−2,248,944 to 1,022,261−182,856−1,365,516 to 827,178

    In this Medicare ACO, we found that reduction in Medicare payments and unplanned hospital events from case management participation were limited to high-risk high-benefit patients. A benefit score [16] was able to identify patients who would benefit from a new program with respect to reducing events, but only among a high-risk population with average HCC scores similar to the population on which the score was developed. The score was not able to successfully identify moderate-risk patients who might benefit.

    There are several possible reasons for our findings on applying a previously developed benefit score prospectively to a new case management program and population. A possible explanation for why the score was able to successfully identify high-risk patients who might benefit from a new (and different) program is that while the historic and new case management programs differed in teams of composition and location, core elements of a program may depend more on what is done and not who does it or where it is done. When both nurses and social workers work in a practice, they tend toward different roles (social workers assess social issues and nurses coordinate hospital transitions) [41], but in solo practice, each may provide all essential elements of case management [42]. Conversely, the benefit score was developed in a high-risk population [16], not in the broader high- and moderate-risk population served by the new program. Even though targeting patients for case management using a risk score alone may be insufficient [5,43], case management programs may still be designed to optimize care for patients at a specific risk level, and enrollment of patients with different risks may mean a mismatch between program goals/activities and patient needs [44]. Our findings suggest that this score may be limited to identifying case management patients who would benefit from a new case management program only among a population similar to that on which the score was developed.

    The latest projections from the Congressional Budget Office are that the Medicare trust fund will run out of money in 2024 [45]. This is the closest the fund has ever come to insolvency since Medicare was established in 1965 and demonstrates the urgent need to understand how to best provide access to high-quality care while simultaneously controlling costs. In order for impactability modeling to help solve the nationwide crisis in caring for high-need high-cost patients [15], “benefit” or “impactability” scores will need to extend beyond the programs and populations in which they were developed. This study provides a first step toward assessing the feasibility and limits of this extension. Although we found evidence that a benefit score could extend to a new program and a similar risk population, caution is warranted as programs vary widely [5,8] and evidence of successful extension to one program does not necessarily indicate that the score could be extended to another program. Unlike risk models, impactability models are intrinsically linked to both the population and the specific program used in their development. Measurement of “similarity” (how similar is similar enough?) is an important open question [46]. More research is needed to understand the core elements of case management (to identify similar programs) and to streamline identification of similar populations.

    There are several limitations to our study. First, we were limited to evaluating the impact of the pandemic in a single large health system with both academic and community clinics. This health system did participate in Medicare ACO programs, indicating that they had a strong base of primary care patients [47]. Examining different programs within a system may mitigate variability in coding and data across systems [48], but could also complicate extension to another system. Moreover, academic systems often serve a different population than community practices, but this health system had a large number of community-based primary care clinics [47]. Second, unmeasured confounding is a limitation of all observational studies. As is the case with any observational study, it is almost never possible to know the direction and magnitude of such unmeasured confounding. However, given our measurement of repeated outcomes both prepandemic and postpandemic, as well as an extensive matching process and the similarity of our matched populations, it is unlikely that any remaining small differences explain our findings. Third, we only followed outcomes for 1 year after enrollment, which may be too short to realize positive outcomes [49]. This may explain our finding in the moderate-risk population, and similar to another study [50], negative findings at 1 year might turn into positive findings at 2 years. Fourth, the study focused only on mortality and unplanned events such as hospitalizations and ED visits. Although we used validated algorithms, the definition of unplanned events likely represented some unavoidable events that may not be directly under the control of the health system. This may have slightly reduced our ability to estimate the impact of the case management programs.

    The use of impactability modeling to match specific case management programs with high-need high-cost patients who might benefit is consistent with the call by Bates et al to make predictions actionable for interventions [12,51]. This approach does not rely on identifying effective case management programs and attempting to standardize their implementation nationwide, a daunting undertaking given the wide variation in programs [5,8], resistance to change within health systems [52], and practical challenges in implementing evidence-based interventions [53]. Yet, impactability modeling brings its own challenges, most importantly the limitation of tailoring each model to a specific case management program and population. Enthusiasm for this approach should be tempered until additional research provides robust strategies to identify case management programs and populations that are sufficiently similar to warrant a score’s application. In the interim, extending a score developed for a specific program and population to a different program and population should be accompanied by ongoing evaluation to confirm its applicability. Over time, policy makers and measure developers should consider impactability modeling when designing new programs and metrics.

    Acknowledgments

    This work was supported by the Health Innovation Program at the UW School of Medicine and Public Health. The sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and the decision to submit the manuscript for publication.

    Authors' Contributions

    MAS had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: MAS, MY, and JH. Acquisition, analysis, or interpretation of data: MAS, MY, JH, XW, and AD. Drafting of the manuscript: MAS. Critical revision of the manuscript for important intellectual content: All authors. Statistical analysis: XW. Obtained funding: MAS. Administrative, technical, or material support: MAS. Study supervision: MAS.

    Conflicts of Interest

    None declared.

    1. Blumenthal D, Chernof B, Fulmer T, Lumpkin J, Selberg J. Caring for high-need, high-cost patients - An urgent priority. N Engl J Med 2016 Sep 08;375(10):909-911. [CrossRef] [Medline]
    2. Long P, Abrams M, Milstein A, Anderson G, Apton K, Dahlberg M, et al. Effective Care for High-Need Patients: Opportunities for Improving Outcomes, Value, and Health. National Academy of Medicine. 2017.   URL: https://nam.edu/wp-content/uploads/2017/06/Effective-Care-for-High-Need-Patients.pdf [accessed 2021-04-06]
    3. Adams AJ, Clark DR, DeLander GE, Mackinnon GE, Malloy M, McGivney MS, et al. Report of the AACP task force on patient-centered medical homes and accountable care organizations. Am J Pharm Educ 2013 Sep 12;77(7):142 [FREE Full text] [CrossRef] [Medline]
    4. Fisher ES, Staiger DO, Bynum JPW, Gottlieb DJ. Creating accountable care organizations: the extended hospital medical staff. Health Aff (Millwood) 2007;26(1):w44-w57 [FREE Full text] [CrossRef] [Medline]
    5. Hickam DH, Weiss JW, Guise JM, Buckley D, Motu'apuaka M, Graham E, et al. Outpatient Case Management for Adults With Medical Illness and Complex Care Needs. Rockville, MD: Agency for Healthcare Research and Quality; 2013.
    6. Stokes J, Panagioti M, Alam R, Checkland K, Cheraghi-Sohi S, Bower P. Effectiveness of case management for 'at risk' patients in primary care: A systematic review and meta-analysis. PLoS One 2015;10(7):e0132340 [FREE Full text] [CrossRef] [Medline]
    7. Smith MA, Vaughan-Sarrazin MS, Yu M, Wang X, Nordby PA, Vogeli C, et al. The importance of health insurance claims data in creating learning health systems: evaluating care for high-need high-cost patients using the National Patient-Centered Clinical Research Network (PCORNet). J Am Med Inform Assoc 2019 Nov 01;26(11):1305-1313 [FREE Full text] [CrossRef] [Medline]
    8. Donelan K, Barreto EA, Michael CU, Nordby P, Smith M, Metlay JP. Variability in care management programs in Medicare ACOs: A survey of medical directors. J Gen Intern Med 2018 Dec;33(12):2043-2045 [FREE Full text] [CrossRef] [Medline]
    9. Nelson L. Lessons from Medicare’s Demonstration Projects on Disease Management and Care Coordination. Congressional Budget Office. 2012.   URL: https:/​/www.​cbo.gov/​sites/​default/​files/​cbofiles/​attachments/​WP2012-01_Nelson_Medicare_DMCC_Demonstrations.​pdf [accessed 2021-04-06]
    10. McDonald K, Sundaram V, Bravata D, Lewis R, Lin N, Kraft S, et al. Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies. Agency for Healthcare Research and Quality. 2007.   URL: https://www.ncbi.nlm.nih.gov/books/NBK44015/pdf/Bookshelf_NBK44015.pdf [accessed 2021-04-06]
    11. Blumenthal D, Abrams MK. Tailoring complex care management for high-need, high-cost patients. JAMA 2016 Oct 25;316(16):1657-1658. [CrossRef] [Medline]
    12. Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff (Millwood) 2014 Jul;33(7):1123-1131. [CrossRef] [Medline]
    13. McWilliams JM. Cost containment and the tale of care coordination. N Engl J Med 2016 Dec 08;375(23):2218-2220 [FREE Full text] [CrossRef] [Medline]
    14. Meek JA. Affordable Care Act: predictive modeling challenges and opportunities for case management. Prof Case Manag 2012;17(1):15-21; quiz 22. [CrossRef] [Medline]
    15. Lewis GH. "Impactibility models": identifying the subgroup of high-risk patients most amenable to hospital-avoidance programs. Milbank Q 2010 Jun;88(2):240-255 [FREE Full text] [CrossRef] [Medline]
    16. Huling JD, Yu M, Smith M. Fused comparative intervention scoring for heterogeneity of longitudinal intervention effects. Ann. Appl. Stat 2019 Jun 1;13(2):1639-1641. [CrossRef]
    17. DuBard CA, Jackson CT. Active redesign of a Medicaid care management strategy for greater return on investment: Predicting impactability. Popul Health Manag 2018 Apr;21(2):102-109 [FREE Full text] [CrossRef] [Medline]
    18. Smith M, Donelan K, Vaughan-Sarrazin M, Yu M, Vogeli C, Wang X, et al. Examining the Effectiveness of Case Management Programs on Preventing Hospital Stays in Older Adults with Multiple Chronic Health Problems -- A PCORnet® Study. Patient-Centered Outcomes Research Institute.   URL: https://www.pcori.org/sites/default/files/Smith023-Final-Research-Report.pdf [accessed 2021-04-06]
    19. Goldberg HI, Neighbor WE, Cheadle AD, Ramsey SD, Diehr P, Gore E. A controlled time-series trial of clinical reminders: using computerized firm systems to make quality improvement research a routine part of mainstream practice. Health Serv Res 2000 Mar;34(7):1519-1534 [FREE Full text] [Medline]
    20. Eccles M, Grimshaw J, Campbell M, Ramsay C. Research designs for studies evaluating the effectiveness of change and improvement strategies. Qual Saf Health Care 2003 Feb;12(1):47-52 [FREE Full text] [CrossRef] [Medline]
    21. Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract 2004 May;10(2):307-312. [CrossRef] [Medline]
    22. Smith MA, Nordby PA, Yu M, Jaffery J. A practical model for research with learning health systems: Building and implementing effective complex case management. Appl Ergon 2020 Apr;84:103023. [CrossRef] [Medline]
    23. IRB QI Self-Certification Decision Tool. HIPxChange.   URL: https://www.hipxchange.org/IRB [accessed 2022-06-03]
    24. Schneeweiss S, Rassen JA, Glynn RJ, Avorn J, Mogun H, Brookhart MA. High-dimensional propensity score adjustment in studies of treatment effects using health care claims data. Epidemiology 2009 Jul;20(4):512-522 [FREE Full text] [CrossRef] [Medline]
    25. Pope GC, Kautter J, Ellis RP, Ash AS, Ayanian JZ, Lezzoni LI, et al. Risk adjustment of Medicare capitation payments using the CMS-HCC model. Health Care Financ Rev 2004;25(4):119-141 [FREE Full text] [Medline]
    26. Lee C, Luo Z, Ngiam K, Zhang M, Zheng K, Chen G, et al. Big Healthcare Data Analytics: Challenges and Applications. In: Khan S, Zomaya A, Abbas A, editors. Handbook of Large-Scale Distributed Computing in Smart Healthcare. Scalable Computing and Communications. Cham: Springer; 2017:11-41.
    27. Rural-Urban Commuting Area Codes (RUCAs). Rural Health Research Center.   URL: http://depts.washington.edu/uwruca/ruca-data.php [accessed 2021-09-08]
    28. Huling JD, Yu M, Liang M, Smith M. Risk prediction for heterogeneous populations with application to hospital admission prediction. Biometrics 2018 Jun;74(2):557-565 [FREE Full text] [CrossRef] [Medline]
    29. Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care 1998 Jan;36(1):8-27. [CrossRef] [Medline]
    30. Hospital-Wide (All-Condition) 30‐Day Risk-Standardized Readmission Measure: DRAFT Measure Methodology Report. Centers for Medicare & Medicaid Services. 2011.   URL: https:/​/www.​cms.gov/​medicare/​quality-initiatives-patient-assessment-instruments/​mms/​downloads/​mmshospital-wideall-conditionreadmissionrate.​pdf [accessed 2022-04-15]
    31. Schwarz G. Estimating the dimension of a model. Ann. Statist 1978 Mar 1;6(2):461-464. [CrossRef]
    32. Rassen JA, Shelat AA, Myers J, Glynn RJ, Rothman KJ, Schneeweiss S. One-to-many propensity score matching in cohort studies. Pharmacoepidemiol Drug Saf 2012 May;21 Suppl 2:69-80. [CrossRef] [Medline]
    33. Stuart EA. Matching methods for causal inference: A review and a look forward. Stat Sci 2010 Feb 01;25(1):1-21 [FREE Full text] [CrossRef] [Medline]
    34. Rubin DB. Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation. In: Matched Sampling for Causal Effects. Cambridge, UK: Cambridge University Press; 2006:365-382.
    35. Estimating Differential Treatment Effects Using the 'personalized' R Package. HIPxChange.   URL: https://www.hipxchange.org/PersonalizedRPackage [accessed 2021-02-05]
    36. Yu M, Huling J, Smith M, Shao J. Developing New Methods for Determining Treatment Benefits Based on Individual Patient Traits. Patient-Centered Outcomes Research Institute. 2020.   URL: https://www.pcori.org/sites/default/files/Yu308-Final-Research-Report.pdf [accessed 2021-04-06]
    37. Greene WH. Econometric Analysis. Upper Saddle River, NJ: Prentice Hall; 1997.
    38. Harrell FE, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996 Feb 28;15(4):361-387. [CrossRef] [Medline]
    39. Berry LL, Rock BL, Smith Houskamp B, Brueggeman J, Tucker L. Care coordination for patients with complex health profiles in inpatient and outpatient settings. Mayo Clin Proc 2013 Feb;88(2):184-194. [CrossRef] [Medline]
    40. Case Management Benefit Scoring System. HIPxChange.   URL: https://www.hipxchange.org/BenefitScore [accessed 2021-02-05]
    41. Donelan K, Chang Y, Berrett-Abebe J, Spetz J, Auerbach DI, Norman L, et al. Care management for older adults: The roles of nurses, social workers, and physicians. Health Aff (Millwood) 2019 Jun;38(6):941-949. [CrossRef] [Medline]
    42. Tahan HA, Campagna V. Case management roles and functions across various settings and professional disciplines. Prof Case Manag 2010;15(5):245-77; quiz 278. [CrossRef] [Medline]
    43. Haime V, Hong C, Mandel L, Mohta N, Iezzoni LI, Ferris TG, et al. Clinician considerations when selecting high-risk patients for care management. Am J Manag Care 2015 Oct 01;21(10):e576-e582 [FREE Full text] [Medline]
    44. Price-Haywood EG, Petersen H, Burton J, Harden-Barrios J, Adubato M, Roberts M, et al. Outpatient complex case management: Health system-tailored risk stratification taxonomy to identify high-cost, high-need patients. J Gen Intern Med 2018 Nov;33(11):1921-1927 [FREE Full text] [CrossRef] [Medline]
    45. The Outlook for Major Federal Trust Funds: 2020 to 2030. Congressional Budget Office.   URL: https://www.cbo.gov/system/files/2020-09/56523-Trust-Funds.pdf [accessed 2021-04-06]
    46. Ippolito M. How similar is similar enough? Semantics and Pragmatics 2016;9(Article 6):1-60 [FREE Full text] [CrossRef]
    47. Schulz J, DeCamp M, Berkowitz SA. Medicare Shared Savings Program: public reporting and shared savings distributions. Am J Manag Care 2015 Aug;21(8):546-553 [FREE Full text] [Medline]
    48. Estiri H, Chan Y, Baldwin L, Jung H, Cole A, Stephens KA. Visualizing anomalies in electronic health record data: The variability explorer tool. AMIA Jt Summits Transl Sci Proc 2015;2015:56-60 [FREE Full text] [Medline]
    49. Berry-Millett R, Bodenheimer T. Care management of patients with complex health care needs. Synth Proj Res Synth Rep 2009 Dec(19):52372. [Medline]
    50. Counsell SR, Callahan CM, Clark DO, Tu W, Buttar AB, Stump TE, et al. Geriatric care management for low-income seniors: a randomized controlled trial. JAMA 2007 Dec 12;298(22):2623-2633. [CrossRef] [Medline]
    51. Smith MA, Adelaine S, Bednarz L, Patterson BW, Pothof J, Liao F. Predictive solutions in learning health systems: The critical need to systematize implementation of prediction to action to intervention. NEJM Catalyst 2021 Apr 21;2(5):0650. [CrossRef]
    52. Coiera E. Why system inertia makes health reform so difficult. BMJ 2011 Jun 23;342:d3693. [CrossRef] [Medline]
    53. Dixon-Woods M, McNicol S, Martin G. Ten challenges in improving quality in healthcare: lessons from the Health Foundation's programme evaluations and relevant literature. BMJ Qual Saf 2012 Oct;21(10):876-884 [FREE Full text] [CrossRef] [Medline]

    ‎
    ACO: accountable care organization
    COPD: chronic obstructive pulmonary disease
    ED: emergency department
    EHR: electronic health record
    HCC: hierarchical condition category
    PCORI: Patient-Centered Outcomes Research Institute

    Edited by T Leung; submitted 08.04.21; peer-reviewed by Y Chu, J Walsh; comments to author 09.12.21; revised version received 15.04.22; accepted 25.04.22; published 13.06.22

    Copyright

    ©Maureen A Smith, Menggang Yu, Jared D Huling, Xinyi Wang, Allie DeLonay, Jonathan Jaffery. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 13.06.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.