This paper is in the following e-collection/theme issue:

Clinical Informatics (1022) mHealth in a Clinical Setting (615)

Published on in Vol 21, No 7 (2019): July

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/13147, first published .
Implementation of a Digitally Enabled Care Pathway (Part 1): Impact on Clinical Outcomes and Associated Health Care Costs

Implementation of a Digitally Enabled Care Pathway (Part 1): Impact on Clinical Outcomes and Associated Health Care Costs

Implementation of a Digitally Enabled Care Pathway (Part 1): Impact on Clinical Outcomes and Associated Health Care Costs

Authors of this article:

Alistair Connell1, 2 Author Orcid Image ;   Rosalind Raine3 Author Orcid Image ;   Peter Martin3 Author Orcid Image ;   Estela Capelas Barbosa3 Author Orcid Image ;   Stephen Morris3 Author Orcid Image ;   Claire Nightingale3, 4 Author Orcid Image ;   Omid Sadeghi-Alavijeh5 Author Orcid Image ;   Dominic King2 Author Orcid Image ;   Alan Karthikesalingam2 Author Orcid Image ;   Cían Hughes2 Author Orcid Image ;   Trevor Back2 Author Orcid Image ;   Kareem Ayoub2 Author Orcid Image ;   Mustafa Suleyman2 Author Orcid Image ;   Gareth Jones5 Author Orcid Image ;   Jennifer Cross5 Author Orcid Image ;   Sarah Stanley5 Author Orcid Image ;   Mary Emerson5 Author Orcid Image ;   Charles Merrick5 Author Orcid Image ;   Geraint Rees6 Author Orcid Image ;   Hugh Montgomery1 Author Orcid Image ;   Christopher Laing5 Author Orcid Image
Article Authors Cited by (13) Tweetations (39) Metrics

    Original Paper

    1Centre for Human Health and Performance, University College London, London, United Kingdom

    2DeepMind Health, London, United Kingdom

    3Department of Applied Health Research, University College London, London, United Kingdom

    4Population Health Research Institute, St George’s, University of London, London, United Kingdom

    5Royal Free London NHS Foundation Trust, London, United Kingdom

    6Faculty of Life Sciences, University College London, London, United Kingdom

    *these authors contributed equally

    Corresponding Author:

    Christopher Laing, MD

    Royal Free London NHS Foundation Trust

    Pond Street

    London,

    United Kingdom

    Phone: 44 020 7794 0500 ext 33322

    Email: chris.laing@nhs.net


    Background: The development of acute kidney injury (AKI) in hospitalized patients is associated with adverse outcomes and increased health care costs. Simple automated e-alerts indicating its presence do not appear to improve outcomes, perhaps because of a lack of explicitly defined integration with a clinical response.

    Objective: We sought to test this hypothesis by evaluating the impact of a digitally enabled intervention on clinical outcomes and health care costs associated with AKI in hospitalized patients.

    Methods: We developed a care pathway comprising automated AKI detection, mobile clinician notification, in-app triage, and a protocolized specialist clinical response. We evaluated its impact by comparing data from pre- and postimplementation phases (May 2016 to January 2017 and May to September 2017, respectively) at the intervention site and another site not receiving the intervention. Clinical outcomes were analyzed using segmented regression analysis. The primary outcome was recovery of renal function to ≤120% of baseline by hospital discharge. Secondary clinical outcomes were mortality within 30 days of alert, progression of AKI stage, transfer to renal/intensive care units, hospital re-admission within 30 days of discharge, dependence on renal replacement therapy 30 days after discharge, and hospital-wide cardiac arrest rate. Time taken for specialist review of AKI alerts was measured. Impact on health care costs as defined by Patient-Level Information and Costing System data was evaluated using difference-in-differences (DID) analysis.

    Results: The median time to AKI alert review by a specialist was 14.0 min (interquartile range 1.0-60.0 min). There was no impact on the primary outcome (estimated odds ratio [OR] 1.00, 95% CI 0.58-1.71; P=.99). Although the hospital-wide cardiac arrest rate fell significantly at the intervention site (OR 0.55, 95% CI 0.38-0.76; P<.001), DID analysis with the comparator site was not significant (OR 1.13, 95% CI 0.63-1.99; P=.69). There was no impact on other secondary clinical outcomes. Mean health care costs per patient were reduced by £2123 (95% CI −£4024 to −£222; P=.03), not including costs of providing the technology.

    Conclusions: The digitally enabled clinical intervention to detect and treat AKI in hospitalized patients reduced health care costs and possibly reduced cardiac arrest rates. Its impact on other clinical outcomes and identification of the active components of the pathway requires clarification through evaluation across multiple sites.

    J Med Internet Res 2019;21(7):e13147

    doi:10.2196/13147

    Keywords

    nephrology; acute kidney injury 


    Background

    Acute kidney injury (AKI)—a sudden decline in kidney function—can be caused by hypovolemia, infection (including severe sepsis), nephrotoxicity, primary renal diseases, and urinary tract obstruction [1]. Affecting more than 18% of hospitalized patients [2], it is associated with prolonged hospital stay, need for acute renal replacement therapy (RRT), or intensive care admission as well as the development of chronic kidney disease and the need for long-term dialysis [3-5]. Although AKI may be a marker of systemic physiological decompensation in acute illnesses (eg, sepsis, trauma, or high-risk surgery), AKI itself might directly cause additional deaths through, for instance, metabolic derangement or extracellular fluid volume overload [6]. Such impacts are expensive; AKI confers excess annual costs of £1 billion to the English National Health Service (NHS) [7]. Similar excess health care costs have been demonstrated in other health systems [8].

    AKI management involves the identification and treatment of life-threatening complications and medical or surgical treatment of underlying cause, supportive care (including RRT, where necessary), and interventions to reduce risk of recurrence [9]. In response to poor outcomes and variations in care delivery [10], automated AKI alerts (using standardized definitions of its presence and severity based on increases in serum creatinine [11]) have been delivered using messages in electronic health record systems or through hospital pagers [12]. In England, this approach has been applied through the embedding of an AKI detection algorithm—The NHS Early Detection Algorithm, NHSEDA (Multimedia Appendix 1)—in laboratory information management systems [13]. However, evidence of the impact of electronic alerts in improving clinical outcomes in AKI is conflicting [14]. The greatest indications of improvement seem to occur when such detection systems are coupled with structured education and clinical intervention packages [15,16]. However, the delivery of such care pathways is challenging: AKI is common and of heterogeneous etiology; it presents in diverse settings; and it is normally, at least in its early stages, managed by a range of nonspecialist teams.

    Objectives

    To address these issues, we developed a digitally enabled care pathway for AKI patients [17]. This uses a mobile app (Streams), which alerts a specialist response team to the presence of AKI in real time, simultaneously providing relevant clinical data in a user-friendly format and allowing communication of key triage decisions among team members. Members of the response team review patients using a care protocol that maps to best practice guidance [9].

    We have reported the clinical impact of this digitally enabled care pathway on patients with AKI at the point of presentation to the emergency department (ED) [18]. The limited impacts we identified in this setting might reflect the difficulty of mitigating harm when AKI is well established or in the context of pathogeneses specific to community-acquired AKI. In this paper, we assess the impact of the care pathway on clinical outcomes for patients who develop AKI following hospital admission and on health care costs.


    The Hospital Sites

    The digital pathway was implemented at the Royal Free Hospital (RFH), a large (839 beds including a 34-bed intensive treatment unit [ITU]) hospital in north London, United Kingdom. It provides acute and emergency care as well as a range of specialist, regional inpatient services (eg, hepatology, HIV and infectious disease, amyloidosis, and vascular surgery) and has a large inpatient nephrology and renal transplant service.

    For the purposes of our evaluation, we used a comparator site managed by the same health care provider organization (Royal Free London NHS Foundation Trust, RFLFT) in which the intervention was not implemented. Barnet General Hospital (BGH) is an acute general hospital with 459 beds. It has a 21-bed ITU that can provide acute RRT and a liaison nephrology service. Tertiary, specialist services are not provided on this site. A number of parallel improvement initiatives were ongoing at the comparator site during the study period, including a sepsis improvement project and an active deteriorating patients improvement program.

    Implementation

    Blood tests, including serum creatinine, are routinely undertaken on hospitalized inpatients across all wards as directed by the treating clinicians. Historically, at both sites, blood tests would be reviewed in batches by the clinicians who ordered them. Results suggesting AKI would be telephoned to relevant wards by laboratory staff. Referral for nephrology assessment would be undertaken at the discretion of the clinical teams and using hospital pagers and phones. Cases would be prioritized and treated by the nephrology teams through assessment of referral information and results on desktop computers and through bedside review. The Patient at Risk and Resuscitation Team (PARRT) provides support to ward teams for patients deemed at risk of deterioration or who trigger existing, physiology-based early warning systems.

    The digitally enabled AKI care pathway and the technical architecture of the Streams app have been described in detail previously [17], and the pre-existing and novel care pathways are shown in Multimedia Appendix 1. Members of the response team undertook training before implementation. Following implementation, Streams continuously applied the NHSEDA to creatinine results for all inpatients. Using iPhones (Apple Inc), the nephrology team was alerted to all potential cases of AKI, with their AKI stage and whether metabolic complications such as hyperkalemia were present. A curated dataset was provided, which included patients’ demographic characteristics, previously coded diagnoses, and relevant results. Filters excluded children, critical care and chronic dialysis patients, and those already under the care of the nephrology inpatient team from producing alerts. Cases were triaged in-app and, when clinical review was warranted, a best-practice care protocol was delivered (Multimedia Appendix 1). This was annotated and entered into the patient’s notes alongside an advisory sticker for key nursing actions (Multimedia Appendix 1). Recovery could be monitored in-app, and repeat AKI alerts were sent if AKI had not recovered after 48 hours or if AKI severity stage increased. Nephrology members received all AKI alerts, and PARRT members received alerts only for patients with stages 2 and 3 AKI. All team members could communicate in-app; triage responses and the outcome of clinical reviews were visible in the app to other team members. Implementation of the care pathway at the RFH used existing RFH PARRT and nephrology staff and did not result in expansion in staff numbers. A diagram outlining the pre- and postintervention care pathways is provided in Multimedia Appendix 1.

    Data Collection

    At both sites, data from RFLFT hospital databases and those supporting Streams app relating to the intervention period (May to September 2017) were compared with those from a predeployment phase (May 2016 to January 2017). Data relating to patients in whom an AKI alert was generated on presentation to the hospital ED are reported elsewhere [18], and such patients were excluded from the analyses reported here.

    Data collected and their sources are detailed in Table 1. The time frame to alert viewing was determined using data recorded by the Streams app. The presence of individual comorbidities and overall patient-specific Charlson comorbidity index score (which categorizes comorbidities based on the International Classification of Diseases diagnosis codes) were derived as per the method by Thygesen et al [19]. Patients were sorted into national quintiles of deprivation (quintile 1, least; quintile 5, most deprived) using Indices of Multiple Deprivation (IMD)—a measure combining 7 domains (income, employment, living environment, health, education skills and training deprivation and disability, barriers to housing and services, and crime) into a single deprivation score for a small area—by cross-referencing patients’ postcodes with the UK Government’s Indices of Deprivation 2015 dataset [20].

    For the economic analysis, we used Payment Level Information and Costing System (PLICS) data supplied by the RFLFT. PLICS is a clinical costing system where costs are derived for each patient spell (ie, admission) by tracing resources used by an individual patient in diagnosis and treatment and calculating the expenditure on those resources using the actual costs incurred by the provider. PLICS has the advantage of including staffing costs and infrastructure absorbed costs. In our study, the PLICS data for hospitalized patients with AKI included the following components: total length of stay (including the length of stay in intensive care unit), pathology and radiology examinations, total theater time, theater cutting time, inpatient dialysis, and overhead costs. These data were analyzed at the spell level. We also obtained data on the costs associated with selected individual components of a spell, which we analyzed separately (ie, length of stay, pathology and radiology examinations, theater total time, and theater cutting time). However, individual cost components were based on tariffs and not fully absorbed costs. Furthermore, we could not obtain individual costs of inpatient dialysis. The final dataset used in the economic analysis comprised total and component-specific spell-level costs at the RFH and BGH, before and after the digitally enabled care pathway was introduced at the RFH.

    Evaluation of Impacts

    The primary outcome was recovery of renal function (return to a serum creatinine concentration within 120% of the baseline, as defined by the NHSEDA) before hospital discharge. Table 1 describes the predefined secondary endpoints. At both sites, NHSEDA was used to identify potential AKI cases. Because the NHSEDA can produce false positives [22], 2 authors (AC and CL) clinically validated all AKI alerts produced from all periods at both hospital sites. Only clinician-confirmed episodes of AKI were included in the analysis. In this paper, we report the outcomes of inpatients producing AKI alerts outside of the ED during the predeployment and deployment phases (Figure 1). The impact of the care pathway on cardiac arrests rate was measured on a hospital level, as it was not possible to ascertain which cardiac arrests occurred among patients with AKI.

    Table 1. Definitions of each outcome and sources of data collected.
    Data category and measureDefinitionSource of data
    Sociodemographic characteristics

    		AgeAge in years at the time of alertHL7a data aggregated within the Streams data processor
    GenderGender codes used in the NHSb Data Dictionary [21]HL7 data aggregated within the Streams data processor
    EthnicityEthnicity category codes used in the NHS Data Dictionary [21]HL7 data aggregated within the Streams data processor
    Comorbid diseasePresence of individual Charlson index comorbidities and overall Charlson scoreHL7 data aggregated within the Streams data processor
    DeprivationIndex of Multiple DeprivationMinistry of Housing, Communities and Local Government database
    Clinical outcomes

    		Recovery of renal functionReturn to <120% index creatinine (as defined by NHSEDAc) by the time of hospital dischargeHL7 data aggregated within the Streams data processor
    Time to recovery of renal functionThe time from AKId alert to recovery of renal function (<120% index creatinine)HL7 data aggregated within the Streams data processor
    MortalityDeath in 30 days following AKI alertHL7 data aggregated within the Streams data processor
    Progression of AKI stageMovement between AKI severity classes following AKI alert and before hospital dischargeHL7 data aggregated within the Streams data processor
    Admission to high acuity or specialist renal inpatient bedAdmission to acute kidney unit/high dependency unit/intensive treatment unit during index admissionHL7 data aggregated within the Streams data processor
    Requirement long-term renal replacement therapyUse of hemofiltration/hemodiafiltration/hemodialysis/peritoneal dialysis in 30 days following hospital discharge dateRFHe Nephrology Clinical Information Management System
    Length of stayTime from AKI alert to hospital dischargeHL7 data aggregated within the Streams data processor
    Re-admission to hospitalRe-admission to hospital in 30 days following index admission discharge dateHL7 data aggregated within the Streams data processor
    Trust-wide metric


    		Cardiac arrest rateNumber of cardiac arrests per 1000 bed daysTrust critical care nursing team logs
    Economic measures


    		Costs per patientCost per patient per hospital spellPayment Level Information and Costing System data and Payment by Results/local tariffs at the trust
    Process of care


    		Time to alert reviewTime from alert generation to alert viewing by a clinicianData aggregated within the Streams data processor

    aHealth Level 7 (HL7) messages are used to transfer information between different health care information technology systems.

    bNHS: National Health Service.

    cNHSEDA: NHS Early Detection Algorithm.

    dAKI: acute kidney injury.

    eRFH: Royal Free Hospital.

    ‎
    Figure 1. Defining the final evaluation sample. AKI: acute kidney injury; ITU: intensive treatment unit.
    View this figure

    Statistical Analysis

    All data were pseudonymized before transfer to the University College London (UCL) for analysis. Analyses were performed using R, version 3.4.3 (R core team) [23], and Stata MP version 14 (StataCorp) [24]. Segmented regression analysis estimated the intervention effect on the primary outcome (return to a serum creatinine concentration within 120% of the baseline, as defined by the NHSEDA) and 5 secondary outcome measures: mortality within 30 days of alert, progression of AKI stage, transfer to renal/intensive care units during admission, re-admission within 30 days of discharge, and dependence on RRT 30 days after discharge. Outcomes were measured as weekly proportions. We used binomial regression models with a logit link predicting the weekly rate of each outcome. Codes 1 and 0 were applied to the period after and before the intervention, respectively. The intervention and comparator sites were coded 1 and 0, respectively. The variable time denoted the week number (1 denoting the first week of the intervention period, and negative numbers denoting weeks in the preintervention period). The statistical model used was:

    logit(y) = β0 + β1int + β2time + β3site + β4int x time + β5int x site + β6time x site β7int x time x site (1)

    where the proportion of interest is denoted by y, the variables intervention, time, and site by int, time, and site, respectively (as defined previously), and the coefficients to be estimated by β0,...,. In addition, 2 coefficients evaluated the evidence for the intervention causing a step change in outcome: the effect of intervention estimates the step change in outcome at the start of the intervention period at the RFH. The interaction site×intervention estimates the difference-in-differences (DID) in the step change between the intervention and comparator hospital sites. We also evaluated evidence for a change in temporal trend in the outcome because of the intervention: the time×intervention interaction estimates the difference in temporal outcome trend between the intervention and preintervention periods at the RFL; the 3-way time×site×intervention interaction estimates the DID in the trend between the intervention and comparator sites.

    For all models, we inspected the autocorrelation function (up to lag 15). No significant autocorrelation was detected in any model. At the point of protocol publication, it was not anticipated that we would be able to collect patient-level data relating to sociodemographic characteristics and comorbid disease.

    To examine the robustness of our primary outcome analysis, we used binary logistic regression to perform a sensitivity analysis: the same model mentioned previously was used, except that (1) the outcome was defined at the patient level and (2) patient-level characteristics (age, sex, ethnicity category, index of multiple deprivation, AKI alert level, the presence of complications at the time of alert, and the presence of individual Charlson score comorbidities) were included as covariates to adjust for any differences in casemix between sites and within sites over time.

    The Wilcoxon rank-sum test was used to analyze the time to creatinine recovery (where this occurred by hospital discharge). To allow for the effects of in-hospital death on this outcome, the effect of the intervention on the length of hospital stay was estimated by competing risk analysis [25]. To determine the effect of the intervention on the time to recognition of AKI, a survival analysis was performed. The Wilcoxon rank-sum and chi-square tests were used to analyze sociodemographic variables as appropriate.

    A total of 500 alerts were selected randomly from all periods, and all sites were reviewed a second time to assess the reliability of case validation. Intra- and interrater reliability was determined using Cohen’s kappa coefficient (Multimedia Appendix 1).

    The number of cardiac arrests was recorded monthly at both hospital sites. Data relating to those which occurred in the hospitals’ ED, cardiac catheterization laboratory, intensive care unit, coronary care unit or in patients who had a formal not for resuscitation order signed were not included in the monthly counts recorded at the hospital level. Poisson regression models with a log link and an offset variable adjusting for the number of admissions per month were used to estimate the intervention effect on this outcome. As data were collected monthly, there was a relative paucity of postintervention data points so that estimating the effect of the intervention on outcome trend was not possible. The statistical model was:

    log(number of cardiac arrests) = β0 + β1int + β2site + β3int x site + log(number of admissions) (2)

    Economic analyses used generalized linear models (GLMs) to estimate DID, where costs were defined at the spell level, and patient-level characteristics (age, sex, ethnicity category, IMD, the presence of complications at the time of alert, and the presence of individual Charlson score comorbidities, such as diabetes mellitus or congestive cardiac failure) were included as covariates so as to allow adjustment for any differences in casemix between sites and within sites over time. A GLM was specified using a gamma family and log link to account for data skewness. The model used was:

    log(cost) = β0 + β1age + β2sex + β3ethnicity + β4imd + β5comp + β6CharlsonScore + β7time + β8site + β9time × site (3)

    where time was defined in relation to the intervention. May to September 2016 was considered preintervention (t1), and May to September 2017 was considered postintervention (t3). For robustness checks, we also carried out a secondary analysis, where the preintervention period was May 2016 to January 2017. The coefficient β9 is the coefficient of interest, measuring the between-site DID, comparing the change over time at the RFH to the change over time at the BGH. We present predictive margins showing adjusted mean costs per spell at the RFH and BGH before and after the intervention was introduced at the RFH. We adjusted for clustering at the patient level to account for the possibility that patients may have had multiple spells.

    Ethical Approval

    The digitally enabled care pathway constituted a new standard service at the RFH. The UCL Joint Research Office reviewed the study protocol and judged that the project fell under the remit of service evaluation as per guidance from the NHS Health Research Authority [26]. As such, no patient consent was required. The evaluation was registered with the RFH Audit Lead and Medical Director. An independent data monitoring committee (which included a patient member) reviewed all analyses before preparation for publication. A full list of committee members is provided in Multimedia Appendix 1.


    Alerts produced for hospitalized patients during the intervention period were reviewed by a member of the specialist response team in a median time of 14.0 min (interquartile range [IQR] 1.0-60.0 min). At the intervention site, clinical validation of the 4392 and 2254 AKI alerts during predeployment (May 2016 to January 2017) and postdeployment (May to September 2017) phases, respectively, yielded 1760 and 919 inpatient AKI episodes in each phase. Of these, 56.5% (994/1960) and 52.2% (480/919), respectively, were located outside the ED. In the predeployment and postdeployment phases at the nonintervention site, clinical validation of the 2866 and 1364 alerts, respectively, yielded 1669 and 772 inpatient AKI episodes, with 39.2% (654/1669) and 45.3% (350/772) being located outside the ED.

    Table 2 summarizes the sociodemographic and clinical characteristics of patients producing AKI alerts at both sites and periods. RFH inpatients were younger (median 72 vs 82 years, P<.001), less likely to be white (P<.001), and less deprived (P<.001) than at BGH. RFH patients had significantly less comorbidity (median [IQR] Charlson score 5.0 [3.0-8.0] vs 5.0 [4.0-8.0], P<.001). The proportion of patients with pre-existing renal disease was also lower (31.5% vs 37.2%, P<.001). Comparing the pre- and postintervention cohorts, there were some significant differences within the comparator site. At BGH, patients in the postintervention period had significantly more severe AKI (P=.01) and a higher burden of comorbid (P<.001) and renal disease (45.1% vs 32.9%, P<.001) than patients in the preintervention period.

    Table 2. Sociodemographic and clinical characteristics of patients producing acute kidney injury alerts.
    VariableHospital site/periodP value
    RFHaBGHbRFH pre vs RFH postBGH pre vs BGH postAll RFH vs all BGH
    PrecPostdPrePost
    AKIe alerts, n994480654350—f——
    Alert severity, n (%)



    		.102.01.32
     AKI1809 (81.4)411 (85.6)571 (87.3)281 (80.3)



    		AKI2127 (12.8)44 (9.2)60 (9.2)47 (13.4)



    		AKI358 (5.8)25 (5.2)23 (3.5)22 (6.3)


    Male, n (%)541 (54.4)257 (53.5)331 (50.6)186 (53.1).74.48.30
    Age (years), median (interquartile range)73.00 (58.00-84.00)7.00 (57.00-83.00)82.00 (73.00-88.00)82.00 (73.25-88.75).14.81<.001
    Ethnicity, n (%)



    		.09.32<.001
     White625 (62.9)281 (58.5)512 (78.3)274 (78.3)



    		Black or Black British76 (7.7)34 (7.1)29 (4.4)12 (3.4)



    		Asian or Asian British110 (11.1)52 (10.8)60 (9.2)25 (7.1)



    		Mixed10 (1.0)2 (0.42)3 (0.5)4 (1.1)



    		Other ethnic groups173 (17.4)111 (23.1)50 (7.7)35 (10.0)



    Index of Multiple Deprivation, n (%)

    		.87.83<.001
     Quintile 1 (least deprived)184 (18.5)84 (17.5)42 (6.42)25 (7.1)



    		Quintile 2216 (21.7)130 (27.1)132 (20.2)60 (17.1)



    		Quintile 3233 (23.4)89 (18.5)183 (28.0)111 (31.7)



    		Quintile 4224 (22.5)111 (23.1)186 (28.4)99 (28.3)



    		Quintile 5 (most deprived)97 (9.8)46 (9.6)108 (16.5)53 (15.1)



    		Unknown40 (4.0)20 (4.2)3 (0.5)2 (0.6)


    Charlson Score, n (%)



    		.49<.001<.001
     0114 (11.5)49 (10.2)10 (1.5)7 (2.0)



    		151 (5.13)11 (2.3)25 (3.8)9 (2.6)



    		263 (6.3)54 (11.2)29 (4.4)13 (3.7)



    		3107 (1.8)43 (9.0)78 (11.9)21 (6.0)



    		4169 (17.0)63 (13.1)150 (22.9)59 (16.9)



    		≥5490 (49.3)260 (54.2)362 (55.4)241 (68.9)


    Pre-existing renal disease present, n (%)303 (30.5)162 (33.8)215 (32.9)158 (45.1).23<.001<.001

    aRFH: Royal Free Hospital.

    bBGH: Barnet General Hospital.

    cPre: May 2016 to January 2017.

    dPost: May 2017 to September 2017.

    eAKI: acute kidney injury.

    fNot applicable.

    Table 3. Descriptive statistics of total cost per spell producing acute kidney injury alerts.
    StatisticsRoyal Free Hospital (£)Barnet General Hospital (£)

    		PreaPostbPrePost
    Mean (SD)12,015.24 (22,732.78)10,154.92 (19,582.30)7391.16 (14,346.27)7108.88 (11,512.95)
    Median5640.504954.003712.503774.00
    1st centile166.00207.00160.00199.00
    25th centile2391.502079.001424.001153.50
    75th centile13,208.5010,567.008466.008897.00
    99th centile111,245.0090,138.0051,991.0045,614.00

    aPre: May 2016 to January 2017.

    bPost: May 2017 to September 2017.

    Table 3 provides descriptive statistics of total costs per spell at each site before and after the intervention. Multimedia Appendix 1 shows the positively skewed distribution of these costs.

    Clinical Outcomes

    Estimates from the models predicting clinical outcomes are reported in Tables 4-7, as far as they relate to the research hypotheses. All estimated model coefficients are reported in Multimedia Appendix 1.

    Table 4. Results of segmented regression analyses for renal recovery and mortality.
    Variable/interaction termRenal recoveryMortality

    		BetaP valueORa (95% CI)BetaP valueOR (95% CI)
    Interventionb.00.991.00 (0.58-1.71).17.671.18 (0.55-2.52)
    Site×interventionc.22.621.24 (0.53-2.92).06.911.07 (0.36-3.15)
    Time×interventiond−.01.610.99 (0.96-1.03).00.891.00 (0.96-1.05)
    Time×site×interventione−.03.290.97 (0.92-1.03)−.03.440.97 (0.91-1.04)

    aOR: odds ratio.

    bThe coefficient intervention provides an estimate of the difference in outcome between the intervention period and the preintervention period at RFH.

    cThe 2-way interaction site×intervention provides an estimate of the difference-in-difference between the 2 hospital sites.

    dThe 2-way interaction time×intervention provides an estimate of the difference in outcome trend over time in the intervention period compared with the preintervention period at RFH.

    eThe 3-way interaction time×site×intervention provides an estimate of the difference-in-difference in the trend between the sites.

    Table 5. Results of segmented regression analyses for progression of acute kidney injury stage and admission to intensive treatment unit/renal unit.
    Variable/interaction termProgression of acute kidney injury stageAdmission to intensive treatment unit/renal unit

    		BetaP valueORa (95% CI)BetaP valueOR (95% CI)
    Interventionb.67.111.96 (0.86-4.47).40.421.50 (0.57-4.00)
    Site×interventionc−.71.270.49 (0.14-1.71)−1.18.180.31 (0.05-1.68)
    Time×interventiond−.01.600.99 (0.93-1.04).02.551.02 (0.96-1.08)
    Time×site×interventione.04.321.04 (0.96-1.13).07.191.08 (0.97-1.20)

    aOR: odds ratio.

    bThe coefficient intervention provides an estimate of the difference in outcome between the intervention period and the preintervention period at RFH.

    cThe 2-way interaction site×intervention provides an estimate of the difference-in-difference between the 2 hospital sites.

    dThe 2-way interaction time×intervention provides an estimate of the difference in outcome trend over time in the intervention period compared with the preintervention period at RFH.

    eThe 3-way interaction time×site×intervention provides an estimate of the difference-in-difference in the trend between the sites.

    Table 6. Results of segmented regression analyses for hospital re-admission and renal replacement therapy use.
    Variable/interaction termRe-admission at 30 daysRenal replacement therapy use at 30 days

    		BetaP valueORa (95% CI)BetaP valueOR (95% CI)
    Interventionb.20.541.22 (0.65-2.29)−3.32.030.04 (0.00-0.62)
    Site×interventionc−.16.770.86 (0.31-2.39)−1.04.990.35 (0-infinity)
    Time×interventiond−.03.230.97 (0.93-1.02).00.981.00 (0.83-1.23)
    Time×site×interventione.01.841.01 (0.94-1.08)−17.62.990.00 (0-infinity)

    aOR: odds ratio.

    bThe coefficient intervention provides an estimate of the difference in outcome between the intervention period and the preintervention period at RFH.

    cThe 2-way interaction site×intervention provides an estimate of the difference-in-difference between the 2 hospital sites.

    dThe 2-way interaction time×intervention provides an estimate of the difference in outcome trend over time in the intervention period compared with the preintervention period at RFH.

    eThe 3-way interaction time×site×intervention provides an estimate of the difference-in-difference in the trend between the sites.

    Table 7. Results of segmented regression analysis for hospital cardiac arrest rate
    Variable/interaction termCardiac arrests

    		BetaP valueOR (95% CI)
    Interventiona−.60<.0010.55 (0.38-0.76)
    Site×interventionb.12.691.13 (0.63-1.99)

    aThe coefficient intervention provides an estimate of the difference in outcome between the intervention period and the preintervention period at RFH.

    bThe 2-way interaction site×intervention provides an estimate of the difference-in-difference between the 2 hospital sites.

    Primary Outcome

    We found no evidence for a step change in renal recovery rate (return to a serum creatinine concentration within 120% of the baseline) following the intervention at the RFH. The estimated odds ratio (OR) for the intervention step change was 1.00 (95% CI 0.58-1.71). There was also no evidence for a significant difference in step change of recovery rate between RFH and BGH (estimated OR 1.24, 95% CI 0.53-2.92; P=.62).

    The model did not estimate a statistically significant change in the trend of renal recovery rates at RFH (estimated OR 0.99, 95% CI 0.96-1.03; P=.61), indicating that the trend in the intervention period at RFH was not significantly different to that in the preintervention period. There was also no significant difference in the trend change between sites (estimated OR 0.97, 95% CI 0.92-1.03; P=.29). The data and model predictions are illustrated in Figure 2. Model estimates from the sensitivity analysis controlling for differences in casemix did not differ substantially from the primary analysis model estimates (Multimedia Appendix 1), and none of the 4 examined estimated ORs were statistically significantly different from 1.

    ‎
    Figure 2. Weekly recovery rate at Royal Free Hospital (RFH) and Barnet General Hospital (BGH) before and after implementation of the care pathway. Individual data points reflect the rate of each outcome for a single week. Solid lines indicate fitted values from the modeling functions.
    View this figure
    Secondary Clinical Outcomes

    We found evidence for a reduction (step change) in the rate of cardiac arrest following the intervention at RFH (estimated OR 0.55, 95% CI 0.38-0.76; P<.001). However, we found no statistically significant difference in the step change between sites (OR 1.13, 95% CI 0.63-1.99; P=.69). The data and model predictions are shown in Figure 3.

    We also found evidence for a reduction (step change) in the rates of RRT use at 30 days at the intervention site (estimated OR 0.04, 95% CI 0.00-0.62, P=.04). However, because RRT was a rare event, estimates for this outcome were not reliable (Tables 4-7 and Multimedia Appendix 1). For all other secondary outcomes, models did not provide statistically significant evidence for an impact of the intervention. The data and model predictions are shown in Multimedia Appendix 1.

    We found no evidence for an effect of the intervention on time to renal recovery. At RFH, the median (IQR) time to renal recovery was 3.00 days (1.00-15.00 days) before and 4.00 days (1.00-12.00 days) after the introduction of the intervention (P=.61). At BGH, the median (IQR) time to renal recovery was 3.00 (1.00-13.00) and 3.00 (1.00-7.00) days, respectively (P=.100). Using competing risk analysis, a significant increase in length of stay was demonstrated at both RFH (P=.046) and BGH (P=.03) after implementation of the care pathway (Multimedia Appendix 1).

    ‎
    Figure 3. Cardiac arrests at Royal Free Hospital (RFH) and Barnet General Hospital (BGH). Individual data points reflect the rate of cardiac arrest per thousand admissions for a single month. Solid lines indicate fitted values from the modeling functions.
    View this figure

    Economic Outcomes

    There was a significant reduction in adjusted mean costs per spell over time at the RFH but not at the BGH (Tables 8-10). There was a significant reduction in mean costs per spell at the RFH in the postimplementation period compared with the preintervention period over and above the (nonsignificant) change seen at the BGH: the DID was −£2123 per spell (95% CI=−£4024 to −£222, P=.03). For the specified secondary analysis including all periods, the DID was −£1631 per spell (95% CI=−£3218 to −£44, P=.04). No significant differences were noted in the analyses of the cost components (Multimedia Appendix 1).

    Table 8. Results of economic analysis: Royal Free Hospital.
    Time periodPreintervention (£)Postintervention (£)Difference (£)P value

    		Mean95% CIMean95% CIMean95% CI
    Periods t1a and t3b only12,176.5210,996.53 to 13,356.509853.378840.91 to 10,865.82−2323.15−3843.90 to −802.41.003
    All periods11,772.6310,936.03 to 12,609.239761.598755.45 to 10,767.72−2011.05−3283.53 to −738.56.002

    at1: May to September 2016.

    bt3: May to September 2017.

    Table 9. Results of economic analysis: Barnet General Hospital.
    Time periodPreintervention (£)Postintervention (£)Difference (£)P value

    		Mean95% CIMean95% CIMean95% CI
    Periods t1a and t3b only7507.886589.77 to 8425.997307.276461.82 to 8152.71−200.62−1370.27 to 969.04.74
    All periods7623.767007.67 to 8239.867243.586413.81 to 8073.35−380.19−1358.56 to 598.19.45

    at1: May to September 2016.

    bt3: May to September 2017.

    Table 10. Results of economic analysis: difference-in-difference analysis of Royal Free Hospital and Barnet General Hospital.
    Time periodMean95% CIP value
    Periods t1a and t3b only (£)−2122.54−4023.37 to −221.70.03
    All periods (£)−1630.86−3217.50 to −44.22.04

    at1: May to September 2016.

    bt3: May to September 2017.


    Principal Findings

    The digitally enabled care pathway for the management of AKI in a large, acute hospital with a complex casemix resulted in no significant impact on the primary outcome of renal recovery or any of the other secondary clinical outcomes measured but was associated with a significant reduction in adjusted mean costs per patient admission. We did not include the costs of providing the technology, and therefore, it is not possible to judge whether or not it would be cost saving overall. Our results suggest that the digitally enabled care pathway would be cost saving, provided provision of the technology costs less than around £1600 per patient spell. The causes of the cost savings are unclear but are likely to be multifactorial, and further research to investigate these would be useful. The most important cost components contributing to this reduction (detailed in Multimedia Appendix 1) were length of stay and theater cutting time (which might itself be expected to play a role among patients requiring surgical intervention for AKI). There was a statistically significant reduction in the need for RRT at 30 days post-AKI; however, our model was not sufficiently reliable, given the low observed event rate of this outcome. The reduction in cardiac arrest rate needs to be viewed with caution because of the large number of hypothesis tests we conducted for our 6 prespecified secondary outcomes, and because this was a hospital-wide measure, this may have been influenced by other concurrently implemented initiatives. Furthermore, cardiac arrest rates also reduced at the comparator site. It is possible that both the RFH digital pathway and BGH’s quality improvement initiative were effective to some extent through different mechanisms.

    There are several possible explanations for the lack of impact on renal recovery. First, this may reflect existing high standards of AKI care before implementation: 30-day mortality for preintervention patients at RFH was 14.9% compared with 18.1% nationally [27]. It is possible that our intervention may have delivered more benefit in hospitals with worse outcomes. Second, AKI arising during inpatient admission has been shown to have worse outcomes than that arising at emergency presentation [28]. This may be because AKI arising during hospital treatment may be harder to modify. Third, AKI detection using NHSEDA depends on an elevation of serum creatinine, the detection of which may lag many hours or even days after the time of renal insult [29]. In consequence, renal injury may be less modifiable by this stage, even using a rapid system of detection such as that described. Finally, it is possible that the Streams app may have had a greater impact were it to have been implemented as part of a different care pathway—perhaps, one that involved general physicians as well as specialty care.

    An explanation for the possible effect of the intervention on rates of cardiac arrest emerged from qualitative data provided in our parallel paper [30]. Here, users suggested the care pathway not only enhanced early access to specialist care for deteriorating patients but also informed treatment escalation plans. The latter included institution of ceilings of care and do not resuscitate orders with patients and relatives. Both would be expected to contribute to a reduction in the recorded unexpected cardiac arrest rate.

    Comparison With Prior Work

    Our data are consistent with recent reports of the benefits of e-alerting systems for AKI for patients and the wider health system. We have reported on the impact of the digitally enabled care pathway on processes of care and clinical outcomes for patients with AKI at the point of presentation to the ED. Implementation of the digitally enabled care pathway for these patients was associated with significant improvement in the reliability of AKI recognition, a reduction in time to recognize and adjust potentially nephrotoxic medications [18]. Our qualitative analysis [30] found that care pathway improved access to patient information and expedited early specialist care. Our results concur with other research findings: a recent study from Korea [31] suggested that e-alerting for inpatients improves AKI recognition and the number of patients receiving specialist review [31]. Moreover, 2 single-site quality improvement projects combining AKI alerts with care bundles and targeted staff education also improved recognition of AKI and the quality of inpatient care [32,33]. In addition, a large multicenter sequential period analysis of an alerting system warning clinicians of the possible presence of AKI next to the display of serum creatinine results resulted in a small but sustained decrease in in-hospital mortality, dialysis use, and length of stay [34]. However, similar to our research, it is unclear which components of these pathways influenced these outcomes. A number of mixed-methods analyses of e-alerting systems for AKI are still underway; results from the qualitative segments of the AKORDD [35] and TACKLING [36] studies are awaited.

    Strengths and Limitations

    Our evaluation had a number of strengths. First, this is, to our knowledge, the first study to define the economic impact of implementing a digital innovation for AKI on health systems. Second, we clinically validated all NHSEDA AKI alerts before analysis and validated this process. Third, our inclusion of a comparator site follows best practice [37], ensuring transparency in the drawing of conclusions about the active components of our intervention.

    Our evaluation also had several limitations. First, longer time frames and the inclusion of multiple intervention and comparator sites would have helped overcome the effect of differences in casemix in the pre- and postintervention period (identified in our single comparator site) and may have helped to clarify any added value of the integration of our digital innovation into the care pathway. This would also have allowed us to investigate the impact on specific patient subgroups and better understand if outcomes differed between different AKI stages. It is possible, for instance, that established severe AKI is far less responsive to intervention than the early disease. It is important that such issues are prospectively addressed in future studies. Longer time frames would also have allowed us to control for any seasonal changes in outcome, which are known to occur [38] and should be borne in mind in the design of future studies. It was not possible to collect cost data relating to the innovation of the intervention site, which should be included in any future cost-benefit analyses. Finally, although time to in-app AKI recognition and virtual review by a specialist was very rapid (median 14.0 min), comparable data from the preimplementation phase could not be collected as this process is integral to the Streams app.

    Conclusions

    The digitally enabled AKI care pathway reduced inpatient health care costs and may also help reduce hospital-wide cardiac arrest rates: this result requires reanalysis in larger, multisite studies. Growing support for greater digitalization of health systems offers the opportunity to improve the quality and safety of care and to reduce its cost. However, prospective evaluation of the clinical and cost impacts of digital innovations within the context in which they are delivered will be key in delivering maximum utility for patients and health systems.

    Acknowledgments

    ECB and RR are supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care North Thames at Barts Health NHS Trust. RR is an NIHR Senior Investigator. GR and HM are funded in part by the NIHR University College London Hospitals Biomedical Research Centre. OSA is partially supported by an NIHR academic clinical fellowship. The authors wish to express their gratitude to the staff and patients of RFLFT. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health.

    Authors' Contributions

    HM, CL, RR, CH, AK, TB, KA, DK, and MS initiated the project and collaboration. CL conceived the care pathway. AC, CL, CM, JC, GJ, SS, and ME supported implementation. AC, CL, GR, RR, HM, PM, and CN were responsible for the design of the evaluation. AC and CL triaged alerts necessary for the evaluation. AC collected all necessary data. Clinical outcomes were analyzed by AC with assistance and oversight from PM and CN. Economic outcomes were analyzed by ECB with assistance from SM. AC, HM, RR, PM, CL, ECB, SM, and GR wrote the paper. All authors read and agreed the final submission.

    Conflicts of Interest

    CL, HM, GR, and RR are paid clinical advisors to DeepMind. AC’s clinical research fellowship was part-funded by DeepMind, where he has been a full-time employee since May 2018. DeepMind remained independent from the collection and analysis of all data. CL was a member of the NICE clinical guideline 169 development group referenced in the article. HM coholds a patent on a fluid delivery device, which might ultimately help in preventing some (dehydration-related) cases of AKI occurring.

    DeepMind was acquired by Google in 2014 and is now part of the Alphabet group. The deployment of Streams app at RFH was the subject of an investigation by the Information Commissioner’s Office in 2017. RFH has since published an audit completed to comply with undertakings following this investigation [39]. In November 2018, it was announced that the Streams app team will be joining Google as part of a wider health effort [40].

    ‎

    Multimedia Appendix 1

    The National Health Service Early Detection Algorithm; acute kidney injury (AKI) care pathway; AKI care protocol; nursing advisory sticker; inter- and intra-operator variability analyses; Royal Free Hospital Data Monitoring Committee; distribution of costs per spell; results of segmented regression analyses; results from binary logistic regression analysis; graphs of secondary outcome data; cost component analyses.

    PDF File (Adobe PDF File), 1MB
    1. Bellomo R, Kellum JA, Ronco C. Acute kidney injury. Lancet 2012 Aug 25;380(9843):756-766. [CrossRef] [Medline]
    2. Zeng X, McMahon GM, Brunelli SM, Bates DW, Waikar SS. Incidence, outcomes, and comparisons across definitions of AKI in hospitalized individuals. Clin J Am Soc Nephrol 2014 Jan;9(1):12-20 [FREE Full text] [CrossRef] [Medline]
    3. Metcalfe W, Simpson M, Khan IH, Prescott GJ, Simpson K, Smith WC, Scottish Renal Registry. Acute renal failure requiring renal replacement therapy: incidence and outcome. QJM 2002 Sep;95(9):579-583. [Medline]
    4. Wang HE, Muntner P, Chertow GM, Warnock DG. Acute kidney injury and mortality in hospitalized patients. Am J Nephrol 2012;35(4):349-355 [FREE Full text] [CrossRef] [Medline]
    5. Chawla LS, Amdur RL, Amodeo S, Kimmel PL, Palant CE. The severity of acute kidney injury predicts progression to chronic kidney disease. Kidney Int 2011 Jun;79(12):1361-1369 [FREE Full text] [CrossRef] [Medline]
    6. Connell A, Laing C. Acute kidney injury. Clin Med (Lond) 2015 Dec;15(6):581-584. [CrossRef] [Medline]
    7. Kerr M, Bedford M, Matthews B, O'Donoghue D. The economic impact of acute kidney injury in England. Nephrol Dial Transplant 2014 Jul;29(7):1362-1368. [CrossRef] [Medline]
    8. Silver S, Chertow G. The economic consequences of acute kidney injury. Nephron 2017;137(4):297-301 [FREE Full text] [CrossRef] [Medline]
    9. National Institute for Health and Care Excellence. Acute kidney injury: prevention, detection and management   URL: https://www.nice.org.uk/guidance/cg169 [accessed 2018-11-14] [WebCite Cache]
    10. Alleway R. NCEPOD. 2009. Acute Kidney Injury: Adding Insult to Injury Report   URL: http://www.ncepod.org.uk/2009aki.html [accessed 2016-03-08]
    11. KDIGO working group. KDIGO. 2012. KDIGO Clinical practice guidelines for acute kidney injury   URL: https://kdigo.org/wp-content/uploads/2016/10/KDIGO-2012-AKI-Guideline-English.pdf
    12. Wilson FP, Shashaty M, Testani J, Aqeel I, Borovskiy Y, Ellenberg SS, et al. Automated, electronic alerts for acute kidney injury: a single-blind, parallel-group, randomised controlled trial. Lancet 2015 May 16;385(9981):1966-1974. [CrossRef] [Medline]
    13. NHS England. 2016. Patient safety alert on standardising the early identification of Acute Kidney Injury   URL: https://www.england.nhs.uk/2014/06/psa-aki/ [accessed 2016-03-08]
    14. Lachance P, Villeneuve P, Rewa OG, Wilson FP, Selby NM, Featherstone RM, et al. Association between e-alert implementation for detection of acute kidney injury and outcomes: a systematic review. Nephrol Dial Transplant 2017 Dec 1;32(2):265-272 [FREE Full text] [CrossRef] [Medline]
    15. Master J, Hammad S, Chamberlain P, Chandrasekar T, Wong C. Reduction in acute kidney injury (AKI) mortality data with the development of a novel AKI management bundle. Nephrol Dial Transplant 2015;30:iii467. [CrossRef]
    16. Chandrasekar T, Sharma A, Chamberlain P, Abraham A. streamlined approach to AKI management leads to reduction of in-hospital mortality, length of stay and AKI progression. Nephrol Dial Transplant 2016;31:170-171. [CrossRef]
    17. Connell A, Montgomery H, Morris S, Nightingale C, Stanley S, Emerson M, et al. Service evaluation of the implementation of a digitally-enabled care pathway for the recognition and management of acute kidney injury. F1000Res 2017;6:1033. [CrossRef] [Medline]
    18. Connell A, Montgomery H, Martin P, Nightingale C, Sadeghi-Alavijeh O, King D, et al. Evaluation of a digitally-enabled care pathway for the management of acute kidney injury in patients admitted to hospital as an emergency (in press). 2019.
    19. Thygesen S, Christiansen C, Christensen S, Lash T, Sørensen H. The predictive value of ICD-10 diagnostic coding used to assess Charlson comorbidity index conditions in the population-based Danish National Registry of Patients. BMC Med Res Methodol 2011 May 28:11. [Medline]
    20. IMD by postcode lookup.   URL: http://imd-by-postcode.opendatacommunities.org/ [accessed 2018-07-23]
    21. NHS Data Dictionary. Attributes   URL: https://www.datadictionary.nhs.uk/data_dictionary/attributes/attributes.asp [accessed 2018-08-14]
    22. Sawhney S, Fluck N, Marks A, Prescott G, Simpson W, Tomlinson L, et al. Acute kidney injury: how does automated detection perform? Nephrol Dial Transplant 2015 Nov 1;30(11):1853-1861. [CrossRef]
    23. R Foundation for Statistical Computing. 2017.   URL: http://www.R-project.org/
    24. Stata: Software for Statistics and Data Science.   URL: https://www.stata.com/
    25. Brock G, Barnes C, Ramirez J, Myers J. How to handle mortality when investigating length of hospital stay and time to clinical stability. BMC Med Res Methodol 2011 Oct 26;11:144. [Medline]
    26. NHS Health Research Authority. Is my study research?   URL: http://www.hra-decisiontools.org.uk/research/redirect.html [accessed 2018-03-23]
    27. Think Kidneys. Reporting the rate of Acute Kidney Injury (AKI) within England: the current state of the NHS AKI Master Patient Index and Registry   URL: https:/​/www.​thinkkidneys.nhs.uk/​aki/​wp-content/​uploads/​sites/​2/​2018/​03/​Reporting-the-rate-of-AKI-January-2018.​pdf [accessed 2018-03-30]
    28. Hsu C, Lee C, Su C, Wang Y, Chen H, Chuang J, et al. Incidence, outcomes, and risk factors of community-acquired and hospital-acquired acute kidney injury: a retrospective cohort study. Medicine 2016 May;95(19):e3674. [Medline]
    29. Thomas M, Blaine C, Dawnay A, Devonald M, Ftouh S, Laing C, et al. The definition of acute kidney injury and its use in practice. Kidney Int 2015 Jan;87(1):62-73 [FREE Full text] [CrossRef] [Medline]
    30. Connell A, Black G, Montgomery H, Martin P, Nightingale C, King D, et al. Implementation of a Digitally Enabled Care Pathway (Part 2): Qualitative Analysis of Experiences of Healthcare Professionals. J Med Internet Res 2019 Jul 31;21(7):e13143. [CrossRef]
    31. Park S, Baek S, Ahn S, Lee K, Hwang H, Ryu J, et al. Impact of electronic acute kidney injury (AKI) alerts with automated nephrologist consultation on detection and severity of AKI: a quality improvement study. Am J Kidney Dis 2017 Jul 25;71(1):9-19. [Medline]
    32. Ebah L, Hanumapura P, Waring D, Challiner R, Hayden K, Alexander J, et al. A multifaceted quality improvement programme to improve acute kidney injury care and outcomes in a large teaching hospital. BMJ Qual Improv Rep 2017 May 25;6(1). [Medline]
    33. Chandrasekar T, Sharma A, Tennent L, Wong C, Chamberlain P, Abraham KA. A whole system approach to improving mortality associated with acute kidney injury. QJM 2017 Oct 1;110(10):657-666. [CrossRef] [Medline]
    34. Al-Jaghbeer M, Dealmeida D, Bilderback A, Ambrosino R, Kellum JA. Clinical decision support for in-hospital AKI. J Am Soc Nephrol 2018 Dec;29(2):654-660 [FREE Full text] [CrossRef] [Medline]
    35. Abdelaziz TS, Lindenmeyer A, Baharani J, Mistry H, Sitch A, Temple RM, et al. Acute Kidney Outreach to Reduce Deterioration and Death (AKORDD) trial: the protocol for a large pilot study. BMJ Open 2016 Dec 19;6(8):e012253 [FREE Full text] [CrossRef] [Medline]
    36. Selby NM, Casula A, Lamming L, Mohammed M, Caskey F, Tackling AKI Investigators. Design and rationale of 'tackling acute kidney injury', a multicentre quality improvement study. Nephron 2016;134(3):200-204 [FREE Full text] [CrossRef] [Medline]
    37. Raine R, Fitzpatrick R, Barratt H, Bevan G, Black N, Boaden R, et al. National Institute for Health Research. Southampton (UK): NIHR Journals Library; 2016. Challenges, solutions and future directions in the evaluation of service innovations in health care and public health   URL: https://njl-admin.nihr.ac.uk/document/download/2004136
    38. Phillips D, Young O, Holmes J, Allen LA, Roberts G, Geen J, Welsh AKI steering group. Seasonal pattern of incidence and outcome of acute kidney injury: a national study of Welsh AKI electronic alerts. Int J Clin Pract 2017 Sep;71(9). [CrossRef] [Medline]
    39. Royal Free London NHS Foundation Trust. Royal Free London publishes audit into Streams app   URL: https://www.royalfree.nhs.uk/news-media/news/royal-free-london-publishes-audit-into-streams-app/ [accessed 2018-10-18]
    40. DeepMind Health. Scaling Streams with Google   URL: https://deepmind.com/blog/scaling-streams-google/ [accessed 2018-11-21]

    ‎
    AKI: acute kidney injury
    BGH: Barnet General Hospital
    DID: difference-in-differences
    ED: emergency department
    GLM: generalized linear model
    IMD: Index of Multiple Deprivation
    IQR: interquartile range
    ITU: intensive treatment unit
    NHS: National Health Service
    NHSEDA: NHS Early Detection Algorithm
    NIHR: National Institute for Health Research
    OR: odds ratio
    PARRT: Patient at Risk and Resuscitation Team
    PLICS: Payment Level Information and Costing System
    RFH: Royal Free Hospital
    RFLFT: Royal Free London NHS Foundation Trust
    RRT: renal replacement therapy
    UCL: University College London

    Edited by G Eysenbach; submitted 14.12.18; peer-reviewed by D O'Donoghue, X Ji, T Kothari; comments to author 16.01.19; revised version received 29.01.19; accepted 30.01.19; published 15.07.19

    Copyright

    ©Alistair Connell, Rosalind Raine, Peter Martin, Estela Capelas Barbosa, Stephen Morris, Claire Nightingale, Omid Sadeghi-Alavijeh, Dominic King, Alan Karthikesalingam, Cían Hughes, Trevor Back, Kareem Ayoub, Mustafa Suleyman, Gareth Jones, Jennifer Cross, Sarah Stanley, Mary Emerson, Charles Merrick, Geraint Rees, Hugh Montgomery, Christopher Laing. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 15.07.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.