Long-Term Impact of a Smartphone App on Prescriber Adherence to Antibiotic Guidelines for Adult Patients With Community-Acquired Pneumonia: Interrupted Time-Series Study

Introduction

Antibiotic stewardship programs in hospitals and community clinics strive to improve rates of appropriate antibiotic prescribing through a wide variety of methods (from clinical decision support tools to educational sessions) both to optimize the treatment of patients with bacterial infections and to reduce inappropriate antibiotic prescribing [1]. Greater adherence to antibiotic guidelines (ie, prescription of antibiotics consistent with guidelines) is associated with better treatment outcomes and reduced antibiotic resistance [1,2], yet rates of adherence remain low [3-7]. Despite the ubiquity and promise of mobile health (mHealth) platforms like smartphone apps to overcome some of the causes of low adherence, such as limited access to guidelines, the long-term impact of mHealth apps on guideline adherence remains unclear [8].

A small number of studies have suggested that apps displaying antibiotic guidelines improve antibiotic prescribing behavior in the short term [8], while the only study to have measured adherence beyond 12 months post app implementation suggested that any improvements are not necessarily sustained [9]. Therefore, the long-term influence of such apps requires further investigation, with implications for their cost-benefit analysis and long-term utility in antibiotic stewardship programs. In 2016, we developed an antibiotic guidelines app, “SCRIPT,” at Auckland City Hospital (ACH) in New Zealand, which displayed antibiotic guidelines for community-acquired pneumonia (CAP) and urinary tract infection to prescribing clinicians on their smartphones and observed improvement in adherence for patients with CAP but not for patients with urinary tract infection [6]. However, this study was limited by only 3 months of follow-up and a short lead-in time for app adoption. Since then, SCRIPT has increased in popularity among local prescribers and become a standard part of the prescribing repertoire, evidenced by 5600 unique users accounting for 600,000 app sessions in 2020 and over 700,000 in 2021 [10]. The app is freely available (as “SCRIPT ADHB” via Google Play and App Store) and sports a simple, user-friendly interface (Figure 1). Given SCRIPT’s widespread adoption in ACH, we aimed to assess its long-term impact on prescriber adherence to antibiotic guidelines in patients with CAP at 3, 12, and 24 months after the SCRIPT guidelines were made available.

‎

Methods

Aims and Study Setting

We performed an interrupted time-series study to test the hypothesis that the provision of the SCRIPT app would increase prescriber adherence to antibiotic guidelines for hospitalized adult patients with CAP. We further hypothesized that adherence to antibiotic guidelines would be higher in cases with chest x-ray evidence of CAP than in cases without chest x-ray evidence of CAP (because the diagnosis of CAP is questionable in cases without chest x-ray evidence [11,12]). The initial impact of SCRIPT’s implementation on adherence to local ACH antibiotic guidelines has been described previously [6]. In the first 2 weeks of the intervention period, educational sessions, posters, and intranet advertisements were employed to socialize the app and facilitate its uptake by ACH clinicians. Thereafter, the app was promoted periodically in newsletters and posters and at each orientation for new rotations of junior doctors.

ACH has a multifaceted approach to antimicrobial stewardship (including formulary management, regular audit and feedback, expert consultation services, and surveillance of antibiotic use), which continued unaltered throughout the study period. The hospital antibiotic guidelines remained on the hospital intranet. No other interventions impacting CAP management were introduced during the study period.

Study Cohort

We retrospectively collected data during 4 periods, as follows: “baseline” pre-app implementation (January 1 to May 31, 2016); “immediate” post-app implementation (June 1 to August 31, 2016); 12-month post-app implementation (June 1 to October 31, 2017); and 24-month post-app implementation (June 1 to October 31, 2018).

Adult patients (aged ≥18 years) admitted to ACH for ≥4 hours with a discharge diagnosis of CAP (International Classification of Diseases-10 codes: J10-18 and J22) were included. Patients were excluded if they were not diagnosed with CAP during the first 24 hours of admission; had incorrectly coded diagnoses (eg, “empyema”); or were transferred from another secondary or tertiary care facility where antibiotics had been administered.

All CAP cases during each period were identified. We used Microsoft Excel’s random number generator to randomly select ≥200 cases per period (200 at “baseline”; 237 in the “immediate” post-app period; 200 at 12 months; and 200 at 24 months). We calculated that inclusion of these case numbers would achieve 90% power to detect an absolute 15% increase in guideline adherence (=.05) [6].

All patients had a chest x-ray at admission to detect radiological features of consolidation, defined as 1 or more opacities in the lung fields consistent with the diagnosis of pneumonia.

Data Collection and Definitions

Electronic health record data were collected using REDCap (version 6.5.15; Vanderbilt University) to record demographic (eg, age, sex, and ethnicity) and clinical data (eg, admission date; diagnostic impression at admission; vital signs at admission—documentation of confusion in the patient, respiratory rate, systolic and diastolic blood pressures; urea; and presence of consolidation on chest x-ray at admission, as reported by a radiologist) as well as antibiotics prescribed (eg, drug name, route, and duration) during the first 24 hours post admission.

Adherence was defined as prescription of antibiotic(s), including dose(s) and route(s) of administration, according to local guidelines, during the first 24 hours post admission. The ACH antibiotic guidelines for CAP vary by the CURB-65 pneumonia severity score, where a point is given for each of the prognostic features (C: confusion, U: increased serum urea concentration, R: respiratory rate ≥30 breaths/min, B: systolic blood pressure <90 mmHg or diastolic blood pressure ≤60 mmHg, and 65: age ≥65 years). Cases with a total CURB-65 score of 0-1 were considered to be at low risk (<10%) of mortality; those with a score of 2 were at intermediate risk (10%-20%) of mortality; and a score of 3-5 indicated high risk (20%-60%) of mortality [13]. In cases whose serum urea concentration had not been measured, CRB-65 scores were calculated. CRB-65 is a validated alternative to CURB-65, shown to be predictive of mortality in patients hospitalized with pneumonia [14]. A CRB-65 score of 0 would equate to a CURB-65 score of 0 at best and 1 at worst; thus, we elected to use CURB-65 score ranges (0-1, 1-2, 2-3, and 3-5). If the attending clinicians had not documented the CURB-65 score in the clinical records, we calculated the patient’s CURB-65 score using relevant data available to the clinicians when selecting antibiotic management. Antibiotic guideline adherence was assessed according to the actual or highest possible CURB-65 score.

Other antibiotic(s), prescribed in addition to guideline-adherent antibiotic(s), were considered unnecessary additional antibiotics. Undertreatment was defined as prescription of an inappropriately narrow-spectrum regimen (eg, prescription of amoxicillin alone for severe CAP).

These definitions were applied by 2 physicians (CHY and SRR) and an infectious diseases specialist pharmacist (EJD) based on the assumption that the patient had CAP, regardless of the presence of pulmonary consolidation on chest x-ray (a defining characteristic of CAP, the absence of which does not preclude the diagnosis of CAP) [15].

Analysis

Statistical analyses were performed using R (version 4.0.3; The R Core Team). Rates of adherence, use of unnecessary additional antibiotics, and undertreatment were compared between study periods and between cases with or without pulmonary consolidation on the admission chest x-ray (based on the reporting radiologist’s assessment), using Pearson chi-square test or Fisher exact test (significance level: α=.05). One case, in the immediate follow-up group, did not have a chest x-ray and was excluded from analyses that compared patients with or without pulmonary consolidation.

Ethics Approval

All analyses were performed in accordance with the study protocol for which ethics approval was granted (New Zealand Health and Disabilities Ethics Committee reference number: 16/STH/6).

Results

Demographic And Clinical Features

The sex, median ages, and ethnicities of the patients in the 4 cohorts were broadly similar (Table 1). The proportions of patients with consolidation on chest x-ray (an initial diagnostic impression of pneumonia) and prescriber-documented CURB-65 scores were higher in the 12-month and 24-month cohorts compared to the baseline cohort. In all 4 cohorts, most patients with consolidation on chest x-ray (43/63, 68% at baseline; 54/77, 70% in the immediate post-app period; 65/92, 71% at 12 months; and 69/102, 68% at 24 months) had an initial diagnostic impression of “pneumonia.” By contrast, in all 4 cohorts, a minority of patients without consolidation on chest x-ray (25/137, 18% at baseline; 14/159, 9% in the immediate post-app period; 31/108, 29% at 12 months; 24/98, 24% at 24 months) had an initial diagnostic impression of “pneumonia” (P<.001).

Overall Antibiotic Guideline Adherence

Compared with the baseline cohort (46/200, 23%), there was a nonsignificant increase in prescriber adherence to the antibiotic guideline in the immediate cohort (73/237, 31%) but a significant increase in adherence in the 12-month cohort (69/200, 34%; P=.01), which was not sustained in the 24-month cohort (62/200, 31%; Table 1).

Antibiotic Guideline Adherence in Patients With Pulmonary Consolidation

For patients with consolidation on chest x-ray, antibiotic guideline adherence increased from 14% (9/63) in the baseline cohort to 30% (23/77) in the immediate cohort—a change that was sustained in the 12-month cohort (32/92, 35%) and in the 24-month cohort (32/102, 31%; P=.04; Table 2). There were no significant differences between cohorts in the prescription of unnecessary additional antibiotics or in undertreatment.

Discussion

In patients with CAP and pulmonary consolidation on chest x-ray, there was a sustained improvement in guideline adherence. However, in patients with CAP without consolidation, where the most common diagnostic impression was “viral illness” or “lower respiratory tract infections (unspecified),” guideline adherence was not sustained. The sustained improvement in adherence to the guidelines for treatment of CAP in patients with consolidation on chest x-ray indicates that clinicians were adapting their use of the guideline to increase their use of it in those patients for whom they thought the guideline was most appropriate. This evolution of prescriber use of the guideline over time is an encouraging feature, particularly given the absence of other initiatives to improve prescribing for CAP, suggesting that prescribers were intellectually engaging with the guideline. An appropriate response by those responsible for maintaining and updating the guideline might be to include the presence or absence of consolidation on the chest x-ray as a decision point in the treatment algorithm.

The only other published study of the long-term impact of an antibiotic guidelines app on prescriber adherence was performed in 3 hospitals in west London, where baseline rates of adherence were high (75%-90%) [9]. The introduction of a smartphone app resulted in a significant increase in the rate of adherence for surgical patients, sustained at 24 months. However, in medical patients, a nonsignificant increase in the rate of adherence was followed by a gradual decline toward preintervention levels. In our study of medical patients with CAP, preintervention rates of adherence were low (46/200, 23%) but improved significantly to 34% (69/200) at 12 months post app implementation, before then declining to 31% (62/200). Our findings are broadly consistent with those of Charani et al [9], who found an initial increase followed by a subsequent decline in guideline adherence in medical patients. It should be noted that adherence to guidelines in our study required that the antibiotic, dose, and mode of administration be as stated in our guidelines; however, the definition used in the Charani et al [9] study required that only the antibiotic were that stated in their guidelines and did not require the dose and mode of administration to be the same as those in the guidelines.

Although very high uptake and use of the app at ACH (>1000 new downloads each year, over half of which are by junior doctors) enabled this real-world evidence study, there were no data directly matching the use of the SCRIPT app by the clinicians whose antibiotic prescriptions were analyzed in this study, that is, we were not able to measure the direct influence of using the app on individual cases of antibiotic prescription but rather the average net effect of making such an app available. Other limitations included the unmeasured impact of team-based decisions (vs individual decisions) for antibiotic prescriptions and of junior doctors changing clinical jobs every few months at ACH, moving to or from other hospitals, which would periodically and variably diminish the proportions of doctors using the app at ACH. We were not able to assess the app’s impact relative to other antibiotic stewardship methods nor to other variables that may influence guideline adherence, such as the prescriber’s level of seniority, where they had previously worked, their specialty, and patient-related factors like comorbidity and illness acuity.

A range of technological advances, including antibiotic guidelines apps and computerized decision support systems appear to offer opportunities to dramatically improve adherence to prescriber guidelines. However, as with our study, it is rare that such advances provide a silver bullet for the widespread, recalcitrant problem of low adherence to antimicrobial prescribing guidelines. Instead, it is common for such advances to provide modest improvements, commonly of a 10%-20% absolute improvement in guideline adherence, when a 30%-50% absolute increase would have been required to achieve adherence rates above 90% [16-19]. Although mHealth solutions have been perceived to be convenient and effective in improving guideline adherence, their high cost would be more justified should their impact be more long-term; this is especially pertinent in multimodal antibiotic stewardship programs, where there would be further opportunity costs.

Causes of failure to achieve large changes in antibiotic guideline adherence include within-team dynamics that may contribute to lack of support for changes in prescriber behavior. Junior clinicians, who write almost all prescriptions, may be more influenced by the entrenched opinions of their senior colleagues than by the advice contained in a guideline [20,21]. Other causes of low adherence may pertain to app-related factors like usability, acceptability, and app fatigue, although SCRIPT was designed using state-of-the-art co-design approaches through interactions between designers and end-user stakeholders [22,23]. Moreover, rates of SCRIPT use at Auckland Hospital have steadily increased rather than declining, suggesting that the app has high usability with no evidence of app fatigue. SCRIPT can only provide guidelines, not actively reinforce them. e-Prescribing may be able to address this gap and could be the subject of future studies in antibiotic guideline adherence.

Overall, our results suggest that a highly used antibiotic guidelines app can help to increase overall rates of prescriber adherence, especially in those patients with the strongest evidence that they fall into the diagnostic group the treatment advice is intended for and in those patients with more severe diseases. Sustaining increased rates of adherence likely requires refinement of the app algorithms in response to evidence that prescribers are selective in their adherence to guidelines and may respond to clinical features that are not included in the app algorithms. As with all innovations, a continuous process of development, testing, analysis, and modification is necessary to achieve the best results.