Automating the Generation of Antimicrobial Resistance Surveillance Reports: Proof-of-Concept Study Involving Seven Hospitals in Seven Countries

Introduction

Generating and sharing antimicrobial resistance (AMR) surveillance reports are fundamental elements of actions against AMR infections at local, national, and international levels. Information on patterns of antimicrobial susceptibility is important to guide empiric choice of therapy, monitor resistance trends, and detect outbreaks of AMR infections at the local level [1-3]. Combining data and reports at the national level provides evidence to inform the implementation of national action plans, decide on resource allocation for interventions, and monitor the impact of those interventions [3-5]. The Review on AMR chaired by Jim O’Neill estimated that 700,000 global deaths are attributable to AMR infections each year (including bacterial infections and tuberculosis) [1,6], and they have an enormous global impact [7,8]. While this represents a very rough estimate subject to well-documented limitations [1,9,10], the report importantly highlighted the need for improved AMR surveillance.

Methods to analyze data and generate AMR surveillance reports are gradually being standardized worldwide [3,11-15]. Recently, the World Health Organization (WHO) launched the Global Antimicrobial Resistance Surveillance System (GLASS) with a defined protocol for AMR surveillance data collection for certain high-priority pathogens and resistance phenotypes [2,11,16]. In general, it is recommended that (1) repeat isolates of a given bacterial species from individual patients should be removed from the calculations and (2) data should be stratified by origin of infection (community or hospital) whenever possible [11,15]. A simple deduplication process is to include only the first isolate of a species per patient per specimen type per survey period in the report [3,11]. The origin of infection is defined by using specimen collection date, location type (inpatient or outpatient setting), and hospital admission date for inpatient isolates as a proxy to define where the infection was likely contracted (community or hospital) [3,11].

Even in areas where AMR surveillance data are available, there are many barriers to utilizing such data [17]. Many hospitals in low and middle-income countries (LMICs) lack the time and resources needed to analyze the data (particularly to deduplicate and validate the accuracy of the summary data), write reports, and release the data or reports [17]. There are open-access laboratory information systems (LISs) [18] and microbiology laboratory database software, including WHONET [19], that are useful for recording and analyzing the data, and generating figures to support the generation of AMR surveillance reports for hospitals in LMICs. Most of these systems have limited ability to generate summary reports for immediate use and be operated by nontechnical staff. In addition, to generate AMR surveillance reports stratified by the origin of infection, additional data on hospital admission are frequently needed [20-23]. This is because hospital admission dates are not generally collected in microbiology laboratory data files. In many hospital settings, microbiology and hospital admission data are held in separate computers or systems with restricted access. Even in high-income countries, many hospitals lack well-trained clinical microbiologists, epidemiologists, or data experts with adequate skills in statistical software (such as R, SAS, SPSS, and STATA) to merge and deduplicate data in the separated databases and generate reports stratified by the origin of infection.

Here, we developed an application termed the AutoMated tool for Antimicrobial resistance Surveillance System (AMASS), which can support a local hospital to independently analyze routinely collected electronic data and rapidly generate AMR surveillance reports. We tested the AMASS in seven hospitals in Cambodia, Lao People’s Democratic Republic, Myanmar, Nepal, Thailand, the United Kingdom, and Vietnam, and deposited the report from each hospital in an open-access platform.

Methods

Design of the Application

The tool operates by reading and processing the raw data files to automatically produce AMR surveillance reports. The application was designed to be open access, user friendly, and highly compatible with readily available data sets at local hospitals, and have high data security. To ensure the tool is fully open access and can run in a standalone inexpensive computer even without internet access, we built the application in R (version 3.6.2; R Project for Statistical Computing), which is a free software environment. We then gave the application a user-friendly interface, which only requires double clicking on the application file to run the automation without the need to understand the R program. We decided to include both R portable (version 3.4.3; R Project for Statistical Computing) and RStudio (version 1.1.423; RStudio, Inc) within the downloadable package so that the application can run without the need to install R or any program prior to running the application. We designed the application so that it reads raw data files in either CSV or Excel format as many hospitals in LMICs can export their data in one of these formats [17]. Hospitals that store microbiology data and hospital data in software, such as WHONET [19], can export data in CSV or Excel format. During the development process, we revised the R codes repeatedly to ensure that the application would produce easy-to-use and easy-to-share outputs. Table 1 documents the features that we focused on when designing the application [24,25].

A schematic overview of input data, data processing, statistical analysis, and output is shown in Multimedia Appendix 1. In brief, input data are microbiology data files (in CSV or Excel format) with or without hospital data files, which can be exported from WHONET or a LIS with data export capacity. The data processing and analysis algorithms, including data deduplication, were developed based on the WHO GLASS recommendations [2,11]. The reports were designed to have a format similar to that of the WHO GLASS 2018 report [14].

The application deduplicates the data by including only the first isolate per sample type per pathogen per survey period for each patient [11]. The application currently includes only blood specimens and the following eight pathogens: Acinetobacter spp., Escherichia coli, Enterococcus spp., Klebsiella pneumoniae, Salmonella spp., Staphylococcus aureus, Streptococcus pneumoniae, and Pseudomonas aeruginosa. Both Enterococcus spp. and P. aeruginosa were added on top of the six priority pathogens described by the WHO GLASS for bacteremia [2,26] because both pathogens are common causes of bacteremia [15,20], and were included in the 2015 global priority list of AMR bacteria [26]. The list of pathogen-antibiotic combinations was modified from the WHO GLASS (Multimedia Appendix 2). Infections are stratified into community-origin or hospital-origin infections using hospital admission dates and specimen collection dates, when available [2,11]. Patients with the first specimen culture positive for the pathogen taken in the outpatient setting or on the first or second day of hospitalization are classified as having community-origin infection [2,11]. Patients with the first specimen culture positive for the pathogen taken on hospital day three or later are classified as having hospital-origin infection [2,11]. Alternatively, in cases where users have data on the origin of infection assigned by the attending physician or infection control team of the hospital, the AMASS can instead use those categorizations to stratify the infection into community-origin or hospital-origin infection. Prevalence and incidence rates are estimated based on the recommendations of the WHO GLASS (Multimedia Appendix 3) [2,11]. An additional report on mortality involving AMR and non-AMR infections is generated when mortality data are available in the hospital admission data file. The term “mortality involving AMR and antimicrobial-susceptible infections” is used because the mortality reported in the hospital admission data represents mortality from any cause, not necessarily attributable to AMR [1]. This measure of mortality includes deaths caused or contributed by other underlying and intermediate causes. Therefore, the term “involving AMR infections” is used, in accordance with the term used by the UK Office for National Statistics [27,28]. The AMASS used the Wilson method to estimate the confidence intervals for proportions.

Example Data Sets

Two example data sets are provided in the downloadable package of the AMASS application. The first example data set is the open-access demonstration file from WHONET [19]. The second example data set is a synthetic data set generated for the AMASS. The second example data set was created to represent a large data set from a 1000-bed hospital, containing both microbiology and hospital admission data. A detailed description on how the second example data set was generated is provided in Multimedia Appendix 4. In brief, two synthetic data files (microbiology_data.xls and hospital_admission_data.xls files) were generated based on the summary data in the AMR surveillance report from 2015 of Sunpasitthiprasong Hospital, Ubon Ratchathani, Thailand. Variables in the microbiology data file include hospital number, specimen type, specimen collection date, culture result, and antibiotic susceptibility testing result, and each row contains information for each specimen. Variables in the hospital admission data file include hospital number, age, sex, admission date, discharge date, and in-hospital discharge outcome. All data were randomly generated using STATA 15.1 (StataCorp).

Testing the AMASS Using Hospital Data

The following seven hospitals participated in the study: Angkor Hospital for Children in Cambodia, Mahosot Hospital in Lao People’s Democratic Republic, North Okkalapa General and Teaching Hospital in Myanmar, Patan Hospital in Nepal, Sunpasitthiprasong Hospital in Thailand, St Thomas’ Hospital in the United Kingdom, and Hospital for Tropical Diseases in Vietnam. The hospitals were selected because microbiology data are collected routinely, and they have prior experience in data quality controls. Moreover, the hospitals varied in the LIS used for data storage and are good examples to demonstrate the practicality and usability of the AMASS in different settings, to a wider audience. The study was approved by the Oxford Tropical Research Ethics Committee, University of Oxford, and local Ethics Committees. CL corresponded with the participating hospitals and demonstrated how the application operates using the example data files. The local hospital staff operated the application using their local data by themselves. All microbiology and hospital data were stored independently within their hospital computers under their local data protection standards. Automatically generated reports and anonymous summary data from each hospital are publicly available and deposited in data repositories with permission from each hospital.

Results

Overview of the AMASS

The AMASS application has been developed to support clinical microbiology laboratories to automatically analyze hospital local data files (in CSV or Excel format) and generate AMR surveillance reports (in PDF and CSV formats) promptly. Six steps are followed to generate the AMR surveillance reports (Figure 1 and Multimedia Appendix 5). First, download the AMASS package from the website [24]. Second, obtain the routinely collected raw microbiology data file and, if available, hospital admission data file, and then, save the data files in the folder of the AMASS application. Third, configure data dictionaries. Two data dictionaries are provided to accommodate different ways of naming variables (eg, sex and gender) and data values (eg, M and F, or male and female). The functioning of the data dictionaries is described in more detail in the “Illustration on How to Use the AMASS” section. Fourth, double click on the AMASS.bat file to run the application. Fifth, review and validate the AMR surveillance reports (generated in PDF format) and the anonymous summary data (generated in CSV format). Sixth, share the reports within the hospital, especially with the local infection control team. The reports and anonymous summary data contain no patient identifiers. Therefore, users may share the reports and anonymous summary data with national and international organizations or make the reports and anonymous summary data open access. The key features of the AMASS are listed in Table 1.

‎View this figure

Illustration on How to Use the AMASS

Input Requirements

To align with formats of commonly exported microbiology data and hospital admission data, the AMASS reads data files in both CSV and Excel formats. The microbiology data need to be in a wide format, meaning that each row should contain data from a single clinical isolate. The key variables required for the microbiology data file include patient identifier, specimen collection date, specimen type, culture result, and antimicrobial susceptibility test interpretation per antibiotic. The hospital admission data file is optional but, if available, also needs to be in a wide format. The key variables needed for the hospital admission data file include patient identifier, admission date, gender, age, and, if possible, in-hospital discharge outcome. The current version of the AMASS uses the Gregorian calendar and requires the date to be in the order of day, month, and year in any format (ie, either text [English] or numeric).

There are two data dictionary files provided for the users to accommodate different ways of naming data variables and data values (Figure 2). The first data dictionary file (dictionary_for_microbiology_data.xls) is for the microbiology data file. For example, the AMASS uses the variable name “hospital_number” as a patient identifier (Row 3, Column A of the data dictionary file). In cases where the raw microbiology data file uses a different name for the patient identifier (eg. hn), users would need to fill “hn” in the data dictionary file (Row 3, Column B of the data dictionary file). This allows the AMASS application to know that the variable “hn” of the raw microbiology data file is the patient identifier (ie, “hospital_number”). The second data dictionary file (dictionary_for_hospital_admission_data.xls) is for the hospital admission data file, which is to be used likewise. Multimedia Appendix 6 presents a step-by-step tutorial on how to use and configure the data dictionaries.

‎View this figure

Outputs Generated by the AMASS

The AMR surveillance reports generated from the AMASS are illustrated using the two open-access example data sets provided within the download package. Multimedia Appendicies 7 and 8 contain AMR surveillance reports generated from the first and second example data sets, respectively. Multimedia Appendix 5 illustrates how to test the AMASS using example data sets in a step-by-step fashion.

In short, the automatically generated report on AMR surveillance contains the following six sections (Figure 3): (1) data overview; (2) an isolate-based report; (3) an isolate-based report with stratification by origin of infection; (4) a sample-based report; (5) a sample-based report without stratification by infection; and (6) mortality involving AMR and antimicrobial-susceptible infections.

The AMASS uses a tier-based approach. When only the microbiology data file with the results of culture positive samples is available, only section one and two are automatically generated for users (as shown in Multimedia Appendix 7, generated from the first example data set). Section three is generated only when data on admission dates are available. This is because these data are required for stratification by the origin of infection. Section four is generated only when data of culture negative specimens (no microbial growth) are available in the microbiology data file. This is because these data are required for the sample-based approach. Section five is section four stratified by the origin of infection and is generated if admission date data are also available. Section six is generated only when mortality data are available (as shown in Multimedia Appendix 8, generated from the second example data set).

The AMASS also generates two log files. The first log file (generated in PDF format) is for users to validate the input data used by the AMASS to generate the AMR surveillance report. It contains information such as the total number of records analyzed, age distribution, number of missing values, and total number of isolates per organism in the raw microbiology data file. The second log file (generated in plain text format) could be used for consultation with R users, statisticians, or the AMASS development team in case of any technical issue when running the AMASS.

‎View this figure

Testing the AMASS Using Hospital Data

The AMASS was tested in seven hospitals in seven countries (Figure 4). The hospitals varied in data availability, data structure, naming of the variables, and definitions for data values (Multimedia Appendix 9). Overall, the proportions of patients having Escherichia coli bacteremia caused by third-generation cephalosporin-resistant isolates ranged from 19% to 85%. The incidence rates of Escherichia coli bacteremia caused by third-generation cephalosporin-resistant isolates ranged from 283 to 2737 per 100,000 tested patients among participating hospitals with available data on negative cultures.

‎View this figure

All participating hospitals deposited their output files and their data dictionary files at figshare [29-35]. Angkor Hospital for Children, North Okkalapa General and Teaching Hospital, and Sunpasitthiprasong Hospital had all six sections of the AMR surveillance report, as microbiology and hospital admission data, data on negative cultures, and in-hospital discharge outcome data were available. Mahosot Hospital and Patan Hospital had sections one and two of the AMR surveillance report, as only microbiology data on positive cultures were available to test in the AMASS. St Thomas’ Hospital had sections one, two, three, and six of the AMR surveillance report because negative culture data were not available to test in the AMASS. Hospital for Tropical Diseases had sections one, two, and four available, as hospital admission data were not available to test in the AMASS.

Data storage systems varied across the hospitals. Angkor Hospital for Children used the ACORN LIS that is based on the Microsoft Access program [36]. Mahosot Hospital used a local LIS that is also based on the Microsoft Access program [37]. Sunpasitthiprasong Hospital used the MLAB program [38]. St Thomas’ Hospital used ICNET [39], which is a commercial clinical surveillance software package. North Okkalapa General and Teaching Hospital used WHONET 2019 (modernized version of WHONET 5.6) [19]. Patan Hospital and Hospital for Tropical Diseases used an in-house LIS based on the MySQL system. Inputting data dictionary files for the first time took about 1 to 3 hours. However, the data dictionary files could be stored and reused when the raw microbiology data file or hospital admission data file was revised or updated. North Okkalapa General and Teaching Hospital used the data dictionary file from the example data set (in the Example_Dataset_1_WHONET folder), which was generated to comply with WHONET exported data (in .XLSX format; Multimedia Appendix 10 illustrates how microbiology data were exported from WHONET 5.6). This saved the time and effort needed to complete the data dictionaries for the hospital.

We found that the AMASS took about 1 to 3 minutes to run and automatically generate an AMR surveillance report using local data and local hospital computers. The AMASS works on Microsoft Windows 10 and with data containing a non-English (Latin and non-Latin characters) language. For example, “blood_specimen” was recorded as “” in the raw microbiology excel file at Hospital for Tropical Diseases, Vietnam. As another example, “male” was recorded as “” and “female” was recorded as “” in the raw hospital admission excel file at Sunpasitthiprasong Hospital, Thailand. By completing data dictionary files, users could use the AMASS to analyze the data and generate the AMR surveillance report even though the raw data were recorded in a local language.

All participating hospitals had previous experience in data verification and in AMR surveillance report validation. All participating hospitals were familiar with the recommendations of the Clinical and Laboratory Standard Institute (CLSI) and European Committee on Antimicrobial Susceptibility Testing (EUCAST) on antimicrobial susceptibility testing (AST) data verification. Additionally, all participating hospitals were familiar with validating summary data by comparing the summary data automatically generated by the AMASS with manual calculations.

Users’ suggestions for improvement were received when the investigation team visited the participating hospitals. The users expressed a need for including other clinical specimen types (eg, urine samples) and a function to support data verification or provide a list of isolates with unusual AST profiles. This is because a data verification function is not currently available in many in-house LISs. Users also expressed the need for a way to import multiple data files and different data formats, particularly the long format (ie, each row represents an AST result obtained from each antibiotic against each isolate). This is because the data exported from automated machines for AST testing are commonly in the long format and it takes time to integrate the data from the automated AST machines and from disk diffusion AST results. Users also noted that the requirement to manually export data from the existing LIS can be tedious and time consuming to perform on a regular basis, and automation that can bypass the data extraction step would be useful.

Discussion

We developed an open-access, offline, and easy-to-use application, AMASS, which allows hospitals, especially in LMICs, to automatically generate AMR surveillance reports from routinely collected electronic microbiology data. Seven hospitals used the AMASS, and shared their reports and summary data files by depositing them in an open-access data repository website. We propose that routine generation and sharing of AMR surveillance reports (ie, cumulative antimicrobial susceptibility reports or antibiograms) in open-access data repositories from hospitals with a microbiology laboratory should be supported worldwide [40].

The AMASS empowers data sharing by reducing the time and effort needed to prepare the summary data and reports without increasing the risks of breaching individual patient data confidentiality. This is because the AMASS can analyze the data and generate the reports promptly without needing to transfer raw data files to any party outside of the hospital. The autogenerated AMR surveillance reports are in PDF format, which can be reviewed and shared locally and immediately. This is important to improve the understanding of the local AMR burden and to inform local patient management. The PDF report and summary CSV files contain no individual patient-level information and can be shared with national and international organizations to support action plans on AMR. Any attempt to compare AMR surveillance reports among different hospitals should be done cautiously, as factors, including the type of hospital, blood culture utilization rates and practices, and patient characteristics, may influence the estimated prevalence and incidence rates of AMR infections [12-14]. An AMR surveillance report should clearly present the denominators used to calculate incidence rates. Selecting an appropriate denominator that represents the local setting is important for estimating the incidence rate of AMR, but remains challenging [41]. The AMASS calculates the total numbers of tested populations and uses them as denominators in section four and section five of the output report based on the WHO GLASS recommendations, and these denominators will not be available without data on negative cultures.

The AMASS uses currently recommended analytical approaches to generate the summary data and reports. The entire code is open access, verifiable, and modifiable. The AMASS can also be expanded and tailored based on local requirements. For instance, notifiable bacterial infections that are important for a local setting can be included by revising the R code provided in the AMASS package. Analytical methods can also be constantly updated and improved over time.

The AMASS is an add-on automatized report-generating tool that can be easily used, even by hospital users with limited information technology skills. It does not replace WHONET, LISs, quality assurance programs, or antimicrobial surveillance systems (including the WHO GLASS). The AMASS therefore differs from the “Macros and Excel Reports” function of WHONET, which allows users to regularly generate reports on screen and export data as Excel files using macro language (ie, a series of written study parameters generated automatically through the WHONET user interface grouped together) [19]. In contrast, the AMASS is designed to automatically compile summary results of multiple organisms into a single report. The current version of the AMASS does not support quality improvement or alert individuals to unexpected antibiogram results, which WHONET is capable of doing [19]. Thus, while WHONET is an appropriate choice for the main software package for microbiology laboratories in LMICs to capture and store microbiology data and to export data for the WHO GLASS [19], the AMASS provides important additional functionality [19].

Verifying individual AST results and validating summary data in the reports (including those generated by the AMASS) remain major barriers to the generation of accurate surveillance data on AMR. Only verified AST results on each patient’s isolate should be used in the data analysis [12-14]. WHONET and few LISs include functions that automatically check the results to ensure that they appear reasonable or alert users to confirm unusual results (eg, amikacin resistance coexisting with gentamicin and tobramycin susceptibility in E. coli is unusual, and data on such cases should be verified) [12-14,19]. In addition, it is important to validate the calculations of any analytical software used to generate the summary data and reports. We recommended that users of the AMASS perform manual validation (such as printing a line listing of all isolates of the species to cross-check with the reports), particularly when the program is used for the first time [12-14]. Users should also understand that the precision of the estimate is based on the sample size, which is illustrated by estimated confidence intervals. In addition, the representativeness of the summary cumulative AST data could be impacted by the sampling strategy used, particularly if blood culture is mainly performed for patients with treatment failure or prolonged hospitalization [11,12,14]. Nonetheless, generating AMR surveillance reports from existing microbiology databases could be the first step for hospitals in LMICs to understand and validate their own data, and support quality control programs at the local level.

Local hospitals must not be discredited because of the statistics in any AMR surveillance report, as possible criticism could be a barrier to data sharing [42]. Negative criticism of the data shared by local hospitals needs to be avoided. It is inevitable for errors to occur in any data set, especially in settings with limited resources and experience on quality assurance and data validation. Tools for data validation should be provided to local hospitals, and the statistics in an AMR report should act as a guide on the direction of improving data quality control, data validation, and infection prevention and control. Credit should be given to the hospitals that share their local AMR data and reports on an open-access data repository platform. Data repository platforms, such as figshare, allow flexibility on updating the data and revising the AMR surveillance reports with appropriate version controls by local hospitals. Publishing open-access AMR data on data repository platforms provides a digital object identifier (DOI) that should be used for data citation [43].

The AMASS has a number of limitations. First, the AMASS is not applicable for hospitals that only store data in paper forms. Second, the AMASS cannot work with raw microbiology data files that are not in a wide format or combine many files in multiple formats. For example, raw microbiology data file where each row contains data of each antibiotic susceptibility result for one single specimen. Third, the current version of the AMASS can only analyze microbiology data that include antimicrobial susceptibility test interpretive categories (susceptible, intermediate, and resistant) based on guidelines that the local hospital uses. Fourth, the AMASS cannot automatically validate the reliability of the data that are imported into the application and used for analysis. Data verification and quality checks will be included in future versions of the application. Fifth, the AMASS was tested with only few data sets that included non-English languages, and further testing on data with other languages may be needed. The current version of the AMASS has been tested on Windows 10 and 7 and may not work on other operating systems. Sixth, the current version of the AMASS cannot provide options to export the AMR surveillance report in different formats (ie, in document or text format). However, users can reuse the summary statistics, which are saved in CSV format under the ResultData folder. Finally, feedback from participating hospitals was not formally evaluated. Studies evaluating the barriers, facilitators, and utilities of supporting applications, such as the AMASS, to generate AMR surveillance reports are needed.

In conclusion, the AMASS can be used to support hospitals with microbiology laboratories in analyzing routinely collected data and generating reports with minimal resources and expertise required. This may empower individual hospitals to contribute to the understanding of and actions on AMR in local settings and maximize the utility of local data.