This paper is in the following e-collection/theme issue:

Viewpoints and Perspectives (514) Safety and Error Prevention in Health (202)

Published on in Vol 21, No 6 (2019): June

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/13576, first published .
Key Issues in the Development of an Evidence-Based Stratified Surgical Patient Safety Improvement Information System: Experience From a Multicenter Surgical Safety Program

Key Issues in the Development of an Evidence-Based Stratified Surgical Patient Safety Improvement Information System: Experience From a Multicenter Surgical Safety Program

Key Issues in the Development of an Evidence-Based Stratified Surgical Patient Safety Improvement Information System: Experience From a Multicenter Surgical Safety Program

Authors of this article:

Xiaochu Yu1 Author Orcid Image ;   Wei Han2 Author Orcid Image ;   Jingmei Jiang2 Author Orcid Image ;   Yipeng Wang1 Author Orcid Image ;   Shijie Xin3 Author Orcid Image ;   Shizheng Wu4 Author Orcid Image ;   Hong Sun5 Author Orcid Image ;   Zixing Wang2 Author Orcid Image ;   Yupei Zhao1 Author Orcid Image
Article Authors Cited by (4) Tweetations (3) Metrics

    Viewpoint

    1Peking Union Medical College Hospital, Chinese Academy of Medical Sciences, Beijing, China

    2Department of Epidemiology and Biostatistics, Institute of Basic Medicine Sciences, Chinese Academy of Medical Science, Beijing, China

    3The First Hospital of China Medical University, Shenyang, China

    4Qinghai Provincial People’s Hospital, Xining, China

    5Xiangya Hospital, Central South University, Changsha, China

    *these authors contributed equally

    Corresponding Author:

    Yupei Zhao, MD

    Peking Union Medical College Hospital, Chinese Academy of Medical Sciences

    No 1 Shuaifuyuan Wangfujing Dongcheng District

    Beijing, 100730

    China

    Phone: 86 010 69155789

    Email: zhao8028@263.net


    Surgery is still far from being completely safe and reliable. Surgical safety has, therefore, been the focus of considerable attention over the last few decades, and there are a growing number of national drives to improve it. There are also a number of large surgical complication reporting systems and system-based interventions, both of which have made remarkable progress in the past two decades. These systems, however, have either mainly focused on reporting complications and played a limited role in guiding practice or have provided nonselective interventions to all patients, perhaps imposing unnecessary burdens on frontline medical staff. We have, therefore, developed an evidence-based stratified surgical safety information system based on a multicenter surgical safety improvement program. This study discusses some critical issues in the process of developing this information system, including (1) decisions about data gathering, (2) establishing and sharing knowledge, (3) developing functions for the system, (4) system implementation, and (5) evaluation and continuous improvement. Using examples drawn from the surgical safety improvement program, we have shown how this type of system can be fitted into day-to-day clinical practice and how it can guide medical practice by incorporating inherent patient-related risk and providing tailored interventions for patients with different levels of risk. We concluded that multidisciplinary collaboration, involving experts in health care (including senior staff in surgery, nursing, and anesthesia), data science, health care management, and health information technology, can help build an evidence-based stratified surgical patient safety improvement system. This can provide an information-intensified surgical safety learning platform and, therefore, benefit surgical patients by delivering tailored interventions and an integrated workflow.

    J Med Internet Res 2019;21(6):e13576

    doi:10.2196/13576

    Keywords

    surgery; patient safety; information system; risk factors; evidence-based practice 


    Globally, each year, more than 230 million operations are performed and at least 7 million patients develop significant surgical complications, including 1 million perioperative deaths (at least half of which are preventable) [1,2]. Preventing harm to patients and improving the safety of surgical patients has, therefore, drawn considerable attention over the last few decades, and there are growing national drives to improve surgical patients’ safety [3]. There are also abundant opportunities for informatics-based improvements in perioperative care linked to the rapid development of information technology use in medical care. Information systems, such as computerized physician order entry, automated dispensing, barcode medication administration, electronic medication reconciliation, and personal health records, are playing an increasingly important role in enhancing patient safety by reducing medication errors [4,5]. Up to 50.2% (470/936) of medical errors can be avoided through the use of information systems [6]. However, most of these systems are not designed specifically for surgery. Their original intention was to regulate clinicians’ daily practice and allow health care providers to carry out routine jobs effectively, preventing potential errors [5]. Patient safety outcomes, which are crucial indicators for evaluating and improving surgical safety practice, cannot usually be obtained reliably through these systems. Extensive and carefully planned specialized information platforms for surgical sectors are, therefore, needed to collect data on patient safety [7]. Several nationwide surgical complication reporting and learning systems have, therefore, been developed in the past two decades, such as the United Kingdom’s National Reporting and Learning System [8]. This was established in late 2003 as a voluntary scheme for reporting patient safety incidents, to support learning from these incidents. Another large-scale Web-based information platform is the American College of Surgeons National Surgical Quality Improvement Program, which dates back to the 1980s and now incorporates hundreds of hospitals across the United States. It was developed to gauge the quality of surgical programs across different hospitals. The primary function of both of these systems is surgical incident reporting. Measuring incidence alone, however, is not enough to guide routine clinical safety behavior and enhance safety. A number of system-based interventions have, therefore, also been developed, mainly focusing on regulating clinical behavior and following the publication of To Err is Human [9]. These included the Surgical Patient Safety System (SURPASS) checklist, which requires 11 forms (nearly 100 items) to be completed and documented by providers for each individual undergoing surgery [10,11]. These intervention strategies regulate the daily clinical practices of health care staff and thus improve patient safety, but they impose a heavy workload in complex clinical settings. This could increase fatigue and undermine the adoption of and compliance with these systems by frontline health care staff [12]. The number of successful system implementations is, therefore, relatively small, with conflicting findings on their effect on patient safety. This tends to lead to skepticism about the true effectiveness of these systems and emphasizes the necessity of developing a patient safety system with high implementation efficiency and low operational complexity [13,14].

    Surgical operations are complex procedures. The perioperative care process is a unique and challenging environment that requires close collaboration among surgeons, anesthetists, and nurses. Medical staff can also encounter sophisticated patient pathophysiological conditions. The length of patients’ stay in the hospital is relatively short in surgical departments, which has posed a significant challenge for surgeons in capturing the key factors influencing surgical outcomes and the timely transfer of key safety information to other members of the surgical team [15]. An information system capable of extracting knowledge from high volume and multi-sourced clinical data and supporting decisions in routine clinical operations could, therefore, improve efficiency and effectiveness. It could also provide a high degree of interoperability as well as information support, process management, and optimization in delivering evidence-based surgical safety interventions [16,17]. The complexities of the perioperative environment, however, can complicate the process of deployment and make technology implementation challenging. Some common issues in this environment must be addressed for successful deployment of information technology [18]. The development of this type of evidence-based patient safety information system (EPSIS), therefore, requires a holistic view. It needs to bring together clinical professionals, health care administrators, data scientists, and information technology engineers. This process can serve as an important catalyst in fostering a safety culture among frontline health workers, constructing a surgical safety ecosystem supported by data scientists, engineers, and administrators [19].

    In 2014, a national project called Modern Surgery and Anesthesia Safety Management System Construction and Promotion (MSCP) was conducted in China. It aimed to improve perioperative patient safety. On the basis of this study, a perioperative surgical safety management information system was developed. This integrated patient data collection, processing, storage, and dissemination to support decision making, work control and documentation, and visualization [20]. To identify key elements and critical issues and formulate a framework to design, develop, and implement an EPSIS, a multidisciplinary panel of experts was assembled during the project period. This panel consisted of 10 medical experts, 5 nurses, 5 medical administrators, 3 data scientists, and 7 computer science engineers. All 30 experts attended several rounds of face-to-face consensus meetings, and widespread suggestions were collected from both the literature and panel members. Notes from project process meetings held by the central project group and the project executive groups from participating hospitals were also reviewed to identify practical challenges faced during the development of the project. These were refined and distributed to the panel members. Key methodological and implementation issues in designing and developing an evidence-based surgical safety information system were discussed, and recommendations on these issues were collected through these meetings. Finally, a summary of the framework was drafted and was circulated to panel members via email. Comments were collected until the group had reached a consensus. This paper discusses these issues (see Figure 1) in detail and uses practical examples from MSCP to illustrate them.

    ‎
    Figure 1. Critical issues in the life-cycle for designing and implementing an evidence-based surgical safety information system.
    View this figure

    Clinical data are usually complex and highly distributed. This poses challenges in collecting data to support the development of an EPSIS, because this type of system relies heavily on the management of high-quality surgical safety data. Using high-quality data also supports workflow management, monitoring, and evaluation of the surgical patient safety system. To satisfy the needs of the whole information system, the accessibility, reliability, and timeliness of data should be ensured.

    Which Surgical Safety Indicators Should Be Collected?

    Surgical safety promotion usually requires integrated interventions involving changes in a set of activities. These have long causal pathways and involve many factors that can influence the causal chain [21]. Recommended surgical safety indicators, therefore, include surgical complications and death and length of stays, which directly measure the observable harm. These indicators, unlike process or surrogate safety indicators (eg, error or culture), are identifiable and quantifiable and are more appropriate as natural endpoints in the story of patient safety [22]. The collection of data on these observable harms provides an easy way to investigate the causal chain. It could, therefore, help support learning about critical surgical safety issues linked to specific contexts as well as developing intuitive and target-sensitive functions for the safety improvement system [22-24].

    How Should Complication Reporting Be Standardized?

    Information about complications is not always readily available. A complication reporting procedure requires accepted principles of accrual, display, and analysis of complication data to be predefined to capture complications in a structural way [22,25,26]. This will allow meaningful comparisons of the incidence of reported complications across different hospitals or different periods within the same hospital [27]. The subsequent data processing could also benefit from this structural reporting. Several classification criteria had been proposed, of which the Clavien-Dindo classification is the most widely used. However, this type of classification system provides limited reference in standardizing complication reporting because it mainly focuses on ranking complication categories in an objective and reproducible manner, on the basis of the therapy used to correct them [28]. In 2002, Martin et al proposed 10 criteria that should be met when reporting complications following surgery [29]. These proposals could serve as a reference in establishing the criteria for reporting information about complications.

    How Can We Retrieve and Integrate Information About Inherent Patient-Related Risk?

    An incomplete data inventory leads to incomplete analyses. Electronic health records allow collection of particular elements of health-related information (eg, obesity, coronary heart disease, and hypertension) that are potentially associated with safety outcomes. However, it remains a challenge to form a multidisciplinary patient safety reference database because it needs to identify and track all data sources [30]. Knowledge of health data attributes, including data definitions, value sets, and other clinical coded content, is required. This means that the data retrieve process needs to involve information technology engineers, medical experts, and data scientists. Once the data have been captured, they can be filtered to support further clinical decision making by task and individual end user requirements. An additional step of data verification, introduced for data quality control, is also necessary. In MSCP, to ensure a trade-off between completeness and efficiency in information collection, a specialist panel was formed including medical experts, information engineers, and data scientists. By drawing on the literature and collecting expert opinions, this panel identified crucial information for inherent patient-related risk and determined how to obtain this information accurately and efficiently. To reduce user workload and transcription errors, a data extraction strategy was established by the clinical experts and information system engineers for data readily obtainable from hospital information systems. For information that is not routinely collected by hospital information systems, or which requires a special reporting mechanism because of its importance (such as complications), a stand-alone electronic data capture system was developed. This included 3 separate subsystems (ward, intensive care unit, and operation room). Patient information was entered by an established data entry team once the patients had been admitted. Complications (using clear definitions) were entered within a week of the patient’s discharge. The data collected from these 3 subsystems were centrally managed. Regular data quality audits were also conducted, and the results were reported monthly.


    Information alone is not enough to improve safety. Knowledge about the spectrum of complications and potential risk information extracted from the data is essential to formulate guidelines to help decision making about which patients to prioritize and what measures to take to prevent surgical complications. Ultimately, these data-driven decisions could play an auxiliary role in supporting clinical decisions and lead to more effective and appropriate use of resources through better procedures. New data obtained through the system are likely to stimulate the updating of knowledge and, therefore, further improve decisions.

    How Should We Prioritize Complications to Target?

    Not all complications are equally important to patient safety. For instance, some complications have an extremely low rate of incidence (eg, pulmonary torsion), are not related to severe harm (eg, subcutaneous hematoma), or are not sensitive to prevention measures (eg, hypothyroidism after thyroidectomy). To ensure that the system is both operationally feasible and cost-effective in routine clinical practice, it is important to prioritize complications. We recommended prioritizing complications based on high incidence and serious prognosis and which are more likely to be preventable. We also suggested that it was important to consider local conditions. Table 1 shows an example from the MSCP Project, using empirical data and expert consensus to identify surgical complications with these 3 characteristics. Complications with low incidence can still be collected through the system. Increased experience and evidence may enable groups to identify underlining patterns for the occurrence of these complications and targeted intervention measures can then be formulated.

    How Can We Translate Surgical Safety Information Into an Evidence-Based Stratifying Strategy?

    An evidence-based stratifying strategy implies that patients with different risk factors (identified using data collected and consensus among the expert panel) will receive hierarchical and targeted interventions. The ability to make a preoperative determination of the overall risk for multiple major complications is a prerequisite for clinical decision making and securing surgical patients’ safety, which is the ultimate goal of EPSIS [31-33]. Using the concept of risk population from epidemiology, we defined all surgical patients as the risk population for surgical complications at the stage before surgery [20]. In other words, every surgical patient could potentially develop any kind of complication and their risk of doing so is determined by factors that vary among the patients. Knowledge about patients’ existing risk factors can be generated using a series of computational models to translate input data [34]. This means that major contributory factors to complications can be identified and intervention measures can then be developed for patients.

    Oinas-Kukkonen argued that intervention should be “tailored to the potential needs, interests, personality, usage context, or other factors relevant to a user group” and that a system that offers personalized content or services has greater effectiveness and efficiency [35]. Patients can be managed through the system on the basis of their identified inherent risk. Specific interventions can be offered to patients with different levels of risk. This can, therefore, determine the appropriate amount of resources for each surgical patient, which is vital for optimizing patient flow. Examples of this concept include the well-known Physiological and Operative Severity Score for the enUmeration of Mortality and Morbidity scoring system and those recently developed by the National Surgical Quality Improvement Program in the United States [25,36]. With the growing tendency for medical professionals to become more specialized, using the risk population concept in the surgical safety field can also provide a macroscopic view across safety issues in multiple specialties. It, therefore, provides a more global picture for system-wide intervention planning.

    How Can We Formulate Targeted Intervention Guidelines?

    A system-integrated intervention guideline for decision making should map out which specific surgical safety intervention measures should be provided to patients with particular baseline risk and how the intervention should be carried out. The evidence from theory, the literature, and empirical research could be brought together and developed through standard procedures, such as the Delphi process [37]. Frontline staff and senior management, including both administrative and clinical leaders, should be involved in formulating the guidelines to get a buy-in from different professional groups (eg, surgeons, nurses, and anesthesiologists) to use the system to deliver interventions [14,38]. For instance, in MSCP, the interdisciplinary team of centralized researchers and clinicians reviewed the relevant research to identify interventions with the greatest benefit and the lowest barriers to use. Figure 2 shows how patients, complications, and interventions are provided in a stratified way in this system.

    Table 1. Examples of prioritization of complications in the modern surgery and anesthesia safety management system construction and promotion project.
    ComplicationDecision
    Surgical site infectionAa
    Delayed healing or nonhealing incisionA
    On ventilator ≥48 hoursA
    Death or confirmed deathA
    HypoparathyroidismBb
    Coma after operation ≥24 hoursB
    Respiratory failureA
    Electrolyte disturbanceB
    Urinary tract infectionA
    Pleural effusionB
    Acute renal failureA
    Skull defectB
    Cerebral edemaB
    Hemorrhage requiring ≥4U RBC infusion 72 hours postoperativelyA
    Iatrogenic pneumothoraxCc
    Esophagus anastomotic fistulaC
    AphasiaC
    Stress ulcerC
    PneumocraniaC
    Myasthenia crisisC
    Vocal cord paralysisC
    Pulmonary torsionC
    Secondary spinal canal stenosisC
    Fracture or loosening or dislocation of prosthesisC
    Internal or external fistula formationC
    Tracheal softening and collapseC
    Incisional herniaC
    Postoperative skin flap or subcutaneous effusionC
    Heterotopic ossificationC

    aHigh incidence (≥0.5%), severe harm, and sensitive to prevention measure.

    bHigh incidence, but not considered severe, or preventable based on the literature and an expert consensus.

    cVery low incidence, based on complications reported through the system.

    ‎
    Figure 2. Stratified intervention for patients at different risk level of complications.
    View this figure

    The data or applications should be presented in a visual form that users can understand. This makes them easy to use to support day-to-day operations and provide high-quality communication among hospital sectors for information- and knowledge-related functions [39]. Frontline medical staff and administrators at different levels should be extensively involved in the development process because adoption needs to be driven by clinicians where significant benefits can be articulated for them, the clinical and administrative teams, and the patients [40].

    How Can We Integrate the System Into the Existing Clinical Workflow and Make the System Sustainable?

    The objective of EPSIS is improving surgical safety by optimizing rather than subverting the current workflow. However, the information system may be frustrating for frontline clinicians and organizations if it does not fit with existing systems. This is particularly true if it causes longer completion times and workflow disruptions [24,41]. The system should, therefore, be embedded into existing clinical routines in line with the way frontline medical staff like to work. It should also take into account the interdependencies among the health care staff, cultural environment, and the infrastructural organization of the hospital. The challenge in achieving this goal is to identify the critical elements in the surgical workflow that influence patient safety management and the need to exchange safety information among medical staff at a minimum cost to existing workflows and thus provide information support for the next task. System use by different people in the clinical setting may be improved by visually representing the workflow of a complex clinical work environment and using user-system interaction analysis and complex design changes. A picture will also help in imposing the necessary workflow control and has been shown to be more successful in changing an unsafe plan or preventing the omission of essential interventions [42,43]. For instance, soft or hard stop functions can be incorporated in the system. Soft stops can alert clinicians if the intervention is not carried out in line with the guidelines, and hard stops alert clinicians and stop the process unless the intervention has been completed or an explanation has been provided to the central control point to override the interception. Soft and hard stops are helpful in promoting a buy-in but can result in variation in practice and poor compliance with safety goals and intervention measures [44]. An example of soft stops from MSCP is an interception function developed to ensure the timely delivery of preoperative patient safety interventions. The surgery submission is not approved if the preoperative intervention has not been completed for the patient.

    How Can We Share Key Surgical Safety Information Smoothly and in a Timely Way?

    Perioperative clinicians and staff have little opportunity to become familiar with surgical patients other than a quick determination of the required procedure. This lack of familiarity with and knowledge about patients could mean that perioperative team members (eg, operation room, postanesthesia care unit, and intensive care unit staff) might omit information that is important for surgical safety, such as allergies or antibiotic use. Integrated care and a high degree of interoperability should be highlighted to ensure high-quality communication of safety information among the various hospital sectors. This will allow quick and adequate responses about surgical patient safety issues [39]. Automatic reminders or alerts can facilitate the seamless transfer of vital information. They can also ensure that appropriate information is delivered to the surgeon at the right time and in a way that will ensure the surgeon receives and acts upon it. For instance, a reminder function was developed in MSCP to alert staff across the surgical ward and operation room when patients had high American Society of Anesthesiologists classifications.

    How Can We Make the User Interface Acceptable to End Users?

    The interface is one of the most significant parts of an information system and helps users to work efficiently, effectively, and satisfactorily [40,45]. The interface design should aim to eliminate complexity, emphasize key elements, and use special colors to mark important areas. One tactic is to make the interface as similar as possible to the previous paper records, so that users do not need to search for the required fields. Only the field relevant to the current task should be displayed to the user, to increase efficiency. Feedback from end users should be sought and welcomed on an ongoing basis for the improvement of interface design and the iteration of development as a result.

    How Can We Achieve a High Level of Interoperability With Existing Information Systems and What is the Benefit?

    Hospitals are information technology–intensive workplaces, incorporating many kinds of information systems (eg, electronic medical records, laboratory information management system, and office automation systems). These form an interoperable digital health ecosystem. Integration of the EPSIS with existing information systems could, therefore, influence several layers of caregivers and provide more convenient workflow control. Caregivers and professionals will be able to send, receive, find, and use digital health and care information in an appropriate, secure, timely, and reliable way and with little additional effort. Data interface standards should be established in the data-sharing process for patient-level data.

    In MSCP, we integrated our intervention system into the existing hospital information system, which, we believe, provided considerable advantages in terms of the extent, depth, and value-added use of the intervention system [46]. For instance, being able to change intervention information electronically makes it easy to continuously monitor and validate the intervention behavior of medical staff. Integration is also helpful in avoiding the inconvenience of switching between different systems and, therefore, decreasing staff resistance to the intervention system [47]. Other benefits include the relative low learning cost, faster adoption, and ease of logging-in. However, there are also some barriers to integration with other information systems, such as the level of investment needed, additional data leakage risk, higher maintenance cost, and administrative resistance.


    There are two main challenges to managing the adoption, implementation, and sustainability of the system. The first is implementing the system organizationally, and the second is shaping the use of the system and related practices to achieve practical alignment with the intervention intention [48,49]. Drawing up an implementation plan and identifying and selecting appropriate methods or techniques that fit the context are considered fundamental to successful implementation [50].

    What Contextual Factors Influence System Implementation?

    Ideas, practices, organizational arrangements, roles, and status in the information system all reflect and are influenced by the wider sociocultural context in which they occur [51]. This is particularly true for the organizational setting within which an information system is implemented, because it forms an integral part of that system. Several frameworks are available to identify contextual factors that are likely to influence the implementation of a given intervention [52,53]. Their use allows attention to be directed toward the contextual factors that are likely to hinder implementation as well as identifying facilitators of success [54,55]. For instance, Meijden et al identified 6 dimensions affecting implementation: (1) system quality, (2) information quality, (3) usage, (4) user satisfaction, (5) individual impact, and (6) organizational impact [56].

    How Should the Surgical Safety Culture and its Improvement Be Assessed?

    Information system initiatives often fail because of mismatching between culture and the information system or a failure to understand culture and its influence on end user adoption of information systems [51]. Changing staff attitudes and views can help the staff understand the reasons for system changes and improve acceptance [57-59]. The use of tools, such as the Healthcare Research and Quality Hospital Survey on Patient Safety Culture, is recommended to inform perceptions about safety and related behavior, as well as to support the adoption of a safety intervention system [60-62]. However, culture change in a health care institution can be a long and intensive process requiring more cross-collaboration and greater user participation at all levels. Studies have suggested that organizational champions who can shepherd system implementation, influence cultural change, and act as a bridge with developers would be a valuable resource. Strategies such as printed educational materials, educational meetings, and educational outreach are also effective in changing attitudes and behaviors and increasing the use of a new information system [38,63].

    Training Strategy

    The implementation of a new information system in a clinical setting often ignores the influence of processes and routines of clinical practice. A lack of understanding of system capabilities can lead to workarounds, with the new system being used in unintended ways [64]. A training plan must be designed and completed before the initial implementation and should include intensive support during implementation. Moreover, providing training shows the organization’s support in system implementation and development [65]. It is often effective to deliver training tailored to different users, for example, nurses, clinicians, and medical administrators. This should, however, include a holistic view of the entire system to strengthen the understanding of its function and goal. Hands-on practice and simulations may provide more benefit than only giving lectures [66]. It is also better to provide both compulsory and voluntary training elements [67]. Finally, training sessions are essential during or before system implementation, but ongoing training and development are also important [68]. Before the start of the MSCP project, training teams involving both clinicians and information technology engineers were established in each hospital. Unit-based education and training sessions were provided for nurses and other clinicians separately, because of the differences in workflow and the system operating interface. The standard operating procedure for efficient operation and compliance was translated into a course to facilitate the training process. Continuous training was also a part of MSCP, to reflect system updates and deployment of new function modules.


    Surgical safety improvement is an iterative procedure. Postimplementation evaluation, feedback, and performance gap assessment can deepen insight into how and why particular changes did or did not occur. This can further increase the benefits of using the system, prolong its sustainability, and support dynamic learning and improvement [69,70].

    What Implementation Outcomes Should Be Used to Assess System Implementation?

    When developing monitoring and evaluation plans for an EPSIS, new system implementation indicators (eg, usability) are recommended in addition to clinical outcomes (eg, complications, death, and length of stay). Different dimensions of outcomes and/or assessment methodologies have been proposed for evaluating the implementation of health care information systems [56,71-73]. For instance, Proctor et al distinguished among 3 distinct but interrelated types of outcomes for assessment in implementation studies: service (eg, safety), implementation (eg, fidelity), and client outcomes (eg, satisfaction) [73]. Hull et al outlined 8 implementation outcomes, defined as “the effects of deliberate and purposive actions to implement new treatments, practices, and services”. The outcomes were acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration diffusion, and sustainability [74]. Correlating these implementation outcomes with implementation success and failure could strengthen the understanding of the mechanisms of the effect. This might further translate into evidence-based interventions to improve surgical safety.

    Monitoring, Auditing, and Formulating a Feedback Channel for Performance Improvement

    A surgical safety information system needs monitoring and continuous auditing to ensure that it adheres to intervention guidelines [75]. Messaging functions could be used to enable real-time recording and transmission of any problems. In MSCP, the completion status of the surgical safety checklist for the operation room is recorded at different stages (sign in, time out, and sign out), and any violation at any stage is recorded and submitted.

    Establishing a mechanism to provide feedback on the results of intervention protocol variations (eg, compliance and completeness) and patient outcomes to frontline medical staff and managers is essential for fostering a culture of surgical patient safety in hospitals [76,77]. These measures can encourage improvements in identifying and sharing information about patient safety incidents. They can also help the staff to identify problems during the implementation process, ultimately supporting continuous learning as well as increasing engagement of medical staff. For instance, case-enhanced learning can use real cases to allow staff to identify problems and solutions. This, therefore, provides learning resources about complications and clinical behaviors to support safety improvement. A redesign-action-feedback closed cycle can be formed among knowledge generation, function and application development, system implementation, and evaluation of effectiveness. This has no defined beginning and end point, and the cycle should not be interpreted as starting with prevention and ending with action.

    In the MSCP project, a graded confirmed awareness subsystem was created. This gradually summarized patient outcomes and the implementation status of the system to improve awareness and confirmation in a specific sequence. This moved from patient to doctor and nurse in charge, to surgeon (daily), to department head (weekly), to medical affairs department (monthly), to hospital dean (quarterly), and then to permanent database. This system established dual duties (for subordinates and superiors) and dual supervision mechanisms (individual and external) for each person. Frontline staff and managers become equal elements in the chain securing patient safety. More importantly, this type of feedback channel could also help create a culture of patient safety and increase the motivation of hospital staff to deliver interventions through the system. In the MSCP, higher implementation rates were observed for all the postoperative prevention packages following the introduction of this subsystem (Table 2).

    Table 2. Effects of implementing a Graded Confirmed Awareness System on surgeons in 4 hospitals in the modern surgery and anesthesia safety management system construction and promotion project.
    Prevention categoryNumber with risk factorsPreoperative prevention measures deliveredPostoperative prevention measures delivered
    Before introduction of the graded confirmed awareness system, n (%)
    Generic intervention11,472 (27.0)10,671 (93.0)5830 (50.8)
    Specific intervention11,641 (31.7)11,453 (98.4)6905 (59.3)
    Intensive intervention4588 (23.8)3245 (70.7)2064 (44.5)
    After introduction of the graded confirmed awareness system, n (%)
    Generic intervention8947 (23.6)7528 (84.1)5535 (61.9)
    Specific intervention8532 (23.2)8270 (96.9)6037 (70.8)
    Intensive intervention3514 (20.1)2241 (63.8)1709 (48.6)

    This study has outlined and discussed several critical issues in the process of developing an EPSIS. The design rationale of the system is summarized in Figure 3. The EPSIS is characterized by several distinct features, including the following:

    1. Optimizing patient flow management and avoiding overload of medical staff by delivering tailored interventions to patients in a target-sensitive and stratified way based on inherent risk.
    2. Incorporating the intervention into routine clinical activities and formulating a surgical safety information circle that ensures that critical safety and management information transfers smoothly between sectors and medical staff, in particular, senior managers, who play a pivotal role in this information circle.
    3. Integrating the intervention information system with the existing hospital information system to construct an interoperable surgical safety improvement ecosystem, incorporating outcome reporting, intervention, monitoring, and feedback and finally developing a surgical safety learning system encouraging a patient safety culture in the hospital.

    The issues we have discussed in this paper might serve as a reference for projects aiming to build an interactive mechanism to combine safety systems supporting wide and continuous improvement in surgical safety.

    There are a number of limitations and challenges in constructing the EPSIS. The EPSIS in MSCP was mainly for patients admitted for elective surgery. Future expansion, for example, to emergency surgery and intensive care units, will present other challenges that have not been considered in this paper. It is difficult to establish evidence-based interventions for complications with low incidence in EPSIS, which is focused on complications. However, EPSIS is both a management system and learning platform. As more information is gathered, this problem will gradually be eliminated.

    ‎
    Figure 3. Design pattern rationale graph of an evidence-based surgical safety information system.
    View this figure

    Acknowledgments

    This study was funded by the National Health and Family Planning Commission of China (grant number: 201402017). The authors thank all the medical staff members and experts for their contributions of time, suggestions, and encouragement in this study.

    Authors' Contributions

    All authors listed in our manuscript have contributed substantially to this study. YZ, as the corresponding author, was responsible for constructing the idea and overarching research goals and aims. XY, WH, and JJ were responsible for designing and coordinating the research, supervising the course, and drafting the manuscript. YW, SX, SW, and HS were responsible for reviewing the paper for its intellectual content before submission. ZW was responsible for the literature review and also took responsibility for reviewing the paper for spelling and grammar.

    Conflicts of Interest

    None declared.

    1. Weiser TG, Makary MA, Haynes AB, Dziekan G, Berry WR, Gawande AA, Safe Surgery Saves Lives Measurement and Study Groups. Standardised metrics for global surgical surveillance. Lancet 2009 Sep 26;374(9695):1113-1117. [CrossRef] [Medline]
    2. Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AH, Dellinger EP, Safe Surgery Saves Lives Study Group. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med 2009 Jan 29;360(5):491-499. [CrossRef] [Medline]
    3. Lindenauer PK, Remus D, Roman S, Rothberg MB, Benjamin EM, Ma A, et al. Public reporting and pay for performance in hospital quality improvement. N Engl J Med 2007 Feb 01;356(5):486-496. [CrossRef] [Medline]
    4. Agrawal A. Medication errors: prevention using information technology systems. Br J Clin Pharmacol 2009 Jun;67(6):681-686 [FREE Full text] [CrossRef] [Medline]
    5. Batra R, Wolbach-Lowes J, Swindells S, Scarsi KK, Podany AT, Sayles H, et al. Impact of an electronic medical record on the incidence of antiretroviral prescription errors and HIV pharmacist reconciliation on error correction among hospitalized HIV-infected patients. Antivir Ther 2015;20(5):555-559. [CrossRef] [Medline]
    6. Stultz JS, Nahata MC. Preventability of Voluntarily Reported or Trigger Tool-Identified Medication Errors in a Pediatric Institution by Information Technology: A Retrospective Cohort Study. Drug Saf 2015 Jul;38(7):661-670. [CrossRef] [Medline]
    7. Padhy RP, Patra MR, Satapathy SC. Design and implementation of a cloud based rural healthcare information system model. UNIASCIT 2012;2(1):149-157 [FREE Full text]
    8. Carter AW, Mossialos E, Darzi A. A national incident reporting and learning system in England and Wales, but at what cost? Expert Rev Pharmacoecon Outcomes Res 2015 Jun;15(3):365-368. [CrossRef] [Medline]
    9. Institute of Medicine (US) Committee on Quality of Health Care in America, Kohn LT, Corrigan JM, Donaldson MS. To err is human: building a safer health system. Washington (DC): National Academies Press (US) 2000:1-287. [CrossRef] [Medline]
    10. Meguid RA, Bronsert MR, Juarez-Colunga E, Hammermeister KE, Henderson WG. Surgical risk preoperative assessment system (SURPAS): II. Parsimonious risk models for postoperative adverse outcomes addressing need for laboratory variables and surgeon specialty-specific models. Ann Surg 2016 Jul;264(1):10-22. [CrossRef] [Medline]
    11. Marang-van de Mheen PJ, Kievit J. Automated registration of adverse events in surgical patients in the Netherlands: the current status. Ned Tijdschr Geneeskd 2003 Jun 28;147(26):1273-1277. [Medline]
    12. Ozrazgat-Baslanti T, Blanc P, Thottakkara P, Ruppert M, Rashidi P, Momcilovic P, et al. Preoperative assessment of the risk for multiple complications after surgery. Surgery 2016 Aug;160(2):463-472 [FREE Full text] [CrossRef] [Medline]
    13. Tingle J. Patient safety at the crossroads. Br J Nurs 2017 Nov 09;26(20):1138-1139. [CrossRef] [Medline]
    14. Urbach DR, Govindarajan A, Saskin R, Wilton AS, Baxter NN. Introduction of surgical safety checklists in Ontario, Canada. N Engl J Med 2014 Mar 13;370(11):1029-1038. [CrossRef] [Medline]
    15. Baek H, Cho M, Kim S, Hwang H, Song M, Yoo S. Analysis of length of hospital stay using electronic health records: a statistical and data mining approach. PLoS One 2018 Apr 13;13(4):e0195901 [FREE Full text] [CrossRef] [Medline]
    16. Pollak VE, Lorch JA. Effect of electronic patient record use on mortality in End Stage Renal Disease, a model chronic disease: retrospective analysis of 9 years of prospectively collected data. BMC Med Inform Decis Mak 2007 Nov 28;7:38. [CrossRef] [Medline]
    17. Abraham C, Junglas IA. From cacophony to harmony: a case study about the IS implementation process as an opportunity for organizational transformation at Sentara Healthcare. J Strategic Inf Syst 2011 Jun;20(2):177-197. [CrossRef]
    18. Schimpff SC. Improving operating room and perioperative safety: background and specific recommendations. Surg Innov 2007 Jun;14(2):127-135. [CrossRef] [Medline]
    19. Ulrich B, Kear T. Patient safety and patient safety culture: foundations of excellent health care delivery. Nephrol Nurs J 2014 Sep;41(5):447-456. [Medline]
    20. Yu X, Jiang J, Liu C, Shen K, Wang Z, Han W, et al. Protocol for a multicentre, multistage, prospective study in China using system-based approaches for consistent improvement in surgical safety. BMJ Open 2017 Jun 15;7(6):e015147 [FREE Full text] [CrossRef] [Medline]
    21. Bärnighausen T, Bloom DE, Humair S. Health systems and HIV treatment in sub-Saharan Africa: matching intervention and programme evaluation strategies. Sex Transm Infect 2012 Mar;88(2):e2. [CrossRef] [Medline]
    22. Lilford RJ, Chilton PJ, Hemming K, Girling AJ, Taylor CA, Barach P. Evaluating policy and service interventions: framework to guide selection and interpretation of study end points. Br Med J 2010 Aug 27;341:c4413. [CrossRef] [Medline]
    23. Archer S, Hull L, Soukup T, Mayer E, Athanasiou T, Sevdalis N, et al. Development of a theoretical framework of factors affecting patient safety incident reporting: a theoretical review of the literature. BMJ Open 2017 Dec 27;7(12):e017155 [FREE Full text] [CrossRef] [Medline]
    24. Sheikh A, Cornford T, Barber N, Avery A, Takian A, Lichtner V, et al. Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from prospective national evaluation in. Br Med J 2011 Oct 17;343:d6054 [FREE Full text] [CrossRef] [Medline]
    25. Khuri SF, Daley J, Henderson W, Hur K, Demakis J, Aust JB, et al. The Department of Veterans Affairs' NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program. Ann Surg 1998 Oct;228(4):491-507. [CrossRef] [Medline]
    26. Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg 1999 Jul;16(1):9-13. [CrossRef] [Medline]
    27. Tsang C, Palmer W, Bottle A, Majeed A, Aylin P. A review of patient safety measures based on routinely collected hospital data. Am J Med Qual 2012;27(2):154-169. [CrossRef] [Medline]
    28. Clavien PA, Sanabria JR, Strasberg SM. Proposed classification of complications of surgery with examples of utility in cholecystectomy. Surgery 1992 May;111(5):518-526. [Medline]
    29. Martin RC, Brennan MF, Jaques DP. Quality of complication reporting in the surgical literature. Ann Surg 2002 Jun;235(6):803-813. [CrossRef] [Medline]
    30. Reiner B. Strategies for medical data extraction and presentation part 2: creating a customizable context and user-specific patient reference database. J Digit Imaging 2015 Jun;28(3):249-255 [FREE Full text] [CrossRef] [Medline]
    31. Schoenfeld AJ, Ochoa LM, Bader JO, Belmont PJ. Risk factors for immediate postoperative complications and mortality following spine surgery: a study of 3475 patients from the National Surgical Quality Improvement Program. J Bone Joint Surg Am 2011 Sep 07;93(17):1577-1582. [CrossRef] [Medline]
    32. Saxton A, Velanovich V. Preoperative frailty and quality of life as predictors of postoperative complications. Ann Surg 2011 Jun;253(6):1223-1229. [CrossRef] [Medline]
    33. Vogel TR, Dombrovskiy VY, Carson JL, Graham AM, Lowry SF. Postoperative Sepsis in the United States. Ann Surg 2010 Dec;252(6):1065-1071. [CrossRef] [Medline]
    34. Dodig-Crnkovic G. Knowledge generation as natural computation. J Syst Cybern Inf 2008;6(2):12-16.
    35. Wahle F, Bollhalder L, Kowatsch T, Fleisch E. Toward the design of evidence-based mental health information systems for people with depression: a systematic literature review and meta-analysis. J Med Internet Res 2017 May 31;19(5):e191 [FREE Full text] [CrossRef] [Medline]
    36. Brooks MJ, Sutton R, Sarin S. Comparison of surgical risk score, POSSUM and p-POSSUM in higher-risk surgical patients. Br J Surg 2005 Oct;92(10):1288-1292. [CrossRef] [Medline]
    37. Jones J, Hunter D. Consensus methods for medical and health services research. Br Med J 1995 Aug 05;311(7001):376-380 [FREE Full text] [CrossRef] [Medline]
    38. Bergs J, Lambrechts F, Simons P, Vlayen A, Marneffe W, Hellings J, et al. Barriers and facilitators related to the implementation of surgical safety checklists: a systematic review of the qualitative evidence. BMJ Qual Saf 2015 Dec;24(12):776-786. [CrossRef] [Medline]
    39. Winter AF, Ammenwerth E, Bott OJ, Brigl B, Buchauer A, Gräber S, et al. Strategic information management plans: the basis for systematic information management in hospitals. Int J Med Inform 2001 Dec;64(2-3):99-109. [Medline]
    40. Scholl J, Syed-Abdul S, Ahmed LA. A case study of an EMR system at a large hospital in India: challenges and strategies for successful adoption. J Biomed Inform 2011 Dec;44(6):958-967 [FREE Full text] [CrossRef] [Medline]
    41. Meyer KE, Altamore R, Chapko M, Miner M, McGann M, Hill E, et al. The need for a thoughtful deployment strategy: evaluating clinicians' perceptions of critical deployment issues. Stud Health Technol Inform 1998;52 Pt 2:854-858. [Medline]
    42. Strom BL, Schinnar R, Aberra F, Bilker W, Hennessy S, Leonard CE, et al. Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med 2010 Sep 27;170(17):1578-1583. [CrossRef] [Medline]
    43. Eslami S, de Keizer NF, Abu-Hanna A. The impact of computerized physician medication order entry in hospitalized patients—a systematic review. Int J Med Inform 2008 Jun;77(6):365-376. [CrossRef] [Medline]
    44. Feldman SS, Buchalter S, Hayes LW. Health information technology in healthcare quality and patient safety: literature review. JMIR Med Inform 2018 Jun 04;6(2):e10264 [FREE Full text] [CrossRef] [Medline]
    45. Jitnupong B, Jirachiefpattana W. Information system user interface design in software services organization: a small-clan case study. MATEC Web Conf 2018 Apr 23;164:01006. [CrossRef]
    46. Desroches CM, Agarwal R, Angst CM, Fischer MA. Differences between integrated and stand-alone E-prescribing systems have implications for future use. Health Aff (Millwood) 2010 Dec;29(12):2268-2277. [CrossRef] [Medline]
    47. McDonald CJ. The barriers to electronic medical record systems and how to overcome them. J Am Med Inform Assoc 1997;4(3):213-221 [FREE Full text] [Medline]
    48. Grol R, Wensing M. What drives change? Barriers to and incentives for achieving evidence-based practice. Med J Aust 2004 Mar 15;180(6 Suppl):S57-S60. [Medline]
    49. Arvidsson V, Holmström J, Lyytinen K. Information systems use as strategy practice: a multi-dimensional view of strategic information system implementation and use. J Strategic Inf Sys 2014 Mar;23(1):45-61. [CrossRef]
    50. Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci 2013 Dec 01;8:139 [FREE Full text] [CrossRef] [Medline]
    51. Jackson S. Organizational culture and information systems adoption: a three-perspective approach. Inf Organ 2011 Apr;21(2):57-83. [CrossRef]
    52. Keith RE, Crosson JC, O'Malley AS, Cromp D, Taylor EF. Using the Consolidated Framework for Implementation Research (CFIR) to produce actionable findings: a rapid-cycle evaluation approach to improving implementation. Implement Sci 2017 Feb 10;12(1):15 [FREE Full text] [CrossRef] [Medline]
    53. Ward MM, Baloh J, Zhu X, Stewart GL. Promoting action on research implementation in health services framework applied to TeamSTEPPS implementation in small rural hospitals. Health Care Manage Rev 2017 Jan;42(1):2-13. [CrossRef] [Medline]
    54. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009 Aug 7;4:50 [FREE Full text] [CrossRef] [Medline]
    55. Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care 2005 Feb;14(1):26-33 [FREE Full text] [CrossRef] [Medline]
    56. Van Der Meijden MJ, Tange HJ, Troost J, Hasman A. Determinants of success of inpatient clinical information systems: a literature review. J Am Med Inform Assoc 2003;10(3):235-243 [FREE Full text] [CrossRef] [Medline]
    57. Borchard A, Schwappach DL, Barbir A, Bezzola P. A systematic review of the effectiveness, compliance, and critical factors for implementation of safety checklists in surgery. Ann Surg 2012 Dec;256(6):925-933. [CrossRef] [Medline]
    58. Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AH, Dellinger EP, Safe Surgery Saves Lives Study Group. Changes in safety attitude and relationship to decreased postoperative morbidity and mortality following implementation of a checklist-based surgical safety intervention. BMJ Qual Saf 2011 Jan;20(1):102-107. [CrossRef] [Medline]
    59. Parand A, Burnett S, Benn J, Iskander S, Pinto A, Vincent C. Medical engagement in organisation-wide safety and quality-improvement programmes: experience in the UK Safer Patients Initiative. Qual Saf Health Care 2010 Oct;19(5):e44. [CrossRef] [Medline]
    60. de Wet C, Johnson P, Mash R, McConnachie A, Bowie P. Measuring perceptions of safety climate in primary care: a cross-sectional study. J Eval Clin Pract 2012 Feb;18(1):135-142. [CrossRef] [Medline]
    61. Flin R, Burns C, Mearns K, Yule S, Robertson EM. Measuring safety climate in health care. Qual Saf Health Care 2006 Apr;15(2):109-115 [FREE Full text] [CrossRef] [Medline]
    62. Nieva VF, Sorra J. Safety culture assessment: a tool for improving patient safety in healthcare organizations. Qual Saf Health Care 2003 Dec;12 Suppl 2:ii17-ii23 [FREE Full text] [CrossRef] [Medline]
    63. Russ SJ, Sevdalis N, Moorthy K, Mayer EK, Rout S, Caris J, et al. A qualitative evaluation of the barriers and facilitators toward implementation of the WHO surgical safety checklist across hospitals in England: lessons from the. Ann Surg 2015 Jan;261(1):81-91. [CrossRef] [Medline]
    64. Cresswell KM, Bates DW, Sheikh A. Ten key considerations for the successful implementation and adoption of large-scale health information technology. J Am Med Inform Assoc 2013 Jun;20(e1):e9-e13 [FREE Full text] [CrossRef] [Medline]
    65. Anderson LK, Stafford CJ. The "big bang" implementation: not for the faint of heart. Comput Nurs 2002;20(1):14-20; quiz 20. [Medline]
    66. Milano CE, Hardman JA, Plesiu A, Rdesinski RE, Biagioli FE. Simulated electronic health record (Sim-EHR) curriculum: teaching EHR skills and use of the EHR for disease management and prevention. Acad Med 2014 Mar;89(3):399-403. [CrossRef] [Medline]
    67. Gegenfurtner A, Könings KD, Kosmajac N, Gebhardt M. Voluntary or mandatory training participation as a moderator in the relationship between goal orientations and transfer of training. Int J Train Dev 2016 Oct 03;20(4):290-301. [CrossRef] [Medline]
    68. Ajami S, Mohammadi-Bertiani Z. Training and its Impact on Hospital Information System (HIS) Success. J Inform Tech Softw Eng 2012 Jan;2:112. [CrossRef]
    69. Berwick DM. Measuring surgical outcomes for improvement: was Codman wrong? J Am Med Assoc 2015 Feb 03;313(5):469-470. [CrossRef] [Medline]
    70. Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012 Jun 13(6):CD000259. [CrossRef] [Medline]
    71. Nelson L, Taylor F, Adams M, Parker DE. Improving pain management for hip fractured elderly. Orthop Nurs 1990;9(3):79-83. [CrossRef] [Medline]
    72. Stoop AP, Berg M. Integrating quantitative and qualitative methods in patient care information system evaluation: guidance for the organizational decision maker. Methods Inf Med 2003;42(4):458-462. [Medline]
    73. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011 Mar;38(2):65-76 [FREE Full text] [CrossRef] [Medline]
    74. Hull L, Athanasiou T, Russ S. Implementation science: a neglected opportunity to accelerate improvements in the safety and quality of surgical care. Ann Surg 2017 Jun;265(6):1104-1112. [CrossRef] [Medline]
    75. Mutale W, Chintu N, Amoroso C, Awoonor-Williams K, Phillips J, Baynes C, Population Health Implementation and Training – Africa Health Initiative Data Collaborative. Improving health information systems for decision making across five sub-Saharan African countries: Implementation strategies from the African Health Initiative. BMC Health Serv Res 2013;13 Suppl 2:S9 [FREE Full text] [CrossRef] [Medline]
    76. Weaver SJ, Lubomksi LH, Wilson RF, Pfoh ER, Martinez KA, Dy SM. Promoting a culture of safety as a patient safety strategy: a systematic review. Ann Intern Med 2013 Mar 05;158(5 Pt 2):369-374 [FREE Full text] [CrossRef] [Medline]
    77. Latif A, Holzmueller CG, Pronovost PJ. Evaluating safety initiatives in healthcare. Curr Anesthesiol Rep 2014 Jun;4(2):100-106 [FREE Full text] [CrossRef] [Medline]

    ‎
    EPSIS: evidence-based patient safety information system
    MSCP: modern surgery and anesthesia safety management system construction and promotion

    Edited by G Eysenbach; submitted 06.02.19; peer-reviewed by F Graur, J Farzi; comments to author 28.03.19; revised version received 18.05.19; accepted 18.05.19; published 24.06.19

    Copyright

    ©Xiaochu Yu, Wei Han, Jingmei Jiang, Yipeng Wang, Shijie Xin, Shizheng Wu, Hong Sun, Zixing Wang, Yupei Zhao. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 24.06.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.