This paper is in the following e-collection/theme issue:

Digital Health Reviews (1289) Adoption and Change Management of eHealth Systems (712) Quality Improvement (253) Health Care Quality and Health Services Research (271) Organizational Issues (154)

Published on in Vol 23, No 5 (2021): May

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/27180, first published .
The Challenge of Integrating eHealth Into Health Care: Systematic Literature Review of the Donabedian Model of Structure, Process, and Outcome

The Challenge of Integrating eHealth Into Health Care: Systematic Literature Review of the Donabedian Model of Structure, Process, and Outcome

The Challenge of Integrating eHealth Into Health Care: Systematic Literature Review of the Donabedian Model of Structure, Process, and Outcome

Authors of this article:

Rosian Tossaint-Schoenmakers1, 2, 3 Author Orcid Image ;   Anke Versluis2, 3 Author Orcid Image ;   Niels Chavannes2, 3 Author Orcid Image ;   Esther Talboom-Kamp2, 3, 4 Author Orcid Image ;   Marise Kasteleyn2, 3 Author Orcid Image
Article Authors Cited by (65) Tweetations (35) Metrics

    Review

    1Saltro Diagnostic Centre, Utrecht, Netherlands

    2National eHealth Living Lab, Leiden University Medical Centre, Leiden, Netherlands

    3Public Health and Primary Care Department, Leiden University Medical Centre, Leiden, Netherlands

    4Unilabs Group, Geneva, Switzerland

    Corresponding Author:

    Rosian Tossaint-Schoenmakers, MSc

    Saltro Diagnostic Centre

    Mississippidreef 83

    Utrecht, 3565 CE

    Netherlands

    Phone: 31 6 55895361

    Email: rtossaint@saltro.nl


    Background: Health care organizations are increasingly working with eHealth. However, the integration of eHealth into regular health care is challenging. It requires organizations to change the way they work and their structure and care processes to be adapted to ensure that eHealth supports the attainment of the desired outcomes.

    Objective: The aims of this study are to investigate whether there are identifiable indicators in the structure, process, and outcome categories that are related to the successful integration of eHealth in regular health care, as well as to investigate which indicators of structure and process are related to outcome indicators.

    Methods: A systematic literature review was conducted using the Donabedian Structure-Process-Outcome (SPO) framework to identify indicators that are related to the integration of eHealth into health care organizations. Data extraction sheets were designed to provide an overview of the study characteristics, eHealth characteristics, and indicators. The extracted indicators were organized into themes and subthemes of the structure, process, and outcome categories.

    Results: Eleven studies were included, covering a variety of study designs, diseases, and eHealth tools. All studies identified structure, process, and outcome indicators that were potentially related to the integration of eHealth. The number of indicators found in the structure, process, and outcome categories was 175, 84, and 88, respectively. The themes with the most-noted indicators and their mutual interaction were inner setting (51 indicators, 16 interactions), care receiver (40 indicators, 11 interactions), and technology (38 indicators, 12 interactions)—all within the structure category; health care actions (38 indicators, 15 interactions) within the process category; and efficiency (30 indicators, 15 interactions) within the outcome category. In-depth examination identified four most-reported indicators, namely “deployment of human resources” (n=11), in the inner setting theme within the structure category; “ease of use” (n=16) and “technical issue” (n=10), both in the technology theme within the structure category; and “health logistics” (n=26), in the efficiency theme within the outcome category.

    Conclusions: Three principles are important for the successful integration of eHealth into health care. First, the role of the care receiver needs to be incorporated into the organizational structure and daily care process. Second, the technology must be well attuned to the organizational structure and daily care process. Third, the deployment of human resources to the daily care processes needs to be aligned with the desired end results. Not adhering to these points could negatively affect the organization, daily process, or the end results.

    J Med Internet Res 2021;23(5):e27180

    doi:10.2196/27180

    Keywords

    eHealth; digital health; blended care; quality; integration; health care organization; structure; process; outcome 


    Health care is changing, whereby patient empowerment, democratization of the internet, and an increasing burden on health care professionals play influential roles [1-3]. In line with these trends, innovations such as eHealth are required to maintain high quality of care [4-6]. eHealth includes a wide range of web-based interventions, for example e-consults, telemonitoring, and web-based viewing of medical records [1,7,8]. However, eHealth is more than a technology; it is another way of working and thinking and requires a change in attitude, which goes beyond the boundaries of a local health care organization [9,10].

    The most comprehensive definition of eHealth with reference to the organizational context is that provided by Eysenbach [11]:

    e-health is an emerging field in the intersection of medical informatics, public health and business, referring to health services and information delivered or enhanced through the Internet and related technologies. In a broader sense, the term characterizes not only a technical development, but also a state-of-mind, a way of thinking, an attitude, and a commitment for networked, global thinking, to improve health care locally, regionally, and worldwide by using information and communication technology.

    In other words, the integration of eHealth into traditional health care requires organizational and behavioral changes for both health care professionals and patients [9,10].

    Organizations are increasingly working with eHealth; however, implementing eHealth into the regular health care system requires organizations to change the way they work [9-11]. eHealth enables patients to have a more active role in managing their health [7,12,13], which affects interactions between the patient and health care professional [14-17]. Furthermore, working with eHealth technology requires workflow adjustments for health care professionals [18,19]. The organization’s structure and care processes need to be adapted to ensure that eHealth supports the attainment of desired outcomes [20,21].

    The challenge of optimally integrating eHealth into health care is thus a complex organizational issue. Several studies have identified elements to promote eHealth adoption, such as the degree of complexity, adaptability of the technology, costs, and stakeholder value [20,22], but uncertainty remains on how digital and traditional health care can blend successfully in the long term. With different definitions of eHealth available in the literature [10,11,23], and unclear barriers for facilitators in the application of eHealth [19], there is a need for further research on how eHealth can successfully be integrated into health care.

    The aim of this study is to analyze how the integration of eHealth can be organized optimally by reviewing studies evaluating real-world eHealth interventions. The Donabedian framework of Structure-Process-Outcome (SPO) [24] was used, allowing the identification of relevant indicators that demonstrate how effective the integration of eHealth is in the organization.

    According to the Donabedian model, the quality of health care can be assessed by three components that are relevant for organizations: structure (ie, requirements of the organization), process (ie, actions to be taken), and outcome (ie, end results), as shown in Figure 1 [24,25]. Structure is defined as the setting in which health care is provided (eg, facilities, equipment, numbers, and qualification of personnel); process, as what is actually done in giving and receiving care (eg, patient and doctor activities, doctor-patient communication and information); and outcome, as the consequence of the provided health care (eg, health status, satisfaction, and costs) [24-26]. Quality of health care is based on different aspects of these three categories and their relationships. As Donabedian eloquently puts it: “A good structure increases the likelihood of good process, and good process increases the likelihood of good outcomes” [24]. The interaction between the categories can be bidirectional, and it is not a simple separation between cause and effect [25]. The movement is an “unbroken chain of antecedents, followed by intermediate ends, which are themselves the means to still further ends” [25].

    ‎
    Figure 1. Donabedian Structure-Process-Outcome framework.
    View this figure

    The aim of this systematic review is twofold: (1) to investigate whether there are identifiable indicators in the structure, process, and outcome categories related to the successful integration of eHealth in regular health care and (2) to investigate which indicators of structure and process are related to outcome indicators.


    Theoretical Framework

    The Donabedian SPO framework was used to identify the indicators of structure, process, and outcome that potentially affect the integration of eHealth into health care organizations. The Donabedian framework covers all relevant aspects of an organization’s structure, process, and outcome and their interrelations, and it combines these aspects with health and social factors. Therefore, it is a suitable model to evaluate the organization of eHealth within health care organizations. The SPO categories are thematically explained in Figure 2 [24,25].

    ‎
    Figure 2. Explanation of the Structure-Process-Outcome categories of the Donabedian model.
    View this figure

    The Donabedian SPO framework was designed in the 20th century before the introduction of eHealth. For this review, the SPO framework was adjusted to be compatible with the current time and incorporated the application of eHealth. The adjustments are described in the themes presented in Textbox 1.

    The adjustments to the SPO framework are shown in Figure 3.

    Adjustment of the Structure-Process-Outcome framework into themes, to integrate the application of eHealth into the health care system.
    • Structure: The setting of provided care can be internal and/or external. Therefore, a distinction was made between inner and outer settings. With regard to resources, technology was added as a separate theme to cover eHealth. This was done because the focus of this research was eHealth. The remaining parts of the resources are covered under inner setting. Human resources, besides health care professionals, included care receivers. Their mutual involvement is required and is therefore also considered a conditional human resource [1]. Organizational structure was split into inner setting and outer setting, in line with the reasons given above, and to take the external stakeholders into account [27].
    • Process: Instead of technical actions, the term health care actions was used, to avoid confusion with the term technology in the structure. Interpersonal actions remained unchanged. Management of the actions was shortened to process management.
    • Outcome: Health status was retained as health status. Satisfaction was broadened to include experience of the health care receiver and experience of the health care provider, as both are pivotal outcome parameters in the health care process [28,29]. Efficiency remained unchanged.
    Textbox 1. Adjustment of the Structure-Process-Outcome framework into themes, to integrate the application of eHealth into the health care system.
    ‎
    Figure 3. Adjustment to the themes of the Structure-Process-Outcome framework, considering eHealth integration.
    View this figure

    Search Strategy

    This systematic review followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines. The research question was as follows: “How are structure indicators and/or process indicators related to eHealth or blended health care outcome indicators?”

    Two authors (RT and MK) searched PubMed, EMBASE, Web of Science, Cochrane, and Emcare databases for relevant studies published up to December 12, 2019. They searched for the following terms in the titles, abstracts, and keywords of the published papers: structure* indicators* or process* indicators* or outcomes* indicators* and [blended care or eHealth* or telehealth*]. Multimedia Appendix 1 contains the full search details. After the search, two authors (RT and AV) screened the titles and abstracts of the relevant articles. Studies were included if they mentioned (1) the use of eHealth or blended care for diagnostics or treatment and (2) structure, process, or outcome indicators. Quantitative, qualitative, and mixed method study designs were included. A study was excluded if (1) it was a protocol, review, meta-analysis, grey literature, book chapter, oral presentation, or poster presentation; (2) it was published in a language other than English or Dutch; (3) full-text of the article was not available; (4) the intervention was not implemented (eg, conducted research regarding the users’ expectations towards a prototype); or (5) the intervention used an analog application via plain-old-telephone lines. Of the remaining articles, RT reviewed the full texts. To ensure reliability, AV randomly selected about 10% of the fully reviewed articles for a blind review. Discrepancies were resolved by discussion. In case of uncertainty, a third author (MK) was consulted.

    Data Extraction

    Data extraction sheets were designed to provide an overview of the (1) study characteristics (eg, title, author, study aim, setting, disease, and quality appraisal); (2) characteristics of the eHealth intervention (eg, technology and function) and description of the intervention; (3) distribution of indicators into themes and categories related to the integration of eHealth into health care; and (4) interaction among the indicators, presented as themes.

    RT designed the first concepts of the data extraction sheets. Authors RT, MK, NC, and ET discussed the design of the data extraction sheets to ensure their usability. Improved sheets were developed accordingly. The blind reviewer (AV) did not discuss the data extraction sheets. The included articles were reread by RT to check whether data clustering was complete and logical and for purposes of data pooling itself. AV selected a sample of 10% of the included articles for data extraction. Discrepancies were resolved by discussion.

    Quality Appraisal

    The Mixed Methods Appraisal Tool (MMAT) was used to appraise the quality of eligible studies in mixed methods systematic reviews—that is, reviews that included qualitative research, randomized controlled trials, nonrandomized studies, quantitative descriptive studies, and mixed methods studies [30]. The MMAT allows determination of the quality of different empirical study designs by using the same measure of five criteria in the chosen category. MMAT scores range on a scale of 1 to 5, with 1 indicating the lowest quality and 5 indicating the highest quality.

    Classification of eHealth Interventions

    eHealth interventions were ordered by type of technology and functionality. For technology, the classification proposed by Nictiz was used, distinguishing websites, apps, video communication, sensors, and wearables, domotics, robotics, and big data (ie, artificial intelligence) [31]. This classification is based on international studies [10,32]. For the present study, eHealth only concerns digital interventions and not analog ones such as analog applications via plain-old-telephone lines; this is in line with the classification proposed by Nictiz. For labelling the functionality, the second and third tiers of the National Institute for Health and Care Excellence (NICE) [33] were used, because these functionalities measure patient outcomes (Tier one consists of system services with no measurable patient outcomes). The functions were classified as communication, self-management, clinical calculation, active monitoring, diagnosis, and treatment [33].

    Organization of Indicators and (Sub)themes of the SPO Framework

    Indicators that had a potential impact on the integration of eHealth in health care were extracted and organized by the relevant theme according to the adjusted SPO framework (Textbox 1). In addition, the reported interactions between the indicators were extracted and organized by the relevant categories and themes. For a clear overview, the indicators within each theme were further divided into two subthemes by RT and ET (Table 1). The creation of subthemes was an iterative process. When reading the full texts, we found some definitions that sharpened some of the subthemes. The full definitions of the themes and subthemes are provided in Multimedia Appendix 2.

    For each of the extracted indicators, the relevant impact on the integration of eHealth was noted. As there is no general standard for when eHealth is successful or effective [3,19], nor did the included studies specify such standards, these indicators were labeled as advantage, disadvantage, or neutral. An advantage in the structure and process categories indicates a positive effect on the integration and/or a positive effect on the outcome. A disadvantage in the structure and process categories indicates a negative effect on the integration and/or a negative effect on the outcome. An indicator that did not turn out to be an advantage nor a disadvantage was labeled neutral. The extracted indicators were noted as advantage, disadvantage, or neutral, in line with the evaluation performed in the corresponding study.

    The following results are presented in this paper: (1) distribution of the indicators into (sub)themes and categories, and the impact on the integration of eHealth into health care; (2) most frequently reported indicators (ie, reported 10 times or more); and (3) interaction among indicators organized into themes and categories.

    Table 1. Themes and subthemes in the structure, process, and outcome categories.
    Category and themeSubtheme
    Structure

    		Inner setting
    • Support of primary process
      • ‎
    • Culture and leadership
      • ‎

    		Health care professional
    • Skills
      • ‎
    • Attitude
      • ‎

    		Care receiver
    • Daily life
      • ‎
    • Baseline characteristics
      • ‎

    		Technology
    • Usability and functionality
      • ‎
    • Interaction with electronic health record
      • ‎

    		Outer setting
    • Finance and legislation
      • ‎
    • Involvement of stakeholders
      • ‎
    Process

    		Health care actions
    • Workflow
      • ‎
    • Patient-centered
      • ‎

    		Interpersonal actions
    • Personal
      • ‎
    • Shifting roles
      • ‎

    		Process management
    • Quality improvement
      • ‎
    • Mistake-proofing
      • ‎
    Outcome

    		Health status
    • Clinical or functional
      • ‎
    • Intrapersonal
      • ‎

    		Experience of care recipient
    • Satisfaction
      • ‎
    • Convenience
      • ‎

    		Experience of health care professional
    • “What’s in it for me”
      • ‎
    • “What’s in it for them”
      • ‎

    		Efficiency
    • Operations
      • ‎
    • Revenues
      • ‎

    Study Selection

    The systematic search led to the identification of 11 eligible articles, selected from a total of 739 articles shortlisted initially (Figure 4).

    ‎
    Figure 4. Flowchart of the systematic review.
    View this figure

    Data Results: Study and eHealth Characteristics

    Study Characteristics

    The included studies cover various study designs, diseases, and health care settings. Most studies were published after 2017 [27,34-41] and were of high quality [27,34,36,37,39,40,42]. Table 2 shows a detailed description of the characteristics of the included studies.

    Table 2. Study characteristics.
    Title, author (year)Aim of the studyStudy designSetting and countryDiseaseSample size (N) and participant typeQuality of studya
    Implementation of the blended care self-management program for caregivers of people with early-stage dementia,
    Boots et al (2017) [34]    		To assess the internal and external validity of the trial and its implementation to inform effect analysisMixed methods,
    nested in an RCTb    		Elderly care, home setting, the NetherlandsEarly-stage dementiaN=62;
    Informal caregivers, psychologists, nurses    		4.5
    Lack of adoption of a mobile app to support patient self-management of diabetes and hypertension in a Federally Qualified Health Centre,
    Thies et al (2017) [35]    		To understand why the trial was unsuccessfulQualitative (interview) analysisPrimary care, USAUncontrolled diabetes and hypertensionN=13;
    Patients, primary care provider, nurses, research assistants    		3.0
    “Sounds a bit crazy, but it was almost more personal:” A qualitative study of patient and clinician experiences of physical therapist-prescribed exercise for knee osteoarthritis via skype, Hinman et al (2017) [36]To explore the experience of therapists and patients using Skype for exercise management of knee osteoarthritisQualitative study, nested in an RCTRehabilitation at home,
    Australia    		Knee osteoarthritisN=20;
    Patients, physical therapists    		5.0
    The challenge of real-world implementation of web-based share care software,
    Lycett et al (2014) [43]    		To highlight the challenges of implementing software and reporting on the extent to which the software met its implementations and user aimsMixed methods study, nested within an RCTChildren’s Hospital, General Practices, AustraliaObesityN=27;
    General Practitioners    		2.5
    Implementation of a multicomponent telemonitoring intervention to improve nutritional status of community-dwelling older adults,
    Van Doorn-van Atten et al (2018) [37]    		To study how PhysioDom Home Dietary Intake Monitoring was delivered and received by participants and nurses, and to study the intervention’s mechanism of impactMixed methods studyHome care and/or lived in a service flat in sheltered accommodation, the NetherlandsAt risk of undernutritionN=105;
    Patients, nurses    		4.5
    Implementation of internet-delivered cognitive behaviour therapy within community mental health clinics, Hadjistavropoulos et al (2017) [27]To understand facilitators and barriers impacting the uptake and implementation of internet cognitive behavior therapyMixed methods studyCommunity Mental Health Clinic,
    Canada    		Depression and anxietyN=33;
    Therapists, managers    		4.5
    Implementation and evaluation of the safety net specialty care program in the Denver Metropolitan Area,
    Fort et al (2017) [38]    		To describe the program, identify aspects that work well, areas for improvement, and offer lessons learnedMixed methods studySafety-net: a non-profit integrated health care system, USAUninsured patientsN=43;
    Patients, primary care clinicians, specialists    		3.5
    Perceived improvement in integrated management of childhood illness implementation through use of mobile technology, Mitchell et al (2012) [42]To examine health care provider and carer perceptions of electronic Integrated Management of Childhood Illness (eIMCI) in diagnosing and treating childhood illnessesQualitative study (semi-structured interviews)Health centers, TanzaniaChildhood illness in children 5 years or youngerN=31;
    Carers, health care providers    		4.5
    High level of integration in integrated disease management leads to higher usage in the e-Vita Study,
    Talboom-Kamp et al (2017) [39]

    		To analyze the factors that successfully promote the use of a self-management platform for chronic obstructive pulmonary disease patientsQuantitative, nonrandomized, parallel cohort designPrimary care, the NetherlandsChronic obstructive pulmonary diseaseN=215;
    Patients    		4.0
    What drives adoption of a computerised, multifaceted quality improvement intervention for cardiovascular disease management in primary healthcare settings?
    Patel et al (2018) [40]    		To identify and explain the underlying mechanisms by which the intervention did and did not have an impactMixed methods studyPrimary care, AustraliaCardiovascular diseaseN=19;
    Patients, general practitioners, nurses, aboriginal health workers    		4.5
    Exploring the challenges of implementing a web-based telemonitoring strategy for teenagers with inflammatory bowel disease,
    Dijkstra et al (2019) [41]    		To evaluate whether the telemonitoring strategy could move from a demonstration project to one that is sustained within existing sitesMixed methods study, nested within an RCT Pediatric gastroenterology centers, the NetherlandsInflammatory bowel diseaseN=27;
    Researchers, clinicians, hospital decision makers, web designer    		2.5

    aMethodological quality of studies assessed with MMAT, ranging from 0 (lowest) to 5 (highest).

    bRCT: randomized controlled trial.

    eHealth Intervention Characteristics, Descriptions, and Results

    The most frequently used digital technology was websites (n=7) [27,34,37-41], and the most frequently reported functions [33] of the technology were self-management (n=6) [34-37,39,41] and communication (n=6) [35,37,39-41,43]. Table 3 shows an overview of the eHealth intervention characteristics, descriptions, and the study results. A detailed description of indicators, sorted according to the structure, process, outcome categories and their respective (sub)themes, are highlighted in the next paragraph.

    Table 3. eHealth intervention characteristics, descriptions of the intervention, and study results.
    First authorTechnology; intervention nameeHealth functionInterventionStudy results (findings)a
    Boots [34]Website; Partner in BalanceSelf-managementFace-to-face coaching with tailored web-based modules.The participation rate of eligible caregivers was 51.9% (80/154). Recruitment barriers included lack of computer and need for support. Young age and employment were considered recruitment facilitators. All coaches attended training and supervision in blended care self-management. Deviations from the structured protocol were reported on intervention time, structure, and feedback. Coaches described an intensified relationship with the caregiver post-intervention. Caregivers appreciated the tailored content and positive feedback. The blended structure increased their openness. Overall, personal goals were attained after the program (t>50). Implementation barriers included lack of financing, time, and deviating target population.
    Thies [35]App; UndisclosedSelf-management, communicationPlatform for active collaboration between patients and their primary care teams.There was a poor fit between the app, end-users, and recruitment and treatment approaches in the setting. Usability testing might have revealed this prior to launch, but this was not an option. There was not sufficient time during routine care for clinical staff to familiarize patients with the app or to check clinical data and messages, which are unreimbursed activities. Some patients did not use the app appropriately. The lack of integration with the electronic health record was cited as a problem for both patients and staff who also said the app was just one more thing to attend to.
    Hinman [36]Video communication; Telerehabilitation via SkypeSelf-management, treatmentIndividualized home-based training strengthening program via Skype delivered physiotherapy.Six themes arose from both patients and therapists. The themes were Structure: technology (ease of use, variable quality, set-up assistance helpful) and patient convenience (time-efficient, flexible, increased access); Process: empowerment to self-manage (facilitated by home environment and therapists focusing on effective treatment) and positive therapeutic relationships (personal undivided attention from therapists, supportive friendly interactions); and Outcomes: satisfaction with care (satisfying, enjoyable, patients would recommend, therapists felt Skype more useful as adjunct to usual practice) and patient benefits (reduced pain, improved function, improved confidence and self-efficacy). A seventh theme arose from therapists regarding process: adjusting routine treatment (need to modify habits, discomfort without hands-on, supported by research environment).
    Lycett [43]Website, HIEb; Shared-Care Obesity Trial in Children (HOPSCOTCH)CommunicateChildren attended a tertiary appointment with a pediatrician and dietician specializing in childhood obesity, followed-up by general practitioner consultations over the following year, supported by shared-care web-based software.Software implementation posed difficult and at times disabling technological barriers. The software’s speed and inability to seamlessly link with day-to-day software was a source of considerable frustration. Overall, general practitioners rated software usability as poor, although most (68%) felt that the structure and functionality of the software was useful.
    Van Doorn-van Atten [37]Website; PhysioDom HDIMcSelf-management, communicationNutritional telemonitoring, education, a follow-up of telemonitoring measurement by a nurse.About 80% of participants completed the intervention. Drop-outs were significantly older, had worse cognitive and physical functioning, and were more care-dependent. The intervention was largely implemented as intended and was received well by participants, but less well by nurses. Participants adhered better to weight telemonitoring than to telemonitoring by means of questionnaires, for which half the participants needed help. Intention to use the intervention was predicted by performance expectancy and social influence. No association was found between process indicators and intervention outcomes.
    Hadjistavropoulos [27]Website; ICBTdTreatmentWeb-based lessons that provide psychoeducation and instructions and therapist support via email or telephone.ICBT implementation was perceived to be most prominently facilitated by intervention characteristics (namely, the relative advantages of ICBT compared to face-to-face therapy, the quality of the ICBT program that was delivered, and evidence supporting ICBT) and implementation processes (namely the use of an external facilitation unit that aided with engaging patients, therapists, and managers and ICBT implementation). The inner setting was identified as the most significant barrier to implementation as a result of limited resources for ICBT combined with greater priority given to face-to-face care.
    Fort [38]Website; Safety Net Specialty Care ProgramDiagnosis, treatmentE-consults between primary care clinicians and specialist, face-to-face visits to the patients from a specialist, and continuing medical education for the primary care clinicians.In the first 20 months of the program, safety-net clinicians at 23 clinics made 602 e-consults to specialists, and 81 patients received face-to-face specialist visits. Of 204 primary care clinicians, 103 made e-consults; 65 specialists participated in the program. Aspects facilitating program use were referral case managers\' involvement and the use of clear, concise questions in e-consults. Key recommendations for process improvement were to promote an understanding of the different healthcare contexts, support provider-to-provider communication, facilitate hand-offs between settings, and clarify program scope.
    Mitchell [42]App; Electronic Integrated Management of Childhood Illness (eIMCI)Diagnosis, treatmentAn electronic handheld device or personal digital assistant, to guide the health care provider through IMCI protocols.Providers expressed positive opinions on eIMCI, noting that the personal digital assistants were faster and easier to use than were the paper forms and encouraged adherence to IMCI procedures. Carers also held a positive view of eIMCI, noting improved service from providers, a more thorough examination of their child, and a perception that providers who used the personal digital assistants were more knowledgeable.
    Talboom-Kamp [39]Website; e-Vita COPDeSelf-management, communicationInsight into personal health data, self-monitoring of health values, education, and a coach for attaining personal goals.Use of a self-management platform was higher when participants received adequate personal assistance about how to use the platform. Blended care, where digital health and usual care are integrated, will likely lead to increased use of the web-based program.
    Patel [40]HIE, website; HealthTrackerCommunication, monitoringReal-time decision support integrated with electronic medical records; CVDf risk communication tool between provider and patient; clinical audit tool; web portal providing peer-ranked performance trends.A complex interaction was found between implementation processes and several contextual factors affecting uptake of the intervention. There was no clear association between team climate, job satisfaction, and intervention outcomes. There were four spheres of influence that appeared to enhance or detract from normalization of the intervention: organizational mission and history, leadership, team environment, and technical integrity of the intervention.
    Dijkstra [41]Website, IBD-liveMonitoring, self-management, communicationFlarometer, platform for direct communication with the IBDg team, module with study questionnaires (Quality of life, absenteeism, health care utilization).The technology and the linked program allowed selection and targeting of teenagers who were most likely to benefit from a face-to-face encounter with their specialist. The value proposition of the technology was clear, with a distinct benefit for patients and an affordable service model, but health providers had plausible personal reasons to resist (double data entry). The organization was not yet ready for the innovation, as it required a shift to new ways of working. Dutch health insurers agreed that screen-to-screen consultations will be reimbursed at a rate equivalent to face-to-face consultations. The technology was considered easy to adapt and evolve over time to meet the needs of its users.

    aResults [27,34-38,40-43] or conclusion [39] as described in the abstracts of the included studies.

    bHIE: Health information exchange.

    cHDIM: Home Dietary Intake Monitoring.

    dICBT: internet cognitive behavior therapy.

    eCOPD: chronic obstructive pulmonary disease.

    fCVD: cardiovascular disease.

    gIBD: inflammatory bowel disease.

    Indicators Organized by (Sub)themes of the SPO Framework

    Overview

    In total, an indicator was reported 347 times: 175 times in the structure category, 84 times in the process category, and 88 times in the outcome category. Of the 347 indicators, 111 were unique indicators (Multimedia Appendix 3). In the structure category, most indicators were labeled as neutral (65/175, 37.1%) or as a disadvantage (70/175, 40%). In the process category, most indicators were labeled as an advantage (30/84, 36%) or neutral (33/84, 39%). In the outcome category, the indicators were mostly classified as a realized advantage (49/88, 56%), as shown in Figure 5.

    Table 4 shows the total distribution of the indicators organized by themes and subthemes of the structure, process, and outcome categories and the extent to which it was reported as an advantage, disadvantage, or neutral to the integration of eHealth and its outcome in regular health care. The themes and subthemes containing the most reported indicators are described below.

    ‎
    Figure 5. Number of indicators reported in the structure, process, and outcome categories. Advantage: in the structure and process categories, advantage indicates a positive effect on the integration. In the outcome category, it indicates a positive effect on the outcome. Disadvantage: in the structure and process categories, disadvantage indicates a negative effect on the integration. In the outcome category, it indicates a negative effect on the outcome. Neutral: Indicator was neither an advantage nor a disadvantage.
    View this figure
    Table 4. Distribution of the indicators according to the themes and subthemes of the structure, process, and outcome categories.
    Category, theme, and subthemeAdvantage (n)Disadvantage (n)Neutral (n)Source
    Structure (n=175)

    		Inner setting (n=51)


    		Support of primary process (n=34)71314[27,34,37-43]


    		Culture and leadership (n=17)791[27,34,37,40]

    		Health care professional (n=28)


    		Skills (n=8)404[27,36,38,40,41,43]


    		Attitude (n=20)884[27,34-41,43]

    		Care receiver (n=40)


    		Daily life (n=18)387[27,34-39]


    		Baseline characteristics (n=22)1516[34-39]

    		Technology (n=38)


    		Usability and functionality (n=33)8178[27,34-43]


    		Interaction with EHRa (n=5)050[35,37,38,41,43]

    		Outer setting (n=18)


    		Finance and legislation (n=10)028[27,34,36,38-41]


    		Involvement of stakeholders (n=8)233[27,38,43]

    		Total structure407065
    Process (n=84)

    		Health care actions (n=38)


    		Workflow (n=18)5112[27,34-39,41-43]


    		Patient-centered (n=20)7013[27,34-39,41,42]

    		Interpersonal actions (n=24)


    		Personal (n=19)1135[27,34-42]


    		Shifting roles (n=5)212[34,36,42]

    		Process management (n=22)


    		Quality improvement (n=11)434[27,34,38,40]


    		Mistake-proofing (n=11)137[27,37-39,41-43]

    		Total process302133
    Outcome (n=88)

    		Health status (n=10)


    		Clinical/functional (n=3)102[36,41,43]


    		Intrapersonal (n=7)601[34,36,37,41,42]

    		Experience of care receiver (n=23)


    		Satisfaction (n=16)1132[34-38,42]


    		Convenience (n=7)700[36,38,42]

    		Experience of health care professional (n=25)


    		“What’s in it for me” (n=15)924[27,34,36-38,40,42]


    		“What’s in it for them” (n=10)1000[27,34,36-38,41-43]

    		Efficiency(n=30)


    		Operations (n=27)4914[34-43]


    		Revenues (n=3)111[27,41,43]

    		Total outcome491524

    aEHR: electronic health record.

    Distribution of Indicators Within the Themes and Subthemes of the Structure Category

    In the structure category, most indicators were reported in the inner setting (51/175, 29.1%), care receiver (40/175, 22.9%), and technology (38/175, 21.7%) themes. The indicators in the inner setting (n=22) and technology (n=23) themes were mainly classified as a disadvantage to the integration, whereas those in the care receiver theme (n=23) were mainly classified as neutral. Regarding the subthemes, most indicators were reported in the support of the primary process subtheme within the inner setting theme (34/175, 19.4%), the baseline characteristics subtheme within the care receiver theme (22/175, 12.6%), and the usability and functionality subtheme within the technology theme (33/175, 18.9%), as shown in Table 4.

    Distribution of Indicators Within the Themes and Subthemes of the Process Category

    Almost half of the indicators were organized within the health care actions theme (38/84, 45%), which were diversely reported as an advantage (n=13), disadvantage (n=11), and neutral (n=15). The subthemes with the most reported indicators were workflow (18/84, 21%), patient-centered (20/84, 24%), both within the health care actions theme, and the personal subtheme (19/84, 23%) within the interpersonal actions theme (Table 4).

    Distribution of Indicators Within the Themes and Subthemes of the Outcome Category

    In the outcome category, the most frequently reported indicators were from the efficiency theme (30/88, 34%), with advantages (n=5) reported for very few indicators. The “experiences” themes of care receivers and health care professionals together accounted for 55% (48/88), both predominated by advantages (n=37). The highest number of indicators were reported in the operations subtheme (n=27/88, 31%; Table 4).

    Most Reported Indicators

    An in-depth examination of the distribution of the indicators showed that the following four indicators were the most reported (ie, reported 10 times or more) among the included studies: “deployment of human resources” (n=11) of the inner setting theme in the structure category; “ease of use” (n=16) and “technical issue” (n=10), both belonging to the technology theme in the structure category; and “health logistics” (n=26) of the efficiency theme in the outcome category. An overview of all indicators is presented in Multimedia Appendix 3.

    Interactions Among Indicators Organized into Themes and Categories

    Overview

    Of the 11 included studies, 10 (91%) reported interactions among indicators organized by themes within the structure, process, and outcome categories. The most frequently reported interaction among indicators at the category level was between the structure and outcome categories (14 times). The most frequently reported interaction among indicators at the theme level was between the care receiver theme within the structure category and the efficiency theme within the outcome category (8 times), as shown in Figure 6.

    ‎
    Figure 6. Interactions among indicators within themes and categories. The numbers within the blue circles represent the number of noted interactions among indicators within the themes. The x-axis represents the antecedent, and the y-axis represents the (intermediate) result.
    View this figure
    Interactions With Themes in the Structure Category

    All themes in the structure category contained indicators as an antecedent to, or as an intermediate result of other indicators. The inner setting (n=16), technology (n=12), and care receiver (n=11) themes represented the highest number of interactions with other themes. Inner setting was noted 7 times as an antecedent and 9 times as an intermediate result. Technology was noted 11 times as an antecedent and once as an intermediate result. Care receiver was noted 10 times as an antecedent and once as an intermediate result. The health care professional (n=3) and outer setting (n=1) themes were noted less frequently (Figure 6).

    Interactions With Themes in the Process Category

    In all themes in the process category, the indicators displayed interactions with indicators of other themes; specifically, health care theme (n=15), noted 7 times as an antecedent and 8 times as an intermediate result; interpersonal actions theme (n=11), 5 times as an antecedent and 6 times as an intermediate result; process management theme (n=9), 8 times as antecedent and once as an intermediate result (Figure 6).

    Interactions With Themes in the Outcome Category

    In the outcome category, the efficiency theme (n=15) contained most of the interacting indicators, all as an intermediate result. The other themes, including health status (n=3), experience of health care receiver (n=1), and experience of health care provider (n=2), were noted less frequently as (intermediate) results (Figure 6).

    Examples of interactions among the indicators and the associated themes are illustrated in Textbox 2.

    Illustrations of reported interactions among indicators and their themes. Indicator names are written in italics as reported in the published studies (followed by the corresponding themes and categories in parentheses).
    • Technical and usability issues (technology theme, structure category) experienced by the health care professional negatively impacted the engagement and the internal collaboration (inner setting theme, structure category) [40] and the health care workflow by causing extra steps and workarounds (health care actions theme, process category) [37,41,43].
    • Technical and usability issues (technology theme, structure category) experienced by the care receiver challenged the care receiver to fit the application of eHealth into their daily lives (care receiver theme, structure category) and caused increased dropouts (efficiency theme, outcome category) [34,39]. Conversely, one study [36] showed that technology that is easy to use (technology theme, structure category), can contribute positively to its application, and fit into the patient’s daily life (care receiver theme, structure category).
    • Insufficient attention to the patient’s burden (care receiver theme, structure category), health literacy (care receiver theme, structure category), and whether the plan fits into their daily life (care receiver theme, structure category) caused dropouts (efficiency theme, outcome category) [36,37,39], and nonadherence to care plans (efficiency theme, outcome category) [34].
    • High workload (inner setting theme, structure category) hindered the incorporation of the application into daily practice (inner setting theme, structure category) [40].
    • Lack of time (inner setting theme, structure category) discouraged health care professionals from their intention to (re)use (experience health care professional theme, outcome category) [37] and health care professionals did not experience an added value for themselves (experience health care professional theme, outcome category) [37].
    • Communicated added value (inner setting theme, structure category) on a corporate level positively influenced the collective engagement (inner setting theme, structure category) [40].
    • Guidelines on the work process (process management, process category) made the work process easier and faster for health care professionals (health care actions theme, process category) but limited the adaptability of the technology for certain recipients (technology theme, structure category) [42].
    • Limited feedback about the quality of care (process management theme, process category) made specialists feel uncertain about the suitability of the technology (health care professional theme, structure category) [38], whereas sharing information (process management theme, process category) to improve program efficiency allowed the program to be a part of the workflow (health care actions theme, process category) [38].
    • Face-to-face contact (health care actions theme, process category) benefitted the personal connection between care receiver and professional (interpersonal actions theme, process category) and the engagement of the care receiver with the treatment (interpersonal actions theme, process category) [34].
    • Personal assistance (health care actions theme, process category) and personalized therapy (health care actions theme, process category) increased the usage of the intervention by the care receiver (efficiency theme, outcome category) [39].
    • Personalized therapy (health care actions theme) also increased the satisfaction of the care receiver (experience of care receiver theme, outcome category) [36].
    • Exceptions to the operational process (health care actions theme, process category) were made too often, such as providing extra support to patients (health care actions theme, process category), or providing less care (health care actions theme, process category), creating new administrative workarounds (health care actions theme, process category) caused by technical issues (technology theme, structure category) [35,37,38,41,43] or high workloads (inner setting theme, structure category) [27].
    • An increase in questioning by professionals (interpersonal actions theme, process category) made carers feel more engaged and knowledgeable (health status theme, outcome category) [42].
    • Recipients’ detailed input (interpersonal actions theme, process category) on the assignments enabled professionals to empathize with their situation and focus on their feedback (interpersonal actions theme, process category) [34].
    Textbox 2. Illustrations of reported interactions among indicators and their themes. Indicator names are written in italics as reported in the published studies (followed by the corresponding themes and categories in parentheses).

    Principal Findings

    This literature review analyzed how eHealth can be organized optimally by using the Donabedian SPO framework. General organizational developments were identified, regardless of the type of illness, setting, or the eHealth application used. A review of the literature of selected cases highlighted three important findings. First, the role of the care recipient needs to be incorporated into the organizational structure and daily care process. Second, the technology must be well attuned to the structure of the organization and daily care process. Third, the deployment of the human resources to the daily processes needs to be aligned with the desired end results. Not adhering to these points could negatively affect the organization, daily process, or the end results. Findings from this research using the Donabedian framework corresponds to the conclusions of other studies using different research methodologies, which is explained below.

    First, the SPO analysis showed that the care recipient plays a crucial role in the successful integration of eHealth. Patient-centered interaction and communication are important, to activate patients in managing their health care and to improve health outcomes in the application of eHealth [5,31,44-46]. Kuipers et al [44] and Rathert [45] demonstrated with systematic literature reviews that patient-centered care and co-creation are positively associated with the physical and social well-being of patients and with satisfaction of patients and health care professionals. These findings are in line with the review of Wildevuur and colleagues [46], demonstrating that organizations that are more patient centered with eHealth interventions achieve better outcomes with regard to patient health and quality of life. Although most health care professionals embrace more patient involvement and engagement, delegating power and responsibilities could be a challenge for health care professionals’ authority [47,48]. Another important issue is knowing who the customers are, what they want, and how the customer’s demand is answered [49]. A previous study reported that eHealth is not suitable for all care receivers [18,50]. Therefore, identifying who benefits most from which kind of therapy is an essential addition to the screening process, and it could lead to more effective targeting and resourcing [51]. Furthermore, insufficiently incorporating the patients’ family, work, and life goals into care plans will likely result in dropouts or nonadherence to care plans [50].

    The second noteworthy finding is the essential role of excellent technology in the integration of eHealth. The way technology is set up has an influence on the working environment of health care professionals [52]. Inflexibility and complexity of the technology comes at the expense of effective daily processes and their quality [53,54]. Several studies demonstrated that the adaptability of eHealth technologies to fit to the local context, its ease of use, and its integration into clinical workflow benefit the users’ acceptance and meaningful use [22,55,56]. This was also reflected in the early phase of the COVID-19 pandemic, where rapid scalable technologies were the easiest to use and quickly implementable [53]. However, the health care system continued to face challenges in adopting digital technology after the first emergency phase of the COVID-19 pandemic, due to inadequate information and communication technology infrastructure and a bad fit of the technology into the clinical workflow that is primarily designed for face-to-face care [53]. Granja et al [54] demonstrated that the application of eHealth is often not fitted to the existing workflow due to time and space constraints and breaking of traditions. Although eHealth is seen as an innovative solution for alleviating the increasing burden for health care professionals [2], it could have a counterproductive effect on the working conditions for employees if the technology is not properly adapted to the structure and processes [57,58]. Third, integrating eHealth into a health care organization requires adjustments of the care processes and utilization of the human resources, with appropriate process monitoring. Working with eHealth also poses logistical challenges; for example, a clear understanding is needed of the expected achievements, processes, and staffing requirements in order to bring about changes and create new capabilities [59]. Vissers and De Vries [49] pointed out that it is necessary to know how the logistical capacities should be assigned to the process, how the processes are measured, and who is responsible for the management of the process. Changes in the workflow are inevitable and necessary for eHealth interventions to be successful [54]. However, integrating eHealth technology into daily care processes is complex, and it needs coordination and process communication [19]. For example, a living laboratory experiment conducted over 3 years with patients, health care professionals, enterprises, and researchers to accelerate the integration of eHealth in daily practice showed that workflow, responsibilities, and roles needed to change, but health care professionals did not know how to approach this and had difficulties in integrating eHealth into their daily care processes [18].

    Strength and Limitations

    The strengths of this research are that international studies were included and represented a wide range of patient groups and settings. The findings were representative for the included studies, and they were not dependent of the study design, disease, target population, setting, or type or function of the eHealth application used. The wide range of settings of the included studies is supportive of a broader application of the present study’s findings. In the Methods, we stated that there is no clear consensus on what constitutes as good eHealth and how it is best organized [3,19]. Nevertheless, we believe that our findings make a significant contribution to improving the integration of eHealth in regular health care by identifying the most common indicators in the organization’s structure, processes, and outcomes. Thus, this research contributes to a new model for integrating organizational, health, and social factors.

    A limitation of this study is that the health outcomes were rarely mentioned in this review. We hypothesized that this is because the main method used in the included studies was process evaluation. Therefore, although the health outcomes played a major role in earlier RCTs, this was not the case in process evaluation studies. The included studies did not define clear standards for the indicators to determine their quality. However, an indicator only becomes meaningful if a standard is specified [60,61]. There are also limitations in the selection procedure. The interrater reliability was not calculated. Due to this complex, broad topic, the predefined inclusion and exclusion criteria were sharpened at the time of selection. It was an iterative process, with a lot of consultation and coordination. In the process, full consensus was reached for all inclusion and exclusion criteria for selection at each step of the research. Another limitation is the classification of indicators into subthemes and themes at the discretion of the authors. It is conceivable that different classifications would reach different conclusions. Yet, the conclusions of each included study fit with the overall conclusion; therefore, the chance of this bias seems to be small. However, the findings of this literature review are dependent on the results of the included studies and may be subject to publication bias. Even though the included publications contain either positive or negative results (eg, a failed randomized trial [35] or interventions with no or less impact [40,43]), a chance of publication bias cannot be precluded automatically [62,63].

    It is also noted that the Donabedian framework itself was designed before the introduction of eHealth and may not include the latest prevailing ideas on the organization of health care. For this reason, the model has been adapted in order to represent eHealth. By doing so, an attempt has been made to reduce the limitation as far as possible. Nevertheless, this literature review confirmed that it is still useful to analyze what contributes to the successful integration of eHealth into traditional health care. Additionally, there are other reputable models for evaluating eHealth interventions, such as the nonadoption, abandonment, scale-up, spread, sustainability (NASSS) framework [20]; Consolidated Framework for Implementation Research (CFIR) [64]; and the holistic framework to improve the uptake and impact of eHealth technologies [19]. These models describe the different phases from the design of the intervention to its adoption and implementation. This literature review focused on quality improvement of the way eHealth is organized, that has already passed the initial phase (of design and adoption). The Donabedian framework covers all relevant aspects for sustaining the integration of eHealth into health care and the interrelations of organization’s structure, processes, and outcomes, as well as integrating these aspects with human and social factors, after the adoption and uptake phase of eHealth.

    Conclusions

    For optimal integration of eHealth into health care, the following main principles should be considered and approached simultaneously. First, the role of the care recipient needs to be incorporated in the organizational structure and daily care process. Second, the technology must be well attuned to the structure of the organization and daily care process. Third, the deployment of human resources to the care process needs to be aligned with the desired end results.

    Thus far, no study has presented a complete overview of the successful and effective organization of eHealth. Therefore, it is desirable to supplement this research with knowledge from other sources, such as in-depth research into the experiences from different perspectives, as this can help us to obtain a complete overview of how eHealth can be successfully integrated into health care organizations.

    Conflicts of Interest

    No conflict specified.

    ‎
    Multimedia Appendix 1

    Search strategy.

    DOC File , 143 KB
    ‎
    Multimedia Appendix 2

    Explanatory notes on structure, process, and outcomes, and the (sub)themes.

    DOCX File , 20 KB
    ‎
    Multimedia Appendix 3

    Unique reported indicators.

    XLSX File (Microsoft Excel File), 22 KB
    1. van Lettow B, Wouters M, Sinnige J. E-health, what is that? Webpage in Dutch. Nictiz.   URL: https://www.nictiz.nl/rapporten/e-health-wat-is-dat/ [accessed 2021-04-26]
    2. Nijland N. Grounding eHealth: towards a holistic framework for sustainable eHealth technologies. University of Twente. 2011 Jan 21.   URL: https:/​/research.​utwente.nl/​en/​publications/​grounding-ehealth-towards-a-holistic-framework-for-sustainable-eh [accessed 2021-04-26]
    3. Ossebaard HC, Van Gemert-Pijnen L. eHealth and quality in health care: implementation time. Int J Qual Health Care 2016 Jun;28(3):415-419. [CrossRef] [Medline]
    4. Christensen CM. Disruptive innovation: can health care learn from other industries? A conversation with Clayton M. Christensen. Interview by Mark D. Smith. Health Aff (Millwood) 2007;26(3):w288-w295. [CrossRef] [Medline]
    5. Heckemann B, Wolf A, Ali L, Sonntag SM, Ekman I. Discovering untapped relationship potential with patients in telehealth: a qualitative interview study. BMJ Open 2016 Mar 02;6(3):e009750 [FREE Full text] [CrossRef] [Medline]
    6. van der Kleij RMJJ, Kasteleyn MJ, Meijer E, Bonten TN, Houwink EJF, Teichert M, et al. SERIES: eHealth in primary care. Part 1: Concepts, conditions and challenges. Eur J Gen Pract 2019 Oct;25(4):179-189 [FREE Full text] [CrossRef] [Medline]
    7. Talboom-Kamp E, Tossaint-Schoenmakers R, Goedhart A, Versluis A, Kasteleyn M. atients' attitudes toward an online patient portal for communicating laboratory test results: Real-world study using the eHealth impact questionnaire. JMIR Form Res 2020 Mar 04;4(3):e17060 [FREE Full text] [CrossRef] [Medline]
    8. Atlas of eHealth country profiles: the use of eHealth in support of universal health coverage. World Health Organization. Geneve, Switzerland: World Health Organization; 2016 Jan 01.   URL: https://www.who.int/publications/i/item/9789241565219 [accessed 2021-04-29]
    9. Chavennes N.H.. eHealth in Disease Management: doel of tool? 2015. https://scholarlypublications.universiteitleiden.nl/handle/1887/51560. Leiden, Netherlands: Leiden University; 2015 Dec 11.   URL: https://hdl.handle.net/1887/51560 [accessed 2021-04-29]
    10. Pagliari C, Sloan D, Gregor P, Sullivan F, Detmer D, Kahan JP, et al. What is eHealth (4): a scoping exercise to map the field. J Med Internet Res 2005 Mar 31;7(1):e9 [FREE Full text] [CrossRef] [Medline]
    11. Eysenbach G. What is e-health? J Med Internet Res 2001;3(2):E20 [FREE Full text] [CrossRef] [Medline]
    12. Matamala-Gomez M, Maisto M, Montana JI, Mavrodiev PA, Baglio F, Rossetto F, et al. The role of engagement in teleneurorehabilitation: A systematic review. Front Neurol 2020 May 6;11:354 [FREE Full text] [CrossRef] [Medline]
    13. Penedo FJ, Oswald LB, Kronenfeld JP, Garcia SF, Cella D, Yanez B. The increasing value of eHealth in the delivery of patient-centred cancer care. The Lancet Oncology 2020 May;21(5):e240-e251. [CrossRef]
    14. Vassilev I, Rowsell A, Pope C, Kennedy A, O'Cathain A, Salisbury C, et al. Assessing the implementability of telehealth interventions for self-management support: a realist review. Implement Sci 2015 Apr 24;10:59 [FREE Full text] [CrossRef] [Medline]
    15. Boers SN, Jongsma KR, Lucivero F, Aardoom J, Büchner FL, de Vries M, et al. SERIES: eHealth in primary care. Part 2: Exploring the ethical implications of its application in primary care practice. Eur J Gen Pract 2020 Dec;26(1):26-32 [FREE Full text] [CrossRef] [Medline]
    16. Talboom-Kamp EP, Verdijk NA, Kasteleyn MJ, Numans ME, Chavannes NH. From chronic disease management to person-centered eHealth; a review on the necessity for blended care. Clinical eHealth 2018 Mar;1(1):3-7. [CrossRef]
    17. Wildevuur SE, Simonse LW, Groenewegen P, Klink A. Information and communication technology enabling partnership in person-centred diabetes management: building a theoretical framework from an inductive case study in The Netherlands. BMJ Open 2019 Jun 16;9(6):e025930 [FREE Full text] [CrossRef] [Medline]
    18. Swinkels ICS, Huygens MWJ, Schoenmakers TM, Oude Nijeweme-D'Hollosy W, van Velsen L, Vermeulen J, et al. Lessons learned from a living lab on the broad adoption of eHealth in primary health care. J Med Internet Res 2018 Mar 29;20(3):e83 [FREE Full text] [CrossRef] [Medline]
    19. van Gemert-Pijnen JEWC, Nijland N, van Limburg M, Ossebaard HC, Kelders SM, Eysenbach G, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res 2011 Dec 05;13(4):e111 [FREE Full text] [CrossRef] [Medline]
    20. Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A'Court C, et al. Beyond adoption: A new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 2017 Nov 01;19(11):e367 [FREE Full text] [CrossRef] [Medline]
    21. Demiris G, Afrin LB, Speedie S, Courtney KL, Sondhi M, Vimarlund V, et al. Patient-centered applications: use of information technology to promote disease management and wellness. A white paper by the AMIA knowledge in motion working group. J Am Med Inform Assoc 2008;15(1):8-13 [FREE Full text] [CrossRef] [Medline]
    22. Ross J, Stevenson F, Lau R, Murray E. Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update). Implement Sci 2016 Oct 26;11(1):146 [FREE Full text] [CrossRef] [Medline]
    23. Boogerd EA, Arts T, Engelen LJ, van de Belt TH. "What is eHealth": Time for an update? JMIR Res Protoc 2015 Mar 12;4(1):e29 [FREE Full text] [CrossRef] [Medline]
    24. Donabedian A. The quality of care. How can it be assessed? JAMA 1988;260(12):1743-1748. [CrossRef] [Medline]
    25. Donabedian A. Evaluating the quality of medical care. 1966. Milbank Q 2005;83(4):691-729 [FREE Full text] [CrossRef] [Medline]
    26. Rademakers J, Delnoij D, de Boer D. Structure, process or outcome: which contributes most to patients' overall assessment of healthcare quality? BMJ Qual Saf 2011 Apr;20(4):326-331. [CrossRef] [Medline]
    27. Hadjistavropoulos HD, Nugent MM, Dirkse D, Pugh N. Implementation of internet-delivered cognitive behavior therapy within community mental health clinics: a process evaluation using the consolidated framework for implementation research. BMC Psychiatry 2017 Sep 12;17(1):331 [FREE Full text] [CrossRef] [Medline]
    28. Bodenheimer T, Sinsky C. From triple to quadruple aim: care of the patient requires care of the provider. Ann Fam Med 2014;12(6):573-576 [FREE Full text] [CrossRef] [Medline]
    29. Sikka R, Morath JM, Leape L. The Quadruple Aim: care, health, cost and meaning in work. BMJ Qual Saf 2015 Oct;24(10):608-610. [CrossRef] [Medline]
    30. Hong Q, Pluye P, Fàbregues S. Mixed Methods Appraisal Tool (MMAT), Version 2018. User guide. Montréal, Canada;: McGill University; 2018 Aug 01.   URL: http:/​/mixedmethodsappraisaltoolpublic.​pbworks.com/​w/​file/​fetch/​127916259/​MMAT_2018_criteria-manual_2018-08-01_ENG.pdf%0Ahttp:/​/mixedmethodsappraisaltoolpublic.​pbworks.com/​ [accessed 2021-04-29]
    31. Krijgsman GKWJ. Ordening in de wereld van eHealth (whitepaper). https://www.nictiz.nl/whitepapers/ordening-in-de-wereld-van-ehealth/. Den Haag, Netherlands: Nictiz; 2012 Aug 24.   URL: https://www.nictiz.nl/whitepapers/ordening-in-de-wereld-van-ehealth/ [accessed 2021-04-29]
    32. Meyer I, Hüsing T, Didero M, Korte W. eHealth Benchmarking (Phase II). Final report. Bonn, Germany; 2009 Mar.   URL: https:/​/joinup.​ec.europa.eu/​sites/​default/​files/​document/​2014-12/​eHealth%20Benchmarking%20%28Phase%20II%29-Final%20Report.​pdf [accessed 2021-04-29]
    33. Cowie MR. National Institute for Health and Care Excellence. Eur Heart J 2015 Jan 21;36(4):195. [Medline]
    34. Boots LM, de Vugt ME, Smeets CM, Kempen GI, Verhey FR. Implementation of the blended care self-management program for caregivers of people with early-stage dementia (Partner in Balance): Process evaluation of a randomized controlled trialmplementation of the blended care self-management program for caregivers of people with early-stage dementia (Partner in Balance): Process evaluation of a randomized controlled trial. J Med Internet Res 2017 Dec 19;19(12):e423 [FREE Full text] [CrossRef] [Medline]
    35. Thies K, Anderson D, Cramer B. Lack of adoption of a mobile app to support patient self-management of diabetes and hypertension in a federally qualified health center: Interview analysis of staff and patients in a failed randomized trial. JMIR Hum Factors 2017 Oct 03;4(4):e24 [FREE Full text] [CrossRef] [Medline]
    36. Hinman RS, Nelligan RK, Bennell KL, Delany C. "Sounds a bit crazy, but it was almost more personal:" A qualitative study of patient and clinician experiences of physical therapist-prescribed exercise for knee osteoarthritis via Skype. Arthritis Care Res (Hoboken) 2017 Dec;69(12):1834-1844. [CrossRef] [Medline]
    37. van Doorn-van Atten MN, de Groot LC, Romea AC, Schwartz S, de Vries JH, Haveman-Nies A. Implementation of a multicomponent telemonitoring intervention to improve nutritional status of community-dwelling older adults: a process evaluation. Public Health Nutr 2019 Feb;22(2):363-374. [CrossRef] [Medline]
    38. Fort M, Namba LM, Dutcher S, Copeland T, Bermingham N, Fellenz C, et al. Implementation and evaluation of a safety net specialty care program in the Denver Metropolitan Area. Perm J 2017;21:16-022 [FREE Full text] [CrossRef] [Medline]
    39. Talboom-Kamp EP, Verdijk NA, Kasteleyn MJ, Harmans LM, Talboom IJ, Numans ME, et al. High level of integration in integrated disease management leads to higher usage in the e-Vita Study: Self-management of chronic obstructive pulmonary disease with web-based platforms in a parallel cohort design. J Med Internet Res 2017 May 31;19(5):e185 [FREE Full text] [CrossRef] [Medline]
    40. Patel B, Usherwood T, Harris M, Patel A, Panaretto K, Zwar N, et al. What drives adoption of a computerised, multifaceted quality improvement intervention for cardiovascular disease management in primary healthcare settings? A mixed methods analysis using normalisation process theory. Implement Sci 2018 Nov 12;13(1):140 [FREE Full text] [CrossRef] [Medline]
    41. Dijkstra A, Heida A, van Rheenen PF. Exploring the challenges of implementing a web-based telemonitoring strategy for teenagers with inflammatory bowel disease: Empirical case study. J Med Internet Res 2019 Mar 29;21(3):e11761 [FREE Full text] [CrossRef] [Medline]
    42. Mitchell M, Getchell M, Nkaka M, Msellemu D, Van Esch J, Hedt-Gauthier B. Perceived improvement in integrated management of childhood illness implementation through use of mobile technology: qualitative evidence from a pilot study in Tanzania. J Health Commun 2012;17 Suppl 1:118-127. [CrossRef] [Medline]
    43. Lycett K, Wittert G, Gunn J, Hutton C, Clifford SA, Wake M. The challenges of real-world implementation of web-based shared care software: the HopSCOTCH Shared-Care Obesity Trial in Children. BMC Med Inform Decis Mak 2014 Jul 24;14:61 [FREE Full text] [CrossRef] [Medline]
    44. Kuipers SJ, Cramm JM, Nieboer AP. The importance of patient-centered care and co-creation of care for satisfaction with care and physical and social well-being of patients with multi-morbidity in the primary care setting. BMC Health Serv Res 2019 Jan 08;19(1):13 [FREE Full text] [CrossRef] [Medline]
    45. Rathert C, Wyrwich MD, Boren SA. Patient-centered care and outcomes: a systematic review of the literature. Med Care Res Rev 2013 Aug;70(4):351-379. [CrossRef] [Medline]
    46. Wildevuur SE, Simonse LWL. Information and communication technology-enabled person-centered care for the "big five" chronic conditions: scoping review. J Med Internet Res 2015 Mar 27;17(3):e77 [FREE Full text] [CrossRef] [Medline]
    47. Tistad M, Lundell S, Wiklund M, Nyberg A, Holmner, Wadell K. Usefulness and relevance of an eHealth tool in supporting the self-management of chronic obstructive pulmonary disease: Explorative qualitative study of a cocreative process. JMIR Hum Factors 2018 Oct 26;5(4):e10801 [FREE Full text] [CrossRef] [Medline]
    48. Stoilkova-Hartmann A, Janssen DJA, Franssen FME, Spruit MA, Wouters EFM. Attitudes of healthcare professionals providing pulmonary rehabilitation toward partnership in care. Heart Lung 2015;44(4):347-352. [CrossRef] [Medline]
    49. Vissers J, Vries, de G. Sleutelen Aan Zorgprocessen. https://www.eur.nl/sites/corporate/files/Oratie_Jan_Vissers-Guus_de_Vries_Sleutelen_aan_Zorgprocessen_0.pdf. Rotterdam, Netherlands: Erasmus University; 2005 Apr 01.   URL: https:/​/www.​eur.nl/​sites/​corporate/​files/​Oratie_Jan_Vissers-Guus_de_Vries_Sleutelen_aan_Zorgprocessen_0.​pdf [accessed 2021-04-29]
    50. Bellows J, Young S, Chase A. Person-focused care at Kaiser Permanente. Perm J 2014;18(1):90-91 [FREE Full text] [CrossRef] [Medline]
    51. Newman S, Steed L, Mulligan K. Self-management interventions for chronic illness. Lancet 2004;364(9444):1523-1537. [CrossRef] [Medline]
    52. van Merode F, van Raak A. Beheersing in de Zorg. Leerboek over Beheersingsvraagstukken Binnen Zorginstellingen En Zorgketens. In: Reed Business BV. Doetinchem, Netherlands: Reed Business BV; 2001.
    53. Golinelli D, Boetto E, Carullo G, Nuzzolese AG, Landini MP, Fantini MP. Adoption of digital technologies in health care during the COVID-19 pandemic: Systematic review of early scientific literature. J Med Internet Res 2020 Nov 06;22(11):e22280 [FREE Full text] [CrossRef] [Medline]
    54. Granja C, Janssen W, Johansen MA. Factors determining the success and failure of eHealth interventions: Systematic review of the literature. J Med Internet Res 2018 May 01;20(5):e10235 [FREE Full text] [CrossRef] [Medline]
    55. Safi S, Thiessen T, Schmailzl KJ. Acceptance and resistance of new digital technologies in medicine Qualitative study. JMIR Res Protoc 2018 Dec 04;7(12):e11072 [FREE Full text] [CrossRef] [Medline]
    56. Bowens FM, Frye PA, Jones WA. Health information technology: integration of clinical workflow into meaningful use of electronic health records. Perspect Health Inf Manag 2010 Oct 01;7:1d [FREE Full text] [Medline]
    57. Wouters M, Huygens M, Voogdt H, Meurs M, Groot, de J, Lamain A, et al. Samen aan zet! eHealth-monitor 2019. https://www.nivel.nl/nl/publicatie/samen-aan-zet-ehealth-monitor-2019. Den Haag and Utrecht, Netherland: Nictiz and het Nivel; 2019 Nov.   URL: https://www.nivel.nl/nl/publicatie/samen-aan-zet-ehealth-monitor-2019 [accessed 2021-04-29]
    58. MacNeill V, Sanders C, Fitzpatrick R, Hendy J, Barlow J, Knapp M, et al. Experiences of front-line health professionals in the delivery of telehealth: a qualitative study. Br J Gen Pract 2014 Jul;64(624):e401-e407 [FREE Full text] [CrossRef] [Medline]
    59. Dansky KH, Thompson D, Sanner T. A framework for evaluating eHealth research. Eval Program Plann 2006 Nov;29(4):397-404. [CrossRef] [Medline]
    60. Zijlstra W. Quality Indicators in Healthcare. Webpage in Dutch. ZBC Kennisbank. Zeist, Netherlands; 2012 Jan 23.   URL: https://zbc.nu/management/ontwikkeling-zorginstelling/kwaliteitsindicatoren-in-de-zorg/ [accessed 2021-04-29]
    61. Harteloh P, Casparie A. Kwaliteit van zorg. Van Een Zorginhoudelijke Benadering Naar Een Bedrijfskundige Aanpak. In: P.M. H, Casparie AM. Maarssen, The Netherlands: Elsevier/ De Tijdstroom; 1998:144.
    62. Müller KF, Briel M, D'Amario A, Kleijnen J, Marusic A, Wager E, et al. Defining publication bias: protocol for a systematic review of highly cited articles and proposal for a new framework. Syst Rev 2013 May 21;2:34 [FREE Full text] [CrossRef] [Medline]
    63. Dickersin K, Min YI. Publication bias: the problem that won't go away. Ann N Y Acad Sci 1993 Dec 31;703:135-46; discussion 146. [CrossRef] [Medline]
    64. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009 Aug 07;4:50 [FREE Full text] [CrossRef] [Medline]

    ‎
    CFIR: Consolidated Framework for Implementation Research
    MMAT: Mixed Method Appraisal Tool
    NASSS: nonadoption, abandonment, scale-up, spread, sustainability
    NICE: National Institute for Health and Care Excellence
    RCT: randomized controlled trial
    PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-analyses
    SPO: Structure-Process-Outcome

    Edited by G Eysenbach; submitted 14.01.21; peer-reviewed by C Basch, X Cheng; comments to author 27.01.21; revised version received 18.02.21; accepted 07.04.21; published 10.05.21

    Copyright

    ©Rosian Tossaint-Schoenmakers, Anke Versluis, Niels Chavannes, Esther Talboom-Kamp, Marise Kasteleyn. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 10.05.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.