Evaluation Criteria of Noninvasive Telemonitoring for Patients With Heart Failure: Systematic Review

Introduction

Heart failure (HF) affects 26 million people worldwide, incurring direct and indirect costs of more than US $100 billion per year [1,2]. HF causes 1% to 2% of all hospitalizations, representing a major burden for patients and the health care system [3]. Number of hospitalizations is a marker of disease instability, and studies have shown that HF management can reduce this number [4].

Telemonitoring is a branch of telemedicine defined as the use of communication technologies to monitor and transmit data on the health status of patients to distant care providers [5,6]. It differs from teleconsultation, where there is a real-time interactive video or audio consultation between the patient and a distant health care provider. It also differs from tele-expertise, where a health care provider presents a patient case and gets advice from a distant colleague through a dedicated system, without direct patient involvement, in real or deferred time.

Systematic reviews of telemonitoring projects have shown reduced hospitalization rates for acute HF [7-9]. Both the European Society of Cardiology and the American Heart Association support the use of telemonitoring to improve the care of HF patients [10,11]. HF telemonitoring can involve the use of invasive or noninvasive monitoring devices. Invasive devices are implanted in the body, and data transmission is not controlled by the patient. Noninvasive devices, like weight scales or blood pressure monitors, are used, mostly by the patient, to self-monitor physiological measurements, signs, or symptoms of the disease.

The information flow is much more intricate with noninvasive than with invasive telemonitoring. First, human action is needed to handle noninvasive telemonitoring devices whereas monitored data is automatically recorded and sent by invasive telemonitoring devices. Second, patients (or family caregivers) are major actors of the noninvasive telemonitoring process whereas their involvement is minimal with invasive telemonitoring. Third, nurses and primary care physicians are usually involved in the noninvasive telemonitoring process whereas they do not take part in invasive monitoring. Fourth, as a result of these differences, noninvasive HF telemonitoring requires readiness for change, education, and training of patients and caregivers whereas invasive HF telemonitoring does not [8]. Due to these major differences, this review focuses on noninvasive HF telemonitoring.

The development of evaluation criteria for electronic tools is considered to be a critical step by the European Society of Cardiology [10]. Standard evaluation frameworks are useful to encourage systematic evaluation and get conclusive results that can be compared or aggregated across programs, allowing the analysis of determinants of success and failure for efficient resource allocation. Standard evaluation frameworks have been used for the evaluation of teleconsultation and tele-expertise [12-16]. To our knowledge, they have not yet been used for the evaluation of telemonitoring projects. Telemonitoring interventions are complex: they involve many different actors with different backgrounds (health care professional, patients and family, technicians, payers), they use technical devices, and they change the usual process of care. They can impact health care on many levels [17]: patient access to care, health and quality of life, patient and care provider education, family and care provider workload, organization of the patient care pathway, health care costs, and more. A comprehensive telemonitoring evaluation framework therefore needs to be multidimensional.

Our aim was to perform a systematic review of criteria used for the evaluation of noninvasive HF telemonitoring projects, describe how they are used in evaluation studies, and organize them into a consistent scheme.

Methods

Information Sources and Eligibility Criteria

We did not submit a review protocol to a prospective register. We searched Medical Literature Analysis and Retrieval System Online (MEDLINE), Excerpta Medica database (EMBASE), and Web of Science for articles published from January, 1990, to the query date (August 15, 2015) using the queries displayed in Multimedia Appendix 1. The search strategy for each bibliographic database was internally discussed, piloted, and refined by the authors but not submitted for external peer review. We checked reference lists of included articles to identify additional studies. We also took advantage of 3 Cochrane reviews on telemonitoring, published shortly after the last query date of our review, to look for missed studies [18-20].

Articles were eligible if they were original reports of a noninvasive telemonitoring project evaluation study for HF using explicit evaluation criteria. Articles were excluded if they did not contain original data (reviews, editorials, position papers, etc), were not written in English, focused on other types of telemedicine (teleconsultation, tele-expertise, etc), and if they reported invasive telemonitoring for HF through implantable devices.

Study Selection and Data Collection

After eliminating duplicate articles, titles and abstracts were independently screened by 2 readers to exclude obviously irrelevant articles. Discordant classifications between the 2 readers were resolved through discussion. The full text of remaining articles was read by 1 investigator, who applied eligibility and exclusion criteria. The final selection was cross-checked by a second investigator. Characteristics of the telemonitoring project, characteristics of the evaluation process, and evaluation criteria were systematically abstracted by 1 investigator and cross-checked by another. The collected data are reported in Multimedia Appendix 2.

Synthesis of Results

A preliminary list of broad evaluation dimensions was adapted by 2 investigators from previously published evaluation frameworks for telemedicine [12-16]. This categorization was then iteratively refined to meld the evaluation criteria found in each reviewed study into a consistent scheme.

Results

Study Selection and Characteristics

The queries of bibliographic databases identified 328 potentially eligible articles, and we included 128 articles in the review (reference list is in Multimedia Appendix 3). The review flowchart describes the process and reasons for exclusion (Figure 1). Characteristics of the 128 studies are reported in Multimedia Appendix 4, and a summary is presented in Table 1.

Europe and the United States contributed the most to the assessment of HF telemonitoring (50% of studies performed in Europe and 41% in the United States). The first study began in 1997 in the United States [21]. Europe started to carry out research to assess HF telemonitoring 3 years later [22-24].

Telemonitoring Characteristics

Some features were highly prevalent across the telemonitoring projects: 80% were carried out by a cardiology team, and the care providers were most often HF nurses (86% of projects). Patients were actively involved in 100% of the projects, but the family of the patients, psychologists, and technicians rarely participated. In 75% of studies, patients were included in the telemonitoring program at discharge from a hospitalization for acute heart failure. The phase of the project lifecycle was clear and identifiable in 125 articles: most of these projects were in the implementation phase (59%), and no project was part of routine clinical care.

The primary monitored data in the telemonitoring projects were weight, HF symptoms, heart rate, and blood pressure. These data were transmitted via telephone (verbal communication or keypad) or Internet (mobile phone or tablet). If the monitored data fell outside predefined boundaries, a warning was triggered and led to corrective actions. However, these actions were described in only 63 articles (49% of studies).

Evaluation Dimensions and Criteria

We retrieved 52 criteria from the 128 studies (Textbox 1) and classified them into 6 main dimensions: economic, clinical, educational, technical, user perspective, and organizational.

Most studies (95%) covered, at most, 3 dimensions, and none covered all 6 (Multimedia Appendix 5). Clinical and economic dimensions were assessed in over 70% of studies, whereas the educational, organizational, and technical dimensions were studied in less than 15% (Table 2).

The evaluation dimensions were not used homogeneously across all phases of project lifecycle. User perceptive was the most often covered dimension in the development phase with clinical and economic dimensions covered most in the later phases (implementation and integration).

Each dimension includes from 2 to 16 criteria (Textbox 1). The total number of criteria used per study ranged from 1 to 11 (Multimedia Appendix 6). The most often used evaluation criteria were cost and resource utilization (71%) and quality of life (51%) (Table 3). The criteria within the same dimension were also not used homogeneously across all phases of the project lifecycle. Concerning user perspective, for example, ease of use of the devices was more often evaluated in the development phase, whereas satisfaction with care was more often evaluated in the later phases of the project lifecycle (Table 3).

‎View this figure

Discussion

We found 128 studies using a total of 52 evaluation criteria categorized into 6 high-level dimensions. No study covered all 6 evaluation dimensions. The evaluation dimensions were not used with the same frequency for all phases of the project lifecycle. The principle focus in the development phase was on user perspective, whereas the focus in the latter phases of the lifecycle was on the clinical and economical dimensions. The technical, organizational, and educational dimensions were poorly evaluated overall.

The 6 dimensions were derived from telemedicine assessment frameworks Grille d'Evaluation Multidisciplinaire Santé Autonomie (multidimensional evaluation grid for health and autonomy) [12], model for assessment of telemedicine [13], Khoja-Durrani-Scott evaluation framework [14], Technologique, Ergonomique, Médicale, Sociale, Économique et Déontologique (technological, ergonomic, medical, social, economic, and ethical) [15], and the 3-dimensional model [16]. These frameworks required adaptations to better fit telemonitoring. First, the role of patients and nurses is prominent in telemonitoring projects, whereas it is more limited in teleconsultation and tele-expertise. Thus, smooth collaboration must be ensured and evaluated between patients and care providers as much as between care providers themselves. Specific organizational and educational evaluation criteria are therefore needed for telemonitoring. Second, telemonitoring stands out from a technical point of view because devices are needed to gather data. The ergonomic assessment, user perception, and technical characteristics of these devices are thus key elements in the assessment of telemonitoring projects [15].

Previous evaluation frameworks are not adapted to all phases of a telemonitoring project lifecycle and do not take into account the perspective of all telemonitoring actors and external stakeholders (manufacturers, payers, etc). For example, a telemonitoring project at the first phase of its lifecycle (pilot) cannot appropriately evaluate clinical outcomes and, at the other end, proper technical evaluation is a prerequisite long before the last phase of a telemonitoring project lifecycle (routine clinical care). Criteria and indicators for a given dimension will differ across lifecycle phases. For example, the user perspective should be evaluated in all 4 phases but with different criteria: “ease of use of the system” is an appropriate criterion in the development phase, “satisfaction with new technology” in the implementation and integration phases, and “overall satisfaction with the process of care” in routine clinical care.

This review is limited by its focus on noninvasive HF telemonitoring. However, aside from disease-specific clinical outcomes, noninvasive telemonitoring services share many technical, economic, organizational, and educational features independently of the target disease. Our framework should therefore be easy to adapt to other health conditions. Our search strategy may have missed some studies that were described by other keywords. However, we found only 7 additional studies in 3 recent Cochrane reviews [18-20]. The number of missing studies is therefore likely to be low. Our 52 criteria cover all outcomes reported in previous systematic reviews on the evaluation of telemonitoring in HF [18,25] but also in other chronic diseases, such as diabetes and chronic obstructive pulmonary diseases [19,20]. We have identified broad evaluation dimensions from previously published evaluation frameworks for telemedicine projects and refined these dimensions iteratively during the review process. The 6 final dimensions accommodate all evaluation criteria used in previously published evaluation studies, but other categorization schemes are certainly possible and should be compared. The data and methods used in this review were not suited to assess the strength, limitations, relevance, and usefulness of each retrieved evaluation criterion. A follow-up to our work is needed to provide more guidance for the use of criteria in future HF telemonitoring evaluation studies.

Comprehensive telemonitoring evaluation frameworks should cover all 6 dimensions and help users choose the appropriate dimensions and evaluation criteria depending on the phase of their telemonitoring project lifecycle and the perspective of telemonitoring actors or external stakeholders they want to adopt. Our next goal is to build such a framework for noninvasive HF telemonitoring, deliberately emphasizing the technical, organizational, and educational dimensions that have been neglected by previous telemonitoring assessment studies. We will test this framework on an ongoing HF telemonitoring project.