Economic Evaluations of Digital Health Interventions for Children and Adolescents: Systematic Review

Introduction

Background

Digital health interventions (DHIs) administered through various technologies have experienced substantial growth and uptake in recent years [1,2]. DHIs are defined as digital technologies such as digital health applications and information and communications technology systems (including SMS text messages) implemented to meet health objectives [3]. DHIs can range from electronic medical records used by providers to mobile apps tailored to patients for remote monitoring, videoconferencing systems for treatment administration training, and SMS text message reminders to promote treatment adherence [4,5]. The diverse roles that DHIs can play in the health system include replacement of face-to-face meetings with health care professionals; provision of patient education and counseling services; data collection and access; health information sharing; promotion of healthy behaviors and prevention; and facilitation of patient monitoring and support through clinical examination, diagnosis, and treatment [2,6]. DHIs compete for scarce National Health Service resources (fixed budget) with other digital and nondigital technologies in the health care system. In this context of scarcity, health economic evaluations provide an assessment of the relative benefits and costs of DHIs and competing options, which is crucial evidence for informing resource allocation decisions [2]. In the context of pediatric and adolescent care settings, the most common uses of DHIs are in mental health, particularly web-based cognitive behavioral therapies (CBTs) that include both children and their families, and in weight management programs involving patient- and caregiver-reported outcomes through mobile apps [7-9].

There is an abundance of literature evaluating the clinical effectiveness of DHIs in both children and adolescents [7-9]. Several studies have found a small but substantial effect of DHIs on health outcomes in these patient groups, primarily for interventions for depression, anxiety, and weight management [7-9]. In contrast, clinical evidence on the effectiveness of DHIs and the ability to compare effectiveness across studies is limited for other diseases such as asthma, attention-deficit/hyperactivity disorder, and eating disorders [8]. This is mainly because of limitations and inconsistencies in clinical trial design, such as small sample size, variable uptake and user engagement with DHIs, lack of blinded outcome assessment, short-term follow-up, and poor specification of the extent of human support (ie, DHIs that are fully self-administered by a patient without any elements of intervention delivery or monitoring by a clinician) [8,9]. As clinical effectiveness data are essential for parameterizing cost-effectiveness models, these inconsistencies in clinical evidence can pose limitations in conducting economic evaluations of DHIs.

The current understanding of the value for money of DHIs for children and adolescents is considerably more limited. Adoption of and engagement with DHIs are likely to be strong in the pediatric and adolescent population, and hence, the potential to be effective and cost-effective (ie, through scale-up) may be high [8]. The cost-saving potential of DHIs is associated in particular with behavioral interventions for chronic conditions in this population, such as the management of obesity and anxiety through web-based CBT [10]. However, the overall cost-effectiveness of DHIs for this population is unclear as evidence seems to differ according to the clinical setting and intervention type [8,11]. For instance, web portals and telephone support programs for obesity and SMS text messaging interventions for mental health and maternal, neonatal, and child care settings are more likely to be cost-effective than telemonitoring and videoconferencing systems for posttraumatic stress disorder and cardiovascular conditions [11].

Most published economic evaluations of DHIs follow standard guidelines for the evaluation of health technologies, such as those for pharmaceutical products and interventions [2]. This includes taking a health system or payer perspective, considering only health-related benefits, and conducting cost-utility analyses (CUAs) [2]. However, economic evaluations of DHIs may raise distinct methodological issues compared with pharmaceuticals and medical devices, including the measurement of non–health care benefits and costs, user involvement, and the choice of comparator [2]. The design of economic evaluations of DHIs tailored to pediatric populations may face additional challenges as DHIs originally designed for adult populations may not offer the type of interactions, self-monitoring features, and user involvement that are appropriate to address the needs of children and adolescents [3,12]. In general, the overall methodological quality of published economic evaluations in pediatric settings is unknown, including whether economic evaluations for this population face distinct methodological issues such as the measurement of costs and effects compared with DHIs for adults.

Objectives

To address these research gaps, we systematically reviewed economic evaluations of DHIs targeted at pediatric and adolescent populations. In addition, we examined and categorized the methodological issues reported in the reviewed economic evaluations with the purpose of informing methodological areas where future research might need to be prioritized.

Methods

Data Sources and Search Strategy

We used free-text terms to search for articles indexed in PubMed (MEDLINE database) from November 2011 to November 2021. This period was selected because of the emerging nature of DHIs, particularly mobile health and eHealth, which have experienced a growth in uptake in recent years. Search term combinations paired “cost-effect*,” “cost benefit,” “cost utility,” “economic evaluation,” “health economic analysis,” “value for money,” “decision model*,” and “cost consequence” with each of “child*,” “paediatric” and “adolescen*,” “infant*,” “neonat*,” “newborn*,” “baby,” and “babies” and each of the following terms: “telemedicine,” “remote* deliver*,” “telehealth,” “digital health,” “mobile health,” “m-health,” “ehealth,” “internet,” and “online” (Textbox 1). The selection of search terms was made based on a review of previously published literature on economic evaluations of DHIs, as well as in the pediatric and adolescent populations [9,10,13,14]. We also manually searched the bibliographies of eligible articles to identify other articles of interest. We conducted a systematic search of articles using the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist and reporting recommendations (Textbox 1 and Multimedia Appendix 1) [15].

Inclusion and Exclusion Criteria and Study Selection

In total, 2 reviewers (TS and TSA) independently screened the titles and abstracts of selected articles according to the prespecified inclusion and exclusion criteria: age range of the population of children and adolescents between 1 and 18 years, type of economic evaluation, intervention (ie, telephone, audiovisual consultation, SMS text messaging, mobile phone app, or web-based portal), and outcomes (effects, costs, and cost-effectiveness results). After screening titles and abstracts, we excluded the following types of studies: studies not covering pediatric and adolescent populations, studies that were not economic evaluations, reviews or systematic reviews, clinical study protocols, feasibility studies, surveys, qualitative studies, case reports, opinion articles, and clinical guidelines (Figure 1 and Table 1). In line with the economic evaluation definition commonly used in the literature [16], cost minimization, cost-consequence, and simple cost comparison studies were excluded from this review (Figure 1 and Table 1). We included articles for data extraction if consensus among the reviewers was reached after reviewing the full texts. A third reviewer was used to help resolve any disagreements. The types of economic evaluations that we included in this review were cost-effectiveness analyses (costs per clinical effect), cost-benefit analyses (costs and effects expressed in monetary terms as net benefit), and CUAs (cost per quality-adjusted life year [QALY] or disability-adjusted life year [DALY]) [16,17]. Although cost-per-QALY analyses are a very common type of economic evaluation for adult interventions, they are often not feasible for children and adolescents, primarily because of the methodological challenges and lack of data on valuing health states and deriving QALYs specific to these populations [18,19].

‎

Data Extraction

We generated a data abstraction procedure (Textbox 2) based on recommendations from the Panel on Cost-Effectiveness in Health and Medicine [20]. A narrative descriptive analysis was used to synthesize the quantitative data. We did not attempt to meta-analyze the results because of the high levels of heterogeneity across the studies. The extracted data of interest included (1) characteristics of the reviewed studies (eg, population, country, disease area, sample size, and DHI type); (2) main results of the studies (costs, outcomes, and incremental cost-effectiveness ratio [ICER]); (3) methodological issues regarding the health economic evaluations; and (4) key cost-effectiveness drivers, which were defined as input parameters that had the most impact on cost-effectiveness results.

To facilitate the summary of the main results of the economic evaluations and comparisons between studies, we converted costs and cost-effectiveness estimates from Great British pound, Euro, and Australian dollar to 2021 US dollar using the following conversion rates: £1=US $1.364, €1=US $1.183, and Aus $1=US $0.751 [21-23].

Methodological Issues

Most economic evaluations of DHIs follow methodology guidelines used for standard health technologies because of a lack of DHI-specific guidelines and published research [2]. However, DHIs are complex interventions with their own features that pose different types of methodological challenges to economic evaluations compared with pharmaceuticals and medical devices [2]. We reported key methodological challenges and issues concerning the conduct of the reviewed economic evaluations. Drawing on previously published literature [2], we grouped all identified issues into 4 distinct categories related to DHIs, data (clinical or and cost), economic evaluation, and study design.

Results

Article Selection

The database search yielded a total of 26,778 articles as of November 2021 (Figure 1). Following the removal of duplicates and studies that did not include populations aged 1 to 18 years, 16.43% (4400/26,778) of the articles remained. We next screened the titles and abstracts and excluded 98.89% (4351/4400) as they were not about pediatric or adolescent populations; were not an economic evaluation; or were a systematic review, clinical study protocol and design, feasibility study, survey and qualitative study, case report, opinion article, or clinical guidelines. We reviewed the full texts of the remaining 49 articles and excluded 29 (59%) as they were not economic evaluations (ie, exclusion of studies that conducted only a simple cost comparison, cost minimization, or cost-consequence analysis). We added 2 articles after screening the references of the reviewed full-text articles. Therefore, we selected 22 articles for inclusion in this review [24-45].

Overview of Key Characteristics

DHIs administered through an web portal were the most frequent among the included studies (8/22, 36%), followed by SMS text messaging (6/22, 27%) and mobile phone apps (3/22, 14%). Other types of DHIs included telephone consultations (2/22, 9%), audiovisual consultations (2/22, 9%), and web-based symptom monitoring (1/22, 5%). Most studies were conducted in Europe (8/22, 36%) and Africa (6/22, 27%). The most frequently evaluated health care specialties were mental health (7/22, 32%) and maternal, newborn, and child health (MNCH; 7/22, 32%). Other disease areas included asthma, malaria, gastrointestinal disorders, sleeping disorders, child immunization, and conditions requiring emergency care.

Of the reviewed articles, 64% (14/22) included economic evaluations that were based on individual patient data (IPD) [23,24,28,30,32,34-39,41-43], with the remaining focusing on decision analytical modeling (7/22, 32%). Only 5% (1/22) of the economic evaluations used a combination of IPD and decision modeling [31]. Of the 14 IPD-based studies, 13 (93%) were randomized controlled trials, and only 1 (7%) was a non-RCT. The average follow-up duration of the IPD-based and model-based economic evaluations was 11 (range 0.5-36) months and 7 (range 3-10) years, respectively. All studies entailed a cost-effectiveness analysis (6/22, 27%), a CUA (9/22, 41%), or a combination of both (7/22, 32%). In Table 2, we provide a detailed overview of the key characteristics of the studies included in this review.

Clinical Effectiveness Estimates

We report key effectiveness outcomes of DHI interventions and comparators in the reviewed studies in Table 3. The most frequently reported health outcomes included the number of deaths averted or lives saved, QALYs, and DALYs. Although most studies (17/22, 77%) suggested that the DHIs were effective, 23% (5/22) of the studies reported not statistically significant differences in health outcomes between the DHI and control groups [25,37,40,41,43]. The most common comparators in the reviewed studies were paper-based, in-person consultations in outpatient settings and the absence of an intervention. Relevant clinical outcomes included care coverage, disease remission rates, immunization rates, treatment response, anxiety and depressive symptoms, and other disease-specific quality-of-life (QoL) metrics. The effectiveness of the DHIs did not seem to differ according to the children’s age cohort (eg, children vs adolescents). Only 5% (1/22) of the studies, which evaluated a web-based asthma monitoring program, reported statistically significant improvements in health outcomes for young children but not for teenagers [45].

Cost Estimates

We summarized the cost estimates associated with DHI implementation and the associated health care costs for children and adolescents. Further details regarding the exact cost components included in each of the reviewed studies can be found in Table 4. Many (9/22, 41%) economic evaluation studies took both health system and societal perspectives, whereas the remaining studies were distributed evenly between the health care system and societal perspectives.

Most studies (18/22, 82%) reported detailed direct cost components broken up into 3 phases of integrating the intervention into the health care system: DHI development, DHI start-up, and DHI implementation. Approximately 55% (12/22) of the studies reported indirect costs, which typically included productivity losses because of missed hours from school for children and from paid and unpaid work for parents and caregivers, parental out-of-pocket expenditures on travel, informal care, and educational expenses associated with tutoring and mentoring of the target population.

DHIs were associated with cost savings compared with the standard of care in 45% (10/22) of the studies [24-26,34,36,38-41,45], ranging from a mean of US $164 to US $5423 per patient (Table 3) [24,39]. Other studies reported the cumulative annual incremental cost of DHIs compared with the standard of care, which ranged from US $184,978 to US $1.1 billion depending on the scale-up of the intervention program [33,35] (Table 3). The relative proportion of each of the DHI-related cost components (reported only in 9/22, 41% of the studies) was as follows: 12% average start-up cost (range 5%-24%), 15% average development cost (range 6%-23%), and 73% average implementation cost (range 63%-89%) [24,27-31,33-35].

Cost-Effectiveness Outcomes

We reported key findings of the economic evaluations in Table 3 grouped by DHI type. In 82% (18/22) of the reviewed studies, DHIs were cost-effective (15/18, 83%) or cost saving (3/18, 17%) compared with the standard of care, with an ICER ranging from US $24 to US $23,602 per DALY averted [29,40,42,43]. In 32% (7/22) of the studies, DHIs were a dominant strategy.

Cost-Effectiveness Drivers

The most frequent drivers of cost-effectiveness across the studies were population coverage, cost components, and intervention effect size (Table 3). In particular, the cost components associated with the implementation of DHI programs and supervision/training of health care professionals had the most impact on cost-effectiveness results [26-29,36,37,40,42-44]. For instance, the cost-effectiveness of internet-delivered universal prevention of major depression was highly sensitive to variations in implementation cost (ie, small changes to the average implementation cost per person led to large changes in the resulting ICER, with the intervention becoming not cost-effective when such cost was >US $90 per person [42]).

The mean differences in health outcomes between DHIs and comparators were most sensitive to factors such as population coverage [28,30,35], assumptions about the number of deaths averted or lives saved [27-29], and number of clinic visits [30,45]. For example, the annual coverage increase from 5% to 10% for antenatal and postnatal care was associated with twice as many lives saved over 10 years of implementation of an SMS text messaging program for pregnant women attending MNCH facilities [28].

Overview of Methodological Issues

DHI-Related Methodological Issues

Challenges directly related to data collection or modeling of the evaluated DHI interventions were reported in 36% (8/22) of the articles and included the following types of issues (Table 5):

1. Lack of a well-defined comparator was observed in 75% (6/8) of the studies [26,35,36,38,40,45]. This included potential biases arising from differences in the delivery format of the DHI and comparator interventions [26] as well as the mixing of DHI and comparator interventions in certain child groups [35,45].
2. Lack of user involvement was reported to weaken the effectiveness of DHI interventions. This was mostly because of difficulties in maintaining the expected level of user involvement during the study period that would be desirable for the specific DHI [24,25,30].
3. Difficulty in measuring the impact of DHIs on costs and outcomes. This was because of either the complexity of DHI programs that was not captured by the economic evaluation [28,31] or the lack of a clear understanding of the pathways to impact, where outcomes may have been influenced by non-DHI components (eg, complementary services) [31].

Issues With Measuring Costs and Outcomes

Limitations of measuring outcomes included biases in self-reported patient information, lack of primary data on resource use, and lack of appropriate QoL measures adjusted to younger age groups (ie, use of generic rather than pediatric EQ-5D and 36-item Short Form Health Survey instruments). The most frequent challenges with cost data were incomplete resource use data, costing intervention scale-up because of difficulties in estimating the likely population size, and uncertainties regarding distinguishing and measuring operational costs versus sunk costs.

Economic Evaluation Methodological Assumptions

We identified methodological issues related to decision modeling in 23% (5/22) of the economic evaluation studies [28,34,40,42,43]. This included failure to appropriately model preventive and curative care in complex treatment pathways, short model time horizons because of lack of evidence on longer-term effects of DHIs, and issues with the validity of the input parameters (Table 5). Another prevalent issue with modeling of DHIs in child settings was the difficulty of summarizing results into a single metric, such as cost per QALY, because of the lack of generic health outcomes in this population [25,32,41] (Table 2).

Study Design Issues

The reviewed studies also faced methodological issues related to the study design (17/22, 77%). These issues primarily related to studies that used individual patient-level data (14/17, 82%) rather than decision analytic models. At the clinical trial initiation/enrollment stage, these issues included selection bias and the absence of blinding to random allocation. For instance, 12% (2/17) of the studies faced potential confounding because of age differences between the intervention and control groups [24,35]. In addition, participants in some of the studies were permitted to self-refer, resulting in self-selection bias [24,26,40]. Other limitations related to the study design included low generalizability/external validity, small sample size because of the difficulty of recruiting children and adolescents for DHI programs, short study follow-up period, and incomplete data because of loss to follow-up (Table 5). Low generalizability was also a common issue in the included studies [24-26,36]. For instance, a narrow age range of the trial population posed a barrier to generalizing the findings of the economic evaluations to a wider health care setting or patient population [26].

Discussion

Principal Findings

This systematic review identified 22 economic evaluations of DHI interventions for children and adolescents. Most studies (14/22, 64%) evaluated interventions delivered through online portals or SMS text messaging, most frequently within the health care specialties of mental health and MNCH. In 82% (18/22) of the reviewed studies, the DHI was cost-effective or cost saving compared with the standard of care. The studies reported various levels of granularity of cost components used in the economic evaluations; however, most studies (18/22, 82%) included direct medical costs and DHI costs broken up into 3 major components: DHI development, DHI start-up, and DHI implementation. More than half (12/22, 55%) of the reviewed studies also included indirect costs, such as productivity losses because of missed hours from school for children and from paid and unpaid work for parents and caregivers. The methodological challenges commonly identified in the reviewed studies suggest that issues with study design were the most prevalent (17/22, 77%), followed by issues with cost data (11/22, 50%) and challenges related to decision modeling (5/22, 23%). The most frequent drivers of cost-effectiveness included population coverage, implementation and user support costs, and intervention effect size.

Comparison With the Existing Literature and Method Guidelines

This study provides the first systematic review of economic evaluations of DHIs targeting pediatric and adolescent populations. In line with the published literature on adult populations, we found that many of the economic evaluations of DHIs for children and adolescents focused on the management of long-term conditions and mental health, primarily in the form of CBT, cardiac monitoring, and weight loss and diabetes management [5,11,46,47]. In these areas, our findings are very much in line with those for adult populations. For example, our review found SMS text message reminders used for weight loss management to be cost-effective, similar to findings for adult populations [11]. However, we identified other areas where DHIs for children and adolescents are often cost-effective or cost saving compared with usual care in contrast to related literature in adult settings [11]. For example, videoconferencing sessions for children and adolescents were found to be cost-effective, whereas similar videoconferencing interventions in adult populations are rarely cost-effective, as well as having no significant impact on health-related QoL [11].

Most of the studies (19/22, 86%) included in this review are unlikely to comprehensively meet the evidence standards framework published by the National Institute for Health and Care Excellence (NICE) and the World Health Organization [3,48]. For example, NICE guidelines suggest the need for generalizability of economic evaluation assumptions beyond the local context, but most of the reviewed studies (15/22, 68%) focused on a narrow patient population with specific characteristics that are not directly generalizable. In addition, many of the reviewed studies (12/22, 55%) did not have a sufficiently long time horizon to capture the relevant health outcomes and costs and did not provide a clear justification for the chosen sensitivity analysis as per NICE recommendations [48].

Strengths

This review adds to the current body of literature by identifying some key methodological issues in the reviewed economic evaluations that are relevant to the pediatric setting. These include mixing of DHI and comparator interventions across treatment arms, inconsistencies in accounting for different cost components of DHIs (eg, maintenance, sunk, and scale-up costs), and difficulties maintaining user involvement.

The methodological issues specific to the nature of DHIs and the difficulties of comparing economic evaluations of DHIs because of heterogeneity align well with those previously identified in the published literature [1,2,11,49,50]. These include lack of established measures for clinical effectiveness of DHIs, difficulty in distinguishing DHIs from “standard of care,” and inability to measure and model broader costs and effects of DHIs beyond the direct impact on the health system (ie, production losses, travel costs, absenteeism, and presenteeism for parents and caregivers). In the context of the published literature, many of the study design issues were similar to those faced in clinical studies of health care interventions more broadly and included selection bias, small sample size, low generalizability, short or no follow-up period, and no blinding to random allocation.

Limitations

This systematic review has some limitations. The database search was limited to studies published between 2011 and 2021 and indexed in PubMed. Considering that DHIs have received increasing attention in the last decade, this time restriction may be reasonable. This review was based on the MEDLINE database and has not been comprehensively combined with other databases. However, when we performed a high-level search in Embase, it did not retrieve any additional relevant studies, suggesting that the PubMed search strategy likely covered the vast majority of the relevant economic evaluation studies.

This review did not assess the reporting quality of the reviewed studies in line with reporting guidelines such as CHEERS (Consolidated Health Economic Evaluation Reporting Standards) [51] for 2 reasons: first, published economic evaluations are typically required to follow the CHEERS checklist, and second, our quality assessment focused on a wider range of methodological issues regarding economic evaluations, which included aspects more relevant to the conduct of economic evaluations of DHIs as well as generic quality aspects such as study design.

Suggestions for Further Research

This review provides a critique of methodological challenges that may arise in economic evaluations of DHIs tailored to pediatric and adolescent populations. Numerous DHIs are produced and adopted at pace, and economic evaluations considering the specific features of DHIs are needed. Economic evaluations with significant limitations may lead to suboptimal decisions and poor use of limited health care resources. In this section, we highlight some methodological areas that, in our view, would constitute interesting avenues for further research.

First, a common issue highlighted across the reviewed studies was the use of health outcome measures that are clinically relevant but less likely to meet the requirements for cost-effectiveness assessments, such as generic QoL measures. The development and validation of generic QoL measures for children and adolescents is beginning to emerge [52], but further research is warranted. For example, it is important to understand whether the measures are consistent across different age groups. Second, a broader perspective capturing indirect and non–health care costs and benefits is generally a preferred perspective in economic evaluations, but our review shows that it has not yet been used consistently for the evaluation of DHIs for children and adolescents. This could be addressed through more carefully designed data collection of indirect costs that account for health- and non–health-related burden to both children and their caregivers [11]. Third, although there have been some efforts to measure user participation (eg, user time) in the evaluation of DHIs [2,49,53], further research is needed to develop a more holistic value framework for formally valuing and incorporating user involvement in the cost and outcome analysis. In addition, following international guidelines and recommendations may improve the quality and consistency of economic evaluations of DHIs.

Conclusions

Our study found that most (18/22, 82%) DHIs for children and adolescents are cost-effective or cost saving compared with the nondigital standard of care from either the health system/payer or societal viewpoint. However, there was a substantial degree of variability in cost-effectiveness results depending on the type of DHI, health care setting, and inclusion of certain input parameters in the economic evaluations. The cost-effectiveness of these DHIs appeared to be sensitive to the inclusion of certain cost components (particularly DHI implementation and supervision/training of health care professionals), intervention effect size, and its potential to be scaled up. This study highlights common methodological challenges directly related to the conduct of economic evaluations of DHIs. These included failure to measure user involvement, generic QoL outcomes, and indirect effects of DHIs, highlighting areas where further methodological research is required.