A Web-Based Computer-Tailored Alcohol Prevention Program for Adolescents: Cost-Effectiveness and Intersectoral Costs and Benefits

Introduction

Excessive alcohol use and alcohol use disorders are major causes of death and disability worldwide [1-4]. In 2012, approximately 3.3 million deaths, or 5.9% of all global deaths, were attributable to alcohol use [4]. In addition, alcohol use led to an estimated total of 139 million disability-adjusted life years, representing 5.1% of the global burden of disease and injury in that year [4]. For all age groups, outliers on the proportion of alcohol-related deaths can be seen in the World Health Organization European Region, varying from 10% for the population aged 80 years and older to 25% for the 20 to 39 age group. It is particularly striking that in this region 10% of adolescent deaths (those aged 15 to 19 years) were attributable to alcohol.

Apart from the impact of alcohol use on morbidity and mortality, the harmful use of alcohol may also lead to significant societal costs [3-7]. For example, in the European Union alone, alcohol-attributable costs were estimated at €125 billion in 2003 [4]. These encompass health care services, such as hospitalizations, home health care, and ambulatory care, but also costs outside the health care sector, such as costs resulting from productivity losses and costs in the criminal justice system. Examples of the latter include vehicle crashes, increased crime, and arrests. Studies have shown that youthful drinkers are at greater risk of being victimized and perpetrating youth violence, low educational attainment, and low college expectations [8,9], putting a financial burden on the criminal justice system and educational sector.

Preventing excessive alcohol use in the whole population and in the young population in particular is thus important, not only to improve the health of individuals and of the whole population, but also to reduce alcohol-related costs inside and outside the health care sector. Computer tailoring could be a means to achieve these goals. Computer tailoring is a behavioral intervention that can be effective in changing health behaviors in general, including the use of alcohol [10-15]. Within a computer-tailored intervention, the content is adapted to individual characteristics of respondents [16]. Often a questionnaire is used as a screening instrument, assessing behavior, relevant sociodemographics, and motivational factors [17,18]. Respondent answers are collected into a data file and automatically matched with tailored feedback messages [16,19]. An advantage of tailored information is that it is perceived as more relevant than nontailored information [20,21]. Moreover, through the Internet, these programs are accessible by a growing percentage of the world population—42.3% had access to Internet in 2014 in comparison to 5.9% in 2000 [22]—and can be accessed wherever and whenever it suits the respondents. For Europe and its younger population (aged 16-24 years), the current Internet penetration is even higher: 70.5% [22] and 91.0% [23], respectively. Taking advantage of this high accessibility by providing Internet-based behavior interventions might significantly limit the need for and burden on health professionals and could, in turn, lead to less administrative costs [19]. Subsequently, the resulting savings of time and resources contribute to a more efficient health care system.

Although the effectiveness of multiple Internet-based and computer-tailored interventions have been studied [24,25], limited information exists on their cost-effectiveness [26]. To date, studies which have included cost-effectiveness analyses (CEAs) focused on adults aged 18 to 65 years [21,27] and 18 to 69 years [3], whereas the cost-effectiveness of such interventions for adolescents younger than 18 years has not yet been studied. Available economic analyses of interventions aimed at reducing alcohol use among adolescents are those of interventions which are not Internet-based and not computer-tailored [28,29]. Furthermore, as acknowledged by Smit et al [3], these studies show a limitation in terms of disregarding many of the costs and benefits in sectors outside the health care sector, also known as intersectoral costs and benefits (ICBs) [30]. Excluding ICBs, such as costs and benefits in the educational and criminal justice sector, may significantly affect the results of the CEAs of interventions [30]. Moreover, including and reporting ICBs within economic evaluations could support decision making regarding the large-scale implementation of such programs. Therefore, the aim of this study is to answer the question of whether a Web-based computer-tailored intervention for adolescents for reducing the use of alcohol is cost-effective from both a health care and societal perspective, and to assess the impact of including ICBs on the outcomes of the analysis.

Methods

Design

Data used was from the Alcoholic Alert study, which was designed as a cluster randomized controlled trial (RCT) with randomization at the level of schools into two conditions [31]. Participants either played a game on alcohol awareness after the baseline assessment (intervention condition) or received care as usual (CAU), meaning they had the opportunity to play the game subsequent to the final measurement (waiting list control condition). Providing this opportunity was due to ethical considerations. Data were recorded at baseline (T0=January/February 2014) and after 4 months (T1=May/June 2014). These were used to conduct two sets of comparative CEAs; one was performed from a health care perspective (including health care costs, excluding ICBs) and one from a societal perspective (including both health care costs and ICBs). Because of a rapid change in Dutch government policy on the minimum legal drinking age (ie, 18 years vs 17 years as of January 1, 2014), the abovementioned time frame and start date differed from the original design [31]. The Alcoholic Alert study was approved by the Medical Ethics Committee of Atrium Orbis Zuyd (METC number: 12-N-104) and was registered in the Dutch Trial Register (NTR 4048).

Randomization

Randomization was performed at school level to prevent contamination between participants. Randomization was conducted by drawing lots. After randomization, 21 schools were assigned to the intervention condition and 23 schools were assigned to the control condition.

Sample

The study population consisted of Dutch adolescents (aged 15-19 years) attending school [31]. Participants included students at schools of higher secondary education, lower secondary education, and lower vocational training. To have enough power for the evaluation, a participant target was made based on the following criteria: 10% reduction in binge drinking occasions (ie, for girls, at least four glasses or, for boys, five glasses of alcohol-containing drinks in one occasion) [32] during the preceding 30 days between the intervention and control group, with an intraclass correlation of .02, a power of 80%, and a significance level of .05. Furthermore, taking into account the drop in power due to an expected dropout of 50% at follow-up, it was estimated that at least 34 schools should be included at T0 [31].

To reach the required number of 34, schools were recruited via several media; schools first received flyers with information about the Alcoholic Alert study, after which they were contacted via telephone and email. If schools enrolled in the study, students of the schools were eligible to participate. However, they could do so only if they provided informed consent by clicking a checkbox, which preceded the Web-based questionnaire at T0 [31].

Intervention

Adolescents in the intervention condition participated in a Web-based computer-tailored alcohol reduction program called Alcoholic Alert [31]. After completing a Web-based questionnaire on the Alcoholic Alert website at T0, the participants entered a game called “Watskeburt” (Dutch slang for “What Happened?!”). In the game, the participant played a character whose goal it was to find out what happened after a night of heavy drinking. Participants received in-game questions concerning alcohol-related sociocognitive factors, including attitude, social influences, self-efficacy expectations, and action plans toward alcohol drinking. These questions were based on the I-Change model, which is an integrated model explaining motivational and behavioral change [33]. Based on their answers, they received computer-tailored feedback on these determinants. They played in three game scenarios within three sessions. A week after playing the third game scenario, the participants were asked to revisit the intervention website to answer several questions. In this fourth session, they were asked about their drinking behavior during the preceding week and then they received computer-tailored feedback on their alcohol use compared to Dutch drinking guidelines. Subsequently, the participants were asked whether they had an event (eg, party, wedding) in the upcoming 30 days then they were challenged to drink less than usual and were asked for the maximum amount they wanted to drink. An email, with a reminder of accepting the challenge, was sent to them a day before the event. Two days after the event, during a fifth session, they were asked to visit the intervention website and fill in their alcohol use. If the participant failed the challenge, they received computer-tailored feedback with tailored advice and had the opportunity to take on a new challenge. If the participant met the challenge, he or she received congratulations and the intervention was over (Figure 1).

Participants receiving CAU also filled in the Web-based questionnaire at T0 and T1, but they did not have access to the game and did not receive computer-tailored feedback until after the final measurement. Further information on the intervention can be found elsewhere [31].

‎View this figure

Measurements and Outcomes

The measurements at T0 and T1 were performed at school after participants received instructions from their teachers [31]. The Web-based questionnaires used for the measurements included items related to alcohol drinking behavior, use of services within the health care sector, and ICBs. In addition, several background variables were measured at T0, including gender, age, educational level (higher secondary level, lower secondary level, and lower vocational), religion (Catholic, Protestant, Muslim, other religion, no religion), and ethnicity (Dutch, Antillean, Belgian, German, Surinamese, Moroccan, Turkish, other) [31].

In this study, the outcome measures were weekly alcohol use and the number of binge drinking occasions in the preceding 30 days. Weekly alcohol use was assessed by asking participants on which days in the past week they had been drinking and, if they did, how many glasses of alcohol they had on these days. Based on this information, the total amount of glasses of alcohol was calculated [17]. From this, as for binge drinking occasions, the weekly alcohol use at T1 was subtracted from the alcohol use at T0. This led to positive scores in case of a reduction in weekly alcohol use or number of binge drinking occasions, and negative scores in case weekly alcohol use or the number of binge drinking occasions increased.

Resource Use and Costing

The following costs related to the Alcoholic Alert intervention were identified as important and measured: (1) intervention costs, (2) health care costs (ie, costs for services inside the health care sector), (3) intersectoral costs (ie, costs for services outside the health care sector), and (4) costs of substance use (eg, use of hard drugs). Because alcohol use was the outcome measure in this study, the costs of alcohol use were not included in the CEA. Doing so would have led to double counting [34].

Costs (in Euros) were measured irrespective of who bore them and were indexed for the reference year 2014 using price indexes from Statistics Netherlands [35]. Calculations for specific costs of service utilization and substance use can be obtained from the first author (RD).

Intervention Costs

Intervention costs were divided into costs made during the development of the intervention and costs for running the intervention. Costs incurred during the development included game development costs (€20,328) and automatic tailoring software license and development costs (€8367.15). These costs and costs for other personnel involved in the development and application of the intervention for this study, such as costs incurred contacting schools, recruiting participants, and analyzing data are sunk costs [36]. Therefore, these are not included in the CEA. This is further justified by the fact that the intervention itself is Web-based and universal for adolescents, meaning the intervention has a wide reach and the development costs per participant drop to a minimum when it is used widely and structurally.

Costs for running the intervention include hosting costs for the website, tailored feedback software, and participants’ time investments. As for the sunk costs, the website hosting costs (€300 per year) per participant drop to a minimum if used widely and are not included in the CEA. Tailored feedback software costs were €7 per participant per week. In case the participant met his or her challenge in the fifth session, he or she would receive tailored feedback over a period of 3 to 6 weeks. The mean intervention duration was 4 weeks, so the tailored feedback costs were €28. The five sessions took up 1.5 hours at school (€8.30 per hour) and 1 hour of free time (€12.50). The total of 2.5 hours was valued at €25 per participant. In sum, the total intervention costs per participant were an estimated €53.

Health Care Costs

Health care costs were calculated by multiplying volumes of health services by related cost prices. Health services measured included contacts with the general practitioner, emergency care, hospital stays, ambulance rides, and mental health services. Cost prices were drawn from the Dutch manual for costing in economic evaluations [37].

Intersectoral Costs

The ICB-related costs were calculated by multiplying volumes of services and time investments outside the health care sector with related cost prices. The ICBs were classified in sectors according to a classification scheme for ICBs by Drost et al [30]. The sectors in this scheme included education, labor and social security, household and leisure, and criminal justice system. The services and time investments measured included school absenteeism and contacts with an attendance officer (education), work absenteeism (labor and social security), failing to perform household and other activities, contacts with youth and family center and family care (household and leisure), and contacts with (youth) police services, court proceedings, and child (health) protection services (criminal justice system). Cost prices were drawn from a Dutch manual for intersectoral costs and benefits of (preventive) interventions [38]. Cost prices not mentioned in the manual were extracted from the Institute for Medical Technology Assessment (iMTA) questionnaire on intensive youth care [39]. For ICBs that required valuation of time, such as failing to perform household activities, some additional information was drawn from the Dutch report “The Netherlands in a Day” (free translation) [40].

Costs of Substance Use

In addition to alcohol use, use of other substances were measured as well. These included packs of cigarettes, use of soft drugs, and use of hard drugs. Cost prices were found on the website of the Jellinek Clinic, which is a renowned Dutch institution specializing in preventing and treating alcohol and substance abuse [41].

Data Preparation

The basis of the analysis was the dataset used for the Alcoholic Alert effect study [24]. However, because cost measures were used for conducting the economic evaluation, some additional data cleaning was required to create a dataset that was suitable for conducting the CEA.

First, because the digital questionnaires contained open-ended questions, participants had the opportunity to fill in unrealistic answers. It was a small subsample that systematically filled in these unrealistic answers, but to improve the validity of the results, these respondents were excluded from analysis. To clear the data of these respondents, limits were set for each variable. Participants breaching these limits by providing unrealistic answers were excluded from analysis. For example, because the 4-month recall period between T0 and T1 amounted to 120 days, any respondent claiming to have stayed more than 120 days in a hospital was excluded. In general, to reduce the chance of wrongfully excluding participants, limits were set high, but within the range of credibility. As for all steps during the process of data preparation and analysis, the limits were discussed in author meetings and were agreed on by all authors. A list of these limits can be obtained in Multimedia Appendix 1.

Second, the dataset was cleared of respondents who at baseline did not answer a single question related to costs. Based on these two steps, the sample at T0, and accordingly at T1, was smaller and different in composition in the CEA than the sample used for the effect study [24].

Finally, to assess the school-based part of the variance in this cluster RCT, intracluster correlation coefficients (ICCs) were calculated for both the weekly alcohol use and binge drinking occasions outcome measures. The ICCs were calculated based on the following formula: ICC or ρ = s_b²/(s_b²+ s_w²), where s_b²= the variance between clusters and s_w²= the variance within clusters [42]. Input for both outcome effect sizes was generated using SPSS version 20 by running linear regression mixed models. These analyses and corresponding calculations resulted in ρ=.06 for binge drinking occasions and ρ=.01 for weekly alcohol use, which shows that the within-cluster variances for both effect sizes were much greater than the between-cluster variances [42]. Based on these results, no re-estimations of effects were required.

Analysis

Descriptive Statistics

Descriptive statistics were used to describe the characteristics of the sample at T0 and at T1. Differences between the intervention and control conditions were assessed in SPSS version 20 using independent samples t tests for continuous variables and chi-square tests for discrete variables. The same software was used to conduct stepwise linear regression analyses to assess the dependence of the outcome measures on these variables.

Cost-Effectiveness Analysis

The base scenario of this study included CEAs from the two perspectives mentioned earlier. We calculated costs of services utilization in three steps: (1) assessment of the services and time consumed in the 4-month period between T0 and T1, (2) calculation of the associated costs in Euros, and (3) calculation of the incremental cost-effectiveness ratio (ICER) using the formula (C_i–C_c)/(E_i–E_c). Here C represents the average total costs per participant during the 4-month period between T0 and T1 and E represents the mean difference in the number of glasses of alcohol or binge drinking occasions at T1 in comparison with the number measured at T0 in the intervention (C_i and E_i) and in the control (C_c and E_c) condition.

Stochastic uncertainty in the data was dealt with using nonparametric bootstraps. By using the bootstrapping technique, means and confidence intervals were calculated and 5000 ICERs were simulated, which were plotted in cost-effectiveness planes. These planes provided a visual representation of the probability of the intervention being cost-effective in comparison with the control condition by showing the distribution of ICERs across four quadrants: (1) more effective and more costly in the northeast quadrant (NE), (2) more effective and less costly in the southeast quadrant (SE), (3) less effective and less costly in the southwest quadrant (SW), and (4) less effective and more costly in the northwest quadrant (NW) [43].

An ICER in the SE and NW is negative, indicating that the intervention is dominant over (SE) or inferior to (NW) the control condition. An ICER in the SW or NE is positive, indicating that from a cost-effectiveness viewpoint the intervention is more favorable than the control condition only when the ICER is lower than the willingness to pay (WTP) per unit effect. Because no threshold (ie, maximum WTP) was available for the weekly alcohol use outcome measure, a cost-effectiveness acceptability curve (CEAC) was created for both perspectives. The CEAC showed the likelihood of the intervention being favorable over CAU for several hypothetical thresholds.

Sensitivity and Subgroup Analyses

Apart from the analyses to deal with stochastic uncertainty, several other analyses were conducted. First, to assess the impact of cost outliers, ICERs were calculated based on data in which cost outliers were excluded. Based on the output of descriptive statistics, it was decided to exclude participants when total costs were greater than €5000. Based on this approach, in the analyses that were conducted from the health care perspective, one participant was excluded. In the analyses that were conducted from the societal perspective, four participants were excluded. Second, to assess the effect of the uptake, the costs of cigarette use, and of the use of soft and hard drugs in the analyses conducted from the societal perspective, additional analyses were conducted without these costs.

Finally, given the heterogenic composition of the study sample, several subgroup CEAs were conducted based on the background variables measured at T0. These included analyses based on dichotomized background variables, including gender (male, female), age (15-16, ≥17), educational level (low, high), religion (religious, not religious), and ethnicity (Dutch, non-Dutch). Again, for all these analyses, stochastic uncertainty was dealt with using nonparametric bootstraps.

Results

Dropout and Sample Characteristics

Figure 2 shows a flowchart with the number of participating schools and adolescents at T0 and T1. In total, 44 schools were randomized into the control condition or intervention condition. Of the schools randomized to the control condition, five withdrew their participation before T0 (one secondary lower education, one lower vocational training, two secondary higher education, one secondary education mixed). In addition, three schools in the control condition (all secondary higher education) and two schools in the intervention condition (one lower vocational education, one higher secondary education) did not start the baseline assessment and did not respond to the emails and phone calls [24].

In total, 2649 adolescents from 34 schools participated in the baseline questionnaire. Of these, and different from the effect study [24], an additional 91 participants (3.4%) were excluded from analysis based on providing unrealistic answers to the cost questions. Another 65 participants were excluded from analysis because they did not answer the cost questions at T0. This resulted in 2493 adolescents who were included in the baseline analysis. Of these, 1538 were in the intervention condition and 955 were in the control condition (Figure 2). The t tests and chi-square tests conducted on the baseline sample showed that the adolescents in the two conditions significantly differed on multiple characteristics. Adolescents in the intervention condition were significantly younger, more often female, had a higher educational level, more often indicated being religious, were less likely to be a drinker, were less often a binge drinker, had less binge drinking occasions, and had a lower weekly alcohol use than adolescents in the control condition (Table 1).

Of the 2493 adolescents, 757 participated in the cost questionnaire at T1 (response rate 30.36%). Of these 757, another 27 participants (3.6%) were excluded from analysis based on having provided unrealistic answers to the cost questions. An additional 40 were excluded from analysis because they did not answer the cost questions at T1, resulting in 690 participants to be analyzed at baseline. Of these, 387 were in the intervention condition and 303 were in the control condition. Here, adolescents in the intervention condition were more often female, had a higher educational level, more often indicated being religious, and had a lower weekly alcohol use than participants in the control condition did (Table 1).

For the T1 sample, results of the linear regression analyses show that weekly alcohol use was dependent on gender, age and educational level (R²=.146). For binge drinking occasions, a significant proportion of the variance could be explained by age and educational level (R²=.136).

‎View this figure

Costs

Table 2 shows that the total health care costs per adolescent were lower in the intervention group (€85.65) than in the control group (€124.49). This difference can largely be explained by the difference in costs for reported hospital stays. However, costs of ICBs (€162.68) and substance use (€36.30) were higher in the intervention group than in the control group (€112.61 and €24.64, respectively). The difference in mean costs for court proceedings is noticeable and explains much of the difference in ICB-related costs. Large differences in specific costs were caused by outliers in either the intervention or the CAU arm. As indicated previously, sensitivity analyses were conducted without these outliers. When the intervention costs were included both from the health care and societal perspective, costs were higher in the intervention group. The z score for each cost category was positive and higher than the reference value of 1.96 [44], indicating that costs were skewed and tailed to the right (Table 2).

Incremental Costs

Table 3 shows costs per condition. Note that these means slightly differ from the costs presented in Table 2, for costs in Table 2 are bootstrapped means, whereas costs in Table 3 are means drawn from raw data. For both perspectives, costs were higher in the intervention condition. The incremental costs (ie, the difference in mean costs per adolescent between the intervention and control condition) varied per perspective, namely €13.76 from the health care perspective and €74.03 from the societal perspective.

Incremental Effects

Table 3 shows the effects per condition. In comparison with the control condition, the intervention was incrementally effective in reducing the weekly use of alcohol and number of binge drinking occasions. At T1, adolescents in the control condition drank a mean 1.51 glasses of alcohol per week more than at T0. In the intervention condition, there was an increase of 0.78 glasses, resulting in a mean incremental effect of 0.73 glasses per week. Furthermore, in the control condition, there was an increase of 0.33 binge drinking occasions. In the intervention condition, there was a decrease of 0.16, resulting in a mean incremental effect of 0.49 binge drinking occasions per 30 days. For both outcome measures, this did not change with perspective; a change of perspective within the base case scenario stipulated only a change in costs.

Incremental Cost-Effectiveness Ratios

From both perspectives, the mean costs were higher for the intervention condition in comparison with the control condition. Since the intervention was more effective than CAU on both outcome measures, this resulted in positive ICERs (Table 3). However, ICERs differed for both perspectives, namely €40 and €79 from the health care perspective, and €62 and €144 from the societal perspective per incremental reduction of one glass of alcohol per week and binge drinking occasion per 30 days, respectively.

The cost-effectiveness planes (Figures 3 and 4, left side) show differences in distributions of the 5000 simulated ICERs across the four quadrants between the CEAs carried out from the two perspectives. Corresponding with the median ICERs presented in Table 3, the majority of simulated ICERs for all base case analyses are located in the NE quadrant. However, the distribution of the simulated ICERs among the quadrants differs between the perspectives. Notable is the shift of the cloud of ICERs toward the SE quadrant in the analysis carried out from the health care perspective (ie, 30% for weekly alcohol use and 34% for binge drinking occasions) in comparison to the analyses carried out from the societal perspective (ie, 23% and 25%, respectively).

The preceding percentages equal the probabilities of the intervention being cost-effective at a WTP max of €0 in the CEACs (Figures 3 and 4, right side). These results show that for low WTP thresholds the probability of the intervention being cost-effective over the control intervention is higher from a health care perspective than it is from the societal perspective. For all base case analyses, the vast majority of simulated incremental effects were in the NE quadrant; therefore, these probabilities increase to approximately 80% when the WTP max increases. The probabilities of the intervention being cost-effective do not differ much between the two perspectives for WTP thresholds greater than €500 (Figures 3 and 4, right side).

Sensitivity and Subgroup Analyses

The results of the sensitivity analyses (ie, excluding cost outliers) attest to the robustness of the base case analyses (Table 3). From the societal perspective, ICERs were close to similar. From the health care perspective, ICERs increased and were higher than those of the societal perspective. However, as for the base case analyses, the probability of the intervention being cost-effective remained dependent on the WTP max. The results of the analyses conducted from a societal perspective minus the costs of drugs and cigarette use were similar to the results of the analyses conducted in which these costs were included (Figures 3 and 4).

Subgroup analyses showed, from both the health care and the societal perspective, and for both outcome measures, that the intervention was cost-effective for the older adolescents and those at a lower educational level (Tables 4 and 5). From a health care perspective, it was found to be cost-effective for the male and nonreligious adolescent subgroups as well. The intervention was not cost-effective for those with a non-Dutch ethnicity or for female adolescents for the weekly alcohol use outcome measure. For all other subgroups, ICERs were positive, meaning the intervention was cost-effective depending on the WTP max. The corresponding cost-effectiveness planes and CEACs of all subgroup analyses can be obtained in Multimedia Appendices 2 and 3.

‎View this figure

‎View this figure

Discussion

Principal Results

To the best of our knowledge, this was the first cost-effectiveness analysis of a Web-based intervention conducted from both the health care and societal perspective that also incorporated the possible impact of ICBs on the cost-effectiveness results. From both a health care and a societal perspective, our study shows the intervention was incrementally more effective and more costly than CAU. This counts for both the analyses in which the weekly alcohol use outcome measure was incorporated and the analyses based on the binge drinking occasions outcome measure.

Although the intervention was incrementally effective in targeting weekly alcohol use, there was an increase in the number of glasses of alcohol between T0 and T1 in both arms of the trial. This can be explained based on the estimation that approximately one-third of the study sample had his or her birthday during this 4-month period between T0 and T1, of which some reached the legal drinking age of 18 years. Furthermore, all adolescents in the sample aged 4 months, which increased the chance of them starting to drink or drink more. This is also true for the younger Dutch adolescents because many start drinking before the legal drinking age [45]. However, contrary to weekly alcohol use, the number of binge drinking occasions did not increase in the intervention arm; a small decrease of a mean 0.16 binge drinking occasions was noticed compared to an increase of a mean 0.33 in the CAU arm. Therefore, relative to the overall alcohol intake and compared to CAU, it can tentatively be concluded that adolescents in the intervention arm became less irresponsible about drinking.

Our research also shows that the inclusion of ICBs in the economic evaluation impacted the cost-effectiveness results of the analysis, especially for certain subgroups. From a health care perspective, the intervention is cost-effective for the male, lower education, older adolescent, and nonreligious subgroups. However, from a societal perspective (which includes ICBs), the intervention is clearly cost-effective only for the lower education and older adolescent subgroups.

The inferiority of the intervention for certain subgroups could, among other reasons, partly be explained based on the finding that the baseline consumption for these subgroups was relatively low compared to that of their counterparts. For example, the baseline mean weekly alcohol use in the female subgroup was 1.49 glasses (SD 3.57) compared to weekly mean 4.18 glasses (SD 8.42) in the male subgroup. In so far as the following can be concluded based on an analysis of the smallest subgroup (n=56), this also goes for the non-Dutch subgroup (mean 2.16, SD 5.00 glasses and mean 1.21, SD 2.44 binge drinking occasions) versus the Dutch subgroup (mean 2.74, SD 6.48 glasses and mean 1.64, SD 2.77 binge drinking occasions). In these subgroups, there was less effect to be gained. Consequently, this could be related to the possibility of these adolescents not identifying themselves as being part of the target group of, and being affected by, the intervention.

Strengths and Limitations

Some of the strengths of this study are its relatively large sample size and its randomized design. The cluster randomization at the school level minimized the risk of contamination between the study conditions. Furthermore, the large heterogenic study sample was a good representation of the Dutch adolescent school-going population and allowed for subgroup analyses on multiple background variables.

The use of a societal perspective along with a health care perspective was a major strength of this study. The societal perspective is argued to be dominant over other perspectives [46-48]. This is because of, but not restricted to, health economics’ foundations in welfare economics, which means that an economic evaluation should include the impact of an intervention on the whole society [46]. However, not only the choice of perspective, but also the way this was implemented in the study design, can be considered a major strength. Because the study population consisted of school-going adolescents, limiting the societal perspective to including merely labor productivity costs would not have properly reflected this impact. For this study, the results show that labor productivity costs make up just a small part of the total costs of ICBs (Table 2). By including costs within the educational sector and criminal justice system, we managed to provide a better reflection of the economic impact of this intervention on society.

Apart from these strengths, the findings of this study need to be placed in the context of the study’s limitations. First, both at T0 and T1, the composition of adolescents in the intervention condition was significantly different from that of the control condition for various characteristics, including gender, educational level, and religion. This might have been caused by (1) the cluster RCT design instead of randomization at the individual level and/or (2) the large dropout before the baseline assessment within the control condition in comparison with the intervention condition. Although the results of the regression analysis showed a relationship between some of the background variables and the outcome measures, uncertainty around the ICERs that were calculated in the base case analyses was dealt with through various strategies. The sensitivity analyses attest to the robustness of the findings. Furthermore, the heterogeneity of the sample was addressed extensively by calculating ICERs and conducting bootstrap analyses for all subgroups based on all background variables.

Second, the follow-up period of this CEA (ie, 4 months) might be regarded as short. Costs and (health) benefits that fall beyond these 4 months were not assessed. Future studies, including additional follow-up measures and cost-effectiveness modeling, could be interesting. Other studies have shown that the cumulative cost savings in the life span of health promotion interventions for adolescents could be high [49].

Third, the authors decided not to further modify the original dataset by imputation and restricted the analysis to complete cases. A missing completely at random analysis (MCAR) in SPSS version 20 based on the N=2493 sample that started at baseline showed 71% to 72% per cost variable at T1. Furthermore, the missing values were not at random (P<.001). Although this was expected considering the large dropout between T0 and T1, the same goes for the n=757 sample that started follow-up (P<.001). Given the nonnormality of cost variables, the nonrandomness of missing values, and the large dropout as is common in many Web-based interventions [19,50-55], it was concluded that additional imputation would have manipulated the original dataset too much. This counts for both the basic imputation methods, such as expectation maximization and last observation carried forward, as well as for the more advanced methods, such as Markov chain Monte Carlo technique with predictive mean matching [56]. As for imputation strategies, the chosen strategy might have led to biased results. Nevertheless, the alternative of replacing more than 70% of the values, which would have been needed in this study, would have increased the risk of a type II error [51]. Imputation would have resulted in an increased chance of underestimating the intervention’s effectiveness and an unrealistic representation of costs.

Fourth, measurements were based on self-reports, which could have led to an underestimation of service use, alcohol use, and use of other goods in comparison with daily diaries [24,57]. As for any measurement based on recalling services or goods used, this is because of forgetting [17]. However, in this study, recall periods were kept short. For example, respondents were asked for their alcohol use in the previous week and not in a typical week. Furthermore, the recall period for the cost measurement questions was only 4 months. In addition, because the groups were randomized, this underestimation is likely to be equally distributed among the intervention and control group. Therefore, it is unlikely that the ICERs were affected.

Finally, within the setting of this study, respondents were free to fill in the answers themselves during the measurements. As mentioned earlier, limits needed to be set to exclude respondents who provided unrealistic answers. The choice made by the authors to exclude whole cases might have affected the outcomes of the analysis. However, this choice is justifiable based on the finding that the vast majority of unrealistic answers were not even close to the limits set by the authors and were far higher than the credible range. For example, there were seven respondents who claimed to have spent more than 200 nights in a hospital bed in the previous 4 months, of which three said to have spent more than 200,000. The 27 adolescents at T1 who filled in unrealistic answers (only 3.6% of the n=757 sample that started follow-up) had a mean 2.9 unrealistic answers. Based on the data, it was clear that the vast majority of these 27 adolescents did this deliberately and systematically. Although the outcome of the analyses might have been affected by the limits that were set by the authors, these limits were carefully considered, discussed, and decided a priori to the analyses. This was done to minimize the chance of biased results. Furthermore, the impact of cost outliers on the outcomes of the base case analyses has been covered in sensitivity analyses in which cost outliers were excluded.

Recommendations

Computer-tailored feedback can be a cost-effective way to target alcohol use and binge drinking among adolescents. In the Netherlands, despite the Dutch government’s change of policy to reduce the minimum legal drinking age, 33.4% of Dutch adolescents were drinkers in 2014 [45]. This is because in practice drinking rules are set not only by Dutch law, but also by parents or caregivers and alcoholic beverages can easily be obtained via family and friends. Therefore, effective and cost-effective interventions targeting adolescent drinking behavior are still very much needed. The high dissemination capabilities of the Alcoholic Alert intervention, combined with its solid basis in the I-Change model and low intervention costs could make it an interesting investment for reducing alcohol use among adolescents.

Because the cost-effectiveness for the whole sample is dependent on the WTP max per effect, it is difficult to make strong recommendations on whether the intervention should be implemented from an economic point of view. Contrary to the generic outcome measure quality-adjusted life year (QALY) [58,59], and as is common for the majority of specific outcome measures, no guidelines are available that provide a reference cost-effectiveness threshold for reducing the consumption of alcohol. However, the CEACs provide decision supportive information because these provide cost-effectiveness probabilities for a wide range of hypothetical thresholds for all analyses. These also show that, from both the health care and the societal perspective, the intervention is cost-effective for older adolescents and for those at a lower educational level, regardless of which threshold is set. From a health economic viewpoint, it is recommended that these specific groups be targeted. When adopting a health care perspective, the same goes for the male and nonreligious adolescent subgroups.

In general, policy makers should be aware of the impact of the perspective chosen for the analysis on its outcomes. Omitting ICBs could negatively affect the reliability and informative value of analyses that are conducted from a societal perspective. Therefore, it is recommended that researchers should carefully make a priori considerations on the costs to be included because leaving out important costs could lead to biased results [60]. Finally, as in this study, high attrition rates could affect the outcomes of CEAs. High attrition rates are common in eHealth interventions [61,62]. It is recommended that more research should be conducted on adherence to eHealth interventions and that these interventions be implemented in practice, thus increasing their effectiveness, cost-effectiveness, and impact on public health.