This paper is in the following e-collection/theme issue:

Web-based and Mobile Health Interventions (3297) Cost-Effectiveness and Economics (252)

Published on in Vol 15, No 1 (2013): January

Evaluating the Translation Process of an Internet-Based Self-Help Intervention for Prevention of Depression: A Cost-Effectiveness Analysis

Evaluating the Translation Process of an Internet-Based Self-Help Intervention for Prevention of Depression: A Cost-Effectiveness Analysis

Evaluating the Translation Process of an Internet-Based Self-Help Intervention for Prevention of Depression: A Cost-Effectiveness Analysis

Authors of this article:

Ove K Lintvedt1 ;   Kathleen M Griffiths2 ;   Martin Eisemann1 ;   Knut Waterloo1
Article Authors Cited by (20) Tweetations (10) Metrics

    Original Paper

    1Faculty of Health Sciences, Department of Psychology, University of Tromsø, Tromsø, Norway

    2Centre for Mental Health Research, The Australian National University, Canberra, Australia

    *all authors contributed equally

    Corresponding Author:

    Knut Waterloo, PhD

    Faculty of Health Sciences

    Department of Psychology

    University of Tromsø

    Huginbakken 32

    Tromsø, N-9037

    Norway

    Phone: 47 77 64 52 90

    Fax:47 77 64 52 91

    Email: knut.waterloo@uit.no


    Background: Depression is common and treatable with cognitive behavior therapy (CBT), for example. However, access to this therapy is limited. Internet-based interventions have been found to be effective in reducing symptoms of depression. The International Society for Research on Internet Interventions has highlighted the importance of translating effective Internet programs into multiple languages to enable worldwide dissemination.

    Objective: The aim of the current study was to determine if it would be cost effective to translate an existing English-language Internet-based intervention for use in a non-English-speaking country.

    Methods: This paper reports an evaluation of a trial in which a research group in Norway translated two English-language Internet-based interventions into Norwegian (MoodGYM and BluePages) that had previously been shown to reduce symptoms of depression. The translation process and estimates of the cost-effectiveness of such a translation process is described. Estimated health effect was found by using quality-adjusted life years (QALY).

    Results: Conservative estimates indicate that for every 1000 persons treated, 16 QALYs are gained. The investment is returned 9 times and the cost-effectiveness ratio (CER) is 3432. The costs of the translation project totaled to approximately 27% of the estimated original English-language version development costs.

    Conclusions: The economic analysis shows that the cost-effectiveness of the translation project was substantial. Hopefully, these results will encourage others to do similar analyses and report cost-effectiveness data in their research reports.

    J Med Internet Res 2013;15(1):e18

    doi:10.2196/jmir.2422

    Keywords

    Internet; Internet intervention; Cognitive Behavior Therapy; Quality-Adjusted Life Years; Cost Effectiveness; Mental health; Depression 


    Computer-aided psychotherapy is a promising way to increase accessibility to evidence-based treatment of many mental disorders, such as mood disorders [1]. Depression is an important global public health issue because of its prevalence [2], that it is associated with a high level of disability and disease burden [3], and its social costs [4]. Depression is also associated with increased risk of physical disorders and early death [5,6].

    In a systematic review of computer-aided psychotherapy programs, Marks et al [7] found 97 computer-aided psychotherapy programs described in 175 studies, of which 103 were randomized controlled trials (RCT). These studies discussed screening, effectiveness, efficacy, cost-effectiveness, and the dissemination of computer-aided psychotherapy programs within health services. Cohen’s d effect sizes for the identified computer-aided psychotherapy systems ranged from 0.2 (small) to 4.3 (extremely large) [7]. For depression, Marks and colleagues [7] identified 9 programs, of which 3 were Internet-based programs evaluated in RCTs (Overcoming Depression on the Internet [8], MoodGYM [9], and netCBT [10]). In their early reviews of Internet-based RCTs of mental disorders, Griffiths and Christensen [11] and Griffiths et al [12,13] identified another program [14]. Since these reviews, at least 16 new Internet-based intervention programs for depression have been deployed. Several of these have been subjected to research evaluation or are in the test phase: Alles onder controle [15], Color your life [16], Deprexis [17], E-couch [18], HealthSteps for Depression [19], Interapy Depression [20], Living Life to The Full On-line [21], MoodCalmer [22], MoodHelper.org [23], MoodMemos [24], myCompass [25], Project Catch-it [26], This Way Up-Depression Course [27], This Way Up-Mixed Depression and Anxiety Course [28], Xanthis [29], and Youth Mental Health [30]. In addition, more than 20 other programs are still being tested and, thus, no published research evidence of the efficacy of the programs are available.

    A recent quantitative meta-analysis on Internet and computerized interventions for adult depression found an overall effect size of d=0.41 for computer-aided psychotherapy compared to control [31]. A recent meta-analysis by Andrews and colleagues [32] found an overall effect size for depression of Hedges’ g=0.78. Most of the research on Internet-based interventions for depression has focused on treatment rather than prevention, and most studies have employed guided rather than pure self-help strategies. One reason for this focus could be that treatment has more immediate benefits than prevention in which the benefits take longer to emerge [33].

    In their meta-analysis, Andersson and Cuijpers [31] reported an average effect size of d=0.25 for unguided Internet and computerized interventions. It has been suggested that such interventions have higher attrition rates than those involving therapist support [31,34-36]. To date, we are not aware of any published studies that directly compare attrition in therapist-guided compared to unguided interventions. However, even if attrition is higher in unguided Internet interventions, they have the potential to provide significant public health benefits due to the degree of dissemination [33].

    Internet-based self-help has the potential to reach target groups with an unmet need for help. In a recent study, we found that two-thirds of the participants completing a trial of an Internet-unguided cognitive behavioral therapy (CBT) program initially reported an unmet need for help with a psychological problem [37]. Briefly, the Internet intervention was associated with a significant reduction in depressive symptoms compared to the control condition, and thus indicated that the intervention facilitated an effective self-help effort among a group of people who normally would not seek professional help. The importance of reaching and engaging this group is obvious given the evidence that low levels of help-seeking intentions among those with mental health problems are associated with suicidal ideation [38].

    There is now sufficient evidence to suggest that Internet interventions can be effective in the prevention and treatment of depression [39,18,40,41]. The International Society for Research on Internet Interventions (ISRII) has highlighted the importance of facilitating the dissemination of Internet applications by providing translation into multiple languages [42]. The World Health Organization (WHO) has also suggested that after establishing their efficacy, Internet-based prevention interventions should be disseminated worldwide [33]. One program that has been subjected to a number of trials and found to be effective over 12 months is the English-language Internet program for depression, MoodGYM [43]. There is also evidence of the efficacy of the English-language website BluePages [44], which provides information on the symptoms of and treatment for depression based on scientific evidence, help resources, and depression and anxiety screening tests. Both programs are available as part of a publicly available, free of charge, e-mental health service delivered by The Australian National University (ANU) and funded by the Australian Government [45]. This paper describes the translation of MoodGYM and BluePages into the Norwegian language by a research group at the University of Tromsø in Norway. Reviews in the field of Internet interventions for mental disorders show that studies are lacking cost-effectiveness data [11,46].

    The aim of this study is to evaluate the translation process of an Internet-based self-help intervention up to dissemination and to study cost and cost-effectiveness of the project. It will also consider the feasibility of providing access to Internet-based interventions in the national language versions. Further, we discuss some of the challenges experienced during this project and give recommendations for future developments. Finally, we attempt to demonstrate the use of disease-specific scales to estimate quality of life.


    Prior Work

    Intention to Use Internet-Based Self-help

    Before translating the Internet-based interventions, a study was undertaken to investigate the need for a Web-based self-help intervention [47]. Nearly 31.9% (115/367) of the respondents reported they felt a need for help, but had not sought help; therefore, they had an unmet need for help. Among these respondents, 91.2% (105/115) reported a positive attitude toward using a service like MoodGYM.

    The Norwegian Version of MoodGYM and BluePages

    In February 2006, Norwegian language versions of BluePages and MoodGYM were made available. The planning of this project started in July 2004 and the project was completed in June 2006. The translation of BluePages was formalized in a licensing deed, specifying ANU as the intellectual property owner and the obligations for the collaborating partners. For MoodGYM, the research collaboration agreement specified a nonexclusive, nontransferable license for the translated version during the period of collaboration and joint ownership of the data emerging from the collaboration. The translation of MoodGYM and BluePages was carried out between October 2005 and January 2006. The Norwegian version of the BluePages website was developed by using a 2-phase process. First, a professional translator prepared a Norwegian version of BluePages. In the second phase, the translation was adjusted by the research group at the University of Tromsø in Norway to ensure that it was culturally and clinically appropriate for Norwegian users. The MoodGYM training program was translated in 4 phases. The research group conducted the first phase of translation. In phase 2, clinical professionals with formal competence in cognitive therapy scrutinized the translation from the first phase and made adjustments to the text when necessary. In phase 3, an expert translator of English compared the Norwegian version of the program with the original Australian version and checked for inconsistencies. In the fourth phase, the research group evaluated all changes and finalized the translation.

    Validating the Norwegian Version

    The Norwegian versions of MoodGYM and BluePages were evaluated in a RCT that compared the effect on depressive symptoms of an unguided Internet-based intervention. The Internet condition consisted of a depression information website and a self-help Web application that delivered automated CBT. The participants in the waiting list condition were free to access formal or informal help as usual. This trial was organized as an unguided quasi-indicated prevention intervention. A total of 163 students (mean age 28.2 years) with elevated psychological distress were enrolled into the trial. The intent-to-treat effect size for depressive symptoms was d=0.63 and d=0.72 for completers [37]. A total of 61.9% (101/163) of the participants remained in the trial at postintervention, including 53% (43/81) of the Internet intervention participants. With respect to adherence, 33% (27/81) of the Internet intervention participants completed on average 63% (3.1/5 modules) of the MoodGYM program without any support or reminders. Among the completers, 63% (27/43) in the experimental condition and 56% (33/59) in the control condition initially reported an unmet need for help. The logs from MoodGYM showed that the 25% of the 81 participants in the experimental condition who did not return the postintervention questionnaire, on average had used almost half of the modules in MoodGYM (2.4/5 modules, 48%). The trial demonstrated a real-world usage of an unguided Internet-based intervention.

    Cost Estimates

    The total cost estimate for the Norwegian version of MoodGYM and BluePages was estimated from the cost of the translation process and the project costs for the Norwegian evaluation trial.

    The translation process was accomplished by using 2 members from the Norwegian research group, 3 students in their final year of clinical psychology training, and external resources (translators, psychiatrists, mailing/printing services). Translation costs for employees and students were calculated based on the estimated time consumed and their salaries. The cost for external resources was based on payments for their services.

    The trial costs were incurred over 8 weeks from initial participant contact until the completion of the validation trial. This estimate for total costs was based on all expenditures of the validation trial for BluePages and MoodGYM and included the direct costs of mailing and printing during the trial, and also salary for employees and students. Working hours for developing funding applications were not included in the estimates. Because the aim of this study was to evaluate the translation process up to dissemination, the trial costs are included even though these costs could be regarded as research costs.

    The initial cost data were in Norwegian kroner (NOK) and dated back to late 2005 and early 2006. For the purpose of this paper, the cost has been adjusted for inflation based on the Norwegian consumer price index into a 2009 price level [48] at an average annual inflation rate of 2.3%. The development cost from ANU dated back to 2001. We have adjusted the cost for inflation based on the Australian consumer price index into a 2009 price level [49], at an average annual inflation rate of 2.9%. The foreign currencies were then converted into Euro (€) based on real exchange rates from January 1, 2010. These rates form the basis of our calculations, rounded up to the nearest €100. The rationale for using the Australian development cost was in the interest of estimating the development cost if the project had been developed from scratch in Norway, and the Australian development cost provided an estimate of this figure. The cost analyses were based on an operating period of 3 years. After such a timeframe, the Internet-based programs need to be maintained, changes made in design, the technology needs to be updated, and so on. Although most of the investment in the intervention can be maintained, there will be some new costs at this point in time.

    Measures

    Center for Epidemiologic Studies Depression Scale

    The Center for Epidemiologic Studies Depression Scale (CES-D) [50] was used as the outcome measure for depressive symptoms. Each item of the CES-D is scored on a 4-point ordinal scale ranging from 0 to 60, with a total score of 16 or higher considered to be depression. Because this criterion has some limitations [51], it will not be used in this paper. For the completers analyses, we used a stratified scale adapted from Rushton and colleagues [52]: subclinical (0-15), mild/moderate (16-23), and moderate/severe (≥24) depression. The internal consistency of the CES-D scale was alpha = .87.

    Kessler Psychological Distress Scale

    The Kessler Psychological Distress Scale (K10) [53] is a scale of psychological distress developed for use in epidemiological surveys. Scores on the K10 scale range from 10 to 50, with higher scores indicating greater distress. People who score under 20 are likely to be well, whereas those scoring between 20 and 24 have a mild mental disorder, those between 25 and 29 have a moderate mental disorder, and those scoring 30 and over have a severe mental disorder. The internal consistency of the K10 scale was alpha = .79.

    Quality of Life

    There has been a growing interest for applying measures for health-related quality of life (QOL) to evaluate health care services, in general. However, in the field of Internet- and computer-based self-help for mental disorders, reviews find that studies lack cost-effectiveness data [11,46]. This is a disadvantage for the field because proper evaluations among interventions and services are not possible. There are several measures available for this purpose: those specific to groups of diseases, and generic measures applied across disease categories [54]. Some produce a single measure as a health index and others assess a profile of scores in different areas of disease. For independent interventions in which it is essential to use average cost-effectiveness ratios and for mutually exclusive interventions, the use of incremental cost-effectiveness ratios is essential [55]. Interventions in which the costs and effects are not affected by the introduction of other interventions are independent interventions. The cost-effectiveness ratio measures how efficiently the intervention can produce an additional quality-adjusted life year (QALY). In this way, the cost-effectiveness of alternative innovations may be compared. The cost-effectiveness ratio (CER) is calculated as the costs of intervention divided by the health gain (eg, life-years gained). The intervention with the lowest CER is preferred [55].

    The Rosser Classification of Illness States Scale

    The Rosser classification of illness states scale (Rosser Index) [56] offers a ratio scale based on QOL for estimating a utility value that captures the degree of improvement in health (∆H). The ∆H is used to calculate the gain in QALY. As one of the earliest utility instruments, the Rosser Index was developed for use in clinical settings [57]. This is a generic measure with a single index measure for health status. The value for each health state is the sum of all products when multiplying each cell in the Rosser health state matrix with corresponding cells in the Rosser valuation matrix (Figure 1).

    In the present study, the Rosser Index was used as a generic instrument to evaluate the observed effects for the Internet intervention. The method is based on classifying outcome data from the trial into 2 components: disability categories (I-VIII) and distress categories (A-D) that define 29 potential health states [56]. The Rosser health state matrix is based on these disability and distress categories.

    The original valuation matrix was based on 70 respondents (doctors, nurses, patients, etc) and was not a random sample of the population. The matrix has been transformed and validated in several publications [58]. The Rosser valuation matrix in its present form is based on the method of magnitude estimation [59] in studies from England [56], but they correspond well with Norwegian data [60]. The process of deciding the appropriate categories for the Rosser health state matrix is according to the method described by Gudex and Kind [61].

    ‎
    Figure 1. Model for the Rosser Index measuring quality of life (QOL). The health state matrix is based on the Center for Epidemiologic Studies Depression scale (CES-D) scores in relation to the Rosser Disability Category and the Kessler Psychological Distress Scale (K10) scores in relation to the Rosser Distress Category.
    View this figure
    Rosser Health State Matrix: Disability Categories

    The CES-D was used as the basis for deciding the appropriate disability categories. No other studies were found that used CES-D as the disability categories in the Rosser health state matrix. The authors have functioned as a reference group of clinical psychologists and found that the CES-D seems to correspond to the Rosser disability dimensions (see Table 1). This is in-line with the recommendations by Gudex and Kind [61]. The first 2 CES-D subgroups (subthreshold and mild/moderate) correspond well with Rosser´s disability categories I (no social disability) and II (slight social disability); the third CES-D subgroup (moderate/severe) corresponds well when dividing the CES-D scores over Rosser´s disability categories III to VI. The disability categories VII to VIII are not used because they represent states of being confined to bed and unconscious. The analogy between CES-D and Rosser Disability category is shown in Table 1.

    Table 1. Center for Epidemiologic Studies Depression scale (CES-D) scores and their corresponding Rosser disability categories.
    CES-DRosser disability category
    SubgroupScoreCategoryDescription
    Subclinical0-15INo social disability
    Mild/moderate16-23IISlight social disability, can continue almost as usual with occupational and home activities
    Moderate/severe24-33IIISevere social disability and/or slight impairment of performance at work

    		34-42IVChoice of work or performance at work very severely limited

    		43-52VUnable to undertake any work/education

    		53-60VIFull-time care or in an institution
    Rosser Health State Matrix: Distress Categories

    The K10 was developed for use in epidemiological surveys and is used to measure psychological distress. No other studies were found that used K10 as the distress category in the Rosser health state matrix. The authors have functioned as a reference group of clinical psychologists and found that the K10 seems to correspond largely with the Rosser distress dimensions (see Table 2). This is in-line with the recommendations by Gudex and Kind [61]. The K10 preassessment scores are used as the basis for deciding the appropriate distress categories for the participants.

    Table 2. Kessler Psychological Distress Scale (K10) scores and their corresponding Rosser distress categories.
    K10Rosser distress category
    SubgroupScoreCategoryDescription
    Being well10-19ANo distress
    Mild mental disorder20-24BMild distress
    Moderate mental disorder25-29CModerate distress
    Severe mental disorder30-50DSevere distress
    Quality-Adjusted Life Years

    A QALY is a year of life adjusted for its quality or its value. A year in perfect health is considered equal to 1 QALY [62]. The value of a year in ill health would be discounted. In the trial, there was a period of 8 weeks between the preassessments and postassessments, in such a way that the ∆H was gained over a 2-month timeframe. The trial did not have a 12-month follow-up, but research from the field shows that the effect over time, from posttest to 12-month follow-up, is maintained [63]. Mackinnon and colleagues [63] conducted a study on 12-month outcome for MoodGYM compared with BluePages and a control group. This gives the best estimate for how to transform our findings to a 12-month outcome. They reported that both the MoodGYM group and the control group had a decline in CES-D scores from posttest to 12-month follow-up, and the control group had the best benefit in reduced symptoms of depression. These transformations of QOL are according to the principles described by Gudex and Kind [61]. The gain in QALYs (QALY-gain) is based on the utility value for improved health ∆H, multiplied by the time interval (T) over which the improvement occurred (measured in years): QALY-gain = ∆H × T [64].

    Cost per QALY Gained

    The National Institute for Health and Clinical Excellence [65,66] in the United Kingdom considers cost per QALY gained among its criteria for coverage recommendations to the National Health Service. This has been converted into an explicit criterion of £30,000 per QALY to be used as a guideline for these recommendations [67]. In Norway, an amount of NOK 500,000 has been proposed (equivalent to €67,100 in 2009) as a temporary best estimate of a QALY [68].


    The Norwegian real cost totalled to €39,900, and the estimated cost to €56,000, adding up to a total of €95,900. Based on data published by Butler and colleagues [69], the Australian development cost was €479,400 plus an average annual maintenance cost of €44,700. The Norwegian costs correspond to almost 27% of the Australian development costs.

    Table 3 summarizes the complete cases analysis for depressive symptoms with change scores and effect size [37]. The between-group difference for the Internet intervention and control groups for the overall complete cases analysis was 7.1 points on the CES-D scale and 7.0 points for the intention-to-treat analysis. The mean scores at baseline for K10 were 33.2 (SD 5.5) for the Internet group and 33.2 (SD 5.0) for the control group. The deterioration in the control group is evident in their total scores and rather high numbers in the CES-D subclinical and moderate/severe subgroups. There is almost no change for the mild/moderate subgroup.

    Table 3. Completers between-group effect size (Hedges’ g) for depression level as measured by the Center for Epidemiologic Studies Depression Scale (CES-D).
    CES-D scale and conditionPretestPosttestContrastaEffect sizeb

    		Mean (SD), nMean (SD), nMean (SD)g (SE)
    Total




    		Intervention22.6 (10.9), 4318.5 (14.0), 434.1 (10.4)0.72 (0.21)

    		Control18.5 (9.6), 5921.4 (13.0), 59–3.0 (9.1)
    Subclinical




    		Intervention11.4 (3.6), 1410.0 (8.6), 221.4 (8.4)0.68 (0.34)

    		Control9.8 (3.4), 2514.4 (9.7), 25–4.6 (8.8)
    Mild/moderate




    		Intervention19.0 (2.7), 1011.0 (9.5), 78.0 (9.8)0.85 (0.42)

    		Control19.2 (2.3), 1719.5 (10.9), 11–0.3 (9.2)
    Moderate/severe




    		Intervention32.7 (7.0), 1928.6 (12.8), 144.1 (11.7)0.67 (0.34)

    		Control30.4 (10.0), 1732.4 (12.1), 23–3.2 (9.2)

    a Positive contrast represents better outcome.

    b Hedges’ g is Hedges’ unbiased effect size; SE is Hedges’ unbiased standard error.

    The following analysis is based on the completers data from the effect trial [37]. For both conditions (Internet intervention and control group), participants’ distress and disability category is given. Table 4 presents an overview for condition and time in which each cell in the matrix represents the number of cases for the combination of distress and disability. Each cell in this descriptive matrix is associated with a corresponding weight in the Rosser valuation matrix (Table 5).

    Table 4. Descriptive matrix with number of participants for each disability and distress state, for condition and test time.
    ConditionDisability categoryDistress category, n


    		BaselinePosttest


    		ABCDABCD
    InternetI185
    		11281

    		II
    		631
    		421

    		III
    		343
    		212

    		IV
    		2
    		5
    		115

    		V


    		2


    		2
    ControlI111103
    		10123

    		II
    		746
    		632

    		III
    		1761315

    		IV
    		1


    		153

    		V
    		1
    		1


    		3

    		VI




    		1

    Table 5. Rosser´s utility valuation scores for health conditions.
    Disability categoryDistress categorya

    		ABCD
    I (no social disability)1.0000.9950.9900.967
    II (slight social disability)0.9900.9860.9730.932
    III (severe social disability)0.9800.9720.9560.912
    IV (severely limited work performance)0.9640.9560.9420.870
    V (unable to work/study)0.9460.9350.9000.700
    VI (total social disability)0.8750.8450.6800.000

    a 1 = healthy; 0 =dead. Table shows only relevant disability categories (for complete table see Kind et al [59]).

    When all cells in the descriptive matrix are multiplied with the corresponding weight in the valuation matrix, the sum of all these products gives the total QOL score for condition and time. The total QOL score is then divided by the number of participants in the group, which gives an average QOL gain (∆H) score for the group [61]. The Internet intervention group (n=43) had a total QOL at pretest of 40.82 (mean 0.95), and 41.02 (mean 0.95) at posttest, with a ∆H of 0.005. For the control group (n=59), the total QOL at pretest was 56.86 (mean 0.96) and 55.96 (mean 0.95) at posttest, with an average ∆H of -0.015. The between-groups gain in QOL (∆H) was 0.020.

    Prediction of a 12-Month Outcome

    By using the change scores from the Australian 12-month follow-up study of MoodGYM and BluePages [63], we calculated a factor to extrapolate our findings to predict a 12-month outcome. Because the results for the control group in our trial were negative, the factors were calculated from posttest to 12-month follow-up. For the MoodGYM group, the change from posttest (mean 15.9) to 12-month follow-up (mean 14.1) was 1.8 points on the CES-D scale, giving a change factor of 1.127. For the control group, the change from posttest (mean 19.5) to 12-month follow-up (mean 16.4) was 3.1 points on the CES-D scale, giving a change factor of 1.19. As the change for the control group was negative, we multiply with the inverse factor at 0.84 (1/1.19). In this way, the difference between groups will diminish.

    The gain in QOL (∆H) for the Internet intervention group was 0.0048, and increased to 0.0054 (0.0048×1.127) after transformation. The negative gain in ∆H for the control group was –0.015 and decreased to –0.013 after transformation. The between-groups ∆H reduced to 0.018. The transformation extrapolates the results to a timeframe of 1 year. This gives a QALY-gain=0.018×1, which equals 0.018 of a QALY.

    Table 6 presents cost analyses assuming an operating period of 3 years, as the Internet-based programs need to be maintained after some time. The total cost for the translation project is €55,000 per year, equal to 27% of the cost of the development project in Australia. For the translation project, we added an annual cost for maintenance (€14,000) and a service fee (€9000) to provide a more realistic estimate for real-world use. The Internet-based programs run on a shared server in Australia, resulting in the lower maintenance cost for the translation project.

    Table 6. Estimated development and translation costs with mean cost (based on 3 years’ operating time).
    Project and costs typeCosts per year (€1000)Mean cost (€1000)


    		Year 1Year 2Year 3
    Development cost (Australia)




    		Development479.4

    		159.8

    		Maintenance44.744.744.744.7

    		Total524.144.744.7204.5
    Translation cost (Norway)




    		Development95.9

    		32.0

    		Maintenance14.014.014.014.0

    		Service fee9.09.09.09.0

    		Total118.923.023.055.0

    Based on data from the validation study, the QALY-gain was found to be 0.018 of a QALY. To obtain one QALY, we need 56 individuals (1/0.018) to use the Internet intervention in the same way as the group of completers did in the trial. This is equal to gaining 1 year of full health for 1 person.

    Table 7 shows a sensitivity analysis for cost and savings, divided on a development and translation project. Estimated savings per QALY is varied between the Norwegian estimate and the UK estimate. For each of these scenarios, Table 7 shows potential savings based on the number of people treated annually. The CER is the ratio between cost and QALY. The development cost for the translation project is less than the cost for 1 QALY gained, and for the developments project just over 3 QALY gained, based on the Norwegian estimate for cost per QALY gained. The most conservative estimate indicates that for every 1000 persons treated, 16 QALYs are gained. The investment is returned 9 times and the CER is 3432 (based on the more conservative UK estimate for cost per QALY gained). Consequently, if 20,000 people are treated per year, 355 QALYs are gained. Annual savings for the translation project are estimated to at least €500,000 per 1000 persons treated.

    Table 7. Estimates for cost and annual savings (based on 3 years’ operating time) for development and translation projects, with cost-effectiveness ratio (CER).
    Project estimatePer QALYAnnual


    		Savings (€1000)PersonsDevelopment cost (€1000)Persons treatedSavings per persona (€1000)Total savings (€1000)QALYsbCERc
    Development








    		Norwegian675620410000.999921513,772

    		Norwegian675620420,0001.1923,725354576

    		UK305620410000.333321118,450

    		UK305620420,0000.5310,510350582
    Translation








    		Norwegian67565510001.201141173228

    		Norwegian67565520,0001.2023.875356154

    		UK30565510000.54481163432

    		UK30565520,0000.5410,659355155

    a Savings per person = (savings per QALY)/(persons per QALY)–(development cost)/(persons treated).

    b QALYs = (total savings)/(savings per QALY).

    c CER = (development cost)/QALYs.

    Sensitivity Analysis

    Our model for estimating the gain in QOL is based on the Rosser Index. Depending on how well the CES-D and K10 scales were fitted to the disability and distress categories, this could affect the output from the model. A sensitivity analysis was conducted to explore how the operationalizing of the scales could have contributed to variation in the outcome (QOL gain). Three scenarios were investigated: (1) the main analysis based on the score for each individual participant, (2) by using the CES-D subgroup mean, and (3) by using the mean for the conditions. In addition, 2 scenarios were investigated to explore the effect of the deterioration in the control group: (4) if there was no change in the control group, and (5) a positive change in the control group. Scenario 1 is already described previously.

    Scenario 2 had only 3 values for each condition and time. In the Internet condition, scores for CES-D and K10 placed the CES-D subgroups as follows at pretest and posttest: subclinical group (disability I/distress B, K10 mean 22.8; disability I/distress B, K10 mean 23.7), the mild/moderate subgroup (disability II/distress B, K10 mean 24.4; disability II/distress C, K10 mean 25.3), and the moderate/severe subgroup (disability III/distress D, K10 mean 30.0; disability III/distress D, K10 mean 30.7). In the control condition, scores for CES-D and K10 placed the CES-D subgroups as follows at pretest and posttest: subclinical group (disability I/distress B, K10 mean 24.5; disability I/distress C, K10 mean 25.4), the mild/moderate subgroup (disability II/distress C, K10 mean 27.5; disability II/distress C, K10 mean 25.1), and the moderate/severe subgroup (disability III/distress C, K10 mean 28.5; disability III/distress C, K10 mean 28.3).

    Scenario 3 was placed in the disability category II for both conditions, based on CES-D scores (Table 3). The K10 mean was 33.2 for both conditions. The same participants were present at both pretest and posttest, making the distress category D at both the pretest and posttest. When looking at the changes in CES-D scores, it is reasonable to assume that the Internet condition, with a reduction of 4.1 points, resulted in a relief of distress. The opposite could be applied for the control condition, as they had an increase of 3.0 points. The manual for the Rosser Index offers no rules for this scenario, so it seems reasonable to apply a reduction corresponding to half of the change between distress categories C and D, at 0.02. The same applies for the control condition.

    Scenario 4 aimed to explore the effect of the deterioration in the control group. If there were no change in the control group during the trial, the QOL gain for the intervention would be equal to the gain for the Internet intervention group. The QOL gain between groups is 0.006, only one-third of the QOL gain from scenario 1.

    Scenario 5 represents a scenario in which the control group had a positive change from baseline to posttest, as one could expect during a time period of 12 months. This scenario uses the pretest assessment for both conditions to calculate a new factor to extrapolate our findings to predict a 12-month outcome. The extrapolating was done directly on the pretest scores for each participant to predict outcome scores at 12 months. The rationale for exploring this is based on the fact that the Norwegian and Australian trials were different, unattended vs minimal contact/attention placebo. The change factors for the Internet intervention and control group were 1.546 and 1.317, respectively. The QOL gain between groups is 0.012, ie, smaller than for scenario 1 (two-thirds), but still notable. An alternative approach for this scenario could be to extrapolate the Norwegian posttest scores and compare to the Australian improvement for the control group. The change factors for the Internet intervention and control group would then be 1.127 (as for scenarios 1-3) with a QOL gain of 0.003. This is only one-sixth of the QOL gain from scenario 1. See Table 8 for the results from the sensitivity analysis.

    Table 8. Quality of life (QOL) sensitivity analysis within and between conditions.
    ScenarioCondition∆Ha∆Hextb
    1. Main analysis


    		Internet QOL gain0.0050.006

    		Control QOL gain–0.015–0.013

    		Between-conditions QOL gain0.0200.018
    2. Subgroup mean


    		Internet QOL gain0.0080.010

    		Control QOL gain–0.004–0.003

    		Between-conditions QOL gain0.0120.013
    3. Condition mean


    		Internet QOL gain0.0200.024

    		Control QOL gain–0.020–0.017

    		Between-conditions QOL gain0.0400.041
    4. Main analysis, unchanged control group

    		Internet QOL gain0.0050.006

    		Control QOL gain0.0000.000

    		Between-conditions QOL gain0.0050.006
    5. Main analysis, gain in control group

    		Internet QOL gain
    		0.024

    		Control QOL gain
    		0.012

    		Between-conditions QOL gain
    		0.012

    a ∆H=QOL gain.

    b ∆Hext = QOL gain extrapolated.


    Conservative estimates indicate that for every 1000 persons treated, 16 QALYs are gained. The investment is returned 9 times and the CER is 3432. The costs of the translation project totalled to approximately 27% of the estimated original English-language version development costs.

    Principal Results

    The scenarios in the sensitivity analyses differ in that they decrease in distribution among the 29 potential health states. Scenario 1 uses up to 15 different states, whereas scenario 2 uses up to 5 different states, and scenario 3 only 1 health state. As the distribution becomes more even, the QOL gain (∆H) between groups decreases, and for scenario 3 there is no gain at all. This is partly an effect of the negative results for the control condition. If the control group is kept stable (no change between pretest and posttest score), the QOL change between groups is equal to the change within the intervention condition. Then there is an opposite effect, in which the QOL gain increases with the decrease in distribution. The result is less sensitivity and gives an inflated result. Between scenarios 1 and 2, this is probably a result of less variance and not necessarily that we had to use judgment, as in scenario 3. The results for the extrapolated QOL gain (∆Hext) offer no additional insight because it is just a fixed factor that scales the scores into a 12-month QOL gain.

    Scenarios 4 and 5 explore the effect of the deterioration in the control group, as one could expect some improvement in that group over a longer time period. The Australian trial is not comparable to the Norwegian because the experimental and control groups are given different focus (minimal contact vs unattended). As scenario 4 shows, we need 3 times as many persons treated to gain 1 QALY. In scenario 5, we need 1.5 as many persons treated to gain 1 QALY.

    As the economic analysis shows, the cost of the translation project is less than 1 gained QALY. In the worst case, it is necessary to treat 46 subjects with symptoms of depression to reach the cost break-even point. The treatment received should, on average, be the same as the average for the completers in the trial. As a translation project, this investment is highly cost-effective as it returns the investment many times. With the Norwegian estimate for gained QALY (savings per QALY), every 1000 persons treated yields 17 QALYs and returns the investment 21 times with a CER at 3228. Based on the UK estimate for gained QALY, every 1000 persons treated gives 16 QALYs, returns the investment 9 times, and the CER is 3432. The cost for the translation projects is less than 1 QALY, so the annual number needed to treat to make the cost break-even is 46 individuals, and the development project needs 171 individuals (based on Norwegian estimate for savings per QALY). This is promising, as we know that people, in general, are positive to Internet-based self-help and that this intervention can serve as a prevention intervention. Results from a meta-analysis show that preventive interventions can reduce the incidence of depressive disorders by 22% [70]. The authors concluded, for people with subthreshold symptoms or with high-risk situations, preventive interventions might be exactly what they need.

    The trial costs are included in development costs, and include testing the Web-based interventions’ effect and effectiveness. The WHO suggests that Internet-based prevention interventions should be disseminated after establishing their efficacy [33]. We consider the process of “establishing efficacy” as a part of the project and recommend others to include trial costs as well.

    We regard the assumptions underlying the cost-effectiveness scenarios as conservative, but at the same time it depends on the kind of marketing, that the potential users find the interventions acceptable, and can trust the service provider. Internet-based interventions are costly to develop and require long-term research to test and evaluate their efficacy and effectiveness before they can be disseminated en masse. Locating funding sources for dissemination is often difficult [33]. This paper shows the potential economic benefits of translating Internet-based interventions into other languages, and that the cultural and linguistic transformation of tools developed in other countries are feasible. Hopefully, this paper also contributes to more research reporting cost-effectiveness data in the future. Our findings should encourage others to undertake similar translation activities. These could comprise other mental disorders for which evidence-based interventions exist, including social anxiety disorder, panic disorder, posttraumatic stress disorder, substance misuse, and eating disorders [71-75]. For small countries, like Norway, with a limited number of researchers in the e-mental health field, it is considerably more cost-effective to translate and implement existing interventions than to develop new ones from scratch. This should enable the relatively rapid introduction and dissemination of new interventions to a broad spectrum of the target population. In addition, such an approach could facilitate global dissemination to reduce health disparities across countries and cultures [76,33]. Although we find it considerably more cost-effective to translate existing interventions than to develop new ones from scratch, there could be situations in which the latter option becomes necessary or appears as more beneficial, ie, if for our area of interest no proven effective Internet-based intervention is available, the content of the program does not use the treatment we are interested in, we want to have control over the content and development of the intervention, and if there is no willingness from intervention developers to give access to their programs.

    Internet-based treatments are more accessible than therapist-delivered treatments and, therefore, have the potential to reach more people with an unmet need [77]. An important further step is to assess the cost-effectiveness of the various Internet-based treatments to allow policy makers and health care providers to compare the benefits of online treatment with other interventions. This paper represents one of the few demonstrating the cost-efficacy of an Internet-based therapeutic intervention to treat or prevent a mental disorder and the only study of the cost-effectiveness of translating an existing program for use in another language, as far as we know.

    Although the current study demonstrates that the translation of interventions can be cost-effective, there are other issues that need to be addressed in establishing a translation agreement. Firstly, the licensing deed or contract will need to specify the degree to which the translator can adapt and customize the translated version to local requirements. Secondly, this agreement will need to specify the extent to which the translating site will have the opportunity to implement new functionalities, such as customized printouts. Thirdly, the agreement will need to specify the extent to which the translating site will have an opportunity to influence and contribute to new content for the program. If the content of the application is altered, the application is no longer the same. However, if the application is run by the original center, as was the case for MoodGYM and BluePages, the proliferation of different versions represents a logistical challenge, particularly if programs, such as the BluePages depression information website, requires updating as new information develops, or changes are introduced on the basis of user feedback. Fourthly, the contract should be explicit about the ownership of data collected from the translator’s national users and trial. Finally, special attention should be paid to the timely updating of the translated version after changes in the original one.

    Limitations

    There are several limitations of this study. The validation part of this paper is based on mixed model repeated measures (MMRM) to handle missing data, which might have introduced some bias. However, the results used in this report were from the complete case analyses as we needed pretest and posttest for all cases to carry out the estimates for QALYs. The effect of the intention-to-treat sample was somewhat smaller than for the complete cases, and our choice of sample could have strengthened the effect and QOL calculations. The deterioration in the control group is not unusual over a short time [78], but over a longer time period, one would expect improvements. The effectiveness study discusses some plausible reasons for this [37], such as that the posttest was conducted just weeks before university examinations started. The usage of completers analysis could have introduced some bias, as well. The effects were considered in a relatively short time span of 8 weeks and should be regarded as potential effect if one completes the trial. There was no long-term follow-up of this trial. How the cost-effectiveness of Internet-based treatment is affected when a longer period of time is used is unknown to us. If an effect is maintained over a longer period, this will increase the QALY gain and thus increase the QALY and reduce the number needed to treat. This assumption was taken into the calculations based on the 12-month outcome data from other studies with MoodGYM and BluePages [63]. All outcome measures are based on self-reports and include only symptoms of depression because no clinical assessment was done. Self-report measures of depression may yield a high rate of false positive and false negative results. This could have compromised the results, identifying persons as depressed who do not meet the criteria for a clinical diagnosis and failing to identify persons with a depressive disorder. No data on sick leave, disability, or other social benefits were collected. The use of such data would have yielded a more complete and realistic cost-effectiveness estimate. However, these benefits would only have increased the estimated savings and present Internet-based interventions in an even more attractive way. Finally, the Rosser Index is just one way to estimate health gains, and other methods could have produced different results [79]. However, this method has been employed previously, and the Rosser Index has its strengths at a policy level often used on existing datasets to foretell health care demands [57]. A similar estimation for QALY gain was conducted to evaluate cost-effectiveness for a face-to-face group intervention designed to decrease depression [80], but they reported only QOL gain on a group level.

    Conclusions

    The economic analysis shows that the cost-effectiveness of the translation project was substantial. This is a natural consequence if the project is proven effective in reducing depressive symptoms and based on unguided self-help. The development cost is fixed and the maintenance cost is minimal. Further, as Internet-based treatments are more accessible than therapist-delivered treatments, when these Web-based interventions are disseminated at a national level, they should reach a significant number of individuals with an unmet need for help. The current findings should encourage others to undertake similar translation activities and facilitate global dissemination of effective programs to reduce health disparities across countries and cultures. When data on societal benefits are taken into account, Internet-based prevention interventions should become even more appealing.

    Conflicts of Interest

    Dr Griffiths is one of the authors and developers of the MoodGYM and BluePages websites, but derives neither personal nor financial benefit from the programs.

    1. Marks IM, Cavanagh K, Gega L. Computer-aided psychotherapy: revolution or bubble? Br J Psychiatry 2007 Dec;191:471-473 [FREE Full text] [CrossRef] [Medline]
    2. Freeman M, Pathare S. WHO resource book on mental health, human rights and legislation. Geneva, Switzerland: World Health Organization; 2005.   URL: http://libdoc.who.int/publications/2005/924156282X.pdf [accessed 2012-10-15] [WebCite Cache]
    3. World Health Organization. The global burden of disease: 2004 update. Geneva, Switzerland: World Health Organization; 2008.   URL: http://www.who.int/healthinfo/global_burden_disease/GBD_report_2004update_full.pdf [accessed 2012-10-15] [WebCite Cache]
    4. Luppa M, Heinrich S, Angermeyer MC, König HH, Riedel-Heller SG. Cost-of-illness studies of depression: a systematic review. J Affect Disord 2007 Feb;98(1-2):29-43. [CrossRef] [Medline]
    5. Katon W, Fan MY, Unützer J, Taylor J, Pincus H, Schoenbaum M. Depression and diabetes: a potentially lethal combination. J Gen Intern Med 2008 Oct;23(10):1571-1575 [FREE Full text] [CrossRef] [Medline]
    6. Smit F, Ederveen A, Cuijpers P, Deeg D, Beekman A. Opportunities for cost-effective prevention of late-life depression: an epidemiological approach. Arch Gen Psychiatry 2006 Mar;63(3):290-296. [CrossRef] [Medline]
    7. Marks IM, Cavanagh K, Gega L. Hands-on Help: Computer-Aided Psychotherapy. Hove, UK: Psychology Press; 2007.
    8. Clarke G, Eubanks D, Reid E, Kelleher C, O'Connor E, DeBar LL, et al. Overcoming Depression on the Internet (ODIN) (2): a randomized trial of a self-help depression skills program with reminders. J Med Internet Res 2005;7(2):e16 [FREE Full text] [CrossRef] [Medline]
    9. Christensen H, Griffiths KM, Jorm AF. Delivering interventions for depression by using the internet: randomised controlled trial. BMJ 2004 Jan 31;328(7434):265 [FREE Full text] [CrossRef] [Medline]
    10. Andersson G, Bergström J, Holländare F, Ekselius L, Carlbring P. Delivering cognitive behavioural therapy for mild to moderate depression via the Internet: predicting outcome at 6-month follow-up. Verhaltenstherapie 2004;14:185-189. [CrossRef]
    11. Griffiths KM, Christensen H. Review of randomised controlled trials of Internet interventions for mental disorders and related conditions. Clinical Psychologist 2006;10:16-29. [CrossRef]
    12. Griffiths KM, Farrer L, Christensen H. InPsych. 2006. Effective treatments a click away   URL: http://www.psychology.org.au/publications/inpsych/click/ [accessed 2013-01-20] [WebCite Cache]
    13. Griffiths K, Farrer L, Christensen H. Clickety-click: e-mental health train on track. Australas Psychiatry 2007 Apr;15(2):100-108. [CrossRef] [Medline]
    14. Patten SB. Prevention of depressive symptoms through the use of distance technologies. Psychiatr Serv 2003 Mar;54(3):396-398. [Medline]
    15. van Straten A, Cuijpers P, Smits N. Effectiveness of a web-based self-help intervention for symptoms of depression, anxiety, and stress: randomized controlled trial. J Med Internet Res 2008;10(1):e7 [FREE Full text] [CrossRef] [Medline]
    16. Spek V, Nyklícek I, Smits N, Cuijpers P, Riper H, Keyzer J, et al. Internet-based cognitive behavioural therapy for subthreshold depression in people over 50 years old: a randomized controlled clinical trial. Psychol Med 2007 Dec;37(12):1797-1806. [CrossRef] [Medline]
    17. Meyer B, Berger T, Caspar F, Beevers CG, Andersson G, Weiss M. Effectiveness of a novel integrative online treatment for depression (Deprexis): randomized controlled trial. J Med Internet Res 2009;11(2):e15 [FREE Full text] [CrossRef] [Medline]
    18. Griffiths KM, Crisp D, Christensen H, Mackinnon AJ, Bennett K. The ANU WellBeing study: a protocol for a quasi-factorial randomised controlled trial of the effectiveness of an Internet support group and an automated Internet intervention for depression. BMC Psychiatry 2010;10:20 [FREE Full text] [CrossRef] [Medline]
    19. Robertson L, Smith M, Castle D, Tannenbaum D. Using the Internet to enhance the treatment of depression. Australas Psychiatry 2006 Dec;14(4):413-417. [CrossRef] [Medline]
    20. Ruwaard J, Schrieken B, Schrijver M, Broeksteeg J, Dekker J, Vermeulen H, et al. Standardized web-based cognitive behavioural therapy of mild to moderate depression: a randomized controlled trial with a long-term follow-up. Cogn Behav Ther 2009 Dec;38(4):206-221. [CrossRef] [Medline]
    21. Living Life to the Full. 2010.   URL: http://www.llttf.com/index.php?section=page&page_seq=13 [accessed 2013-01-18] [WebCite Cache]
    22. CCBT Limited Healthcare Online. 2010. MoodCalmer, online self help cCBT program for people experiencing low mood and depression   URL: http://www.ccbt.co.uk/ [accessed 2012-10-15] [WebCite Cache]
    23. Clark DM, Layard R, Smithies R, Richards DA, Suckling R, Wright B. Improving access to psychological therapy: initial evaluation of two UK demonstration sites. Behav Res Ther 2009 Nov;47(11):910-920 [FREE Full text] [CrossRef] [Medline]
    24. Morgan AJ, Jorm AF, Mackinnon AJ. Email-based promotion of self-help for subthreshold depression: Mood Memos randomised controlled trial. Br J Psychiatry 2012 May;200(5):412-418. [CrossRef] [Medline]
    25. Harrison V, Proudfoot J, Wee PP, Parker G, Pavlovic DH, Manicavasagar V. Mobile mental health: review of the emerging field and proof of concept study. J Ment Health 2011 Dec;20(6):509-524. [CrossRef] [Medline]
    26. Van Voorhees BW, Fogel J, Reinecke MA, Gladstone T, Stuart S, Gollan J, et al. Randomized clinical trial of an Internet-based depression prevention program for adolescents (Project CATCH-IT) in primary care: 12-week outcomes. J Dev Behav Pediatr 2009 Feb;30(1):23-37. [CrossRef] [Medline]
    27. Titov N, Andrews G, Kemp A, Robinson E. Characteristics of adults with anxiety or depression treated at an internet clinic: comparison with a national survey and an outpatient clinic. PLoS One 2010;5(5):e10885 [FREE Full text] [CrossRef] [Medline]
    28. Titov N, Dear BF, Schwencke G, Andrews G, Johnston L, Craske MG, et al. Transdiagnostic internet treatment for anxiety and depression: a randomised controlled trial. Behav Res Ther 2011 Aug;49(8):441-452. [CrossRef] [Medline]
    29. McKeown A, Potts HWW. Computerised cognitive-behavioural therapy for prevention early intervention in anxiety and depression: a case study of Xanthis. 2009.   URL: http://discovery.ucl.ac.uk/18338/1/18338.pdf [accessed 2013-01-20] [WebCite Cache]
    30. Deitz DK, Cook RF, Billings DW, Hendrickson A. A web-based mental health program: reaching parents at work. J Pediatr Psychol 2009 Jun;34(5):488-494 [FREE Full text] [CrossRef] [Medline]
    31. Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn Behav Ther 2009 Dec;38(4):196-205. [CrossRef] [Medline]
    32. Andrews G, Cuijpers P, Craske MG, McEvoy P, Titov N. Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: a meta-analysis. PLoS One 2010;5(10):e13196 [FREE Full text] [CrossRef] [Medline]
    33. World Health Organization, Department of Mental Health and Substance Abuse, Prevention Research Centre of the Universities of Nijmegen and Maastricht. Prevention of mental disorders: effective interventions and policy options: summary report. Geneva, Switzerland: World Health Organization; 2004.   URL: http://www.who.int/mental_health/evidence/en/prevention_of_mental_disorders_sr.pdf [accessed 2012-10-15] [WebCite Cache]
    34. Eysenbach G. The law of attrition. J Med Internet Res 2005;7(1):e11 [FREE Full text] [CrossRef] [Medline]
    35. Marks I, Cavanagh K. Computer-aided psychological treatments: evolving issues. Annu Rev Clin Psychol 2009;5:121-141. [CrossRef] [Medline]
    36. Titov N. Internet-delivered psychotherapy for depression in adults. Curr Opin Psychiatry 2011 Jan;24(1):18-23. [CrossRef] [Medline]
    37. Lintvedt OK, Griffiths KM, Sørensen K, Ostvik AR, Wang CE, Eisemann M, et al. Evaluating the effectiveness and efficacy of unguided internet-based self-help intervention for the prevention of depression: a randomized controlled trial. Clin Psychol Psychother 2013 Jan;20(1):10-27. [CrossRef] [Medline]
    38. Carlton PA, Deane FP. Impact of attitudes and suicidal ideation on adolescents' intentions to seek professional psychological help. J Adolesc 2000 Feb;23(1):35-45. [CrossRef] [Medline]
    39. Calear AL, Christensen H, Mackinnon A, Griffiths KM, O'Kearney R. The YouthMood Project: a cluster randomized controlled trial of an online cognitive behavioral program with adolescents. J Consult Clin Psychol 2009 Dec;77(6):1021-1032. [CrossRef] [Medline]
    40. Ritterband LM, Thorndike FP, Gonder-Frederick LA, Magee JC, Bailey ET, Saylor DK, et al. Efficacy of an Internet-based behavioral intervention for adults with insomnia. Arch Gen Psychiatry 2009 Jul;66(7):692-698. [CrossRef] [Medline]
    41. Spek V, Cuijpers P, Nyklícek I, Riper H, Keyzer J, Pop V. Internet-based cognitive behaviour therapy for symptoms of depression and anxiety: a meta-analysis. Psychol Med 2007 Mar;37(3):319-328. [CrossRef] [Medline]
    42. Ritterband LM, Andersson G, Christensen HM, Carlbring P, Cuijpers P. Directions for the International Society for Research on Internet Interventions (ISRII). J Med Internet Res 2006;8(3):e23 [FREE Full text] [CrossRef] [Medline]
    43. Australian National University. The MoodGYM training program. 2012.   URL: https://www.moodgym.anu.edu.au/welcome/faq [accessed 2012-10-15] [WebCite Cache]
    44. Australian National University. BluePages Depression Information. 2012.   URL: http://www.bluepages.anu.edu.au/about_bluepages/ [accessed 2012-10-15] [WebCite Cache]
    45. Bennett K, Reynolds J, Christensen H, Griffiths KM. e-hub: an online self-help mental health service in the community. Med J Aust 2010 Jun 7;192(11 Suppl):S48-S52. [Medline]
    46. Tate DF, Finkelstein EA, Khavjou O, Gustafson A. Cost effectiveness of internet interventions: review and recommendations. Ann Behav Med 2009 Aug;38(1):40-45 [FREE Full text] [CrossRef] [Medline]
    47. Lintvedt OK, Sørensen K, Østvik AR, Verplanken B, Wang CE. The need for web-based cognitive behavior therapy among university students. Journal of Technology in Human Services 2008;26(2-4):239-258. [CrossRef]
    48. Statistics Norway. 2012. Consumer Price Index   URL: http://www.ssb.no/vis/kpi/kpiregn.html [accessed 2012-10-15] [WebCite Cache]
    49. Reserve Bank of Australia. 2012. Inflation calculator   URL: http://www.rba.gov.au/calculator/annualDecimal.html [accessed 2012-10-15] [WebCite Cache]
    50. Radloff LS. The CES-D scale: A self-report depression scale for research in the general population. Applied Psychological Measurement 1977 Jun;1(3):385-401. [CrossRef]
    51. Santor D, Zuroff DC, Ramsay JO, Cervantes P, Palacios J. Examining scale discriminability in the BDI and CES-D as a function of depressive severity. Psychological Assessment 1995 Jun;7(2):131-139. [CrossRef]
    52. Rushton JL, Forcier M, Schectman RM. Epidemiology of depressive symptoms in the National Longitudinal Study of Adolescent Health. J Am Acad Child Adolesc Psychiatry 2002 Feb;41(2):199-205. [CrossRef] [Medline]
    53. Kessler RC, Andrews G, Colpe LJ, Hiripi E, Mroczek DK, Normand SL, et al. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychol Med 2002 Aug;32(6):959-976. [Medline]
    54. Gater RA, Kind P, Gudex C. Quality of life in liaison psychiatry. A comparison of patient and clinician assessment. Br J Psychiatry 1995 Apr;166(4):515-520. [Medline]
    55. McCabe C. What is cost-utility analysis?. London: Hayward Medical Communications; 2009.   URL: http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/What_is_cost-util.pdf [accessed 2012-10-15] [WebCite Cache]
    56. Rosser R, Kind P. A scale of valuations of states of illness: is there a social consensus? Int J Epidemiol 1978 Dec;7(4):347-358. [Medline]
    57. Hawthorne G, Richardson J, Olsen JA, Mortimer D, Smith R. Critical review of the multi-attribute utility literature. West Heidelberg, Australia: Centre for Health Program Evaluation, Monash University; 2001.   URL: http://www.buseco.monash.edu.au/centres/che/pubs/wp103.pdf [accessed 2012-10-15] [WebCite Cache]
    58. Gudex C, Kind P, van Dalen H, Durand MA, Morris J, Williams A. Comparing scaling methods for health state valuations: Rosser revisited (Discussion paper 107). York, UK: Centre for Health Economics, University of York; 1993.   URL: http://www.york.ac.uk/media/che/documents/papers/discussionpapers/CHE%20Discussion%20Paper%20107.pdf [accessed 2012-10-15] [WebCite Cache]
    59. Kind P, Rosser R, Williams A. Valuation of quality of life: some psychometric evidence. In: Jones-Lee MW, editor. The Value of Life and Safety. Amsterdam: North-Holland Pub Co; 1982:159-170.
    60. Nord E. [A table of reference values for cost-benefit-analyses in health care]. Tidsskr Nor Laegeforen 1996 Nov 10;116(27):3246-3249. [Medline]
    61. Gudex C, Kind P. The QALY toolkit (Discussion paper 38). York, UK: University of York Centre for Health Economics; 1988.   URL: http://www.york.ac.uk/media/che/documents/papers/discussionpapers/CHE%20Discussion%20Paper%2038.pdf [accessed 2012-10-15] [WebCite Cache]
    62. Torrance GW, Feeny D. Utilities and quality-adjusted life years. Int J Technol Assess Health Care 1989;5(4):559-575. [Medline]
    63. Mackinnon A, Griffiths KM, Christensen H. Comparative randomised trial of online cognitive-behavioural therapy and an information website for depression: 12-month outcomes. Br J Psychiatry 2008 Feb;192(2):130-134 [FREE Full text] [CrossRef] [Medline]
    64. Olsen JA. In: Olsen J, editor. Principles in Health Economics and Policy: Distributing Health Care. Oxford, UK: Oxford University Press; 2009.
    65. National Institute for Health and Clinical Excellence. NICE Guideline Manual. 2007. Incorporating health economics in guidelines and assessing resource impact   URL: http://www.nice.org.uk/niceMedia/pdf/GuidelinesManualChapter8.pdf [accessed 2012-10-15] [WebCite Cache]
    66. National Institute for Health and Clinical Excellence. 2010. Measuring effectiveness and cost effectiveness: the QALY   URL: http://www.nice.org.uk/newsroom/features/measuringeffectivenessandcosteffectivenesstheqaly.jsp [accessed 2012-10-15] [WebCite Cache]
    67. Towse A, Pritchard C. National Institute for Clinical Excellence (NICE): Is economic appraisal working? Pharmacoeconomics 2002;20 Suppl 3:95-105. [Medline]
    68. Norwegian Directorate for Health and Social Affairs. Helseeffekter i samfunnsøkonomiske analyser [Health effects of economic analyses]. 2007.   URL: http://www.shdir.no/vp/multimedia/archive/00020/IS-1435_20969a.pdf [accessed 2012-10-15] [WebCite Cache]
    69. Butler J, Christensen H, Bartlett LE, Groves C. Cost-effectiveness analysis of MoodGYM in the management of clinical depression. 2005.   URL: http://www.acerh.edu.au/Materials/WPA_MoodGYM_Abstract.pdf [accessed 2012-10-15] [WebCite Cache]
    70. Cuijpers P, van Straten A, Smit F, Mihalopoulos C, Beekman A. Preventing the onset of depressive disorders: a meta-analytic review of psychological interventions. Am J Psychiatry 2008 Oct;165(10):1272-1280. [CrossRef] [Medline]
    71. Botella C, Hofmann SG, Moscovitch DA. A self-applied, Internet-based intervention for fear of public speaking. J Clin Psychol 2004 Aug;60(8):821-830. [CrossRef] [Medline]
    72. Kypri K, Hallett J, Howat P, McManus A, Maycock B, Bowe S, et al. Randomized controlled trial of proactive web-based alcohol screening and brief intervention for university students. Arch Intern Med 2009 Sep 14;169(16):1508-1514. [CrossRef] [Medline]
    73. Lange A, van de Ven JP, Schrieken B. Interapy: treatment of post-traumatic stress via the internet. Cogn Behav Ther 2003;32(3):110-124. [CrossRef] [Medline]
    74. Richards J, Klein B, Carlbring P. Internet-based treatment for panic disorder. Cogn Behav Ther 2003;32(3):125-135. [CrossRef] [Medline]
    75. Winzelberg AJ, Eppstein D, Eldredge KL, Wilfley D, Dasmahapatra R, Dev P, et al. Effectiveness of an Internet-based program for reducing risk factors for eating disorders. J Consult Clin Psychol 2000 Apr;68(2):346-350. [Medline]
    76. Reynolds CF. Prevention of depressive disorders: a brave new world. Depress Anxiety 2009;26(12):1062-1065 [FREE Full text] [CrossRef] [Medline]
    77. Warmerdam L, Smit F, van Straten A, Riper H, Cuijpers P. Cost-utility and cost-effectiveness of internet-based treatment for adults with depressive symptoms: randomized trial. J Med Internet Res 2010;12(5):e53 [FREE Full text] [CrossRef] [Medline]
    78. Titov N, Andrews G, Choi I, Schwencke G, Mahoney A. Shyness 3: randomized controlled trial of guided versus unguided Internet-based CBT for social phobia. Aust N Z J Psychiatry 2008 Dec;42(12):1030-1040. [CrossRef] [Medline]
    79. Saxena S, Carlson D, Billington R, WHOQOL Group. World Health Organisation Quality Of Life. The WHO quality of life assessment instrument (WHOQOL-Bref): the importance of its items for cross-cultural research. Qual Life Res 2001;10(8):711-721. [Medline]
    80. Nord E, Dalgard OS. [Economic evaluation of a course in coping with depression]. Tidsskr Nor Laegeforen 2006 Feb 23;126(5):586-588 [FREE Full text] [Medline]

    ‎
    ANU: Australian National University
    CBT: cognitive behavior therapy
    CER: cost-effectiveness ratio
    CES-D: Center for Epidemiologic Studies Depression Scale
    d: Cohen’s d
    g: Hedges’ g
    ∆H: degree of improvement in health
    ∆Hext: degree of improvement in health, extrapolated
    ISRII: International Society for Research on Internet Interventions
    K10: Kessler Psychological Distress Scale
    MMRM: mixed model repeated measures
    NICE: National Institute for Clinical Excellence
    NOK: Norwegian kroner
    QALY: quality-adjusted life year
    QALY-gain: QALYs gained over a timeframe of 1 year
    QOL: quality of life
    RCT: randomized controlled trial
    WHO: World Health Organization

    Edited by G Eysenbach; submitted 05.11.12; peer-reviewed by S Maloney, D Kavanagh; comments to author 27.11.12; revised version received 03.12.12; accepted 20.12.12; published 23.01.13

    Copyright

    ©Ove K Lintvedt, Kathleen M Griffiths, Martin Eisemann, Knut Waterloo. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 23.01.2013.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.