This paper is in the following e-collection/theme issue:

Electronic/Mobile Data Capture, Internet-based Survey & Research Methodology (832) Digital Mental Health Interventions, e-Mental Health and Cyberpsychology (1514) Research Instruments, Questionnaires, and Tools (576) New Methods (944) Methods and New Tools in Mental Health Research (454)

Published on in Vol 18, No 6 (2016): Jun

Web-Based Assessment of Mental Well-Being in Early Adolescence: A Reliability Study

Web-Based Assessment of Mental Well-Being in Early Adolescence: A Reliability Study

Web-Based Assessment of Mental Well-Being in Early Adolescence: A Reliability Study

Authors of this article:

Christoph Hamann1 Author Orcid Image ;   Frauke Schultze-Lutter1 Author Orcid Image ;   Leila Tarokh1, 2, 3 Author Orcid Image
Article Authors Cited by (9) Tweetations (7) Metrics

    Short Paper

    1University Hospital of Child and Adolescent Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland

    2Institute of Pharmacology and Toxicology, University of Zurich, Zurich, Switzerland

    3Psychiatry and Human Behavior, The Alpert Medical School of Brown University, Providence, RI, United States

    Corresponding Author:

    Leila Tarokh, PhD

    University Hospital of Child and Adolescent Psychiatry and Psychotherapy

    University of Bern

    Bolligenstr. 111, Haus A

    Bern, 3000

    Switzerland

    Phone: 41 31 932 8554

    Fax:41 31 932 8569

    Email: leila_tarokh@brown.edu


    Background: The ever-increasing use of the Internet among adolescents represents an emerging opportunity for researchers to gain access to larger samples, which can be queried over several years longitudinally. Among adolescents, young adolescents (ages 11 to 13 years) are of particular interest to clinicians as this is a transitional stage, during which depressive and anxiety symptoms often emerge. However, it remains unclear whether these youngest adolescents can accurately answer questions about their mental well-being using a Web-based platform.

    Objective: The aim of the study was to examine the accuracy of responses obtained from Web-based questionnaires by comparing Web-based with paper-and-pencil versions of depression and anxiety questionnaires.

    Methods: The primary outcome was the score on the depression and anxiety questionnaires under two conditions: (1) paper-and-pencil and (2) Web-based versions. Twenty-eight adolescents (aged 11-13 years, mean age 12.78 years and SD 0.78; 18 females, 64%) were randomly assigned to complete either the paper-and-pencil or the Web-based questionnaire first. Intraclass correlation coefficients (ICCs) were calculated to measure intrarater reliability. Intraclass correlation coefficients were calculated separately for depression (Children’s Depression Inventory, CDI) and anxiety (Spence Children’s Anxiety Scale, SCAS) questionnaires.

    Results: On average, it took participants 17 minutes (SD 6) to answer 116 questions online. Intraclass correlation coefficient analysis revealed high intrarater reliability when comparing Web-based with paper-and-pencil responses for both CDI (ICC=.88; P<.001) and the SCAS (ICC=.95; P<.001). According to published criteria, both of these values are in the “almost perfect” category indicating the highest degree of reliability.

    Conclusions: The results of the study show an excellent reliability of Web-based assessment in 11- to 13-year-old children as compared with the standard paper-pencil assessment. Furthermore, we found that Web-based assessments with young adolescents are highly feasible, with all enrolled participants completing the Web-based form. As early adolescence is a time of remarkable social and behavioral changes, these findings open up new avenues for researchers from diverse fields who are interested in studying large samples of young adolescents over time.

    J Med Internet Res 2016;18(6):e138

    doi:10.2196/jmir.5482

    Keywords

    early adolescence; online assessment; reliability 


    Adolescent development is long, complex, and highly individual (eg, [1-4]). In order to capture the dynamic process that is adolescent development, large sample sizes or longitudinal assessment is often required [5]. Access to large samples of adolescents over extended periods can be challenging because of factors such as changing of schools, moving, or unstable family environments [6]. Thus, data collection is often cumbersome, expensive, and can result in dropouts that may influence results [7,8].

    One way to circumvent some of these difficulties is through the administration of Web-based questionnaires. Adolescents today have regular access to the Internet either at school or at home. For example, according to the Pew Research Center, 95% of teenagers (12-17 years) have Internet access in the United States [9], and on average a European teenager uses online media for 66 minutes per day [10]. The ever-increasing use of the Internet among adolescents represents an emerging opportunity for researchers to gain access to large samples, which can be queried over several years longitudinally. Subjects can complete such questionnaires wherever and whenever it suits them using different media, for example, tablets, laptops, or mobile phones, and researchers can monitor data quality instantaneously. Furthermore, the Internet offers diverse options for communication between participants and researchers, which can be used to bolster participation and to minimize dropouts [8].

    Previous studies have shown that adolescents (13-20 years) [11,12] can accurately and reliably fill out Web-based questionnaires about their mental and physical well-being. However, few studies have addressed the reliability of Web-based questionnaires in young adolescents between the ages of 11 and 13 years in a naturalistic setting (ie, at home, unmonitored). This age range is of particular interest to clinicians and researchers, because it is a transitional stage [13]. This transition from childhood to adolescence—accompanied by increased independence, a new school environment, the onset of puberty, and shifting peer relationships—can be highly stressful and may lead to the emergence of depressive and anxiety symptoms. The influence of stressors on psychopathology in youth is under intensive investigation and conclusions are still difficult to draw [14]. Understanding the etiology of psychiatric disorders is critical to early intervention and most disorders have their onset during adolescence [15-18]. Therefore, surveying adolescents early on in development and following them longitudinally will further our understanding of adolescent development.

    Thus, the aim of this study was to examine whether young adolescents can accurately fill out Web-based questionnaires about depression and anxiety, the two most common psychiatric disorders among adolescents (lifetime prevalence of 25% of anxiety disorders and 13% of mood disorders in 13- to 18-year-olds [18]). We accomplish this by comparing Web-based with standard paper-and-pencil versions of depression and anxiety questionnaires.


    Sample and Design

    Twenty-eight children between the ages of 11 and 13 years (mean 12.78 years and SD 0.78; 18 females, 64%) participated in this study. The study was briefly introduced to 2 classes at a Swiss secondary school. Students who wished to participate sent back their contact information in addition to a consent form signed by their parents or legal guardian and themselves. Once recruited into the study, participants were randomly assigned to complete either the paper-and-pencil version or the Web-based version first in a randomized crossover design. In the paper-and-pencil condition, participants received the questionnaires by mail and a self-addressed stamped envelope was provided for returning the questionnaires. In the Web-based condition, participants received an identification number, a password, and a link to the survey website. Parents were instructed to leave the child alone to fill out the Web-based and paper-and-pencil forms. After completing the first condition, participants received the questionnaires of the alternate condition so that the second assessment was not more than 2 weeks from the first. This rather short lag time was chosen to limit the influence of changes in mental state on the completion of the state-sensitive questionnaires. Participants received compensation in the form of gift vouchers for taking part in the study, which was approved by the Ethics Committee in Bern, Switzerland.

    Questionnaires

    The German version (Depressions-Inventar für Kinder und Jugendliche, DIKJ) of the Children’s Depression Inventory (CDI; [19]) was used to measure depressive symptoms. This scale is a well-established self-report measure of depressive symptoms appropriate for children between the ages of 7 and 17 years. This scale consists of 26 items, which are each scored from 0 to 2, and thus the total score of this scale yields values ranging between 0 and 52.

    The German version of the Spence Children’s Anxiety Scale (SCAS) was used to measure anxiety symptoms. This scale was designed to assess anxiety symptoms in individuals between the ages of 8 and 15 years and consists of 38 items, which are scored on a 4-point scale—thus scores range between 0 and 114. This scale also permits calculation of subscores, allowing for the evaluation of anxiety across specific domains [20-22].

    Statistical Analysis

    Statistical analysis was conducted with SPSS version 23.0.0.0. Intraclass correlation coefficients (ICCs) were used to examine the degree of correspondence between paper and Web-based versions. Intraclass correlation coefficient values range between 0 and 1 and are conventionally categorized as follows: 0-.2 poor, .2-.4 fair, .4-.6 moderate, .6-.8 substantial, and .8-1.0 almost perfect [23]. The Web-based version was considered reliable when an ICC value equal to or greater than .8 (minimum of substantial agreement) was obtained. Additionally, Kendall tau correlations were used to measure the equivalence between the paper-and-pencil and Web-based versions, and Wilcoxon tests were performed to test for statistically significant differences between the two conditions.


    On average, it took 17 minutes (SD 6) to fill out the Web-based questionnaires, which consisted of the SCAS, CDI, and an additional 52 questions about sleep behavior and quality resulting in a total of 116 questions. Because paper-and-pencil questionnaires were filled out at home, no data on time taken to complete the forms were available. The evaluation of the paper-and-pencil and Web-based versions resulted in an overall ICC of .88 (P<.001) for the CDI and an ICC of .95 (P<.001) for the SCAS (Table 1). According to the criteria of Landis and Koch [23], ICC values higher than .8 fall into the category of “almost perfect” indicating the highest degree of reliability. Kendall tau correlation was also significant with correlation coefficients (r) of .58 (P<.001) for the CDI and .72 (P<.001) for the SCAS. Additionally, Wilcoxon tests comparing paper-and-pencil with the Web-based condition showed no significant difference (Table 1), with the exception of the subscore for Physical Injury Fear of the SCAS.

    Table 1. The degree of correspondence between the paper-and-pencil and Web-based versions of depression and anxiety questionnaires filled out by 11- to 13-year-old children.
    QuestionnairePaper-and-pencil,
    mean (SD)    		Web-based, mean (SD)ICCa (P)Zb (P)τc (P)
    Total SCASd19.46 (12.64)21.11 (13.41).95 (<.001)−1.36 (.173).72 (<.001)

    		Separation Anxiety2.43 (2.77)2.75 (2.99).92 (<.001)−0.82 (.410).75 (<.001)

    		Social Phobia4.68 (2.94)4.61 (3.01).91 (<.001)−0.43 (.666).66 (<.001)

    		Obsessive Compulsivness4.07 (3.03)4.07 (2.98).92 (<.001)−1.50 (.881).71 (<.001)

    		Panic Agoraphobia2.26 (2.80)1.78 (2.74).90 (<.001)−1.40 (.163).49 (.002)

    		Physical Injury Fear2.32 (2.18)3.04 (2.78).90 (<.001)−3.06 (.002).86 (<.001)

    		Generalized Anxiety4.36 (2.36)4.82 (2.61).86 (<.001)−1.45 (.148).60 (<.001)
    Total CDIe (DIKJf)6.61 (4.46)7.39 (4.41).88 (<.001)−1.09 (.277).58 (<.001)

    aICC: intraclass correlation coefficient.

    bZ: Wilcoxon-Mann-Whitney test.

    cτ: Kendall tau-b.

    dSCAS: Spence Children’s Anxiety Scale.

    eCDI: Children\'s Depression Inventory.

    fDIKJ: Depressions-Inventar für Kinder und Jugendliche.


    In this randomized study, we show that Internet-based data collection of mental health questionnaires is feasible and reliable in early adolescence outside a highly supervised environment. A number of studies have reported feasibility and acceptance of Web-based questionnaires evaluating health among children [24-27], teenagers [11,12,26-29], and adults [30-40]; however, very few studies have examined the reliability of Web-based questionnaires under unsupervised conditions in a sample of young adolescents. For example, Mangunkusumo et al [25] found comparable responses on an Internet versus paper mode of health questionnaires in an elementary school cohort (ages 10-12 years), using a randomized within-subject design. However, the children in this study were intensively supervised by their teachers and changing modalities happened during the same school lesson with only a 5-minute break in between, thus making a memory bias in favor of high correspondence likely. Our within-subject results show a similar outcome in an unsupervised setting and with a longer lag time between assessments, where young adolescents filled out questionnaires during their free time without close supervision—an important modification for the feasibility of longitudinal study designs.

    Furthermore, our results not only show high ICC values in the overall sum scores of CDI and SCAS, but they also show high ICC and correlation scores in the subscores of the SCAS. Only the Physical Injury Fear subscale differed between the Web-based and the paper-and-pencil groups and may be due to the small number of items that are included in each subscore.

    Additionally, the adolescents in our study were able to answer a relatively large number of items (total of 116 items) in a short time, which shows the feasibility of collecting large datasets. Furthermore, despite our modest sample size, we were able to demonstrate high ICC values, which indicate high intrarater reliability in comparison to interrater variability. Nevertheless, the small sample size is a limitation of our study.

    In summary, our study suggests that Internet-based data collection in young adolescents in the field of mental health and beyond is feasible and reliable. Consequently, Internet-based questionnaires can be implemented in larger longitudinal studies in the future in order to develop a better understanding of adolescent behavior and follow the emergence of psychiatric disorders, which can lead to prevention and better treatment.

    Acknowledgments

    The authors thank Daniela Rupp, Nathaline Margot, and Laura Tüshaus for help with data collection, entry, and management and Dr Chantal Michel for assistance with the ethics application for this study. We are also grateful to Philipp Wettstein and the study participants for their help and cooperation.

    Conflicts of Interest

    None declared.

    1. Andersen SL. Trajectories of brain development: point of vulnerability or window of opportunity? Neurosci Biobehav Rev 2003;27(1-2):3-18. [Medline]
    2. Giedd JN. The teen brain: insights from neuroimaging. J Adolesc Health 2008 Apr;42(4):335-343. [CrossRef] [Medline]
    3. Giedd JN, Rapoport JL. Structural MRI of pediatric brain development: what have we learned and where are we going? Neuron 2010 Sep 9;67(5):728-734 [FREE Full text] [CrossRef] [Medline]
    4. Berk LE. Development Through the Lifespan. 5th edition. Boston, MA: Pearson Education, Inc; 2009.
    5. Farrington DP. Longitudinal research strategies: advantages, problems, and prospects. J Am Acad Child Adolesc Psychiatry 1991 May;30(3):369-374. [CrossRef] [Medline]
    6. Howe LD, Tilling K, Galobardes B, Lawlor DA. Loss to follow-up in cohort studies: bias in estimates of socioeconomic inequalities. Epidemiology 2013 Jan;24(1):1-9. [CrossRef] [Medline]
    7. Gustavson K, Borren I. Bias in the study of prediction of change: a Monte Carlo simulation study of the effects of selective attrition and inappropriate modeling of regression toward the mean. BMC Med Res Methodol 2014;14:133 [FREE Full text] [CrossRef] [Medline]
    8. David MC, Alati R, Ware RS, Kinner SA. Attrition in a longitudinal study with hard-to-reach participants was reduced by ongoing contact. J Clin Epidemiol 2013 May;66(5):575-581. [CrossRef] [Medline]
    9. Pew Research Center. 2013. Internet User Demographics   URL: http://www.pewinternet.org/data-trend/teens/internet-user-demographics/ [accessed 2016-03-04] [WebCite Cache]
    10. Hermida M, Signer S. EU kids online. 2013. Willkommen bei EU Kids Online: Schweiz   URL: http://www.eukidsonline.ch/ [accessed 2016-03-05] [WebCite Cache]
    11. Wang Y, Lee C, Lew-Ting C, Hsiao CK, Chen D, Chen WJ. Survey of substance use among high school students in Taipei: web-based questionnaire versus paper-and-pencil questionnaire. J Adolesc Health 2005 Oct;37(4):289-295. [CrossRef] [Medline]
    12. Ramo DE, Liu H, Prochaska JJ. Reliability and validity of young adults' anonymous online reports of marijuana use and thoughts about use. Psychol Addict Behav 2012 Dec;26(4):801-811 [FREE Full text] [CrossRef] [Medline]
    13. Grills-Taquechel AE, Norton P, Ollendick TH. A longitudinal examination of factors predicting anxiety during the transition to middle school. Anxiety Stress Coping 2010 Oct;23(5):493-513 [FREE Full text] [CrossRef] [Medline]
    14. Kushner SC. A Review of the Direct and Interactive Effects of Life Stressors and Dispositional Traits on Youth Psychopathology. Child Psychiatry Hum Dev 2015 Oct;46(5):810-819. [CrossRef] [Medline]
    15. Thapar A, Collishaw S, Pine DS, Thapar AK. Depression in adolescence. Lancet 2012 Mar 17;379(9820):1056-1067 [FREE Full text] [CrossRef] [Medline]
    16. Cummings CM, Caporino NE, Kendall PC. Comorbidity of anxiety and depression in children and adolescents: 20 years after. Psychol Bull 2014 May;140(3):816-845 [FREE Full text] [CrossRef] [Medline]
    17. Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry 2005 Jun;62(6):593-602. [CrossRef] [Medline]
    18. Merikangas KR, He J, Burstein M, Swanson SA, Avenevoli S, Cui L, et al. Lifetime prevalence of mental disorders in U.S. adolescents: results from the National Comorbidity Survey Replication--Adolescent Supplement (NCS-A). J Am Acad Child Adolesc Psychiatry 2010 Oct;49(10):980-989 [FREE Full text] [CrossRef] [Medline]
    19. Kovacs M. The Children's Depression, Inventory (CDI). Psychopharmacol Bull 1985;21(4):995-998. [Medline]
    20. Spence SH. Structure of anxiety symptoms among children: a confirmatory factor-analytic study. J Abnorm Psychol 1997 May;106(2):280-297. [Medline]
    21. Spence SH. A measure of anxiety symptoms among children. Behav Res Ther 1998 May;36(5):545-566. [Medline]
    22. Spence SH, Barrett PM, Turner CM. Psychometric properties of the Spence Children's Anxiety Scale with young adolescents. J Anxiety Disord 2003;17(6):605-625. [Medline]
    23. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977 Mar;33(1):159-174. [Medline]
    24. McCabe SE, Boyd CJ, Young A, Crawford S, Pope D. Mode effects for collecting alcohol and tobacco data among 3rd and 4th grade students: a randomized pilot study of Web-form versus paper-form surveys. Addict Behav 2005 May;30(4):663-671. [CrossRef] [Medline]
    25. Mangunkusumo RT, Duisterhout JS, de GN, Maarsingh EJ, de Koning Harry J, Raat H. Internet versus paper mode of health and health behavior questionnaires in elementary schools: asthma and fruit as examples. J Sch Health 2006 Feb;76(2):80-86. [CrossRef] [Medline]
    26. Goodman R. Online child and adolescent mental health surveys can be good enough. Soc Psychiatry Psychiatr Epidemiol 2013 Aug;48(8):1317-1325. [CrossRef] [Medline]
    27. Hohwü L, Lyshol H, Gissler M, Jonsson SH, Petzold M, Obel C. Web-based versus traditional paper questionnaires: a mixed-mode survey with a Nordic perspective. J Med Internet Res 2013;15(8):e173 [FREE Full text] [CrossRef] [Medline]
    28. Raat H, Mangunkusumo RT, Landgraf JM, Kloek G, Brug J. Feasibility, reliability, and validity of adolescent health status measurement by the Child Health Questionnaire Child Form (CHQ-CF): internet administration compared with the standard paper version. Qual Life Res 2007 May;16(4):675-685 [FREE Full text] [CrossRef] [Medline]
    29. Raghupathy S, Hahn-Smith S. Reliability of the high school Youth Risk Behavior Survey when administered online. Int Q Community Health Educ 2011 Sep;32(2):135-148. [CrossRef] [Medline]
    30. Coles M, Cook L, Blake T. Assessing obsessive compulsive symptoms and cognitions on the internet: evidence for the comparability of paper and Internet administration. Behav Res Ther 2007 Sep;45(9):2232-2240. [CrossRef] [Medline]
    31. Holländare F, Andersson G, Engström I. A comparison of psychometric properties between internet and paper versions of two depression instruments (BDI-II and MADRS-S) administered to clinic patients. J Med Internet Res 2010;12(5):e49 [FREE Full text] [CrossRef] [Medline]
    32. Khazaal Y, Chatton A, Monney G, Nallet A, Khan R, Zullino D, et al. Internal consistency and measurement equivalence of the cannabis screening questions on the paper-and-pencil face-to-face ASSIST versus the online instrument. Subst Abuse Treat Prev Policy 2015;10:8 [FREE Full text] [CrossRef] [Medline]
    33. Kongsved SM, Basnov M, Holm-Christensen K, Hjollund NH. Response rate and completeness of questionnaires: a randomized study of Internet versus paper-and-pencil versions. J Med Internet Res 2007;9(3):e25 [FREE Full text] [CrossRef] [Medline]
    34. Ritter P, Lorig K, Laurent D, Matthews K. Internet versus mailed questionnaires: a randomized comparison. J Med Internet Res 2004 Sep 15;6(3):e29 [FREE Full text] [CrossRef] [Medline]
    35. Thorén ES, Andersson G, Lunner T. The use of research questionnaires with hearing impaired adults: online vs. paper-and-pencil administration. BMC Ear Nose Throat Disord 2012;12:12 [FREE Full text] [CrossRef] [Medline]
    36. Touvier M, Méjean C, Kesse-Guyot E, Pollet C, Malon A, Castetbon K, et al. Comparison between web-based and paper versions of a self-administered anthropometric questionnaire. Eur J Epidemiol 2010 May;25(5):287-296. [CrossRef] [Medline]
    37. Vallejo MA, Mañanes G, Isabel Comeche M A, Díaz MI. Comparison between administration via Internet and paper-and-pencil administration of two clinical instruments: SCL-90-R and GHQ-28. J Behav Ther Exp Psychiatry 2008 Sep;39(3):201-208. [CrossRef] [Medline]
    38. Verkuil B, Brosschot JF. The online version of the Dutch Penn State Worry Questionnaire: factor structure, predictive validity and reliability. J Anxiety Disord 2012 Dec;26(8):844-848. [CrossRef] [Medline]
    39. Vleeschouwer M, Schubart CD, Henquet C, Myin-Germeys I, van Gastel Willemijn A, Hillegers Manon H J, van Os Jim J, Boks Marco P M, et al. Does assessment type matter? A measurement invariance analysis of online and paper and pencil assessment of the Community Assessment of Psychic Experiences (CAPE). PLoS One 2014;9(1):e84011 [FREE Full text] [CrossRef] [Medline]
    40. Whitehead L. Methodological issues in Internet-mediated research: a randomized comparison of internet versus mailed questionnaires. J Med Internet Res 2011;13(4):e109 [FREE Full text] [CrossRef] [Medline]

    ‎
    CDI: Children's Depression Inventory
    DIKJ: Depressions-Inventar für Kinder und Jugendliche
    ICC: Intraclass Correlation Coefficient
    SCAS: Spence Children’s Anxiety Scale

    Edited by G Eysenbach; submitted 28.12.15; peer-reviewed by L Hohwü, S Kendal; comments to author 11.02.16; revised version received 22.03.16; accepted 17.04.16; published 15.06.16

    Copyright

    ©Christoph Hamann, Frauke Schultze-Lutter, Leila Tarokh. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 15.06.2016.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.