This paper is in the following e-collection/theme issue:

Web-based and Mobile Health Interventions (3297) Registered Report (560) Innovations in Mental Health Systems (390)

Published on in Vol 22, No 10 (2020): October

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/20790, first published .
Effects of a Mobile and Web App (Thought Spot) on Mental Health Help-Seeking Among College and University Students: Randomized Controlled Trial

Effects of a Mobile and Web App (Thought Spot) on Mental Health Help-Seeking Among College and University Students: Randomized Controlled Trial

Effects of a Mobile and Web App (Thought Spot) on Mental Health Help-Seeking Among College and University Students: Randomized Controlled Trial

Authors of this article:

David Wiljer1, 2, 3, 4 Author Orcid Image ;   Jenny Shi2 Author Orcid Image ;   Brian Lo1, 2, 3, 5 Author Orcid Image ;   Marcos Sanches6 Author Orcid Image ;   Elisa Hollenberg2 Author Orcid Image ;   Andrew Johnson2 Author Orcid Image ;   Alexxa Abi-Jaoudé2 Author Orcid Image ;   Gloria Chaim4, 7 Author Orcid Image ;   Kristin Cleverley4, 7, 8 Author Orcid Image ;   Joanna Henderson4, 7 Author Orcid Image ;   Wanrudee Isaranuwatchai3, 9 Author Orcid Image ;   Andrea Levinson4, 10 Author Orcid Image ;   Janine Robb10 Author Orcid Image ;   Howard W Wong2, 3, 11 Author Orcid Image ;   Aristotle Voineskos4, 11 Author Orcid Image
Article Authors Cited by (31) Tweetations (12) Metrics

    Original Paper

    1UHN Digital, University Health Network, Toronto, ON, Canada

    2Office of Education, Centre for Addiction and Mental Health, Toronto, ON, Canada

    3Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada

    4Department of Psychiatry, Faculty of Medicine, University of Toronto, Toronto, ON, Canada

    5Information Management Group, Centre for Addiction and Mental Health, Toronto, ON, Canada

    6Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health, Toronto, ON, Canada

    7Margaret and Wallace McCain Centre for Child, Youth & Family Mental Health, Centre for Addiction and Mental Health, Toronto, ON, Canada

    8Lawrence S Bloomberg Faculty of Nursing, University of Toronto, Toronto, ON, Canada

    9Centre for Excellence in Economic Analysis Research, St Michael’s Hospital, Toronto, ON, Canada

    10Health and Wellness, University of Toronto, Toronto, ON, Canada

    11Slaight Family Centre for Youth in Transition, Centre for Addiction and Mental Health, Toronto, ON, Canada

    Corresponding Author:

    David Wiljer, PhD

    UHN Digital

    University Health Network

    190 Elizabeth Street

    R. Fraser Elliott Building RFE 3-411

    Toronto, ON

    Canada

    Phone: 1 416 340 4800 ext 6322

    Email: david.wiljer@uhn.ca


    Background: Mental health disorders are the most prevalent health issues among postsecondary students, yet few solutions to this emerging crisis exist. While mobile health technologies are touted as promising solutions for the unmet mental health needs of these students, the efficacy of these tools remains unclear. In response to these gaps, this study evaluates Thought Spot, a mobile and web app created through participatory design research.

    Objective: The goal of the research is to examine the impact of Thought Spot on mental health and wellness help-seeking intentions, behaviors, attitudes, self-stigma, and self-efficacy among postsecondary students in Canada.

    Methods: A 2-armed randomized controlled trial involving students from three postsecondary institutions was conducted. Students were eligible if they were aged 17 to 29 years, enrolled in full-time or part-time studies, functionally competent in English, and had access to a compatible digital device. The usual care group received a mental health services information pamphlet. The intervention group received the Thought Spot app on their digital device. Thought Spot is a standalone app that allows users to add, review, and search crowdsourced information about nearby mental health and wellness services. Users can also track their mood on the app. Outcomes were self-assessed through questionnaires collected at baseline and 3 and 6 months. The primary outcome was change in formal help-seeking intentions from baseline to 6 months, measured by the General Help-Seeking Questionnaire. A mixed-effects model was used to compare the impact of usual care and intervention on the primary outcome (formal help-seeking intentions). Secondary outcomes included changes in informal help-seeking intentions and help-seeking behaviors, help-seeking attitudes, self-stigma, and self-efficacy.

    Results: A total of 481 students were randomized into two groups: 240 to usual care, and 241 to the intervention group. There were no significant differences in help-seeking intentions between the usual care and intervention groups over 6 months (F2,877=0.85; P=.43, f=0.04). Both groups demonstrated similar increases in formal help-seeking intentions at 3 and 6 months (F2,877=23.52; P<.001, f=0.21). Compared with males, females sought more help from formal resources (OR 1.86; 95% CI 1.22 to 2.83, P=.001). Females were less likely to seek help from informal sources than males (OR 0.80; 95% CI 0.22 to 0.73, P<.001).

    Conclusions: Prompting postsecondary students about mental health and help-seeking appears to increase help-seeking intentions. mHealth interventions may be as effective as information pamphlets in increasing formal help-seeking but may confer a small advantage in driving help-seeking from informal sources. Although there is enthusiasm, developers and health policy experts should exercise caution and thoroughly evaluate these types of digital tools. Future studies should explore the cost-effectiveness of digital interventions and develop strategies for improving their efficacy.

    Trial Registration: ClinicalTrials.gov NCT03412461; https://clinicaltrials.gov/ct2/show/NCT03412461

    International Registered Report Identifier (IRRID): RR2-10.2196/resprot.6446

    J Med Internet Res 2020;22(10):e20790

    doi:10.2196/20790

    Keywords

    crowdsourcing; help-seeking behavior; mental health; mobile applications; randomized controlled trial; school mental health services; social support; young adult 


    Mental health disorders among postsecondary students are a global public health concern [1-4]. Youth aged 16 to 29 years face many challenges in the transition from childhood to adulthood, and 70% of all mental health conditions have their onset during this period [5-8]. Postsecondary education is an experience that can pose many challenges for transition-aged youth due to social, financial, and academic stressors [3,9-11]. Over the last 10 years, rates of mental health disorders among postsecondary students have increased [12]. Yet 35% of youth who are experiencing mental health problems do not seek formal help (eg, clinical services, health professionals) [13] or informal help (eg, friends and family, religious leaders, self-help) [13] due to barriers such as perceived stigma, difficulty expressing their concerns, difficulty accessing help, and a preference for self-reliance [14,15]. Amid the “campus mental health crisis” [1], there is a desperate need for interventions that facilitate help-seeking and access to mental health and wellness services [16].

    The proliferation of mobile devices and their ubiquity in the lives of transition-aged youth has encouraged developers, postsecondary institutions, and health care organizations to focus their efforts on online and mobile health (mHealth) interventions that address the mental health challenges faced by this population [17-20]. The evidence to support mHealth solutions has not been established, but they remain a major hope for transition-aged youth, given the extraordinary challenges and costs of addressing the mental health needs of postsecondary students [21-23]. To date, assessments of most mHealth interventions have been limited to case studies, pilot studies, and randomized controlled trials (RCTs) with small sample sizes [22]. Few studies involve user-centered design processes or end users as co-creators, evidence-based strategies that help mHealth tools achieve sustained engagement, effectiveness, and behavior change [24].

    Applying principles from participatory action and participatory design research [25,26], the research team produced Thought Spot, an mHealth intervention co-created with transition-aged youth that aims to improve help-seeking behavior related to mental health services for postsecondary settings [27,28]. Thought Spot serves as a map-based database (via a mobile app and website) that allows users to search and geolocate health, mental health, and wellness resources (Figure 1). Additional features of the app include mood and thought tracking, reviews about nearby resources and services, and the ability to bookmark searched information. Thought Spot users can also participate in crowdsourcing by adding new resources and writing reviews. Accordingly, the main objective of this RCT was to assess the impact of Thought Spot on intentions to seek formal help. The secondary objective was to examine the impact on intentions to seek help from informal sources and on help-seeking behaviors, help-seeking attitudes, self-stigma, and self-efficacy [17]. The research team hypothesized that Thought Spot would be superior to school-specific mental health services information pamphlets in increasing formal help-seeking intentions.

    ‎
    Figure 1. Features on the Thought Spot mobile app and online platform, designed by students for students to allow users to find and share health, mental health, and wellness resources (spots) using a map-based database of crowdsourced resources and a self-contained search feature.
    View this figure

    Study Design

    The team conducted a 2-armed RCT using participants who were students at three postsecondary institutions in the Greater Toronto Area (University of Toronto, Ryerson University, and George Brown College). The study staff and biostatistician were blinded throughout the study. The protocol was approved by the research ethics boards of each participating postsecondary institution and the Centre for Addiction and Mental Health and has been previously published (Multimedia Appendix 1) [27]. The trial was registered at ClinicalTrials.gov [NCT03412461]. Digital informed consent was obtained from each participant through Research Electronic Data Capture (REDCap) before participants could be included in the trial [29].

    Recruitment

    Participants were recruited using the most effective methods identified by two focus groups and the Thought Spot youth advisory committee during earlier stages of the project [28]. Participants were recruited through institutional and student-related listservs, bulletin boards, websites, social media, and class presentations.

    Full-time and part-time students aged 17 to 29 years who were enrolled at any of the three postsecondary institutions, who were functionally competent in English, and who had access to a digital device compatible with the intervention were eligible to participate in the study. Active suicidality was the sole exclusion criterion, but no participants met this criterion during the screening process (Figure 2; Multimedia Appendix 2). Participants also did not have knowledge of or access to Thought Spot prior to participating in the study.

    ‎
    Figure 2. Consolidated System of Reporting Trials flow diagram detailing the order of screening, randomization, and follow-up procedures.
    View this figure

    Randomization and Treatment

    Participants were randomly assigned using a 1:1 allocation ratio to either receive access to Thought Spot or to receive a school-specific mental health services information pamphlet. Randomization and allocation was performed using REDCap, a secure, browser-based, electronic data capture system [29]. A randomization module within REDCap provided computer-generated random allocation of participants in blocks of 10. All participants were enrolled by EH. All authors were blind to the allocation except for JS, who oversaw the randomization process. REDCap was configured to collect data through online questionnaires at baseline, 3 months and 6 months. We asked participants in both groups to use their intervention (Thought Spot or pamphlet) as needed over 6 months. Intervention group participants received detailed instructions via email on how to download and access the Thought Spot app on their personal electronic devices. Participants were also sent two study reminders, at 6 weeks and 18 weeks, in anticipation of the 3-month and 6-month questionnaires. At the end of the study, participants in the intervention group completed an adapted version of the Usefulness, Satisfaction, and Ease of Use questionnaire [30] and were invited to participate in qualitative interviews to evaluate Thought Spot’s usability and user experience. Full details on the randomization process and interventions are provided in the trial protocol in Multimedia Appendix 1. There were no changes to methods or major changes to the features and functionalities of the app. The content on the app changed by design because participants could add their own crowdsourced information about mental health resources and services during the trial. There were multiple bug fixes to address usability issues, such as malfunctioning buttons, slowness of search features, and log-in credentials not being saved. There were no major downtimes during the study period. Participants were provided with a small honorarium (up to CAD $40 [USD $30.08]) for participating in the study.

    Assessment Measures

    The primary outcome was a change in formal help-seeking intentions from baseline to 6 months measured by the General Help-Seeking Questionnaire (GHSQ) [31]. This scale comprises ten 7-point Likert scale questions that ask participants how likely they are to seek help from different sources, including friends, family members, mental health professionals, and helplines [31]. The formal help-seeking intentions scores were calculated by summing the scores of the 3 questions measuring formal help-seeking intentions [31]. Secondary outcomes of the study included changes in informal help-seeking intentions and help-seeking behaviors, help-seeking attitudes, self-stigma, and self-efficacy, measured, respectively, by the Actual Help-Seeking Questionnaire (AHSQ) [31], Attitudes Toward Seeking Professional Psychological Help Scale–Short Form (ATSPPH-SF) [32], Self-Stigma of Seeking Help Scale (SSOSH) [33], and the Youth Efficacy/Empowerment Scale–Mental Health (YES-MH) [34].

    Statistical Analysis

    Statistical analysis was performed by a biostatistician (MS) who was blinded to randomization. To determine the required sample size, power calculations were conducted using the primary outcome, the average formal help-seeking score on the GHSQ [27]. Monte Carlo simulations were conducted in SAS 9.4 for Windows (SAS Institute Inc) using the means, standard deviations, and within-subject associations reported in previous research using the GHSQ [35,36]. Monte Carlo simulations were chosen because our hypothesis required a model that accounted for dropouts, random effects, and the specific use of linear contrast to compare pre with post. The calculation accounted for a 40% dropout rate and assumed the use of a mixed-effects model to test the primary hypothesis at a significance level of .05 (2-tailed). A sample of 472 participants (236 per arm) was found to provide 80% power to detect a small effect size (Cohen d=0.25), which is equivalent to a 15% difference in change in GHSQ score from baseline to 6 months.

    Univariate analyses were conducted to describe groups at baseline (Table 1) and compare completers and noncompleters of the questionnaires. Completers were defined as participants who finished the survey at all 3 points of the study [27]. To evaluate the primary hypothesis, the research team used mixed-effects models using the intention-to-treat method, where time, group, and gender were entered as fixed effects and participants were entered as random effects. A linear contrast of estimated marginal means from the fixed-effect interaction between time and group was conducted to compare changes from baseline to 6 months across groups. Missing values were handled by maximum likelihood estimation, which is able to incorporate all available information in the data under the missing at random assumptions [37]. The team conducted sensitivity analyses, in which we added model variables found to have significant bivariate associations with dropouts, defined as participants who were randomized but who did not complete all 3 surveys, at a significance level of α=.20.

    Table 1. Baseline characteristics of participants who underwent randomizationa.
    CharacteristicsControl (n=240)Treatment (n=241)
    Gender, n (%)

    		Female188 (78.3)190 (78.6)

    		Male47 (19.6)44 (18.9)

    		Nonbinary5 (2.1)7 (2.5)
    Age in years, mean (SD)23.2 (3.1)22.9 (3.4)
    Student status, n (%)

    		Domestic student207 (86.3)207 (85.9)

    		International student31 (12.9)33 (13.7)

    		Don’t know2 (0.8)1 (0.4)
    Education level, n (%)

    		High school diploma144 (60.0)140 (58.1)

    		College diploma23 (9.6)9 (3.7)

    		Bachelor’s degree59 (24.6)74 (30.7)

    		Master’s degree11 (4.6)9 (4.2)

    		Doctoral degree00

    		Other3 (1.3)9 (3.7)
    Type of postsecondary institutionb, n (%)

    		College110 (45.8)114 (47.3)

    		University130 (54.2)125 (51.9)

    		Did not answer02 (0.8)
    Experience with mental health or substance use concerns, n (%)

    		Yes168 (70.0)172 (70.8)

    		No55 (23.0)63 (26.1)

    		Don’t know16 (6.7)6 (4.6)

    		Did not answer1 (0.4)0
    GHSQc formal sources, mean (SD)8.6 (3.9)8.3 (4.1)
    GHSQ informal sources, mean (SD)36.6 (8.5)36.1 (9.4)

    aPercentages may not total 100 due to rounding. There was no significant difference between the trial groups.

    bPostsecondary institution was missing for two participants.

    cGHSQ: General Help-Seeking Questionnaire.

    Similar models were conducted to explore the secondary outcomes of this study except for help-seeking behavior. This outcome was measured using the binary-scaled AHSQ and analyzed using a mixed binomial logistic regression model. The P values for the secondary analysis were not adjusted for multiple comparison. In addition to the prespecified analysis plan, the research team conducted an exploratory per-protocol analysis of the primary and secondary outcomes. Participants who logged on to Thought Spot more than once during the study were considered compliant.

    For F tests, the study report standardized effect sizes using Cohen f [38], reported as f, which is calculated by the R package sjstats (R Foundation for Statistical Computing) [39]. For contrasts that represent simple changes or difference in changes, Cohen d was used, which is reported as d. The respective d values are calculated using estimated marginal means and two formulas described by Morris and DeShon [40]. The biostatistician applied their recommended formula for repeated-measures design that focuses on change within a person relative to the variability of change scores and another formula used for independent groups pretest/posttest designs (Multimedia Appendix 3) [40].


    Recruitment and Participant Characteristics

    From March 2018 to January 2019, 481 participants were randomized from three Canadian postsecondary institutions into a 6-month trial on a rolling basis. Of these participants, 240 were assigned to the control group and 241 were assigned to the intervention group (Figure 2). The trial ended after all participants received their 6-month follow-up on June 30, 2019. Table 1 shows a comparison of study groups at baseline. Prior experience with mental health concerns was reported by 70.7% (340/481) of participants, and 61.3% (294/480) of participants reported having experienced suicidal ideation in their lifetime. There were no reported adverse events or harms during the trial.

    Primary Outcome: Change in Formal Help-Seeking Intentions

    The mixed-effects model found a significant time effect (F2,877=23.52; P<.001, f=0.21) but not a significant group-by-time interaction (F2,877=0.85; P=.43, f=0.04). Linear contrasts exploring the change in formal help-seeking intentions between baseline and 6 months did not find significant differences between the intervention and control group (estimate mean difference in change 0.39; t894=1.05; 95% CI –0.34 to 1.12; P=.29; d=0.09). Both the control group (estimated mean change 1.32; 95% CI 0.81 to 1.84, d=0.33) and the intervention group (estimated mean difference 0.93, 95% CI 0.41 to 1.45, d=0.24) showed a significant increase in the intention to seek formal sources of help. A sensitivity analysis revealed that group-by-time interaction was not significant after controlling for factors that were associated with dropout (F2,145=1.18, P=.31, f=0.11). There was also no difference in changes between groups (estimate 1.35, 95% CI –0.51 to 3.21, d=0.32).

    Secondary Outcomes

    An analysis of secondary outcomes found no significant group-by-time interactions for help-seeking intentions from informal sources (GHSQ), help-seeking behavior from formal sources (AHSQ), help-seeking attitudes toward professional help (ATSPPH), self-efficacy (SSOSH) and self-stigma (YES-MH; Table 2). However, some evidence of a group-by-time interaction was found for help-seeking from informal sources (Wald χ2=4.5, P=.11), but it is not significant (Table 3). Linear contrasts of the interaction indicated a decrease in help-seeking behavior related to informal sources in the control group, whereas an increase from 3 months to 6 months was found for the intervention group (odds ratio [OR] 0.86; 95% CI 0.71 to 1.02, P=.09), but it did not meet the significance threshold (Multimedia Appendix 4).

    Table 2. Main results from mixed models.
    Participant questionnairesLinear contrastaGroup-by-time interactionb

    		Estimatea95% CIP valueF statisticDFcP value
    GHSQd (intention to treat)

    		Formal resources0.39–0.34 to 1.12.300.852/877.43

    		Informal resources0.36–1.21 to 1.92.650.702/811.50
    GHSQ (per-protocol) – formal resources0.18–0.62 to 0.98.660.142/757.87
    GHSQ (dropout)b – formal resources1.35–0.51 to 3.21.151.182/145.31
    GHSQ (controlled for site)e – formal resources0.390.34 to 1.13.290.852/877.43
    ATSPPHf–0.15–0.91 to 0.61.701.392/850.25
    SSOSHg0.08–1.13 to 1.28.900.032/876.97
    YES-MHh: self–0.09–0.73 to 0.55.790.122/801.89
    YES-MH: service0.47–0.40 to 1.34.290.602/823.55
    YES-MH: system0.62–0.26 to 1.50.171.312/818.27
    YES-MH: total0.16–1.52 to 1.83.850.082/738.92

    aThe linear contrast tests the change from baseline to end of the trial across groups (primary outcome); (6 months – baseline) in control minus intervention. Positive values indicate larger increases in the intervention group.

    bThe group-by-time interaction tests for any difference in the group trajectories (not primary outcome). Sensitivity analysis controls for all variables at α= .20. These variables are listed in Multimedia Appendix 7.

    cDF: degree of freedom.

    dGHSQ: General Help-Seeking Questionnaire.

    eAn additional sensitivity analysis controlling for site as fixed effects was conducted, but site was not significant and results were similar to those for the main model.

    fATSPPH: Attitudes Toward Seeking Professional Psychological Help Scale.

    gSSOSH: Self-Stigma of Seeking Help Scale.

    hYES-MH: Youth Efficacy/Empowerment Scale–Mental Health.

    Table 3. Main results from binomial models.
    Participant questionnairesLinear contrastaGroup-by-time interactionb

    		Odds ratio95% CIP valueWald χ2DFcP value
    AHSQd – formal resources0.800.48-1.34.391.32.53
    AHSQ – informal resources0.860.71-1.02.094.52.11

    aIn a binomial model, the contrast is the ratio of odds ratios. A ratio lower than 1 indicates a larger increase in the probability of positive answers in the intervention group.

    bThe statistic used in binomial models is the chi-square statistic.

    cDF: degree of freedom.

    dAHSQ: Actual Help-Seeking Questionnaire.

    The sex effect was significant for help-seeking behavior related to formal and informal sources, help-seeking attitudes toward professional help, and self-efficacy (Multimedia Appendix 5 and 6). Compared with males, females exhibited higher formal AHSQ scores (OR 1.86; 95% CI 1.22 to 2.83, P=.001). Females also exhibited significantly lower AHSQ informal scores than males (OR 0.80; 95% CI 0.22 to 0.73, P<.001). Similarly, for attitudes toward health care professionals, females had significantly lower ATSPPH scores than males (estimate 0.80; 95% CI 0.22 to 1.38, P=.003).

    Per-Protocol Analysis

    Of the participants in the intervention group, 70.1% (169/241) met the definition of compliance. In comparing these participants to the control group, the per-protocol analysis led to the same conclusion for all primary and secondary outcomes with nonsignificant linear contrast (difference in change 0.18, t711=0.439, P=.66, d=0.03). In the intervention group, compliance significantly moderated the GHSQ trajectory (P=.006, F=0.15), with compliant participants increasing their help-seeking more than noncompliant participants. This change mostly happened between baseline to 3 months.

    App Use Data

    Of the 241 people randomized to the intervention group, 168 visited Thought Spot, resulting in 3696 clicks recorded between March 2018 and June 2019. Overall, users viewed 190 Spots, conducted 293 searches, and created 74 Thoughts. Spots are locations of mental health and wellness resources. Thoughts are users’ personal journal entries on the app. Details of the user data will be published independently of these findings.


    Principal Findings

    In the analysis of our primary outcome, there were no significant differences in the formal help-seeking intentions of postsecondary students between control and intervention groups. However, both groups experienced a similar increase in formal help-seeking intentions during the 6-month study period, as assessed by the GHSQ. These findings suggest that prompting youth about mental health, regardless of the delivery method (eg, information pamphlet or mHealth intervention), may increase help-seeking intentions. Our results are consistent with findings from previous RCTs involving online mental health services, in which the interventions did not lead to significant differences in formal help-seeking compared with an active control group [41,42]. However, the similar increase in help-seeking in both arms observed in our study supports the emerging discussion in that these technologies may have a “role as another option in their toolkit” when in need of mental health and wellness resources [41].

    Analyses of secondary outcomes revealed no significant group-by-time interactions for help-seeking intentions from informal sources, attitudes toward seeking professional help, self-efficacy, or self-stigma. Although not significant, there is some evidence suggesting a small increase in help-seeking behavior related to informal sources between 3 and 6 months in the intervention group, whereas a small decrease was seen in the control group. This difference contrasts with findings from previous studies of help-seeking interventions, which reported no effect on informal help-seeking [42]. Although the attitudes and intentions regarding informal help-seeking did not differ between groups, the sustained increase in informal help-seeking behavior observed between 3 and 6 months in the intervention group may indicate that Thought Spot is effective in converting intention into action [43]. The difference may also be due to previously reported advantages of online interventions over typical information pamphlets and formal sources of help, such as being anonymous, less stigmatizing, easier to access and use, and more trustworthy [41,44-49]. Nonetheless, these findings must be contextualized using data from qualitative interviews, usability data, and use data collected throughout the RCT.

    When looking at the differences between AHSQ scores, the results suggest that females tend to seek help from formal sources more than males. However, when looking at the informal AHSQ scores, males sought more help from informal sources than females. This observation complements the existing literature on gender differences in transition-aged youths’ help-seeking behaviors. Based on a cross-section survey study, Findlay and Sunderland [50] found that females reported contacting formal and informal resources more than males. However, the data from this RCT show that there may be a difference in preference between formal and informal resources between genders. Another interesting finding was related to the help-seeking behavior for the nonbinary gender group. While not significant due to the small sample size, the considerable odds ratio for AHSQ formal scores suggest there may be a small effect where participants in the nonbinary gender group have higher help-seeking behavior than males.

    The per-protocol analysis indicated that compliant participants increased their help-seeking more than noncompliant participants, which suggests that repeated visits may contribute to changes in help-seeking intentions. While these exploratory findings should be approached with caution, it supports the emerging interest on the impact of repeated app use on study outcomes [51]. The app evaluation framework, developed by the American Psychiatric Association, highlights the importance of user engagement in the evaluation of apps for clinical use [52]. However, while use of the app is required to enable the expected benefits, the significance of the repeated use in the context of help-seeking and whether use can be representative as meeting their needs remains unclear. As such, this finding warrants further investigation into the use data and the experiences of users.

    Comparison With Prior Work

    One of the noteworthy strengths of our trial is its standing as one of the few RCTs with large sample sizes that evaluate the effectiveness of mHealth interventions for help-seeking among youth in postsecondary settings [41]. The research team saw higher than predicted retention rates of at least 73% at both 3 and 6 months. The baseline characteristics between the control and intervention groups were balanced, and the research team controlled for variables associated with the outcome (eg, GHSQ). Another strength of this study is the use of participatory action research and co-creation methodology to engage transition-aged youth throughout the development and execution of the trial [53]. This methodology has been shown to increase participation in mental health care, better address youth concerns, and produce more relevant results [54,55]. The research team was successful in engaging youth across three postsecondary institutions in a longitudinal mental health study. Given the many barriers that make meaningful youth engagement a challenge [56], obtaining 1291 individual responses during our recruitment is notable in its magnitude, especially given the short time frame (relative to other mental health trials). The high level of engagement in the study suggests that our commitment to co-creating solutions resonates with the postsecondary student population [28]. It may also reflect students’ enthusiasm for mental health solutions and mHealth-related research, suggesting a need for continued engagement with this population at a time when mental health concerns and suicide rates among youth continue to rise at postsecondary institutions [57,58].

    Limitations

    The study had some limitations. Group assignment could not be blinded for participants. There were also software bugs that led to an inconsistent app environment and usability issues during the trial. These issues could have affected the level of user engagement and compliance with the intervention and ultimately the effectiveness of Thought Spot because some participants had difficulty accessing key functions of the app during certain points of the trial.

    There was also a number of participants who did not complete the 3-month survey packages. There were no software bugs or reported issues that could have led to the drop in participation and the reason for this observation is unknown.

    Additionally, the effect size for the change in informal help-seeking behavior was small and was noted only in the 3- to 6-month period. Similarly, due to sample size restrictions, we were unable to compare the impact of Thought Spot among international and domestic students. Given the unique barriers associated with attending college in an unfamiliar location [59], future investigations should explore whether these apps can facilitate help-seeking for international students. Further investigation and a longer follow-up period are required to evaluate the sustained impact of mHealth solutions on informal help-seeking. Finally, the development of Thought Spot and the findings from this trial are based on the unique experiences of transition-aged youth enrolled in Canadian postsecondary institutions and may not be generalizable to youth outside Canadian academic settings.

    Conclusions

    In summary, there were no significant differences in formal help-seeking intentions between the control and intervention groups. Female participants sought help from formal resources more often than males, whereas males were more likely to seek help from informal sources than females. There was some evidence of a small increase in informal help-seeking behavior between 3 and 6 months in the intervention group. Both groups experienced a similar increase in formal help-seeking intentions over 6 months. These findings suggest a need to further explore the effectiveness of mHealth technologies in supporting the mental health help-seeking needs of transition-aged youth. It is increasingly important as a next step to compare the cost-effectiveness of Thought Spot and information pamphlets for understanding the feasibility and sustainability of mHealth tools compared with existing strategies [60].

    Acknowledgments

    This project was supported by eHealth Innovation Partnership Program grant from the Canadian Institute for Health Research (EH1-143558). The Canadian Institute for Health Research was not involved in the study design, data analysis, data interpretation, or writing of the report.

    We thank all students who participated in the trial and in our youth engagement activities. We also thank all Centre for Addiction and Mental Health (CAMH) staff, collaborators, youth advisory committee members, and practicum students who were involved and supported the Thought Spot project, including: Daryl Boshart, Almerinda Collela, Paulysha De Gannes, Dr Gunther Eysenbach, Genevieve Ferguson, Olivia Heffernan, Tyson Herzog, Dr Jeffrey Hoch, Dr Huan Jiang, Mandee Kaur, Dr Marcus Law, Julia Roy, Shehab Sennah, Sarah Sharpe, Angie Solomon, Chelsea Stunden, Vivetha Thambinathan, Tim Tripp, Nicole VanHeerwaarden, Julie Zasada, and Hema Zbogar. We thank the CAMH REDCap Operations Committee for their assistance in implementing the e-consent process and survey delivery procedures. QoC Health served as the software developers of the Thought Spot platform.

    Conflicts of Interest

    None of the authors have conflicts of interest to disclose.

    ‎
    Multimedia Appendix 1

    Thought Spot randomized controlled trial protocol and statistical analysis plan.

    PDF File (Adobe PDF File), 355 KB
    ‎
    Multimedia Appendix 2

    Automated process for participants self-reporting active suicidality.

    DOCX File , 15 KB
    ‎
    Multimedia Appendix 3

    Formulas to calculate effect sizes for repeated measures and pretest/posttest research designs.

    DOCX File , 18 KB
    ‎
    Multimedia Appendix 4

    Group-by-time interaction for help-seeking behavior from informal sources.

    DOCX File , 90 KB
    ‎
    Multimedia Appendix 5

    Gender effects for help-seeking behaviors, intentions, and attitudes toward professional help and self-efficacy.

    DOCX File , 17 KB
    ‎
    Multimedia Appendix 6

    Gender effects for help-seeking behaviors, intentions, and attitudes toward professional help and self-efficacy when comparing females and nonbinary to males.

    DOCX File , 191 KB
    ‎
    Multimedia Appendix 7

    Control variables or interactions that were added to the dropout model aside from the baseline General Help Seeking Questionnaire and sex.

    DOCX File , 15 KB
    ‎
    Multimedia Appendix 8

    CONSORT-EHEALTH checklist (V 1.6.1).

    PDF File (Adobe PDF File), 1349 KB
    1. Auerbach RP, Mortier P, Bruffaerts R, Alonso J, Benjet C, Cuijpers P, WHO WMH-ICS Collaborators. WHO World Mental Health Surveys International College Student Project: prevalence and distribution of mental disorders. J Abnorm Psychol 2018 Oct;127(7):623-638. [CrossRef] [Medline]
    2. Ketchen Lipson S, Gaddis SM, Heinze J, Beck K, Eisenberg D. Variations in student mental health and treatment utilization across US colleges and universities. J Am Coll Health 2015;63(6):388-396. [CrossRef] [Medline]
    3. Cunningham S, Duffy A. Investing in our future: importance of postsecondary student mental health research. Can J Psychiatry 2019 Feb;64(2):79-81 [FREE Full text] [CrossRef] [Medline]
    4. Gunnell D, Kidger J, Elvidge H. Adolescent mental health in crisis. BMJ 2018 Jun 19;361:k2608. [CrossRef] [Medline]
    5. The human face of mental health and mental illness in Canada. Ottawa: Government of Canada; 2006.   URL: https://www.phac-aspc.gc.ca/publicat/human-humain06/pdf/human_face_e.pdf [accessed 2020-09-28]
    6. MacKinnon N, Colman I. Factors associated with suicidal thought and help-seeking behaviour in transition-aged youth versus adults. Can J Psychiatry 2016 Dec;61(12):789-796 [FREE Full text] [CrossRef] [Medline]
    7. Caspi A, Houts RM, Ambler A, Danese A, Elliott ML, Hariri A, et al. Longitudinal assessment of mental health disorders and comorbidities across 4 decades among participants in the dunedin birth cohort study. JAMA Netw Open 2020 Apr 01;3(4):e203221 [FREE Full text] [CrossRef] [Medline]
    8. Patton GC, Sawyer SM, Santelli JS, Ross DA, Afifi R, Allen NB, et al. Our future: a Lancet commission on adolescent health and wellbeing. Lancet 2016 Jun 11;387(10036):2423-2478. [CrossRef] [Medline]
    9. Zarrett N, Eccles J. The passage to adulthood: challenges of late adolescence. New Dir Youth Dev 2006(111):13-28. [CrossRef] [Medline]
    10. Ruberman L. Challenges in the transition to college: the perspective of the therapist back home. Am J Psychother 2014;68(1):103-115. [CrossRef] [Medline]
    11. Chan V, Moore J, Derenne J, Fuchs DC. Transitional age youth and college mental health. Child Adolesc Psychiatr Clin N Am 2019 Jul;28(3):363-375 [FREE Full text] [CrossRef] [Medline]
    12. Lipson SK, Lattie EG, Eisenberg D. Increased rates of mental health service utilization by U.S. college students: 10-year population-level trends (2007-2017). Psychiatr Serv 2019 Jan 01;70(1):60-63. [CrossRef] [Medline]
    13. Rickwood D, Thomas K. Conceptual measurement framework for help-seeking for mental health problems. Psychol Res Behav Manag 2012;5:173-183 [FREE Full text] [CrossRef] [Medline]
    14. Salaheddin K, Mason B. Identifying barriers to mental health help-seeking among young adults in the UK: a cross-sectional survey. Br J Gen Pract 2016 Oct;66(651):e686-e692. [CrossRef] [Medline]
    15. DiPlacito-DeRango ML. Acknowledge the barriers to better the practices: support for student mental health in higher education. Can J Scholarship Teaching Learning 2016 Dec 11;7(2):1. [CrossRef]
    16. Ng P, Padjen M. An overview of post-secondary mental health on campuses in Ontario: challenges and successes. Int J Ment Health Addiction 2018 Oct 24;17(3):531-541. [CrossRef]
    17. Seko Y, Kidd S, Wiljer D, McKenzie K. Youth mental health interventions via mobile phones: a scoping review. Cyberpsychol Behav Soc Netw 2014 Sep;17(9):591-602. [CrossRef] [Medline]
    18. Bratu S. Fear of missing out, improper behavior, and distressing patterns of use. Linguis Philos Invest 2018;17:130-140. [CrossRef]
    19. Roess A. The promise, growth, and reality of mobile health: another data-free zone. N Engl J Med 2017 Nov 23;377(21):2010-2011. [CrossRef] [Medline]
    20. Becker TD, Torous JB. Recent developments in digital mental health interventions for college and university students. Curr Treat Options Psych 2019 Jun 14;6(3):210-220. [CrossRef]
    21. Supporting the mental health of emerging adults in Ontario’s post-secondary system. Ontario University and College Health Association. 2017.   URL: http://oucha.ca/pdf/mental_health/Oucha-Paper-April-2017.pdf [accessed 2020-09-28]
    22. Tomlinson M, Rotheram-Borus MJ, Swartz L, Tsai AC. Scaling up mHealth: where is the evidence? PLoS Med 2013;10(2):e1001382 [FREE Full text] [CrossRef] [Medline]
    23. Mohr DC, Burns MN, Schueller SM, Clarke G, Klinkman M. Behavioral intervention technologies: evidence review and recommendations for future research in mental health. Gen Hosp Psychiatry 2013 Aug;35(4):332-338 [FREE Full text] [CrossRef] [Medline]
    24. McCurdie T, Taneva S, Casselman M, Yeung M, McDaniel C, Ho W, et al. mHealth consumer apps: the case for user-centered design. Biomed Instrum Technol 2012;Suppl:49-56. [CrossRef] [Medline]
    25. Whyte W, Greenwood D, Lazes P. Participatory action research: through practice to science in social research. Particip Action Res 1991:19-55. [CrossRef]
    26. Hagen P, Collin P, Metcalf A, Nicholas M, Rahilly K, Swainston N. Participatory Design of evidence-based online youth mental health promotion, prevention, early intervention and treatment. Melbourne: Young and Well Cooperative Research Centre. 2012.   URL: http://about.au.reachout.com/wp-content/uploads/2015/05/Young_and_Well_CRC_IM_PD_Guide.pdf [accessed 2020-09-28]
    27. Wiljer D, Abi-Jaoude A, Johnson A, Ferguson G, Sanches M, Levinson A, et al. Enhancing self-efficacy for help-seeking among transition-aged youth in postsecondary settings with mental health and/or substance use concerns, using crowd-sourced online and mobile technologies: the thought spot protocol. JMIR Res Protoc 2016 Nov 04;5(4):e201 [FREE Full text] [CrossRef] [Medline]
    28. VanHeerwaarden N, Ferguson G, Abi-Jaoude A, Johnson A, Hollenberg E, Chaim G, et al. The optimization of an eHealth solution (thought spot) with transition-aged youth in postsecondary settings: participatory design research. J Med Internet Res 2018 Mar 06;20(3):e79 [FREE Full text] [CrossRef] [Medline]
    29. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap): a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009 Apr;42(2):377-381 [FREE Full text] [CrossRef] [Medline]
    30. Gao M, Kortum P, Oswald F. Psychometric evaluation of the use (usefulness, satisfaction, and ease of use) questionnaire for reliability and validity. Proc Hum Fact Ergonom Soc 2018 Sep 27;62(1):1414-1418. [CrossRef]
    31. Rickwood D, Deane FP, Wilson CJ, Ciarrochi J. Young people's help-seeking for mental health problems. Adv Mental Health 2005 Dec;4(3):218-251. [CrossRef]
    32. Fischer E, Farina A. Attitudes toward seeking professional psychologial help: a shortened form and considerations for research. J Coll Stud Dev 1995;36(4):368-373.
    33. Vogel DL, Wade NG, Haake S. Measuring the self-stigma associated with seeking psychological help. J Couns Psychol 2006;53(3):325-337. [CrossRef]
    34. Walker J, Powers L. Introduction to the youth self-efficacy scale/mental health and the youth participation in planning scale. Research and Training Centre on Family Support and Children's Mental Health. 2007.   URL: https://www.pathwaysrtc.pdx.edu/pdf/pbCompleteSurveyPacket.pdf [accessed 2020-09-28]
    35. Smith CL, Shochet IM. The impact of mental health literacy on help-seeking intentions: results of a pilot study with first year psychology students. Int J Mental Health Promot 2011 Jan;13(2):14-20. [CrossRef]
    36. Costin DL, Mackinnon AJ, Griffiths KM, Batterham PJ, Bennett AJ, Bennett K, et al. Health e-cards as a means of encouraging help seeking for depression among young adults: randomized controlled trial. J Med Internet Res 2009;11(4):e42 [FREE Full text] [CrossRef] [Medline]
    37. Geert M, Geert V. Models for Discrete Longitudinal Data. New York: Springer; 2005.
    38. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd Edition. Hillsdale: Lawrence Erlbaum; 1988.
    39. Lüdecke D. sjstats: statistical functions for regression models version 0.17.6. 2018 Nov 18.   URL: https://zenodo.org/record/1489175 [accessed 2020-02-14]
    40. Morris SB, DeShon RP. Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychol Methods 2002 Mar;7(1):105-125. [Medline]
    41. Kauer SD, Mangan C, Sanci L. Do online mental health services improve help-seeking for young people? A systematic review. J Med Internet Res 2014;16(3):e66 [FREE Full text] [CrossRef] [Medline]
    42. Xu Z, Huang F, Kösters M, Staiger T, Becker T, Thornicroft G, et al. Effectiveness of interventions to promote help-seeking for mental health problems: systematic review and meta-analysis. Psychol Med 2018 Dec;48(16):2658-2667. [CrossRef] [Medline]
    43. Levin ME, Stocke K, Pierce B, Levin C. Do college students use online self-help? A survey of intentions and use of mental health resources. J Coll Student Psychother 2017 Oct 09;32(3):181-198. [CrossRef]
    44. Collin PJ, Metcalf AT, Stephens-Reicher JC, Blanchard ME, Herrman HE, Rahilly K, et al. ReachOut.com: the role of an online service for promoting help-seeking in young people. Adv Mental Health 2014 Dec 17;10(1):39-51. [CrossRef]
    45. Shandley K, Austin D, Klein B, Kyrios M. An evaluation of "Reach Out Central": an online gaming program for supporting the mental health of young people. Health Educ Res 2010 Aug;25(4):563-574. [CrossRef] [Medline]
    46. Bradley KL, Robinson LM, Brannen CL. Adolescent help-seeking for psychological distress, depression, and anxiety using an Internet program. Int J Mental Health Promotion 2012 Feb;14(1):23-34. [CrossRef]
    47. Burns JM, Davenport TA, Durkin LA, Luscombe GM, Hickie IB. The internet as a setting for mental health service utilisation by young people. Med J Aust 2010 Jun 7;192(11 Suppl):S22-S26. [Medline]
    48. Kummervold PE, Gammon D, Bergvik S, Johnsen JK, Hasvold T, Rosenvinge JH. Social support in a wired world: use of online mental health forums in Norway. Nord J Psychiatry 2002;56(1):59-65. [CrossRef] [Medline]
    49. Horgan A, Sweeney J. Young students' use of the Internet for mental health information and support. J Psychiatr Ment Health Nurs 2010 Mar;17(2):117-123. [CrossRef] [Medline]
    50. Findlay L, Sunderland A. Professional and informal mental health support reported by Canadians aged 15 to 24. Health Reports 2014:1. [Medline]
    51. Huckvale K, Nicholas J, Torous J, Larsen ME. Smartphone apps for the treatment of mental health conditions: status and considerations. Curr Opin Psychol 2020 Dec;36:65-70. [CrossRef]
    52. Torous JB, Chan SR, Gipson SYT, Kim JW, Nguyen T, Luo J, et al. A hierarchical framework for evaluation and informed decision making regarding smartphone apps for clinical care. Psychiatr Serv 2018 Feb 15;69(5):498-500. [CrossRef] [Medline]
    53. Leask CF, Sandlund M, Skelton DA, Altenburg TM, Cardon G, Chinapaw MJM, GrandStand‚ Safe StepTeenage Girls on the Move Research Groups. Framework, principles and recommendations for utilising participatory methodologies in the co-creation and evaluation of public health interventions. Res Involv Engagem 2019;5:2 [FREE Full text] [CrossRef] [Medline]
    54. Orlowski SK, Lawn S, Venning A, Winsall M, Jones GM, Wyld K, et al. Participatory research as one piece of the puzzle: a systematic review of consumer involvement in design of technology-based youth mental health and well-being interventions. JMIR Hum Factors 2015;2(2):e12 [FREE Full text] [CrossRef] [Medline]
    55. Anderson JE, Lowen CA. Connecting youth with health services: systematic review. Can Fam Physician 2010 Aug;56(8):778-784. [Medline]
    56. O'Donoghue JL, Kirshner B, McLaughlin M. Introduction: moving youth participation forward. New Dir Youth Dev 2002(96):15-26. [CrossRef] [Medline]
    57. Curtin SC, Heron M. Death rates due to suicide and homicide among persons aged 10-24: United States, 2000-2017. NCHS Data Brief 2019 Oct(352):1-8. [Medline]
    58. Jaworska N, De Somma E, Fonseka B, Heck E, MacQueen GM. Mental health services for students at postsecondary institutions: a national survey. Can J Psychiatry 2016 Dec;61(12):766-775 [FREE Full text] [CrossRef] [Medline]
    59. Mori SC. Addressing the mental health concerns of international students. J Couns Develop 2000;78(2):137-144. [CrossRef]
    60. Kaur A, Isaranuwatchai W, Jaffer A, Ferguson G, Abi-Jaoude A, Johnson A, et al. A web- and mobile-based map of mental health resources for postsecondary students (thought spot): protocol for an economic evaluation. JMIR Res Protoc 2018 Mar 29;7(3):e83 [FREE Full text] [CrossRef] [Medline]

    ‎
    AHSQ: Actual Help-Seeking Questionnaire
    ATSPPH-SF: Attitudes Toward Seeking Professional Psychological Help Scale–Short Form
    CAMH: Centre for Addiction and Mental Health
    GHSQ: General Help-Seeking Questionnaire
    mHealth: mobile health
    OR: odds ratio
    RCT: randomized controlled trial
    REDCap: Research Electronic Data Capture
    SSOSH: Self-Stigma of Seeking Help Scale
    YES-MH: Youth Efficacy/Empowerment Scale–Mental Health

    Edited by G Eysenbach; submitted 17.06.20; peer-reviewed by N Wang, AM Duncan; comments to author 19.07.20; revised version received 10.08.20; accepted 16.08.20; published 30.10.20

    Copyright

    ©David Wiljer, Jenny Shi, Brian Lo, Marcos Sanches, Elisa Hollenberg, Andrew Johnson, Alexxa Abi-Jaoudé, Gloria Chaim, Kristin Cleverley, Joanna Henderson, Wanrudee Isaranuwatchai, Andrea Levinson, Janine Robb, Howard W Wong, Aristotle Voineskos. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 30.10.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.