Comparison of YouthCHAT, an Electronic Composite Psychosocial Screener, With a Clinician Interview Assessment for Young People: Randomized Trial

Introduction

Psychosocial problems and risky health behaviors are significant issues for young people worldwide. In New Zealand, one-third of adolescents are affected by anxiety and depression [1]; the highest rate of suicide is among youth aged 15 to 24 years [2], and approximately one-fourth of high school students engage in hazardous alcohol use [3]. Mental health issues and risky health behaviors can lead to costly long-term health and social outcomes [4-6], and as such, local and World Health Organization policies emphasize the value of developing more effective tools and appropriately targeted and accessible services to identify and address the needs of young people [7-9]. At the same time, young people want a greater say in how services are designed and delivered and expect services to be diverse, contemporary, and responsive [10].

Home, Education, Eating, Activities, Drugs and Alcohol, Sexuality, Suicide and Depression, Safety (HEEADSSS) assessment is a clinician-administered interview-based assessment of young people that can identify mental health and substance use problems [11,12]. Currently, all year 9 (usually 13-year-old) students in low-decile schools (those with the highest proportion of students from low socioeconomic communities) and some attendees at primary care and pediatric services in New Zealand are screened for psychosocial problems via HEEADSSS assessment. Although HEEADSSS offers a straightforward, holistic, and gradual approach to assessing young people across many domains, it is a psychosocial interview rather than a screening tool. Drawbacks include its lack of validation for problem identification, the cost of resourcing, time required for administration (up to an hour per person), and variable quality depending on the skill and experience of the assessor.

The Youth version, Case-finding and Help Assessment Tool (YouthCHAT) [13,14] is a self-report, electronic screener that covers the following domains: smoking, drinking, recreational drug use (based on the Substances and Choices Scale, SACS) [15], problematic gambling, depression (based on the Patient Health Questionnaire-Adolescent Version, PHQ-A) [16,17], anxiety (based on the Generalized Anxiety Disorder-7 scale, GAD-7), sexual health, general stresses, exposure to abuse, behavior problems, anger management problems, eating problems, and physical activity [18]. For each positive domain screened, there is a help question that asks participants if they would like help either today or later. Responses to the help question support conversations between young people and their health providers about the issues they would like addressed, which facilitates shared decision-making, with increased likelihood that real sustained changes will be made (Figures 1 and 2).

Students complete YouthCHAT electronically on a device, and a summary report is immediately available for the nurse or other health provider through the electronic health record at the point of care. This includes positive or negative responses for each module, and where positive, the score and its interpretation when applicable (eg, depression: PHQ-A score 24=severe depression [20-27 out of a possible 27]) as well as whether help is wanted either now or later is included. A positive PQA-9 question triggers a red alert for self-harm. Where YouthCHAT indicates serious issues such as suicidal ideation, the nurse will intervene even when the students indicate they do not want help. Health providers using YouthCHAT are provided with stepped care resources for each module, tailored to their setting. Although detection of positive issues may increase referral rate and hence workload, this applies equally to HEEADSSS assessment.

YouthCHAT was developed via co-design with young people in primary care, youth, and school settings [13,14,19-22], and previous research has demonstrated its acceptability among young people of New Zealand European, Māori, and Pacific Island ethnicities [23] and identified that some students prefer disclosing sensitive information via electronic means rather than face-to-face means [14,24-27].

Electronic screening has been shown to provide consistent results and can lead to more disclosures and reduce staff time [28,29]. Arguments have been made for and against screening for mental health issues such as depression; however, screening has been found to be effective as long as it is linked to evidence-based interventions, not conducted as a stand-alone activity [30].

This study aimed to compare the performance and acceptability of YouthCHAT screening and HEEADSSS assessment for 13-year-old students attending a nurse-led clinic in a high school setting.

‎View this figure

‎View this figure

Methods

Trial Design

A randomized trial using a counterbalanced design was employed to deliver YouthCHAT screening either before or after face-to-face HEEADSSS assessment.

Mapping of Youth Version, Case-Finding and Help Assessment Tool and Home, Education, Eating, Activities, Drugs and Alcohol, Sexuality, Suicide and Depression, Safety

Although YouthCHAT and HEEADSSS contained similar areas of assessment, it was necessary to map specific domains to each other for comparison of results. HEEADSSS includes domains on substance misuse, problem eating, sexual health, and physical activity, which are approximately equivalent to YouthCHAT modules. Questions regarding problems at home and safety were mapped for comparison (see Table 1). Although HEEADSSS mental health domain includes a number of nonspecific items such a sadness, grief, and difficulty sleeping, YouthCHAT mental health modules only comprised screening measures for depression and anxiety.

For HEEADSSS assessment, there are no threshold scores—the assessor decides whether the response is positive for that domain or not. For YouthCHAT, responses are clearly positive or negative for each domain. Where there are added tools, cutoff points are as follows:

• ASSIST: 3 to 26 at risk of health and other problems from current pattern of smoking and greater than 26 at high risk of experiencing severe problems (health, social, financial, legal, and relationship) as a result of current pattern of smoking and likely to be dependent;
• SACS: 2 to 3 low-level problems with alcohol or drugs requiring further assessment and more than 3 problems with alcohol or drugs probably requiring treatment;
• PHQ-A: 10 to 14 moderate depression, 15 to 19 moderately severe depression, and 20 to 27 severe depression; and
• GAD-7: greater than 9 general anxiety disorder.

Participants

All year 9 (13- to 14-year-old) students at a low-decile high school in Auckland, New Zealand, were invited to participate following the provision of written information about the study at the start of the school year and the completion of paired informed parental consent (using an opt-out process) and individual participant assent (as all students were aged <16 years). No students were excluded from the study. HEEADSSS assessment is mandatory for all year 9 students regardless of the study.

Recruitment and Randomization

Participants were randomized to receive either HEEADSSS assessment by a school nurse followed by an electronic YouthCHAT screen (condition 1) or YouthCHAT followed by HEEADSSS assessment (condition 2). Clinic staff were provided with a random list from a computer-generated random numbers table, with consecutive sampling until all enrolled students had completed assessment. This took place during a planned break from class time when students receive their annually required HEEADSSS assessment. Review of results and any necessary follow-up was arranged by the school nurse immediately following the completion of YouthCHAT screening and HEEADSSS assessment.

Outcomes

Primary outcome measures were (1) the time taken to complete YouthCHAT and HEEADSSS, (2) comparative detection rates for YouthCHAT and HEEADSSS for each issue, and (3) acceptability of YouthCHAT to students and staff. YouthCHAT data were collected electronically (completed on an electronic tablet by students), and encrypted results were securely stored on a central database. HEEADSSS results were entered into the electronic health record by school nurses. A subset of students completed paper-based acceptability questionnaires, and the 3 school nurses were interviewed individually.

Analyses

Quantitative data were analyzed using Microsoft Excel 2013 and Statistical Package for the Social Sciences version 25 (SPSS v25). Analyses included basic descriptive statistics, between-intervention analyses undertaken with paired t tests (for numeric variables) or McNemar tests (for categorical variables), and between-condition nonparametric analyses undertaken with Mann-Whitney U tests. Distributions were checked for normality throughout.

Qualitative data were analyzed using a general inductive approach [31], with collated text independently coded by 2 researchers (HT and FG) to identify emerging themes. Discrepancies were resolved through an adjudication session.

Further methodological details are described in our trial protocol [32]. The study was approved by the New Zealand Northern Region Ethics Committee (16/CEN/137/AM03).

Results

Description of Participants

From the 139 eligible students, 129 assented. Electronic screening and face-to-face assessments were conducted between March and November 2017. There were incomplete or missing data for 19 students, giving a total sample size of 110 for analysis (81%; see Figure 1).

In addition, 63% (71/113) of the participants were of Pacific ethnicity, 29% (33/113) were of Māori ethnicity, and the remaining 8% (9/113) were of New Zealand European or other ethnicity. Moreover, 51% (58/113) of the participants were male, and the randomized condition numbers were similar, with 49% in condition 1 and 51% in condition 2. From the 32 students invited to participate in a focus group (8 during each term), 21 (66%) attended, with 3 groups of 5 participants and 1 group of 6 participants.

Time Taken to Complete Youth Version, Case-Finding and Help Assessment Tool and Home, Education, Eating, Activities, Drugs and Alcohol, Sexuality, Suicide and Depression, Safety Assessment

HEEADSSS time data were missing for 19 students; therefore, the comparative time to complete analyses were conducted for 94 students. YouthCHAT took an average of 8 min 57 seconds (range 1 min 45 seconds to 54 min 15 seconds) to complete, compared with HEEADSSS with an average nearly double at 17 min 22 seconds (range 3 min to 45 min), giving a mean difference of 8 min 25 seconds (range 6 min 20 seconds to 11 min 10 seconds; P<.01). For several students, the Wi-Fi connection was lost for YouthCHAT, which may be reflected in the outlier durations of 25 min to 54 min, whereas the vast majority took 10 min or less.

Detection Rates of Complete Youth Version, Case-Finding and Help Assessment Tool and Home, Education, Eating, Activities, Drugs and Alcohol, Sexuality, Suicide and Depression, Safety Assessment

The comparative detection rates are presented in Table 2. The 2 assessments had roughly similar detection rates for substance misuse and problems at home, but YouthCHAT detected significantly more issues around problems with eating or body image perception, safety, physical inactivity, and sexual health (all P<.01). HEEADSSS mental health category had a greater detection rate when compared with YouthCHAT-detected positive responses to the depression and anxiety tests (P<.01); however, no direct comparison between the assessments for only depression and anxiety rates was possible.

Effects of Randomization Order

The order in which students received YouthCHAT and HEEADSSS (condition 1 vs condition 2) made no significant difference to the duration of YouthCHAT assessment. For YouthCHAT, condition 1 took an average of 8 min 37 seconds (SD 6 min 54 seconds) and condition 2 took an average of 9 min 24 seconds (SD 8 min 26 seconds). For HEEADSSS, condition 1 took an average of 16 min 13 seconds (SD 8 min 42 seconds) and condition 2 took an average of 18 min 7 seconds (SD 9 min 49 seconds).

Similarly, the order made no significant difference to YouthCHAT-positive depression PHQ-A screen rates (both condition 1 and condition 2 had 5 positive screens; χ²_0.95=0.0) or to anxiety GAD-7 rates (condition 1 had 3 screens and condition 2 had 2 screens; χ²_0.68=0.25).

Acceptability of Youth Version, Case-Finding and Help Assessment Tool and Home, Education, Eating, Activities, Drugs and Alcohol, Sexuality, Suicide and Depression, Safety Assessment

The results of the student acceptability survey completed by 21 students are shown in Table 3. There were no significant differences.

A total of 4 key themes emerged from the analysis of the 3 nurse interviews (Table 4): (1) valuable tool, (2) difficulties with use, (3) comparing YouthCHAT with HEEADSSS, and (4) additional uses for YouthCHAT. In summary, students found YouthCHAT easy to understand, nurses liked its look and feel, it helped identify students at risk, and nurses found the summary report and the help question useful (Figures 2 and 3). Students did identify some difficulties with use, including Wi-Fi connectivity problems at times, and some students had literacy, language, or cognitive ability issues that were a barrier to its use. In comparison with HEEADSSS, nurses found YouthCHAT easier for students to answer, faster to administer, and helped students subsequently answer face-to-face questions. Nurses suggested additional uses, including repeating annually for a longitudinal picture and using opportunistically with at-risk students in all year groups.

‎View this figure

Discussion

Principal Findings and Comparison With Prior Work

Our results indicate that YouthCHAT is a timesaving, effective, and acceptable psychosocial screener for use in a high school setting. Even with the occasional Wi-Fi glitch, it was significantly faster to use.

In general, YouthCHAT had similar or significantly higher detection rates than HEEADSSS. Although the detection rate for mental health problems or distress was higher with HEEADSSS, this reflects poor mapping of the 2 assessments for this issue. YouthCHAT mental health consisted solely of positive scores for depression or anxiety on the PHQ-A or GAD-7, whereas the HEEADSSS domain also included many nonspecific issues such as low mood, distress, unresolved grief, sadness about a historical event, and difficulty sleeping; hence, it is to be expected that more students will score higher for mental distress than score positive for depression or anxiety with YouthCHAT. Rates of depression, anxiety, and substance use problems identified via YouthCHAT were in line with expectations for this age group [33,34].

Students revealed significantly more concerns via YouthCHAT than in their HEEADSSS assessment about sensitive issues such as their body weight and sexual health and safety issues such as bullying, violence, and anger. This is consistent with evidence that youth prefer to disclose sensitive information via electronic means, without fear of being judged [24-27]. Electronic screening helps them structure their thoughts and prioritize the issues for which they want help [24]. Despite the difference between groups not being statistically significant, students’ concerns regarding privacy of information and being asked questions about sensitive issues suggest that screening for psychosocial issues should always be undertaken in a careful manner and an appropriate setting.

To date, no other screening instrument has been shown to be effective for comprehensively identifying multiple psychosocial problems in young people. Reviews of individual instruments for identifying common psychological problems in young people have identified strengths and weaknesses of different psychometric tests and recommended that these instruments are reserved for targeted use within clinical settings [35,36]. Overall, 3 HEEADSSS-based electronic screeners, TickIT [37], myAssessment [38], and the Headspace Assessment Interview [39], have recently been demonstrated to be acceptable to users in hospital and youth clinic-based settings but have not been evaluated regarding their detection rates.

Given the temporally evolving and fluctuating nature of psychosocial issues during adolescence [40,41], routine (eg, annual) YouthCHAT screening is likely to increase the chance of early detection and intervention. Conducting YouthCHAT before a scheduled HEEADSSS assessment means that the latter needs to focus only on domains where YouthCHAT is positive. Such a targeted HEEADSSS approach will reduce the time taken and hence the cost of this assessment. Embedding YouthCHAT screening within a regular holistic school health check may also increase mental health literacy [42], normalize the management of psychosocial issues, and reduce stigma about help seeking [43]. Downstream benefits of early intervention may include improved social relationships, better engagement in education and employment, reduced involvement with the justice system, and lower rates of youth suicide [44].

Strengths and Limitations

Strengths of this study include the comparison of YouthCHAT with an existing means of evaluating young people for psychosocial problems, the high response rate, and the collection of both student and staff perspectives on the use of electronic screening within a school environment. The restriction of participants to 13- to 14-year-olds and 3 school nurses from a single high school limits the generalizability of our findings. Owing to the variability in the time taken to complete both tests being considerably less than anticipated (ie, smaller SDs for the time taken to complete YouthCHAT screening and HEEADSSS assessment than expected), our power to detect a difference between the 2 interventions was higher than anticipated. Furthermore, there is a clear statistical difference between the interventions based on this sample; therefore, our final sample size was sufficient to answer our primary research questions. The inclusion of predominantly Māori and Pacific Island participants is both a strength and weakness of this study. Māori and Pacific people comprise 20% and 11%, respectively, of New Zealanders aged 10 to 17 years; hence, these ethnicities are oversampled. However, Māori and Pacific Island youth have higher rates of emotional difficulties [45,46], including depression [47] and suicide [48], yet they access specialist services at lower rates than other ethnicities [49], so early identification and intervention for these youth is key. Finally, the inability to directly map all the YouthCHAT modules to the HEEADSSS assessment domains limited the scope of comparison.

Conclusions

YouthCHAT has been shown to be significantly quicker than HEEADSSS to administer, has a high detection rate of a range of psychosocial issues, and is acceptable to both students and staff. Its potential use is for both opportunistic and routine annual screening of high school students, especially those of low socioeconomic status. Next steps include its evaluation with students of different ages and in different types of school settings. Current evidence supports its use as a first-line screening instrument, which can be followed by a targeted HEEADSSS assessment where indicated.