This paper is in the following e-collection/theme issue:

Research Instruments, Questionnaires, and Tools (540) Mobile Health (mhealth) (2748) mHealth for Data Collection and Research (907) mHealth for Wellness, Behavior Change and Prevention (2729) Diagnostic Tools in Mental Health (220) Posttraumatic Stress Disorder (PTSD) (106)

Published on in Vol 22, No 9 (2020): September

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/19716, first published .
Mobile Insight in Risk, Resilience, and Online Referral (MIRROR): Psychometric Evaluation of an Online Self-Help Test

Mobile Insight in Risk, Resilience, and Online Referral (MIRROR): Psychometric Evaluation of an Online Self-Help Test

Mobile Insight in Risk, Resilience, and Online Referral (MIRROR): Psychometric Evaluation of an Online Self-Help Test

Authors of this article:

Merel Marjolein van Herpen1, 2 Author Orcid Image ;   Manon A Boeschoten3 Author Orcid Image ;   Hans te Brake1 Author Orcid Image ;   Niels van der Aa3 Author Orcid Image ;   Miranda Olff2, 4 Author Orcid Image
Article Authors Cited by (2) Tweetations (7) Metrics

    Original Paper

    1ARQ Centre of Expertise for the Impact of Disasters and Crises, Diemen, Netherlands

    2Department of Psychiatry, Amsterdam Neuroscience & Public Health, Amsterdam University Medical Center, Amsterdam, Netherlands

    3ARQ Centre ‘45, Oegstgeest/Diemen, Netherlands

    4ARQ National Psychotrauma Centre, Diemen, Netherlands

    *these authors contributed equally

    Corresponding Author:

    Merel Marjolein van Herpen, MA

    ARQ Centre of Expertise for the Impact of Disasters and Crises

    Nienoord 5

    Diemen,

    Netherlands

    Phone: 31 610082023

    Email: m.van.herpen@impact.arq.org


    Related ArticleThis is a corrected version. See correction statement in: https://www.jmir.org/2021/6/e29776

    Background: Most people who experience a potentially traumatic event (PTE) recover on their own. A small group of individuals develops psychological complaints, but this is often not detected in time or guidance to care is suboptimal. To identify these individuals and encourage them to seek help, a web-based self-help test called Mobile Insight in Risk, Resilience, and Online Referral (MIRROR) was developed. MIRROR takes an innovative approach since it integrates both negative and positive outcomes of PTEs and time since the event and provides direct feedback to the user.

    Objective: The goal of this study was to assess MIRROR’s use, examine its psychometric properties (factor structure, internal consistency, and convergent and divergent validity), and evaluate how well it classifies respondents into different outcome categories compared with reference measures.

    Methods: MIRROR was embedded in the website of Victim Support Netherlands so visitors could use it. We compared MIRROR’s outcomes to reference measures of PTSD symptoms (PTSD Checklist for DSM-5), depression, anxiety, stress (Depression Anxiety Stress Scale–21), psychological resilience (Resilience Evaluation Scale), and positive mental health (Mental Health Continuum Short Form).

    Results: In 6 months, 1112 respondents completed MIRROR, of whom 663 also completed the reference measures. Results showed good internal consistency (interitem correlations range .24 to .55, corrected item-total correlations range .30 to .54, and Cronbach alpha coefficient range .62 to .68), and convergent and divergent validity (Pearson correlations range –.259 to .665). Exploratory and confirmatory factor analyses (EFA+CFA) yielded a 2-factor model with good model fit (CFA model fit indices: χ219=107.8, P<.001, CFI=.965, TLI=.948, RMSEA=.065), conceptual meaning, and parsimony. MIRROR correctly classified respondents into different outcome categories compared with the reference measures.

    Conclusions: MIRROR is a valid and reliable self-help test to identify negative (PTSD complaints) and positive outcomes (psychosocial functioning and resilience) of PTEs. MIRROR is an easily accessible online tool that can help people who have experienced a PTE to timely identify psychological complaints and find appropriate support, a tool that might be highly needed in times like the coronavirus pandemic.

    J Med Internet Res 2020;22(9):e19716

    doi:10.2196/19716

    Keywords

    potentially traumatic events; mobile mental health; self-help; online; resilience; posttraumatic stress disorder 


    Most people will experience at least one potentially traumatic event (PTE) in their lives [1-5]. The impact of PTEs is not the same for every individual. Research shows that most individuals are able to maintain a healthy level of functioning or resilience after experiencing a PTE and psychological complaints usually diminish over time without professional support [1,6-10]. However, a small but significant group of individuals develops psychological complaints such as posttraumatic stress disorder (PTSD) that require care [2].

    Experiencing psychological complaints a few days to weeks after a PTE is often considered normal [11-13]. The National Institute for Health and Care Excellence (NICE) advises to consider active monitoring—also known as watchful waiting—following a PTE (ie, regular monitoring of people with some PTSD symptoms within 1 month of the event) [14]. The European Network for Traumatic Stress (TENTS) guideline for post-disaster psychosocial care advises against formal screening of everyone affected by a PTE but stresses the importance of identifying individuals in need of support. Once PTSD has been diagnosed, early treatment is advised [14-18]. It could be concluded, then, that support for people who have experienced a PTE is necessary, preferably early, and easily accessible.

    Unfortunately, the small but significant group that develops persisting psychological complaints is often not detected in time or guidance to care is suboptimal [19,20]. Guidance to care can be hindered due to people not recognizing their symptoms or having self-stigma, which prevents them from seeking help [21-24]. In addition, health care facilities may lack the resources to be able to reach people who have experienced a PTE and identify the ones who need support [23,25]. Also, general practitioners may not recognize PTSD symptoms [26] or other psychological complaints [27].

    In order to prevent the development and persistence of trauma-related complaints, timely and accurate identification is needed [23,28]. Short and easy-to-use screening instruments could enable individuals at risk of developing psychological complaints to self-identify and monitor possible symptoms after PTEs. Moreover, providing online or mobile self-help tests can aid in timely identification of symptoms in people who have experienced a PTE, providing more information regarding normal psychological responses and encouraging help seeking [29,30].

    Multiple studies show that when one chooses to assist people who have experienced a PTE, it is important to support self-reliance and resilience [1,11,14]. Normalizing and validating emotional responses can promote the capacity to deal with these emotions [11]. Also, the extent to which individuals identify themselves as being resilient is considered to positively influence post-trauma outcomes [31,32]. Several self-report screening instruments are available to predict PTSD, such as the Trauma Screening Questionnaire, Impact of Event Scale–Revised or PTSD Checklist for the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (PCL-5) [33,34]. However, most instruments only screen for complaints and do not inquire about protective factors such as psychological resilience and psychosocial functioning [33,34]. In addition, most screening instruments do not consider the time period that has passed since the event. Such information is necessary to determine whether reported complaints can be appraised as normal given the stressful event just happened or whether referral to care is needed [14]. By not including time in classifying responses, screening can overlook or misappraise the different response trajectories that have been found after PTEs [9].

    To incorporate above guideline advice and address the aforementioned concerns in the early support of people who have experienced a PTE, Mobile Insight in Risk, Resilience, and Online Referral (MIRROR) was developed. MIRROR is a web-based self-help test with the potential to reach large groups of people who are seeking reassurance on how they are coping. MIRROR takes an innovative approach since it integrates both negative and positive outcomes of PTEs and time since the event. This was realized by creating a new questionnaire based on existing measures on resilience, functioning, and PTSD, and by developing a new algorithm that takes into account multiple factors. In compliance with NICE, TENTS, and DSM-5 guidelines [14,15,35], MIRROR’s algorithm includes the following as main weight factors: severity of complaints, time passed since event, and level of psychosocial functioning. MIRROR provides users with personal advice based on respondent answers with relevant follow-up support options such as a reminder for self-monitoring and contact information for consultation. Giving personal feedback to users is recommended to augment the use of mobile self-tests after PTEs [36]. Also, arranging active monitoring with follow-up within 1 month is advised [14]. Of relevance, no difference has been found between responses on a PTSD self-report administered via a mobile device versus paper administration [37]. MIRROR aims to contribute to the early identification of those likely to develop psychological complaints and encourage them to seek help. At the same time, MIRROR aims to support self-reliance by facilitating self-monitoring and self-recovery through follow-up support options.

    While it is recognized that mobile apps have the potential to improve timely identification of complaints and delivery of mental health support after PTEs, there is very little research on their validity, reliability, and effectiveness [29,30,38,39]. Therefore, the aims of this study were to assess MIRROR’s use, examine MIRROR’s psychometric properties (factor structure, internal consistency, and convergent and divergent validity) and evaluate how well MIRROR classifies respondents into different outcome categories compared with reference measures.


    Mobile Insight in Risk, Resilience, and Online Referral (MIRROR)

    A multidisciplinary team of professionals in the fields of psychotrauma (clinicians, researchers, and policy officers) and victim and crisis support developed MIRROR. The items and algorithm were based on existing protocols—DSM-5 and the International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) [35,40]—best practices and recommendations of the Dutch National Multidisciplinary Guideline on Psychosocial Support in Disasters and Crises [41], and international guidelines for PTSD and postdisaster psychosocial care [14,15].

    MIRROR consists of 2 parts. Part 1 includes items regarding event-related characteristics: type of event, measured with all events of the Dutch version of the Life Events Checklist for the DSM-5 (LEC-5) [42], time passed since the event (measured in weeks), and relation to the event (happened to me, learned about it, witnessed it, part of my job). Part 2 consists of 8 items divided in 3 sections. The first concerns PTSD core symptoms (4 items in total; 1 about intrusion, 2 about avoidance, and 1 about arousal). The items are developed based on the clusters in the DSM-IV, DSM-5, ICD-10, and ICD-11. Higher scores reflect more PTSD symptoms. The second concerns the item “how would you rate your present functioning (at work/home),” based on the widely used Global Assessment of Functioning (GAF) score for which higher scores reflect a higher level of functioning. The third concerns resilience (3 items in total; about social support, self-reliance, and problem solving), based on the resilience concept as introduced by Van der Meer et al [43]. Higher scores reflect more resilience. PTSD and resilience items are answered on a 5-point response scale, ranging from 1 (never) to 5 (all the time). Functioning is rated on a scale from 1 to 10.

    MIRROR’s algorithm aims to identify PTSD symptoms, psychosocial functioning, and resilience; normalize complaints (ie, reassuring users that it is normal to experience distress shortly after a PTE); and stimulate seeking support in users with persisting complaints. See Multimedia Appendix 1 for an overview of the possible outcomes of the algorithm. In the algorithm, MIRROR’s PTSD scale and functioning item are classified in 3 levels: low, moderate, and high. Resilience is categorized as either low or high. The categorizations are based on the aforementioned existing protocols and best practices. MIRROR’s algorithm differentiates 3 phases of time passed since the event: (1) less than 1 week ago, (2) between 1 and 4 weeks, and (3) more than 4 weeks or reoccurring. These were based on the assumption that complaints after PTEs may occur but will generally diminish over time, as most people recover on their own [6]. Therefore, the occurrence of PTSD core complaints with moderate to low functioning shortly after an adverse event can be seen as normal [11-13], but if complaints and moderate to low functioning are present after 1 month, guidance to care is needed [14-18].

    MIRROR summarizes the outcome of its algorithm to respondents as either green, orange, or red. Together with this color outcome, respondents receive personal advice. The color outcome is based on the level of complaints, functioning, and time passed since the event. MIRROR’s resilience scale is not included in the color outcome because based on current research it is unclear precisely how resilience interacts with the development of PTSD complaints and functioning after PTEs. Nonetheless, resilience is integrated in the personal advice to stimulate the use of social support. If respondents score low on resilience they are encouraged to seek support from those close to them and individuals who have experienced similar events.

    A green outcome indicates few complaints and/or sufficient functioning, and the accompanying advice states no further action is needed. An orange outcome indicates complaints and moderate functioning in combination with a PTE that happened only recently (ie, less than 1 month). The accompanying advice is directed at normalizing complaints combined with promoting watchful waiting and encouraging setting a reminder to use MIRROR again in 2 weeks to assess if complaints have diminished. The red outcome indicates significant complaints (ie, low functioning or complaints with moderate to low functioning for a longer period or due to a reoccurring event) which have persisted for more than 1 month. Therefore, the advice aims to encourage the user to seek consultation with a general practitioner or to contact Victim Support Netherlands. MIRROR provides follow-up support options with its advice, such as the opportunity to get in touch with people who have had similar experiences, reading information about dealing with stress reactions, or setting a reminder to use MIRROR again in 2 weeks.

    Participants and Procedure

    MIRROR was available in the Dutch language and open for each visitor on the website of Victim Support Netherlands (Slachtofferhulp Nederland). The specifically targeted sample consisted of website visitors who were automatically led to MIRROR when searching for information regarding stress reactions following a PTE. MIRROR is a responsive website; respondents did not have to download it. MIRROR can be used on mobile and nonmobile devices. To evaluate the psychometric properties of MIRROR, we added a research survey with reference measures (see details in Measures) after the MIRROR questions. Data collection took place during a period of 6 months. We tested the usability and technical functionality of MIRROR and the research survey before making it available. Each item was presented on a new webpage.

    Before starting MIRROR, respondents were invited to participate in the research survey. Participants were informed regarding the purpose of the study, duration time of the survey, and data storage. Participation was voluntary and completely anonymous. Respondents received no incentive for completing MIRROR or the research survey. They were asked for informed consent to use their data for research purposes, in accordance with the European General Data Protection Regulation. The Medical Ethical Committee of Amsterdam University Medical Center exempted this study from formal review (W18_364 #18.435).

    Data collection took place between February and August 2019. Only original answers were saved in the database. That is, if respondents went back to change their answers once they already received their advice, changes were not saved. We followed data cleaning recommendations by Birnbaum [44] and Wood et al [45]. Data were discarded when respondents did not complete all survey items. In case of identical answers on all items of the different reference measures, other systematic answering patterns, or obvious unusual missing answers on certain measures, we reviewed individual results thoroughly and discarded the data in case of doubt.

    Measures

    Posttraumatic Stress Disorder Symptoms

    To measure PTSD symptoms, we used the Dutch version of the PCL-5 [46,47]. The PCL-5 consists of 20 items and measures symptoms of intrusion (cluster B, 5 items), avoidance (cluster C, 2 items), negative alterations in cognitions and mood (cluster D, 7 items), and alterations in arousal and reactivity (cluster E, 6 items) in the past month. All items are answered on a 5-point scale, ranging from 0 (not at all) to 4 (extremely). The PCL-5 showed good psychometric properties in different languages [48-50]. The total score was calculated by adding all item scores. Scale scores per cluster were calculated by adding the scores of the corresponding items. Higher scores reflect more severe symptoms. Cronbach alphas in our sample ranged between .77 and .86 for the B, C, D, and E clusters. The DSM-5 rule to determine a provisional PTSD diagnosis was followed. This entails treating each item with a minimum score of 2 as a symptom endorsed and requiring at least one B symptom, one C symptom, two D symptoms, and two E symptoms [46].

    Depression, Anxiety, and Stress

    To assess other common psychological complaints after PTEs, we used the Dutch short version of the Depression Anxiety Stress Scale (DASS-21) measuring depression (7 items), anxiety (7 items), and stress (7 items) [51,52]. The DASS-21 is a valid and reliable measure [53,54]. Item scores were summed to calculate scale scores and the total score. Higher scores reflect more severe symptoms. In our sample, Cronbach alphas were .92, .86, and .86 for depression, anxiety, and stress scales, respectively. A 4-point response scale measures the extent to which each state has been experienced over the past week ranging from 0 (did not apply to me at all) to 3 (applied to me very much, or most of the time). To determine cutoff values, DASS-21 scale scores were multiplied by two, in accordance with the scale’s manual [52]. The manual provides cutoff scores for a Dutch clinical sample. These discriminate the following categories: normal (depression <9, anxiety <7, stress <14), mild (depression 10-13, anxiety 8-9, stress 15-18), moderate (depression 14-20, anxiety 10-14, stress 19-25), severe (depression 21-27, anxiety 15-19, stress 26-33) and extremely severe (depression >28, anxiety >20, stress >34).

    Psychological Resilience

    We used the Resilience Evaluation Scale (RES) to assess psychological resilience [43]. The 9 items are rated on a 5-point scale ranging from 0 (strongly disagree) to 4 (strongly agree). We calculated the total score by adding all items. Higher scores reflect more psychological resilience. The RES is a valid and reliable measure [43]. In this sample, Cronbach alpha of the total scale was .88.

    Positive Mental Health

    We assessed positive mental health with the Dutch version of the Mental Health Continuum Short Form (MHC-SF) [55,56]. The MHC-SF measures emotional well-being (3 items), social well-being (5 items), and psychological well-being (6 items). Items were rated on a 6-point scale ranging from 0 (never) to 5 (every day). The MHC-SF is a valid and reliable instrument [56,57]. We calculated the total score by summing all item scores. Higher scores reflect more positive mental health. In this sample, Cronbach alpha of the total scale was .93.

    Google Analytics

    Google Analytics data were collected between March and August 2019 to examine MIRROR’s use. Due to technical problems, data from February 2019 were missing. The data provide information on the number of unique visits per page, type of device used, and number of visitors who have started MIRROR (defined as a unique page visit on MIRROR’s start page) and who have finished MIRROR (defined as unique page visit on MIRROR’s outcome and advice page). Google Analytics cannot determine to what extent the follow-up options were used, but it can detect how many respondents have visited the follow-up support option pages.

    Statistical Analyses

    Sample and Use

    Since participation in the research survey was optional, this resulted in two 2 samples. The MIRROR-only sample consists of respondents who only completed MIRROR. The validation sample includes respondents who completed MIRROR and the accompanying survey with reference measures before receiving their advice. The total sample combines these two samples, consisting of all respondents. To examine if the validation sample was representative of the MIRROR user, we used independent-samples t tests in SPSS Statistics version 23 (IBM Corporation) to compare the MIRROR-only sample with the validation sample based on their MIRROR scores and event-related characteristics.

    We used the total sample to evaluate MIRROR’s use and examine MIRROR’s factor structure and internal consistency because for these analyses only data from MIRROR were needed. We used the validation sample to examine MIRROR’s convergent and divergent validity and evaluate how well MIRROR classifies respondents into different outcome categories because for these analyses data from MIRROR as well as reference measures from the accompanied survey were needed.

    Factor Structure

    We used Mplus version 8 (Muthen & Muthen) to conduct exploratory factor analysis (EFA) using geomin rotation and confirmatory analysis (CFA). EFA assumes that any item may be associated with any factor. CFA specifies expected relationships between items and their underlying latent factors. Because items of MIRROR’s PTSD and resilience section were categorical, they were treated as ordinal and therefore the means and variance adjusted weighted least square (WLSMV) estimator was used. An underlying normal distribution was assumed for each ordinal item, where the 5 response categories were divided by 4 thresholds estimated from the data. MIRROR’s functioning item has 10 response categories and was treated as continuous. Because MIRROR’s factor structure was not tested before, several models with different numbers of latent factors were examined using EFA. To assess the model with the optimal number of latent factors needed to adequately account for the correlations among item scores, we used Kaiser criterion (ie, eigenvalues of the latent factors >1) and model fit statistics. The model with the best balance between model fit, parsimony, and conceptual interpretability was selected as the most optimal model. Subsequently, CFA was used to test the optimal model based on EFA. The difference in goodness-of-fit between nested models was evaluated with the difftest option in Mplus for appropriate chi-square difference testing with the WLSMV estimator [58]. The chi-square difference test is highly sensitive to sample size such that even trivial differences between two nested models may be significant [59]. Therefore, we also assessed the difference in comparative fit index (CFI). A difference in CFI <0.01 indicates a better fit of the nested model compared with the more complex model [59]. For EFA and CFA, the model fit indices CFI, Tucker-Lewis index (TLI), and root mean square error of approximation (RMSEA) were used to evaluate model fit. Model fit can be considered good when CFI and TLI are close to .95, and RMSEA <.06 [60]. If RMSEA <.08, model fit can be considered adequate [60].

    Internal Consistency

    We evaluated internal consistency of MIRROR’s PTSD and resilience section with interitem correlations, corrected item-total correlations, and Cronbach alpha in SPSS Statistics version 23. Internal consistency of MIRROR’s functioning section could not be evaluated since it is represented by only one item. When most interitem correlations are in the recommended range of .15 to .50 (moderate magnitude) and Cronbach alpha for the scale is >.80, internal consistency can be considered good [61]. Cronbach alpha is a function of scale length and therefore is likely to be lower for MIRROR’s scales since they consist of 3 or 4 items [61]. Corrected item-total correlations were computed to assess whether item scores regarding PTSD and resilience are associated with overall PTSD and resilience scores.

    Convergent and Divergent Validity

    To evaluate MIRROR’s convergent and divergent validity, we calculated Pearson correlations between the MIRROR scales and reference measures. Convergent and divergent validity can be considered good when the correlations between a scale and equivalent measure (eg, MIRROR’s PTSD scale and the PTSD scale of the PCL-5) are significant and high while correlations between this scale and other related measures (eg, MIRROR’s PTSD scale and depression scale of the DASS-21) are lower and moderate or modest in magnitude.

    Classification Quality

    To evaluate how well MIRROR classifies respondents into a red, orange, or green outcome, we tested whether respondents in these three outcome categories differed on related reference measures by using cross-tabs and analysis of variance (ANOVA). If the assumption of equal variances was violated, we used the Welch F-test and Games-Howell post hoc test. MIRROR’s PTSD scale score was calculated by summing the 4 PTSD items. Higher scores reflect more severe symptoms. MIRROR’s resilience scale score was calculated by a summing the 3 items. Higher scores reflect more resilience. Provisional PTSD diagnosis based on PCL-5 were used to classify respondents. To examine the distribution on depression, anxiety and stress symptoms, respondents were classified by comparing their scores to a Dutch clinical reference group. Respondents with normal and mild complaints compared with the reference group were classified into one group representing subclinical complaints. Respondents with average, severe, and very severe complaints compared with the reference group were classified into another group, representing clinical complaints. Since no reference groups were available with regard to the RES and MHC-SF, the sample was divided into tertiles (ie, 3 groups of equal size divided by the 33rd and 66th percentile) based on the total scores of the RES and MHC-SF. With regard to the RES, the first tertile (scores ≤17) was assumed to represent relatively low psychological resilience, the second tertile (scores from 18 to 24) relatively moderate psychological resilience, and the third tertile (scores ≥25) relatively high psychological resilience. With regard to the MHC-SF, the first (scores ≤23), second (scores from 24 to 47), and third tertile (scores ≥48) were assumed to represent relatively low, moderate, and high positive mental health, respectively.


    Sample and Use

    MIRROR was completed 1314 times in the study period of 6 months. In total, 51.90% (682/1314) of respondents started the research survey. We deleted 51 respondents who indicated they used MIRROR on behalf of a family member, partner, friend, or colleague who experienced a PTE. We deleted 37 repeated measurements, completed by respondents who set a reminder. We excluded 95 respondents because they did not complete all research survey items. After thorough investigation of the answering patterns, we deleted 19 respondents because of unusual answering patterns. A total of 84.63% (1112/1314) of respondents were retained in the total sample, of whom 59.62% (validation sample, 663/1112) also completed all questionnaires of the accompanying research survey.

    Table 1 presents the MIRROR scores, outcomes, and event-related characteristics for the MIRROR-only and validation sample. We found no significant difference between the samples on MIRROR’s PTSD scale: t1110=–.401, P=.69; resilience scale: t1110=.752, P=.45; or level of functioning t1110=1.547, P=.12. We found a significant association between sample and MIRROR outcome: χ22,n=1112=18.99, P<.001; the validation sample consisted of more respondents with the red MIRROR outcome than the MIRROR-only sample. The event-related characteristics for both samples were similar, see Table 1. Overall, the validation sample can be considered representative of all MIRROR users in this study period. In the validation sample, 74.2% (492/663) of respondents were female. Almost half (300/663, 45.3%) of respondents were aged between 21 and 40 years. Tables 2 and 3 present the frequency distributions for MIRROR’s response categories.

    Table 1. Mobile Insight in Risk, Resilience and Online Referral (MIRROR) scores, outcomes, and event-related characteristics for the validation sample and MIRROR-only sample.
    MIRRORaValidationb (n=663)MIRRORb only (n=449)
    MIRROR scores, mean (SD)


    		MIRROR PTSDc scale14.88 (3.39)14.80 (3.28)

    		MIRROR functioning4.92 (1.96)5.11 (1.94)

    		MIRROR resilience scale10.08 (2.36)10.91 (2.37)
    MIRRORa outcomeb, n (%)


    		Red409 (61.7)224 (49.9)

    		Orange230 (34.7)214 (47.7)

    		Green24 (3.6)11 (2.4)
    Type of event (LEC-5d), n (%)


    		Another very stressful event or experience216 (32.6)150 (33.4)

    		Transportation accident115 (17.4)107 (23.8)

    		Physical assault109 (16.5)50 (11.1)

    		Sudden accidental death38 (5.7)20 (4.5)

    		Serious accident at work, home, or during recreation33 (5.0)28 (6.2)

    		Sexual assault33 (5.0)18 (4.0)

    		Assault with a weapon30 (4.5)25 (5.6)

    		Other unwanted or uncomfortable sexual experience30 (4.5)14 (3.1)

    		Sudden violent death24 (3.6)16(3.6)

    		Severe human suffering14 (2.1)5 (1.1)

    		Life-threatening illness or injury10 (1.5)5 (1.1)

    		Fire or explosion9 (1.4)4 (0.9)

    		Combat or exposure to a war zone1 (0.2)0 (0)

    		Captivity0 (0)4 (0.9)

    		Serious injury, harm, or death caused by you to someone else0 (0)3 (0.7)

    		Natural disaster0 (0)0 (0)
    Relation to the event, n (%)


    		Event happened to me480 (72.5)311 (69.3)

    		I witnessed the event129 (19.5)94 (20.9)

    		I learned about the event42 (6.3)35 (7.8)

    		Othere11 (1.7)9 (2.0)
    Work-related, n (%)


    		No586 (88.4)379 (84.4)

    		Yes77 (11.6)70 (15.6)
    Time since the event, n (%)


    		Less than 1 week241 (36.3)218 (48.6)

    		Over 4 weeks214 (32.3)113 (25.2)

    		Between 1 and 4 weeks144 (21.7)90 (20.0)

    		It happens repeatedly64 (9.7)28 (6.2)

    aMIRROR: Mobile Insight in Risk, Resilience, and Online Referral.

    bSignificant association between sample and MIRROR outcome, P<.001.

    cPTSD: posttraumatic stress disorder.

    dLEC-5: Life Events Checklist for DSM-5.

    eIf respondents could not select one of the event relations (happened to me, witnessed it, learned about it, work-related), they are asked to specify their relation to the event.

    Table 2. Frequency distribution in percentages of Mobile Insight in Risk, Resilience and Online Referral (MIRROR) item response categories, items 1-4 and 6-8 (n=1112).
    Scale and item numberNeverRarelySometimesOftenAll the time
    PTSDa





    		12.75.716.638.536.5

    		25.18.519.327.439.6

    		39.313.826.922.927.1

    		48.511.426.726.826.6
    Resilience





    		65.38.521.735.329.3

    		77.315.635.230.511.5

    		85.215.145.628.15.9

    aPTSD: posttraumatic stress disorder.

    Table 3. Frequency distribution in percentages of Mobile Insight in Risk, Resilience and Online Referral (MIRROR) item response categories, item 5 (n=1112).
    Scale and item number12345678910
    Functioning










    		54.96.910.415.819.820.912.16.31.61.3

    A detailed overview of the scores of the validation sample on the reference measures can be found in Multimedia Appendix 2. Overall, these show a high level of complaints in our sample and rather low levels of psychological resilience and positive mental health (also see Table 7 and Figure 1 for reference measures of each MIRROR outcome category).

    Google Analytics data provided insight into MIRROR’s use. The number of visitors who started MIRROR was 2555, of whom 2247 (87.95%) finished it. The original database contained 1314 entries. This discrepancy can be explained by users having the opportunity to refuse to have their data saved before starting. Of all users, 47.59% (1216/2555) chose this option. Furthermore, of the follow-up support options, the “seek contact with Victim Support Netherlands” page had most views (411 unique views), followed by “more information” (293 unique views), “send your advice to yourself or someone else” (235 unique views), “seek contact with people who have had similar experiences” (209 unique views), and “set a reminder” (161 unique views). A total of 28.7% (113/394) of respondents who received the orange outcome and were advised to complete MIRROR again in 2 weeks immediately set a reminder to complete MIRROR again in 2 weeks. A total of 22.1% (25/113) did so at the time of data analyses. The most often used device was the smartphone (1566/2555, 61.29%), followed by desktop (794/2555, 31.08%), and tablet (195/2555, 7.63%).

    Factor Structure

    Table 4 presents the factor loadings for the 2-factor and 3-factor solution model of MIRROR as estimated by EFA. EFA yielded a 3-factor solution with good model fit based on all fit indices. The Kaiser criterion was met for the first 2 factors, eigenvalues of the third through eighth factor were <1. The 3-factor solution separated MIRROR’s PTSD items into 2 factors: 1 with the intrusion item and 1 with the avoidance and arousal/reactivity items. However, item 2 (“have you become jumpy and/or vigilant since the event?”) cross-loaded significantly on 2 factors within the model, with only a small difference between the 2 factor loadings (λ=0.030). This indicates that item 2 did not sufficiently distinguish between both factors. The 3-factor solution clustered the functioning item with the resilience items into a third factor.

    Table 4. Geomin rotated factor loadings for the 2-factor and 3-factor solution model of Mobile Insight in Risk, Resilience and Online Referral (MIRROR) as estimated by exploratory factor analysis (n=1112).
    MIRRORa items2-factor solutionb3-factor solutionc

    		F1F2F1F2F3
    1. Are you troubled by images of or thoughts about the event?d0.525*–0.0040.813*0.0150.018
    2. Have you become jumpy and/or vigilant since the event?e0.585*–0.0090.308*0.338*–0.012
    3. Do you try to avoid things that are related to the event?f0.789*0.071–0.0001.078*0.245*
    4. Do you try to avoid thinking about the event?g0.648*–0.0160.208*0.459*–0.019
    5. How would you rate your present functioning (at work/home)?h–0.153*0.354*–0.213*0.0040.360*
    6. Do you experience support from those close to you?i0.081*0.388*0.160*–0.0640.374*
    7. Are you confident in yourself?j0.0060.827*0.010–0.0210.827*
    8. Are you able to deal with any problems you encounter?k–0.0150.730*–0.0740.0180.718*

    aMobile Insight in Risk, Resilience and Online Referral.

    bModel fit indices for the 2-factor solution: χ213=88.7, P<.001, CFI=.969, TLI=.933, RMSEA=.072.

    cModel fit indices for the 3-factor solution: χ27=12.6, P=.084, CFI=.998, TLI=.991, RMSEA=.027.

    dEigenvalue 2.777,

    eEigenvalue 1.466.

    fEigenvalue .927.

    gEigenvalue .715.

    hEigenvalue .668.

    iEigenvalue .640.

    jEigenvalue .437.

    kEigenvalue .369.

    *P<.05.

    EFA yielded a 2-factor solution with adequate model fit. The RMSEA and TLI indicated adequate model fit and CFI indicated good model fit (Table 4). The Kaiser criterion was met for the first 2 factors; eigenvalues of the third through eighth factor were <1. The first factor of the 2-factor solution consisted of the PTSD items and the second factor consisted of the functioning and resilience items. No cross-loadings were observed in this model.

    Next, we conducted CFA to further compare the 2- and 3-factor model that resulted from EFA. Table 5 presents the model fit indices based on CFA of both aforementioned models. The model fit indices were similar for both models; the CFI and TLI indicated good model fit, the RMSEA acceptable model fit. As indicated by the significant χ2 difference test, the 2-factor model has worse model fit compared with the 3-factor model (χ22,n=1112=13.63, P=.001). However, the difference in CFI is <0.01, indicating the 2-factor model does not have worse model fit. We selected the 2-factor model as the best-fitting model to our data, given the χ2 difference test is sensitive to sample size, the CFI difference is <.001, and it is more parsimonious and better interpretable at a conceptual level compared with the 3-factor model. The 2-factor model represents a clear distinction between negatively formulated outcomes (PTSD complaints) and positively formulated outcomes (psychosocial functioning and resilience) of PTEs. The positively formulated outcomes combine psychosocial functioning, social support, self-reliance and problem solving. We therefore propose to rename this factor psychosocial resources.

    Table 5. Confirmatory factor analysis model fit indices (n=1112).
    Modelχ2P valuedfaCFIbTLIcRMSEAd
    Two-factor solution107.78<.001190.9650.9480.065
    Three-factor solution95.868<.001170.9690.9490.064

    adf: degree of freedom.

    bCFI: comparative fit index.

    cTLI: Tucker-Lewis index.

    dRMSEA: root mean square error of approximation.

    Internal Consistency

    Interitem correlations of MIRROR’s PTSD complaints scale ranged between .28 and .48 with a mean of .34. All of the interitem correlations of the PTSD scale were in the recommended range of moderate magnitude of .15 to .50, indicating that this scale has high internal consistency in combination with a differentiated item set. Corrected item-total correlations for this scale ranged between .39 and .54 with a mean of .46, indicating that high scores on the PTSD items are associated with high scores on the overall PTSD scale of MIRROR. Cronbach alpha coefficient for MIRROR’s PTSD scale was .68.

    Interitem correlations of MIRROR’s resilience scale ranged between .24 and .55, with a mean of .36. In addition, 1 out of 3 interitem correlations was higher than the recommended range of moderate magnitude of .15 to .50 (between “are you confident in yourself” and “are you able to deal with any problems you encounter”), indicating that this scale has high internal consistency in combination with a differentiated item set. Corrected item-total correlations ranged between .30 and .52 with a mean of .44, indicating that high scores on the resilience items are associated with high scores on the overall resilience scale of MIRROR. Cronbach alpha coefficient for MIRROR’s resilience scale was .62.

    Convergent and Divergent Validity

    Pearson correlations between MIRROR and reference measures are presented in Table 6. MIRROR’s PTSD scale showed strongest correlations with PTSD as measured with the PCL-5, followed by a lower but still substantial correlation with psychological complaints as assessed with the DASS-21. The weakest correlations were observed between PTSD symptom severity as assessed with MIRROR and psychological resilience and positive mental health. MIRROR’s resilience scale showed strongest correlation with psychological resilience (RES), followed by a slightly lower correlation with positive mental health, psychological complaints (DASS-21), and PTSD (PCL-5). MIRROR’s functioning item showed strongest correlations with psychological complaints (DASS-21) followed by PTSD (PCL-5) with lower correlations with positive mental health (MHC-SF) and psychological resilience (RES). In conclusion, the correlational structure indicates good convergent and divergent validity of MIRROR’s PTSD subscale. The correlational structure with regard to MIRROR’s resilience scale and functioning item indicates adequate convergent and divergent validity.

    Table 6. Correlations between Mobile Insight in Risk, Resilience and Online Referral (MIRROR) subscales and reference measures (n=663).
    MIRRORPTSDaP valueResilienceP valueFunctioningP value
    PCL-5b.665<.001–.507<.001–.442<.001
    DASS-21c.486<.001–.539<.001–.449<.001
    RESd–.265<.001.612<.001.279<.001
    MHC-SFe–.259<.001.603<.001.319<.001

    aPTSD: posttraumatic stress disorder.

    bPCL-5: PTSD Checklist for DSM-5.

    cDASS-21: Depression Anxiety Stress scale.

    dRES: Resilience Evaluation Scale.

    eMHC-SF: Mental Health Continuum Short Form.

    Classification Quality

    We expected respondents with the red MIRROR outcome to report more PTSD symptoms and depression, anxiety, and stress complaints; lower psychological resilience; and positive mental health compared with respondents with the green and orange MIRROR outcome. Table 7 presents the means and standard deviations on the reference measures for each MIRROR outcome category. Figure 1 shows the classification percentages on reference measures for each MIRROR outcome category. Both Table 7 and Figure 1 show that respondents with the red MIRROR outcome category report higher complaints and lower psychological resilience and positive mental health compared with the orange and green MIRROR outcome category.

    Table 7. Means and standard deviations of reference measures for each Mobile Insight in Risk, Resilience and Online Referral (MIRROR) outcome category (n=663).
    MIRRORa outcome category (n)Green (n=24), mean (SD)Orange (n=200), mean (SD)Red (n=439), mean (SD)
    PTSDb (PCL-5c)18.04 (12.49)36.09 (15.77)46.13 (14.04)
    Depression (DASS-21d)4.08 (8.10)11.73 (11.54)19.66 (11.54)
    Anxiety (DASS-21)5.25 (6.72)14.03 (10.27)18.04 (10.30)
    Stress (DASS-21)10.42 (7.32)17.60 (9.20)22.49 (9.37)
    Psychological resilience (RESe)25.58 (5.11)22.04 (6.02)18.82 (7.15)
    Positive mental health (MHC-SFf)50.0 (12.05)43.11 (14.89)31.42 (14.28)

    aMIRROR: Mobile Insight in Risk, Resilience and Online Referral.

    bPTSD: posttraumatic stress disorder.

    cPCL-5: PTSD Checklist for DSM-5.

    dDASS-21: Depression Anxiety Stress Scale.

    eRES: Resilience Evaluation Scale.

    fMHC-SF: Mental Health Continuum Short Form.

    ‎
    Figure 1. Classification percentages on reference measures of each Mobile Insight in Risk, Resilience, and Online Referral (MIRROR) outcome category.
    View this figure

    We conducted several 1-way between-groups ANOVAs to investigate the difference in mean scores on the reference measures between MIRROR outcome categories. As can be seen, negative outcomes were highest for the red MIRROR outcome category and positive outcomes highest for the green outcome category. The ANOVA results are shown in Table 8. We found significant differences in PTSD symptoms; depression anxiety, and stress; psychological resilience; and positive mental health between groups. Post hoc tests revealed that PTSD symptoms and depression, anxiety, and stress complaints were significantly different between all groups (P<.001). Psychological resilience was significantly higher for the green and orange MIRROR outcome category compared with the red category (P<.001). It was also significantly higher for the green category compared with the orange category (P=.01). Positive mental health was significantly higher for the green and orange category compared with the red category (P<.001). There was no significant difference between the green and orange category (P=.07).

    Table 8. One-way between-groups analyses of variance with Mobile Insight in Risk, Resilience and Online Referral (MIRROR) outcome categories and reference measures.
    Analysis of varianceF-testCohen ddfa between groupsdf within groupsP value
    PTSDb symptomsc73.32.168262.90<.001
    Depressionc65.21.136265.81<.001
    Anxietyc42.48.072267.37<.001
    Stress34.15.0942660.0<.001
    Psychological resiliencec30.13.068265.44<.001
    Positive mental health57.79.0692660.00<.001

    adf: degree of freedom.

    bPTSD: posttraumatic stress disorder.

    cThe assumption of equal variances was violated. Therefore, the Welch F-test and Games-Howell post hoc test were used.


    Principal Findings and Comparison With Prior Work

    The purpose of this study was to evaluate the use and psychometric and classification properties of MIRROR. MIRROR is an innovative web-based self-help test to identify individuals who develop psychological complaints after a PTE, encourage them to seek help, and support self-reliance. Our results indicated that MIRROR is a valid and reliable self-help test to identify negative outcomes (PTSD core symptoms) and positive outcomes (psychosocial functioning and resilience). MIRROR is able to correctly classify respondents according to their PTSD complaints and scores on reference measures. During the study period, 87.95% (2247/2555) of respondents who started MIRROR completed it.

    We found that MIRROR’s presupposed model of 3 factors (PTSD symptoms, psychosocial functioning, and resilience) did not fit our data best. Instead, a 2-factor solution showed good model fit, conceptual meaning, and maximum parsimony. This model separates MIRROR’s PTSD items from the functioning and resilience items (social support, self-reliance, and problem solving). In retrospect, the grouping of the functioning and resilience items is not entirely surprising. If we assume stress to be the result of an imbalance between perceived external and internal demands and perceived personal and social resources [62], it is likely that this distinction between demands and resources is reflected in the way people cope with PTEs. We propose to call the factor psychosocial resources. In accordance with this distinction, the 2-factor model clearly separates negative (PTSD complaints) and positive (psychosocial resources) outcomes of PTEs. This is in line with the general notion that PTSD and psychosocial resources are separate constructs [63-65].

    The convergent and divergent validity of MIRROR is supported by the correlations that were found between MIRROR and the reference measures. The results indicate good convergent and divergent validity for MIRROR’s PTSD items. As expected, MIRROR’s PTSD showed strongest correlations with PTSD (assessed with the PCL-5), followed by a lower but substantial correlation with psychological complaints (measured with the DASS-21). MIRROR’s PTSD items showed low correlations with positive reference measures (assessed with the RES and MHC-SF). The results indicate adequate convergent and divergent validity for MIRROR’s resilience items but less distinct than MIRROR’s PTSD. MIRROR’s resilience items showed strongest correlations with psychological resilience, followed by slightly lower but substantial correlations with the other reference measures. The results in this study correspond with the finding of Van der Meer et al [43] who found the RES total scale to be positively associated with established measures for resilience, self-esteem, self-efficacy, and global functioning and negatively associated with PTSD symptoms. Furthermore, the different patterns of correlations for MIRROR’s PTSD and resilience scales agrees with the notion that PTSD and resilience are two separate constructs [63-65]. MIRROR’s functioning item showed the strongest correlation with psychological complaints and PTSD and lower correlations with the positive reference measures. This indicates adequate convergent and divergent validity. The factor analyses revealed that functioning belongs to the resilience items of MIRROR. However, the correlation between MIRROR’s functioning item and psychological complaints and PTSD is in line with studies that show that psychosocial functioning can be impaired by psychological complaints [64,66,67].

    We found that both MIRROR’s PTSD and resilience scales show good internal consistency. The Cronbach alpha coefficients for these scales are relatively low (.68 and .62, respectively), but this is not unusual given the (intentionally) short scales of MIRROR and given that Cronbach alpha is a function of scale length [61]. Because MIRROR contains only few items, we calculated interitem and item-total correlations. The results indicate that both scales have high internal consistency and that high scores on the items are associated with high scores on the overall scales.

    MIRROR was able to correctly classify respondents into green (no further action needed), orange (encourage self-monitoring), or red (encourage seeking consultation) outcome categories and advice compared with the other measures. Results showed that respondents with a red outcome reported having more severe PTSD symptoms; more severe depression, anxiety, and stress complaints; and lower psychological resilience and positive mental health compared with respondents with a green or orange outcome. The occurrence of PTSD and other stress-related complaints like depression following traumatic exposure is in line with former results [68]. It is important to recognize that MIRROR is specifically evaluating the risk of developing PTSD instead of other mental health outcomes of PTEs such as depression, anxiety, and substance abuse. If a respondent experiences low functioning, they will receive advice to seek consultation with their general practitioner despite the level of their PTSD complaints. This is based on the assumption that low functioning but no PTSD complaints may indicate that other problems could be at hand such as depression, anxiety, or substance abuse. Importantly, MIRROR appears to adequately identify users with more severe complaints and validly advises them to seek help. Our results seem to underline the relevance of including the factor “time since the event” in MIRROR’s algorithm. According to the PCL-5, 54.5% (109/200) of the respondents with the orange outcome had a provisional PTSD diagnosis. However, their complaints could still diminish, considering the event happened only recently for these respondents and research has shown that in most individuals complaints usually diminish over time [1,2,11]. Therefore, in accordance with international guidelines [14], respondents with the orange outcome are advised to monitor how their complaints develop (by setting a reminder to use MIRROR again in 2 weeks).

    The evaluation of MIRROR’s use with Google Analytics showed that the number of users of MIRROR was substantial (n=2555), and the completion rate was high (2247/2555, 87.95%). These results are in line with former studies on apps assessing and monitoring mental health after PTEs indicating high use [29,30,36] and high completion rate [49]. In general, the follow-up options were visited less frequently (161 to 411 unique visits) than the outcome and advice page (2247 unique visits). A reason for this could be that receiving MIRROR’s outcome and advice is sufficient initial support for people who have experienced a PTE, providing insight into how they are coping. A total of 28.7% (113/194) of respondents who were advised to complete MIRROR again in 2 weeks immediately set a reminder, suggesting MIRROR is able to support self-monitoring. Unfortunately, this study’s design and considerations of ethical nature did not enable us to assess use in more depth.

    Future Research and Limitations

    Although guidelines on screening for PTSD complaints and postdisaster psychosocial care are widely available [7,15,69-71], the challenge remains how to reach and identify people at risk of developing psychological complaints after a PTE on a large scale. Future research could focus on investigating the implementation of MIRROR on a larger scale—for example, after terrorist attacks or natural disasters. Literature is inconclusive about the benefits versus disadvantages of formal screening of an entire population after a disaster or crisis [14,15,69,72]. Because of limited evidence of effectivity and sensitivity of screening, organizational efforts related to screening, and the often scarce resources available [25,73], it is generally not recommended to perform formal screening of complaints among all involved people following incidents. At the same time, we know that early recognition and timely referral to help are essential for preventing and treating traumatic stress symptoms. This is supported by evidence of the effectiveness of early psychological interventions for individuals prescreened with traumatic stress symptoms shortly following trauma and no benefits in those not prescreened for these symptoms [16]. Mobile apps such as MIRROR can make a contribution to solving the screening dilemma by supporting low key, accessible, and easy-to-use self-assessment and -monitoring. In this view, MIRROR could be implemented as a first step in the support for people who have experienced a PTE, before having to consult professional care [29,36]. MIRROR might lower the barrier to seek help given its open accessibility and anonymity. Future research could focus on acquiring longitudinal data of MIRROR to assess the development of complaints, functioning, and resilience over time and establish MIRROR’s ability to correctly classify users accordingly. Also, qualitative research might clarify what actions users take as a result of MIRROR’s personal advice.

    Our study has some limitations. In our validation sample, 74.2% (492/663) of respondents were female, and 45.3% (300/663) of respondents were aged between 21 and 40 years. This could lead to selection bias and limited generalizability of the results, which is common with open internet surveys [74]. However, our sample is a specifically targeted sample because it consisted of visitors of the website of Victim Support Netherlands. Considering website visitors were automatically led to MIRROR when searching for information regarding stress reactions following a PTE, a high prevalence of psychological complaints after traumatic exposure in our sample could be expected. Moreover, research has shown that women have a higher risk of developing PTSD compared with men [75], they are more likely to seek medical or health-related information online [76], and young people use the internet as their main source of information, and this is also true for mental health concerns [77,78]. This demonstrates that the targeted sample was reached. The main strength of this study is by comparing MIRROR to more broadly used reference measures, it contributes to the highly needed evidence base of mobile apps with the potential to improve timely identification of psychological complaints [29,30,79].

    Conclusions

    This study shows that MIRROR is a psychometrically sound, anonymous, and easily accessible self-help test for people who have experienced a PTE. It is able to identify both negative (PTSD symptoms) and positive (psychosocial resources) outcomes of PTEs and classify respondents in accordance with reference measures. This study will hopefully contribute to enhancing adequate and timely identification of people who suffer from psychological complaints after PTEs.

    Acknowledgments

    We would like to thank Victim Support Netherlands (Slachtofferhulp Nederland) for their help in facilitating this study. This study was partially funded by the Interreg North-West Europe Programme which invested in the e-mental health innovation and transnational implementation platform North West Europe (eMEN) project, an EU-wide platform for e-mental health innovation and implementation formed by private and public partners in northwest Europe [80]. This study was also funded by ARQ National Psychotrauma Centre and Victim Support Netherlands.

    Conflicts of Interest

    This study has been conducted by the independent research center ARQ Centre of Expertise for the Impact of Disasters and Crises and ARQ Centre ’45. The funders (ARQ National Psychotrauma Centre, Interreg North-West Europe, and Victim Support Netherlands) had no influence on the outcomes of this study.

    ‎
    Multimedia Appendix 1

    Overview of outcomes in the Mobile Insight in Risk, Resilience, and Online Referral (MIRROR) instrument.

    DOCX File , 24 KB
    ‎
    Multimedia Appendix 2

    Sample characteristics on reference measures and demography (n=663).

    DOCX File , 28 KB
    1. Bonanno GA, Westphal M, Mancini AD. Resilience to loss and potential trauma. Annu Rev Clin Psychol 2011;7:511-535. [CrossRef] [Medline]
    2. de Vries G, Olff M. The lifetime prevalence of traumatic events and posttraumatic stress disorder in the Netherlands. J Trauma Stress 2009 Aug;22(4):259-267. [CrossRef] [Medline]
    3. Benjet C, Bromet E, Karam EG, Kessler RC, McLaughlin KA, Ruscio AM, et al. The epidemiology of traumatic event exposure worldwide: results from the World Mental Health Survey Consortium. Psychol Med 2016 Jan;46(2):327-343 [FREE Full text] [CrossRef] [Medline]
    4. Kessler RC, Aguilar-Gaxiola S, Alonso J, Benjet C, Bromet EJ, Cardoso G, et al. Trauma and PTSD in the WHO World Mental Health Surveys. Eur J Psychotraumatol 2017;8(sup5):1353383 [FREE Full text] [CrossRef] [Medline]
    5. Knipscheer J, Sleijpen M, Frank L, de Graaf R, Kleber R, Ten Have M, et al. Prevalence of potentially traumatic events, other life events and subsequent reactions indicative for posttraumatic stress disorder in the Netherlands: a general population study based on the trauma screening questionnaire. Int J Environ Res Public Health 2020 Mar 06;17(5):1725 [FREE Full text] [CrossRef] [Medline]
    6. Bonanno G. Resilience in the face of potential trauma. Curr Dir Psychol Sci 2005;14(3):135-138 [FREE Full text] [CrossRef]
    7. Silver RC, Holman EA, McIntosh DN, Poulin M, Gil-Rivas V. Nationwide longitudinal study of psychological responses to September 11. JAMA 2002 Sep 11;288(10):1235-1244. [CrossRef] [Medline]
    8. Forbes D, Creamer M, Phelps A, Bryant R, McFarlane A, Devilly GJ, et al. Australian guidelines for the treatment of adults with acute stress disorder and post-traumatic stress disorder. Aust N Z J Psychiatry 2007 Aug;41(8):637-648. [CrossRef] [Medline]
    9. Galatzer-Levy IR, Huang SH, Bonanno GA. Trajectories of resilience and dysfunction following potential trauma: a review and statistical evaluation. Clin Psychol Rev 2018 Jul;63:41-55. [CrossRef] [Medline]
    10. Olff M, Amstadter A, Armour C, Birkeland MS, Bui E, Cloitre M, et al. A decennial review of psychotraumatology: what did we learn and where are we going? Eur J Psychotraumatol 2019;10(1):1672948 [FREE Full text] [CrossRef] [Medline]
    11. Hobfoll SE, Watson P, Bell CC, Bryant RA, Brymer MJ, Friedman MJ, et al. Five essential elements of immediate and mid-term mass trauma intervention: empirical evidence. Psychiatry 2007;70(4):283-315. [CrossRef] [Medline]
    12. Bryant RA, Harvey AG, Guthrie RM, Moulds ML. Acute psychophysiological arousal and posttraumatic stress disorder: a two-year prospective study. J Trauma Stress 2003 Oct;16(5):439-443. [CrossRef] [Medline]
    13. Bryant RA. Acute stress reactions: can biological responses predict posttraumatic stress disorder? CNS Spectr 2003 Sep;8(9):668-674. [CrossRef] [Medline]
    14. Post-traumatic stress disorder NICE guideline NG116. National Institute of Health and Care Excellence.   URL: https://www.nice.org.uk/guidance/ng116/resources/posttraumatic-stress-disorder-pdf-66141601777861 [accessed 2020-09-02]
    15. Bisson JI, Tavakoly B, Witteveen AB, Ajdukovic D, Jehel L, Johansen VJ, et al. TENTS guidelines: development of post-disaster psychosocial care guidelines through a Delphi process. Br J Psychiatry 2010 Jan;196(1):69-74. [CrossRef] [Medline]
    16. Roberts NP, Kitchiner NJ, Kenardy J, Lewis CE, Bisson JI. Early psychological intervention following recent trauma: a systematic review and meta-analysis. Eur J Psychotraumatol 2019;10(1):1695486 [FREE Full text] [CrossRef] [Medline]
    17. Boelen PA, Olff M, Smid GE. Traumatic loss: mental health consequences and implications for treatment and prevention. Eur J Psychotraumatol 2019 Apr 15;10(1):1591331. [CrossRef]
    18. Oosterbaan V, Covers MLV, Bicanic IAE, Huntjens RJC, de Jongh A. Do early interventions prevent PTSD? A systematic review and meta-analysis of the safety and efficacy of early interventions after sexual assault. Eur J Psychotraumatol 2019;10(1):1682932 [FREE Full text] [CrossRef] [Medline]
    19. Stene LE, Wentzel-Larsen T, Dyb G. Healthcare needs, experiences and satisfaction after terrorism: a longitudinal study of survivors from the Utøya attack. Front Psychol 2016;7:1809 [FREE Full text] [CrossRef] [Medline]
    20. Jeavons S. Long-term needs of motor vehicle accident victims: are they being met? Aust Health Rev 2001;24(1):128-135. [CrossRef] [Medline]
    21. Cheng H, Wang C, McDermott RC, Kridel M, Rislin JL. Self-stigma, mental health literacy, and attitudes toward seeking psychological help. J Couns Devel 2018 Jan 04;96(1):64-74. [CrossRef]
    22. Corrigan P. How stigma interferes with mental health care. Am Psychol 2004 Oct;59(7):614-625. [CrossRef] [Medline]
    23. Shalev AY, Ankri YLE, Peleg T, Israeli-Shalev Y, Freedman S. Barriers to receiving early care for PTSD: results from the Jerusalem trauma outreach and prevention study. Psychiatr Serv 2011 Jul;62(7):765-773. [CrossRef] [Medline]
    24. Stuber J, Galea S, Boscarino JA, Schlesinger M. Was there unmet mental health need after the September 11, 2001 terrorist attacks? Soc Psychiatry Psychiatr Epidemiol 2006 Mar;41(3):230-240. [CrossRef] [Medline]
    25. Brewin CR, Fuchkan N, Huntley Z, Robertson M, Thompson M, Scragg P, et al. Outreach and screening following the 2005 London bombings: usage and outcomes. Psychol Med 2010 Dec;40(12):2049-2057 [FREE Full text] [CrossRef] [Medline]
    26. Rosenbaum L. Post-traumatic stress disorder: a challenge for primary care—misunderstood and incognito. Brit J Gen Pract 2004;54(499):83-85. [CrossRef] [Medline]
    27. Verhaak PFM, Schellevis FG, Nuijen J, Volkers AC. Patients with a psychiatric disorder in general practice: determinants of general practitioners' psychological diagnosis. Gen Hosp Psychiatry 2006 Apr;28(2):125-132. [CrossRef] [Medline]
    28. Dekkers AMM, Olff M, Näring GWB. Identifying persons at risk for PTSD after trauma with TSQ in the Netherlands. Community Ment Health J 2010 Feb;46(1):20-25 [FREE Full text] [CrossRef] [Medline]
    29. Olff M. Mobile mental health: a challenging research agenda. Eur J Psychotraumatol 2015;6:27882 [FREE Full text] [CrossRef] [Medline]
    30. Price M, Yuen EK, Goetter EM, Herbert JD, Forman EM, Acierno R, et al. mHealth: a mechanism to deliver more accessible, more effective mental health care. Clin Psychol Psychother 2014;21(5):427-436 [FREE Full text] [CrossRef] [Medline]
    31. Connor KM. Assessment of resilience in the aftermath of trauma. J Clin Psychiatry 2006;67 Suppl 2:46-49 [FREE Full text] [Medline]
    32. Windle G, Bennett KM, Noyes J. A methodological review of resilience measurement scales. Health Qual Life Outcomes 2011 Feb 04;9(8):1-18 [FREE Full text] [CrossRef] [Medline]
    33. Mouthaan J, Sijbrandij M, Reitsma JB, Gersons BPR, Olff M. Comparing screening instruments to predict posttraumatic stress disorder. PLoS One 2014;9(5):e97183 [FREE Full text] [CrossRef] [Medline]
    34. Brewin CR. Systematic review of screening instruments for adults at risk of PTSD. J Trauma Stress 2005 Mar;18(1):53-62. [CrossRef] [Medline]
    35. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5th Edition. Arlington: American Psychiatric Publishing; 2013.
    36. Price M, Sawyer T, Harris M, Skalka C. Usability evaluation of a mobile monitoring system to assess symptoms after a traumatic injury: a mixed-methods study. JMIR Ment Health 2016 Jan;3(1):e3 [FREE Full text] [CrossRef] [Medline]
    37. Price M, Kuhn E, Hoffman JE, Ruzek J, Acierno R. Comparison of the PTSD checklist (PCL) administered via a mobile device relative to a paper form. J Trauma Stress 2015 Oct;28(5):480-483. [CrossRef] [Medline]
    38. Donker T, Petrie K, Proudfoot J, Clarke J, Birch M, Christensen H. Smartphones for smarter delivery of mental health programs: a systematic review. J Med Internet Res 2013;15(11):e247 [FREE Full text] [CrossRef] [Medline]
    39. Rodriguez-Paras C, Tippey K, Brown E, Sasangohar F, Creech S, Kum H. Posttraumatic stress disorder and mobile health: app investigation and scoping literature review. JMIR Mhealth Uhealth Oct 26 2017;5(10):e156. [CrossRef] [Medline]
    40. International Statistical Classification of Diseases and Related Health Problems: Tenth Revision, 2nd Edition. World Health Organization. 2004.   URL: https://apps.who.int/iris/handle/10665/42980 [accessed 2020-09-02]
    41. ARQ Centre of Expertise for the Impact of Disasters and Crises. Dutch Multidisciplinary Guideline for Post-Disaster Psychosocial Support. Diemen: ARQ Centre of Expertise for the Impact of Disasters and Crises; 2014.
    42. Boeschoten MA, Bakker A, Jongedijk RA, Olff M. Life Events Checklist for DSM-5: Nederlandstalige versie [Dutch version]. ARQ Centre '45. 2014.   URL: https://www.psychotraumadiagnostics.centrum45.nl/en [accessed 2020-09-02]
    43. van der Meer CAI, Te Brake H, van der Aa N, Dashtgard P, Bakker A, Olff M. Assessing psychological resilience: development and psychometric properties of the English and Dutch version of the Resilience Evaluation Scale (RES). Front Psychiatry 2018;9:169 [FREE Full text] [CrossRef] [Medline]
    44. Birnbaum MH. Human research and data collection via the internet. Annu Rev Psychol 2004;55:803-832. [CrossRef] [Medline]
    45. Wood RTA, Griffiths MD, Eatough V. Online data collection from video game players: methodological issues. Cyberpsychol Behav 2004 Oct;7(5):511-518. [CrossRef] [Medline]
    46. Weathers FW, Litz BT, Keane TM, Palmieri PA, Marx BP, Schnurr PP. The PTSD Checklist for DSM-5 (PCL-5).   URL: https://www.ptsd.va.gov/professional/assessment/documents/PCL5_Standard_form.PDF [accessed 2020-09-02]
    47. Eidhof MB, Ter Heide FJJ, van Der Aa N, Schreckenbach M, Schmidt U, Brand BL, et al. The dissociative subtype of PTSD interview (dsp-i): development and psychometric properties. J Trauma Dissociation 2019;20(5):564-581. [CrossRef] [Medline]
    48. Bovin MJ, Marx BP, Weathers FW, Gallagher MW, Rodriguez P, Schnurr PP, et al. Psychometric properties of the PTSD Checklist for Diagnostic and Statistical Manual of Mental Disorders-Fifth Edition (PCL-5) in veterans. Psychol Assess 2016 Nov;28(11):1379-1391. [CrossRef] [Medline]
    49. van der Meer CAI, Bakker A, Schrieken BAL, Hoofwijk MC, Olff M. Screening for trauma-related symptoms via a smartphone app: the validity of Smart Assessment on your Mobile in referred police officers. Int J Methods Psychiatr Res 2017 Sep;26(3):1 [FREE Full text] [CrossRef] [Medline]
    50. Krüger-Gottschalk A, Knaevelsrud C, Rau H, Dyer A, Schäfer I, Schellong J, et al. The German version of the Posttraumatic Stress Disorder Checklist for DSM-5 (PCL-5): psychometric properties and diagnostic utility. BMC Psychiatry 2017 Nov 28;17(1):379 [FREE Full text] [CrossRef] [Medline]
    51. Lovibond PF, Lovibond SH. The structure of negative emotional states: comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behav Res Ther 1995 Mar;33(3):335-343. [CrossRef] [Medline]
    52. de Beurs E, van Dyck R, Marquenie L, Lange A, Blonk R. De DASS: een vragenlijst voor het meten van depressie, angst en stress. Gedragstherapie 2001;34(1):35-54. [CrossRef]
    53. Henry JD, Crawford JR. The short-form version of the Depression Anxiety Stress Scales (DASS-21): construct validity and normative data in a large non-clinical sample. Br J Clin Psychol 2005 Jun;44(Pt 2):227-239. [CrossRef] [Medline]
    54. Lee D. The convergent, discriminant, and nomological validity of the Depression Anxiety Stress Scales-21 (DASS-21). J Affect Disord 2019 Dec 01;259:136-142. [CrossRef] [Medline]
    55. Keyes CLM. Mental health in adolescence: is America's youth flourishing? Am J Orthopsychiatry 2006 Jul;76(3):395-402. [CrossRef] [Medline]
    56. Lamers SMA, Westerhof GJ, Bohlmeijer ET, ten Klooster PM, Keyes CLM. Evaluating the psychometric properties of the Mental Health Continuum-Short Form (MHC-SF). J Clin Psychol 2011 Jan;67(1):99-110. [CrossRef] [Medline]
    57. Keyes CLM, Wissing M, Potgieter JP, Temane M, Kruger A, van Rooy S. Evaluation of the mental health continuum-short form (MHC-SF) in Setswana-speaking South Africans. Clin Psychol Psychother 2008;15(3):181-192. [CrossRef] [Medline]
    58. Muthén L, Muthén B. Mplus User's Guide. Eighth Edition. Los Angeles: Muthén & Muthén; 2017.
    59. Cheung G, Rensvold R. Evaluating goodness-of-fit indexes for testing measurement invariance. Struct Eq Mod 2002;9(2):233-255 [FREE Full text] [CrossRef]
    60. Hu L, Bentler P. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Eq Mod 1991;6:1-55 [FREE Full text] [CrossRef]
    61. Clark L, Watson D. Constructing validity: basic issues in objective scale development. Psychol Assess 1995;7(3):309 [FREE Full text] [CrossRef]
    62. Folkman S, Lazarus RS. Stress, Appraisal, and Coping. New York: Springer Publishing Company; 1984.
    63. Olff M. Bonding after trauma: on the role of social support and the oxytocin system in traumatic stress. Eur J Psychotraumatol 2012;3:1 [FREE Full text] [CrossRef] [Medline]
    64. Kuhn E, Blanchard EB, Hickling EJ. Posttraumatic stress disorder and psychosocial functioning within two samples of MVA survivors. Behav Res Ther 2003 Sep;41(9):1105-1112 [FREE Full text] [CrossRef] [Medline]
    65. Ozer EJ, Best SR, Lipsey TL, Weiss DS. Predictors of posttraumatic stress disorder and symptoms in adults: a meta-analysis. Psychol Bull 2003 Jan;129(1):52-73. [CrossRef] [Medline]
    66. Palyo SA, Beck JG. Post-traumatic stress disorder symptoms, pain, and perceived life control: associations with psychosocial and physical functioning. Pain 2005 Sep;117(1-2):121-127 [FREE Full text] [CrossRef] [Medline]
    67. Fried EI, Nesse RM. The impact of individual depressive symptoms on impairment of psychosocial functioning. PLoS One 2014;9(2):e90311 [FREE Full text] [CrossRef] [Medline]
    68. O'Donnell ML, Creamer M, Pattison P. Posttraumatic stress disorder and depression following trauma: understanding comorbidity. Am J Psychiatry 2004 Aug;161(8):1390-1396. [CrossRef] [Medline]
    69. Te Brake H, Dückers M. Early psychosocial interventions after disasters, terrorism and other shocking events: is there a gap between norms and practice in Europe? Eur J Psychotraumatol 2013;4(1):19093 [FREE Full text] [CrossRef] [Medline]
    70. Te Brake H, Dückers M, De Vries M, Van Duin D, Rooze M, Spreeuwenberg C. Early psychosocial interventions after disasters, terrorism, and other shocking events: guideline development. Nurs Health Sci 2009 Dec;11(4):336-343. [CrossRef] [Medline]
    71. Jacobs J, Oosterbeek M, Tummers LG, Noordegraaf M, Yzermans CJ, Dückers MLA. The organization of post-disaster psychosocial support in the Netherlands: a meta-synthesis. Eur J Psychotraumatol 2019;10(1):1544024 [FREE Full text] [CrossRef] [Medline]
    72. Australian Centre for Posttraumatic Mental Health. Australian Guidelines for the Treatment of Acute Stress Disorder and Posttraumatic Stress Disorder. Melbourne, Victoria: ACPMH; 2013.
    73. Bisson JI, Weltch R, Maddern S, Shepherd JP. Implementing a screening programme for post-traumatic stress disorder following violent crime. Eur J Psychotraumatol 2010;1:1 [FREE Full text] [CrossRef] [Medline]
    74. Eysenbach G, Wyatt J. Using the Internet for surveys and health research. J Med Internet Res 2002 Nov;4(2):E13 [FREE Full text] [CrossRef] [Medline]
    75. Olff M. Sex and gender differences in post-traumatic stress disorder: an update. Eur J Psychotraumatol 2017 Jul 27;8(sup4):1351204. [CrossRef]
    76. Smail-Crevier R, Powers G, Noel C, Wang J. Health-related internet usage and design feature preference for e-mental health programs among men and women. J Med Internet Res 2019 Mar 18;21(3):e11224 [FREE Full text] [CrossRef] [Medline]
    77. Pretorius C, Chambers D, Coyle D. Young people's online help-seeking and mental health difficulties: systematic narrative review. J Med Internet Res 2019 Nov 19;21(11):e13873 [FREE Full text] [CrossRef] [Medline]
    78. Pretorius C, Chambers D, Cowan B, Coyle D. Young people seeking help online for mental health: cross-sectional survey study. JMIR Ment Health 2019 Aug 26;6(8):e13524 [FREE Full text] [CrossRef] [Medline]
    79. Ennis N, Sijercic I, Monson CM. Internet-delivered early interventions for individuals exposed to traumatic events: systematic review. J Med Internet Res 2018 Nov 14;20(11):e280 [FREE Full text] [CrossRef] [Medline]
    80. e-mental health innovation and transnational implementation platform North West Europe (eMEN). Interreg NWE.   URL: https:/​/www.​nweurope.eu/​projects/​project-search/​e-mental-health-innovation-and-transnational-implementation-platform-north-west-europe-emen/​ [accessed 2020-09-02]

    ‎
    ANOVA: analysis of variance
    CFA: confirmatory analysis
    CFI: comparative fit index
    DASS-21: Depression Anxiety Stress scale
    DSM-5: Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition
    EFA: exploratory factor analysis
    eMEN: e-Mental health innovation and transnational implementation platform North West Europe
    GAF: Global Assessment of Functioning
    ICD-10: International Statistical Classification of Diseases and Related Health Problems, Tenth Revision
    LEC-5: Life Events Checklist for DSM-5
    MHC-SF: Mental Health Continuum Short Form
    MIRROR: Mobile Insight in Risk, Resilience and Online Referral
    NICE: National Institute for Health and Care Excellence
    PCL-5: PTSD Checklist for DSM-5
    PTE: potentially traumatic event
    PTSD: posttraumatic stress disorder
    RES: Resilience Evaluation Scale
    RMSEA: root mean square error of approximation
    TENTS: The European Network for Traumatic Stress
    TLI: Tucker-Lewis index
    WLSMV: means and variance adjusted weighted least square

    Edited by T Rashid Soron; submitted 30.04.20; peer-reviewed by A Teles, A Mavragani, C Benjet; comments to author 25.05.20; revised version received 08.07.20; accepted 17.07.20; published 25.09.20

    Copyright

    ©Merel Marjolein van Herpen, Manon A Boeschoten, Hans te Brake, Niels van der Aa, Miranda Olff. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 25.09.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.