Role of Moderators on Engagement of Adolescents With Depression or Anxiety in a Social Media Intervention: Content Analysis of Web-Based Interactions

Background The Supporting Our Valued Adolescents (SOVA) intervention aims to use a moderated social media website to encourage peer discussion about negative health beliefs, which may prevent treatment uptake. Web moderators with a background in behavioral health are used to facilitate peer conversation to promote a sense of community, provide social support, and ensure safety. Objective Although moderation is a core component of this intervention, little is known on best practices for moderators to ensure safety while encouraging engagement. This study sought to describe interactions between moderators and peer users and understand moderator experiences through individual interviews. Methods Adolescents and young adults aged 14 to 26 years with depression or anxiety history were recruited for a usability study of the SOVA intervention. During this study, 14 moderators were trained to regularly review comments to blog posts for safety, facilitate conversation, and correct misinformation. A total of 110 blog posts and their associated comments were extracted and coded using a codebook based on items from the supportive accountability model and a peer social support analysis. Closing interviews with 12 moderators assessing their experience of moderating were conducted, recorded, and transcribed. Blog post text and comments as well as transcripts of moderator interviews were assessed using a thematic analysis approach, and blog posts were examined for trends in content of moderator comments comparing blog posts with differences in comment contributor order. Results There were no safety concerns during the study, and moderators only intervened to remove identifiable information. Web moderators exhibited elements of supportive accountability (such as being perceived as experts and using verbal rewards as well as offering informational and emotional support). When the moderators provided the last comment under a blog post, thereby potentially ending contribution by users, they were at times found to be commenting about their own experiences. Moderators interviewed after completing their role expressed challenges in engaging users. A cohort of moderators who received more extensive training on supportive accountability and peer social support felt their ability to engage users improved because of the training. Conclusions Moderators of a Web-based support site for adolescents with depression or anxiety were able to ensure safety while promoting user engagement. Moderators can elicit user engagement by offering gratitude and encouragement to users, asking users follow-up questions, and limiting their own opinions and experiences when responding to comments.


Introduction
Web-based interventions enhanced with social media components offer a novel approach to intervening on key mechanisms which may increase adolescent use of mental health services. These key mechanisms include addressing parents' and adolescents' health beliefs [1,2], offering emotional support [3,4], and facilitating communication about mental health with other adolescents and other parents. There has been a generational shift regarding the approach to medicine, such that patients consult Web materials before visiting a physician in person [5]. Furthermore, adolescents with depression often use social media to discuss their mental health beliefs to obtain emotional support from peers [6]. Although research shows that depressed adolescents may benefit from talking to others on the Web about their mood [7], they can also encounter negative content and feedback from others, including cyberbullying [6,8]. Website interventions employing social media components therefore should be moderated to first and foremost ensure safety and encourage positive Web-based interactions between users [9]. Moderators have been used successfully to protect users [10], and moderators who exhibit skills and knowledge can promote engagement in Web-based social support groups [11].
Social media allows interventions to target individuals who may otherwise be hard to reach, with moderators as a means to reduce risk [12]. For example, those who are shy/quiet may be more likely to share their feelings on the Web [13]. Several studies have demonstrated the success of using social media as an intervention for depressed young people [7], using moderation to limit adverse events [14]. According to Kraut and Resnick's book Building Successful Online Communities: Evidence-Based Social Design, moderation of Web-based content is particularly important in ensuring safety and blocking inappropriate messages [15]. Furthermore, users are more satisfied with the moderator if screening decisions are fair, criteria are clear/consistent, and moderation is provided by impartial members of the community. The supportive accountability model, created by David Mohr, provides a structured framework for Web-based coaches to simultaneously support and hold users accountable when actively engaging in electronic health (eHealth) behavior change interventions [16]. Mohr and Kraut's work dictates that users are more likely to adhere to treatment when users view coaches as individuals who are benevolent and trustworthy and as having expertise. Coaches should provide user-centered benefits, be specific about expectations, involve the user in making decisions, tie outcomes to a user's larger life goals, focus on process not outcome, and always justify actions. Mohr warns against external rewards, such as money, which may undermine intrinsic motivation and limit long-term success. In contrast, verbal rewards such as positive feedback are encouraged as a means to enhance intrinsic motivation. The supportive accountability model also considers mimicking the communication style of users in content, tense, and tone, known as paralinguistic mirroring, as useful for increasing comfort and trust between users and coaches.
The Supporting Our Valued Adolescents (SOVA) social media website is a Web-based intervention for adolescents and young adults aged 14 to 26 years with depression or anxiety. The site uses daily blog posts to address mental health and increase the perceived need [17] for services with the goal of creating a Web-based community which mainly has anonymous discourse via comments to blog posts [18]. Stakeholder feedback and established user-centered methods informed the design of SOVA, with subsequent usability testing showing user-friendliness and no safety concerns, with high importance placed by users on the need for website moderation [19]. The greatest concern for stakeholders in using social media for suicide prevention was that the moderators would not have sufficient training to be able to intervene in a safety situation [20]. Thus, one of the key aspects in SOVA site development was ensuring round-the-clock site moderation by behavioral health professionals or trainees. Given that anonymity was important to stakeholders, the moderation was crucial to maintain trustworthy content. One of the challenges surrounding the SOVA project involved identifying strategies a moderator can use to strike an equilibrium between the dichotomy of a disciplinary, and possibly intrusive, role (ensuring user safety and accurately shared information), with a role to build a Web-based community (facilitating conversation and user engagement). This study explored the moderator process itself and how moderators can balance these seemingly opposing roles.
Few previous studies assessed how Web moderators can engage users, especially in adolescent internet support groups. As a part of a usability study of the SOVA intervention [18], we had an additional opportunity to examine the role of the Web moderator through observational (comments) and self-report (interview) data. This study sought to understand the role of the moderator more clearly through 2 specific aims: (1) to describe the extent to which moderator interaction ensures safety, facilitates or deters peer conversations, exhibits principals of supportive accountability, and provides social support (and types provided); and (2) to understand and describe the experience and feedback from moderators on the SOVA website to inform future changes to moderator training.

Sample
Adolescents and young adults aged 14 to 26 years with a self-report of experiencing symptoms of depression or anxiety were recruited for a usability study of the SOVA intervention which is described in detail in a previous paper [18]. Briefly, during that study, users who logged on to the website were able to comment on daily blog posts, all of which were written by moderators at that time, except for 2 written by one user who was piloting peer blogging. Of the 96 participants recruited, 41 users ever logged in, and only 16 users ever commented, with 5 users commenting more than 5 times.

Blog Post Comment Extraction and Coding Process
The blog post and their associated comments were extracted from the website database for 110 blog posts from April 2015 to February 2017 and downloaded into NVivo (QSR International) qualitative software. A codebook was developed which included labelling of comment author (user vs moderator) and author order, and individual codes based on items retrieved from the supportive accountability model [16] and peer social support analysis [21] (Multimedia Appendix 2). Although user comments were coded for social support provided to other users, the analysis focused on moderator comments and moderator-user interactions.
The first 50 blog posts were coded by 2 coders independently using an initial codebook. They then met with the PI to review codes and modified the codebook based on this feedback, mostly regarding the definition of specific codes. The units of coding used were one entire comment; although at times, parts of a blog post could be included as a comment if it included codes perceived to be the blog post writer talking to the community. For example, moderators would often include questions at the end of a blog post for the community. Individual comments could be coded with multiple codes if fitting under multiple categories. The modified codebook was used by 2 coders to recode the initial 50 blog posts until a percent agreement score of 90% was achieved (our criterion was at least 80%, there were 3 iterations of coding), after which 1 coder coded the rest of the blog posts. Mismatches were adjudicated by the PI. The assumption was made that the user who sees an article written by the moderator also perceives this as an interaction with the moderator when commenting on that post. Blog posts were then broken up into categories based on the order of individuals commenting (eg, user, moderator, and user). Blog posts were separated into categories for which conversation continued after moderator response and those for which conversation ended after moderator response. Codes within these categories were compared for overlying themes.

Moderator Interviews and Coding Process
As moderators discontinued their work with SOVA, usually because of a finishing graduate school practicum or job transition, they were asked to participate in a closing interview conducted by a research assistant. Individual interviews were conducted with 12 moderators, all of whom had moderated for a period of 6 months or more. Interview questions assessed the experience of moderating and perceptions about training. These were recorded, transcribed, and double coded until greater than 80% agreement was achieved across all nodes. Interviews were transcribed verbatim excluding filler words (eg, like and um). A prespecified codebook (Multimedia Appendix 3), updated during coding, was utilized. This interview study was found to be exempt from IRB approval.

Analysis
We used a template analysis approach for both analysis of the blog post comments as well as the moderator interviews, using a prespecified codebook and a hierarchical approach to the coding, being open to future changes in codes as coding progressed [22]. For blog post comments, an exploratory analysis was conducted to assess differences between the content of comments for when conversation stopped after the moderator commented versus when it continued. Researchers examined the blog posts from this final category where both moderator and user responded and explored trends within them to ascertain if moderator involvement or the order of involvement seemed related to the amount of further user comment contributions as a proxy for engagement. For moderator interviews, thematic content analysis was conducted to understand overarching themes of moderator experiences and opinions. Findings were discussed with previous moderators as a method of member checking [23].

Ethical Approval
This work was approved by the IRB of the University of Pittsburgh (PRO15060158).

Blog Post Comment Analysis
Out of 363 total blog posts, the 110 that received comments (2 of which were written by a user) were coded and assessed. First, there were no safety concerns throughout moderating. There were fewer than 5 instances that moderators intervened to remove identifiable information from comments or correct misinformation. There were no instances of cyberbullying or crisis situations. Moderators most often commented to facilitate conversation or offer feedback and advice.
In blog posts where conversation stopped, moderators were found to be self-disclosing, commenting about their own experiences. Alternatively, the conversations that continued (eg, user, moderator, and user) revealed several trends. These trends included the moderator utilizing emotional support including thanking the user, asking them a question, asking for their tips, and offering them encouragement. As further detailed below, moderators commonly displayed elements of Mohr's supportive accountability model in conversations which continued after moderator input by (1) sharing or displaying moderator expertise, (2) verbally rewarding user, and (3) mirroring (eg, mimicking the same emoticon as a user) [16].
An example of a blog post where the conversation continued after the moderator responded was one called, "What Depression Really Looks Like," which addressed the stereotype that people who are depressed will look depressed, when often that is not the case. The question asked at the end of the blog post was, "Do you think the pictures used to portray depression play a role in the stigma around it?" A user responded after which the moderator responded, and a conversation followed (Figure 1). In this case, the moderator asked a follow-up question. They also verbally rewarded the user, which was one of Mohr's motivational constructs for increasing engagement and motivation.
Another example of continued conversation addressed negative emotions, specifically, the loss of a loved one. In this case, a user commented, a moderator responded, and a conversation continued ( Figure 2). Here, the moderator displayed expertise and a potential solution to the user's problem. The moderator also provided support, both emotionally and in the form of informational resources.
A final example of continued conversation was one surrounding a post that discussed depression and how it manifests in the individual. The same user commented back to the moderator in this case ( Figure 3). The moderator first provided emotional support by thanking the user for their post and then asked a follow-up question. The same user responded to the moderator's post by elaborating on the situation and discussing relaxation techniques.
An example of a blog post where the moderator stopped the conversation was one called Calm.com, which provided information about a relaxation website. The question asked at the end of the article was, "Do you have the calm app? What do you think of it? What do you think of having something that allows you to put away everything else for just a few minutes to have some time for just yourself? Tell us about it in the comments!" A user responded after which the moderator responded, and there was no further conversation ( Figure 4). In this example, the moderator was commenting on their own experiences using positive self-disclosure, and the conversation ceased to continue.    In the first example, the moderator's question was not answered by the original user but did not prevent further commenting. In the second example, the moderator provided both emotional and informational support, and neither comment deterred further user interaction. In the third example, the moderator asked the user a question, to which the same user responded, thus fostering further engagement. In the final example where the conversation stopped, perhaps the moderator commenting about their own experiences may have deterred further user response.
An analysis of the coded comments revealed several trends in the way moderators engaged on SOVA. All codes occurring more than 10 times in blogpost comments can be found in Multimedia Appendix 4. In 22 blog post comments, moderators provided expertise in the form of either recommending local resources such as crisis centers or education material or sharing professional knowledge on a variety of topics related to mental health or use of social media. In 3 comments, users perceived expertise from moderators and responded with acknowledgement and appreciation, including statements such as "these are really helpful." For example, in response to a quote about friendship, a user replied, "this happened to be just the right time that I needed to hear a quote like this. Thank you." In 14 comments, the moderator responded to users' comments by asking follow-up questions such as "Was it as difficult for you to share with those you trust? Do you or anyone else have any tips on how to approach the subject?" to try to increase user interest and engagement. Out of these 14 comments, 11 resulted in a user responding to the question asked by the moderator, whereas 3 did not. However, those 3 that did not incite a user response to the posed question did not limit different users from commenting further within the same thread.
Moderators also offered potential solutions to problems users expressed in 10 comments, often suggesting ways to avoid rumination and adopt more positive attitudes. An example of a moderator providing a solution was when giving advice to someone struggling with sleep, encouraging the user to practice "controlling your exposure to light, creating bedtime rituals that help with relaxation, and keeping the bedroom cool and quiet." Users expressed that they learned something new from the site on 10 occasions, though not overlapping with moderators' solutions but in response to informational posts or tips from other users. In 51 comments, moderators provided users with verbal rewards including phrases such as "thanks for sharing!" and "great point!" Users occasionally shared desire for emotional support or information (2 and 4 comments, respectively). Of these 6 cases, the moderator responded 5 times, and another user responded once. For example, when a user asked, "How would the average person know when someone's having a panic attack?" the moderator responded with details and a link to a website outlining mental health first aid guidelines. Moreover, moderators responded thoroughly to all aspects of the users' comments-thanking them for sharing, addressing the story they shared with positive affirmations, answering any questions they may have posed, and occasionally sharing some new information or asking a follow-up question to promote discussion. For example, when a user stated, "I have a hard time wording things," the moderator asked, "What are some ways you might try to word what you want to say from reading this article?" which prompted the user to continue the conversation. This demonstrates the moderator reacting to a disclosure by the user and using an open-ended question to further discussion.
Emotional support was provided most commonly by users, but also by moderators, for a total of 57 times. This support was most often in the form of moderators' or users' acknowledgment of what the original user posted ("I agree, journaling can be a great way to deal with things"), or from users sharing similar stories of one's own, saying things such as "I completely agree [and I, too] want to share my struggle with this." Informational support was also provided by both users and moderators 167 times. This involved sharing resources and strategies with fellow users, including providing emotional coping strategies, such as links to websites as mentioned or advice on how to deal with therapists, physicians, stressful situations, or negative thoughts.
In 63 comments, users made positive remarks highlighting the value of the SOVA website, most often responding to blog posts, but also to other users' comments or moderator's comments, such as "This was very insightful and helpful" and "I'm definitely going to try [that]."

Moderator Interview Analysis
Moderators shared challenges in engaging users in thoughtful conversation during the usability testing phase. They believed this was because of a limited number of users on the site at one time, and iterative updates to website functionality (eg, new article notifications not working). They also shared challenges engaging users because of the nature of users themselves, "I think it just takes a unique type of adolescent to want to engage online with this type of site...it really makes sense that our adolescents do use a lot of online communities and that this is something they would be interested in, but I think it takes a lot of forethought and insight that adolescents might not have about their own mental health." One moderator mentioned some of the challenging aspects surrounding engaging users, "How do you create a conversation...if you are responding to a comment and you say, '...That's a really good point...' how do you make it into a conversation so that they want to respond more? Or how do you get other users to interact with each other? That is an important role of the moderator to create that space where conversations can happen." Following concerns from moderators that more guidance on peer engagement and crisis training were needed, this was updated in training procedures, and later moderators perceived that training was adequate and valuable to performing their duties, especially when previous or existing moderators were accessible and approachable. Often, it took the moderators several weeks to become fully comfortable with their duties and those with a stronger background in mental health reported feeling more prepared. They felt their role in commenting was relatively unimportant when compared with keeping the site and users safe. Taking that into account, there were no safety concerns, and moderators found using a research study mobile phone with all new site content emailed to the mobile phone a feasible way to incorporate moderating with balancing their other daily tasks, so they could be available 24×7. Overall, the moderators stated that they most enjoyed interacting with other users (including replying to comments), gaining mental health experience and training, and feeling as though they were making a difference through their role. The moderators had many positive things to say about the study, "I like that the role [of the moderator] is important, even though I do not necessarily always have a lot of interactions with the users. I know that the subscribers know that someone is there to make sure the information is accurate and that it is a safe environment to discuss and talk is good. I like knowing my role is needed."

Principal Findings
In this study, after examining the role of the moderator in a Web-based intervention for adolescents with depression or anxiety, we found that moderating such an intervention was feasible and resulted in no safety concerns. Additionally, moderators exhibited various approaches that may impact user engagement. Moderators themselves expressed satisfaction with receiving training on techniques which may enhance user engagement and keep users safe on the Web, stating that they found the experience valuable. The findings of this study influenced changes to current moderator training including incorporating more feedback on emergency and safety protocols as well as enhancing feedback on how to increase user engagement by limiting moderator self-disclosure.
The role of the moderator in Web-based behavior interventions for adolescents and young adults is a necessary one to ensure the safety of users and quality of Web content. Moderation has been found to foster a welcoming and safe environment, prevent cyberbullying [24], and may even increase respect among users [11]. Moderation also improves quality and accuracy of information shared by users [11,25] and helps direct users to specific site content to address their unique needs [26]. In a study looking at YouTube TED Talk comments, moderators were mainly used to screen offensive content [27]. However, excessive moderation limited nonoffensive comments as well-demonstrating that it is important to be prudent when screening content to avoid limiting conversation. Along with promoting site safety and quality content, the moderating role in the context of our study involved facilitating Web-based peer-peer interactions, providing social support, and encouraging adolescents to communicate about mental health. This was achieved through follow-up questions and reliable responses to users' comments within 24 hours. In our study and other eHealth interventions, periodic prompting by moderators can be an important tool to encourage participation [28]. Achieving this multiplicity of roles may at times be conflicting and complex, especially when attempting to convey a message through brief text. This challenge was often acknowledged and reflected upon in our moderator interviews. Nevertheless, in a variety of instances, moderators have been perceived positively with appreciation from users while serving the roles of encouraging user participation and filtering undesired content [29][30][31]. In the technological usability study of SOVA, we found no concerns from users regarding moderator interactions [18]. The supportive accountability model considers some moderator behaviors that may be exhibited in an interactive Web coaching scenario where traditional goal setting and rapport building may occur. As no such expectations were stated to users in our intervention, we did not expect to find some of the code families including bond, trustworthiness, benevolence, reciprocity, process expectations, and mirroring. Other code families from supportive accountability for which we did expect to find codes included: expertise, definition, identifying a problem, interest, reward, cues, social support, and seeking and providing emotional and informational support.
We found that varying moderator approaches may impact user engagement. Our findings led us to conclude that our concern of moderators deterring further conversation did not happen most of the time, but that at times when moderators exhibited self-disclosure, conversation from users would stop. Moderators may experience success by omitting their own opinions and experiences when responding to comments as this behavior may deter further discussion among users. One explanation for this finding is derived from the theory of Rogerian client-centered therapy, whereby person-centered dialogue is favored, such as encouraging patients to share or asking clarifying questions [32]. Working in this model, moderators would be expected to have more success when they avoid self-disclosures. Another explanation for this finding is that the user may perceive disregard from the moderator if the moderator shares a similar experience which they overcame. A study found that people who have experienced hardships appear less sensitive to those currently suffering and may be less likely to provide compassion [33]. The third possible explanation is derived from the concept of acquiescence to authority, whereby an uneven power dynamic between individuals may lead to acquiescence from the perceived person of lesser power when an authority figure makes a declarative statement [34]. In the context of this study, moderators are in a position of power as they oversee the website. Thus, the users may be less likely to respond to declarations, such as self-disclosures, from moderators and more likely to respond to moderators requesting feedback.
User engagement is essential for the success of Web-based interventions [15], which can be influenced by content, moderators, and users themselves. A study examining user engagement between different types of Facebook posts (eg, pictures, polls, multimedia, and questions) found that posts requiring a simple response, particularly polls, generated the greatest engagement [35]. Similarly, in our study, moderator-initiated posts that offered simple questions and moderator follow-up comments did incite a response from users, suggesting that users felt comfortable engaging with content sourced directly from the moderator.
We found that moderators exhibited techniques aligned with supportive accountability and social support and that users did seem to engage when these techniques were used, for example, when moderators would offer encouragement, informational or emotional support, display expertise, or offer verbal rewards. Despite this information on moderator strategies, the way users interact on the Web is, not surprisingly, user-dependent and varies significantly from person to person [27,36]. A study examining ratings of fictitious political candidates revealed that internal perception of self-efficacy plays a large role in how people are influenced by others on social media [37]. Stronger internal self-efficacy has been demonstrated to be a more powerful predictor of user response than the opinions of others, and individuals with depression and anxiety may have lower self-efficacy. This suggests that some users may be more susceptible to moderator influence than others. Moderators may also find success in altering their conversation style to directly target individuals, such as by mirroring the way in which the users they engage (ie, use of emoticons) [16]. Achieving this multiplicity of roles may at times be conflicting and complex, especially when attempting to convey a message through brief text.
Many therapy-based Web interventions reveal several important aspects of how humans may provide support. First, supported interventions, either by a therapist or nonhealth professional administrator, are more successful than unsupported mechanisms even when human resources are low [38]. Additionally, the method of support offered is not as important as consistency in meeting expectations and needs [39]. When Web-based therapists exhibit flexibility with guideline adherence and tailor support to individual client needs, adherence improves [40,41]. This challenge was also acknowledged and reflected upon in our moderator interviews. In some Web-based interventions, peer moderators acted as a bridge between adolescents using an intervention and expert moderators [24,26]. These peer moderators were often used to welcome new users and helped facilitate user engagement. Data suggest that peers can provide quality support with minimal training [42]. In a study utilizing peer support and assessing a social media intervention to prevent relapse in youth depression, moderation was found to foster a welcoming and safe environment and prevent cyberbullying [24]. Thus, a future improvement to moderation of the SOVA website could be made to include experienced fellow users as peer moderators.
This study has several limitations. First, the study was not designed to empirically examine the impact of utilizing specific techniques to increase user engagement. The data presented here are highly explorative and observational in nature, and a future experimental study would be needed to confirm our results, as we only describe trends in coding for when conversation continues and conversation stops. Supplementary analysis of moderator-user order on other social media platforms could be used to further investigate the trends found in this study. Thus, more qualitative research regarding moderator techniques in other behavioral interventions could further inform intervention design. In addition, the study was conducted during a feasibility and usability study with a smaller sample of users engaged on the site. If there were a larger sample of active users, there may be additional findings not accounted for in this study. Regardless, initial stakeholder feedback for the design of SOVA raised many concerns about safety [19], and we have been pleased that through social norm setting, we believe moderators have demonstrated the goal and purpose of the site and that this has limited negative interactions and any safety concerns. The role of a moderator involves delicate balance, as too much moderation can be perceived as surveillance and become detrimental to the success of the intervention [43]. Further research could assess the effect of moderators sending private messages to users in response to blog post comments, as an alternative to publicly responding back, and whether this promotes conversation among users. Feedback about moderator techniques such as this can have far-reaching effects for effectively increasing user engagement in potentially valuable social media interventions.

Conclusions
The high rate of suicide and low rate of mental health treatment among adolescents highlight the need for social media interventions such as SOVA. Moderation is key for this sort of intervention to be both effective and safe. Moderators on the SOVA site elicited user engagement by offering gratitude and encouragement to users, asking users follow-up questions and limiting their own opinions and experiences when responding to comments. Users commenting on SOVA perceived that it had positive effects on increasing their adoption of healthy attitudes and behaviors. The research described is innovative specifically in investigating strategies a moderator can use to balance a potentially punitive and interfering role (enforcing site rules) with a supportive role (providing social support and facilitating conversation to promote peer-peer social support) for an adolescent Web-based support group intervention, all while effectively promoting user engagement.  26.09.2019 This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.
Objective: The aim of this paper is to describe our implementation science protocol, which serves as a foundation by which to systematically guide the implementation of technology-enabled solutions in traditional face-to-face and Web-based mental health services, allowing for revisions over time on the basis of retrospective review and constructive feedback from the services in which the technology-enabled solutions are implemented.

Methods:
Our implementation science protocol comprises four phases. The primary objective of the scoping and feasibility phase (Phase 1) is to determine the alignment between the service partner and the quality improvement goals supported by the technology-enabled solution. This is followed by Phase 2, the local co-design and preimplementation phase, which aims to utilize co-design methodologies, including service pathway modelling, participatory design, and user (acceptance) testing, to determine how the solutions could be used to enhance the service. In Phase 3, implementation, the accepted solution is embedded in the mental health service to achieve better outcomes for consumers and their families as well as health professionals and service managers. Using iterative evaluative processes throughout Phase 3, the solution is continuously developed, designed, and refined during implementation to adapt to the changing needs of the stakeholders, including consumers with lived experience and their families as well as the service. Thus, the primary outcome of Phase 3 is the optimized technology-enabled solution that can be maintained in a service during the sustainment and scalability phase (Phase 4) for the purposes of mental health services reform.
Results: Funding for the protocol was provided by the Australian Government Department of Health in June of 2017 for a period of 3 years. At the time of this publication, the protocol had been initiated in 11 services, serving three populations, all of which are currently operating in Phase 3. The first results are expected to be submitted for publication in 2020.

Conclusions:
With the aim of improving mental health service quality, our implementation science protocol aids in the identification of factors that predict the likelihood of implementation success, as well as the development of strategies to proactively mitigate potential barriers to achieve better implementation outcomes. Putting in place a theoretically sound implementation science protocol is essential to facilitate the uptake of novel technology-enabled solutions and evidence-based practices into routine clinical practice for the purposes of improved outcomes.

Health Information Technologies and Mental Health Services Reform
Globally, the mental health system is plagued by fundamental shortcomings, including delays in service provision, limitations in access, fragmented services, failure to utilize routine outcome monitoring, and provision of care that does not match the consumer's level of need. New health information technologies (HITs) are being rapidly developed to improve the delivery of mental health care for both health professionals and consumers, as well as to better support self-management of care. For example, cognitive-behavioral therapy approaches have been incorporated into several apps and websites, such as MoodMission [1] and CBT-i Coach [2], to help consumers better self-manage their health and well-being, provide psychoeducation about areas of concern or difficulty, and enhance traditional face-to-face care. Unfortunately, it is all too common for both traditional clinical and more novel electronic or Web-based interventions found to be efficacious in research studies not to be associated with meaningful outcomes for consumers in clinical settings [3]. This may partly relate to fundamental differences in the factors frequently associated with successful clinical trials (eg, highly standardized, homogenous participant sample, and control of possible confounding factors), relative to those that facilitate effectiveness in clinical practice or community settings (eg, flexibility in the intervention for providers and consumers, appropriateness for a broad consumer group across multiple clinical settings, and applicability for multiple conditions). However, it is also important to consider possible failures in the implementation of evidence-based interventions in practice. For example, a recent systematic review highlighted the failings of HIT to support the management of heart failure because of the complexities of providing care to consumers who are often older, with multiple comorbidities, more vulnerable with less support, and less technically savvy [4]. Furthermore, Bont and Bal describe an HIT as being set up for success by the clinical sponsors but failing because of the impact on traditional notions of what it meant to be a good health professional [5].

Implementation of Health Information Technologies: Barriers and Facilitators
As it relates to health care, implementation science is defined as the scientific study of methods to promote the systematic uptake of research and development (R&D) findings and evidence-based practices into routine clinical practice, with the aim of improving the quality and effectiveness of health services and the care provided [6]. There is increasing appreciation among leading research organizations regarding the need to develop and utilize effective methods of implementing efficacious or effective interventions for the purposes of improving health care quality and efficiency [7][8][9]. To facilitate successful implementation, our implementation science protocol serves as a strategic, high-level, long-range plan with 3 primary aims: (1) describe the implementation process, (2) explain the facilitators and barriers to implementation outcomes, and (3) evaluate implementation [10].

Potential Facilitators and Barriers to Successful Implementation
The results of a recent systematic review highlighted the benefits of HITs on the quality and efficacy of health care, partly by facilitating adherence to guidelines or protocol-based care with the aid of embedded electronic decision support functions [11]. Specific examples from the mental health field include consistent support for phone calls or short message service text messages about medications, resulting both in improved adherence and reductions in hospitalizations, as well as for the use of Web-based self-management tools to improve quality of life, mood, and social functioning [12]. Handheld devices to support health care consultations have also been shown to improve satisfaction with care [12]. Despite the unique and additive benefits of HITs to mental health service quality, previous studies have shown that a range of facilitators, barriers, and contextual conditions impact on implementation adoption and sustainment [13][14][15][16], potentially resulting in underutilization. Specifically, several recent reviews have documented potential barriers and facilitators to implementation processes [17][18][19][20][21], which can be divided into service-level, health professional, and individual factors. Notably, there is often a reciprocal relationship between facilitators and barriers, such that a facilitator that is not championed can quickly become a barrier and vice versa.

Service Factors
The importance of leadership from the senior service management, as well as at the local service level, is consistently highlighted as a potential facilitator to successful implementation [17,18,21,22]. Indeed, our experience implementing a prototype technology-enabled solution [23], described in detail below, across 5 youth mental health services highlighted the benefit of strong local leadership to promote service innovation, as well as R&D within a service [24]. Senior leaders help ensure alignment between the technology and the service mission and help foster an organizational culture that is open to change in the service model and receptive to the technology-enabled solution [17]. In this regard, involvement of local champions (ie, prominent and well-respected individuals in a service) in the co-design and co-development of the solution and its implementation is frequently emphasized as a facilitator of successful implementation [18,25]. In addition, executive sponsorship is essential to emphasize the organizational commitment and demonstrate support for service change and redevelopment, especially given that the underestimation of change management can also act as a barrier to successful implementation [17,20]. Service leaders engender a positive attitude toward change and encourage adoption of the technology-enabled solution by frontline staff and consumers. Contingencies need to be put in place to ensure that the local champion roles are always filled, regardless of staff turnover.
Consideration should also be given to the degree of alignment between conventional service models and workflows and the solution, as misalignment represents a barrier to successful implementation, with higher degrees of misalignment increasing the complexity of the implementation [26]. Furthermore, in a systematic review of electronic health implementations, workflow disruptions were found to be the most cited factor in determining the success or failure of an implementation [20]. Collaboration and R&D between researchers and service staff (eg, health professionals, administrators, and service managers), as well as consumers with lived experience and their families, in relation to the iterative co-design and co-development of the technology-enabled solution (including service model), are critically important components to implementation success and sustainability for mental health services reform [15,18,20,27,28].
Resource limitations, including lack of appropriate personnel and equipment, are routinely reported as barriers to implementation [17,21]. Interestingly, however, the availability of adequate resources within a service has not been found to be a specific facilitator of implementation. As summarized by Vis et al [21], reliable access to information and communications technology (ICT), as well as any required interoperability with other existing technology within the service, such as an electronic medical record or client management system, is a key determinant of implementation.

Health Professional Factors
A number of factors related to health professionals have been identified as facilitators for successful implementation. For example, the co-design and configuration of the solution to fit the needs of health professionals foster buy-in and acceptance [18,21,22]. Successful implementation is also facilitated by effective education and training of health professionals, which nurtures self-efficacy and capacity in the context of continuous on-the-ground support [17,18,21,22]. Indeed, research indicates that learning outcomes are enhanced when initial education and training are supplemented with ongoing implementation support [29]. Several potential barriers to implementation at the level of health professionals have also been consistently reported in the literature, including negative staff attitudes, resistance to change, and changes to work practices [17,[20][21][22].

Individual Factors
Implementation theories consistently state the importance of considering consumers' needs when designing any implementation processes intended to improve outcomes [30,31]. As such, the involvement of consumers with lived experience and their families in the co-design process is key to implementation success [20]. Consideration of consumer preferences for and disparities in the use of digital devices and modes of technology (eg, email, app, and website) to connect with health services is essential [32] when iteratively designing and developing technology-enabled solutions. Other identified facilitators include consideration of the convenience and appropriateness of the solution in addressing consumers' health care needs [21], the user friendliness of the solution [22,33], and the appropriate adaptations (ie, configuration) of the solution to fit the specific cultural needs of the populations seeking care [18]. As drivers of acceptance [21,22] and empowerment for consumers with lived experience [20], the combination of these factors is likely to promote successful implementation. Conversely, individual resistance or nonparticipation, recruitment and retention issues, individual concerns about privacy, confidentiality, and information security, and a failure to adapt solutions over time to meet individual needs [17,20,22] are consistent barriers to successful implementation and, in turn, mental health services reform.

Co-Design of the Technology-Enabled Solution
As described in detail in a study by Davenport et al [23], the co-designed InnoWell Platform was developed through Project Synergy (by InnoWell Pty Ltd) to collect, store, and report clinical data back to a consumer and their health professional to promote person-centered care, self-management, early intervention, shared decision making, and routine outcome monitoring (see Textbox 1). Textbox 1. Description of the InnoWell Platform as it is listed on the Australian Register of Therapeutic Goods (software as a medical device, class 1, ARTG ID 315030).
The InnoWell Platform is a customizable digital tool that assists assessment, monitoring and management of mental health issues, and maintenance of wellbeing. It does this by collecting personal and health information from consumers and their service providers. This information is stored, scored, and reported back to consumers and their health professionals to promote collaborative care. The clinical content is determined in collaboration with the service provider who invited the consumer to use the platform. Importantly, the platform does not provide stand-alone medical or health advice, diagnosis, or treatment. Instead, it guides and supports (but does not direct) consumers and their health professionals to decide what may be suitable care options. Importantly, all care aligns with the existing clinical governance (eg, policies and procedures) of the service provider.

An Implementation Science Protocol for Local Mental Health Services Reform
For the purposes of the technology-enabled solution, quality implementation is achieved when the deployment of the technology and service model into health care services meets the requirements and standards to achieve the desired outcome [34], namely technology-enabled mental health services reform. With this aim in mind, our implementation science protocol incorporates elements from 3 sources, namely the Quality Implementation Framework [29] and the Accelerated Creation-to-Sustainment model [15], as well as learnings from our experience co-designing and implementing a prototype technology-enabled solution into primary mental health care settings with young people [24,35]. The primary objective is to design a standardized yet flexible implementation science protocol to serve as a foundation by which to systematically guide implementation efforts, allowing for revisions over time on the basis of retrospective review and constructive feedback from services in which the technology-enabled solution is to be implemented and the consumers who will be engaging with the solution. Our implementation science protocol comprises 4 phases. The objectives and outcomes of each phase are described in detail below.

Phase 1: Scoping and Feasibility
The primary objective of Phase 1 is to determine the fit between the aims of the potential mental health service partner and the quality improvement goals supported by the technology-enabled solution. To begin with, it is essential to ensure that the service leadership is invested in quality improvement at the individual level and to demonstrate the relationship between this aim and the components of the technology-enabled solution. Our protocol emphasizes the importance of engaging organizational leaders from the earliest steps of the process. Furthermore, it is crucial to understand the basic attributes of the service, including the following: the characteristics of the consumers who access the service for care, the qualifications and occupancy of the service health professionals, administrators and service managers, and the current hardware and ICT infrastructure (eg, electronic medical record, client management system, and availability of Wi-Fi). It is also important to identify those persons who will be responsible for facilitating the implementation process at the service, including making decisions related to configuration and customization of the technology-enabled solution and overseeing change management processes (eg, changes to the clinical pathway, user journey, and workflows) on the ground. Provided the key tasks outlined in the scoping and feasibility phase are addressed, transition to Phase 2 then occurs. Alternatively, the service may choose to address the internal service gaps identified in this phase before proceeding to the next phase.

Phase 2: Co-Design and Preimplementation
A key feature of the technology is that the functionality and content are configurable, which allows it to adapt easily to local contexts, as well as specialist clinical and population groups, and it thus meets the person-centered care needs of a wider range of people presenting to diverse mental health providers. The primary objective of this phase is to utilize novel and innovative co-design methodologies to develop the technology-enabled solution and service model, as well as to user (acceptance) test these solutions for mental health services reform.

Participatory Design of the Technology Solution
As previously described in detail by our team [35][36][37], the potential end users (eg, general population, consumers with lived experience and their families, health professionals, administrators, and service managers) help inform the development of the technology through the continuous and iterative use of participatory design (or co-design), knowledge translation, user (acceptance) testing, and rapid prototyping methodologies. These user-centered methodologies emphasize the involvement of individuals with lived experienced (eg, consumers, family members, and peer support workers) in the co-design process of the technology and the implementation process. As outlined by Mohr et al [15], interactive and iterative participatory design methodologies with key stakeholders (eg, information gathering, clarification of user requirements, workflow observations, co-design workshops, and user testing) help adapt the technology to local needs. Furthermore, as outlined in Figure 1, our protocol incorporates iterative evaluation methods to promote the continuous development and design of the technology, as well as the implementation process for the service and consumers for whom they provide care. The process of refining the technology can occur iteratively through the Implementation and Sustainment phases [15].

Co-Designing Technology-Enabled Service Models
In parallel, a process of co-designing service models occurs with representative stakeholders. The preexisting service model and related staff roles and responsibilities are mapped by using a process of service modelling developed through R&D. As illustrated in the hypothetical example shown in Figure 2, each aspect of an individual's journey through the service is delineated, starting from the perspective of the individual seeking help and expanding when other service stakeholders interact with that individual through the care journey. Following the presentation of an individual's experience of a prototypic version of the technology-enabled solution, the gaps between the current model offered by the service and the key features of high-quality mental health care are explored. A hypothesized technology-enhanced service model is then co-designed, intertwining established processes with the additional technology elements and processes (eg, Web-based intake assessment, video visit). The changes to workflows and practices for each of the affected stakeholders (eg, health professional, administrator, and service manager) are also noted throughout the model. Finally, existing service metrics used to report on service quality are reviewed (as available) across pre-established quality domains (eg, safety, efficiency, and accessibility); thereafter, agreement is reached on the metrics used to monitor service quality using the technology solution [38].

Education and Training
The education and training needs of service staff across all roles (eg, health professionals, administrators, and service managers), as well as consumers, are scoped during Phase 2. First, education and training are provided in relation to the evidence-based digital, clinical, service, and safety elements essential to improving service quality. Second, training in the use of the technology-enabled solution includes all necessary information to use the technology effectively for all staff members, including both a comprehensive overview of the functionality and the components relevant to the roles in the service. As outlined in Table 1, education and training are provided before implementation, with ongoing education, training, and technical assistance available throughout Phase 3.

Discovering Facilitators and Barriers Before Implementation
In Phase 2, ongoing interactions with the team of stakeholders involved in the implementation and change management process, present opportunities for the active and passive identification of barriers to, and facilitators of, successful implementation not previously uncovered. Table 1 highlights some of the potential barriers identified in previous studies and the mitigation strategies for each. Table 1. Potential service-specific barriers and mitigation strategies to implementation of our technology-enabled solutions.

Mitigation strategies Barriers
Engagement and buy-in fostered through participatory design and user (acceptance) testing; co-design of service model to identify gaps between new and existing workflows and foster collaborative problem-solving approaches to customization and configuration of the technology-enabled solution for the benefit of service quality improvement; development of a communication strategy to assist the service with messaging within internal stakeholders and service users Negative staff attitudes [20,21]; staff resistance to change [17,[20][21][22]; changes to work practices, such as increased workload [20][21][22] Implementation Officer embedded in service to maintain continuity of support; contingency plan for training new staff Staff turnover and lack of staff resources [17,18] Ongoing evaluation of the technology-enabled solution; continuous and iterative refinement of the technology-enabled solution and service model throughout Phases 3 and 4 (implementation and sustainment) Innovation not able to adapt over time to meet staff needs [17,20,22] Service provider readiness assessment to determine compatibility of the technology-enabled solution with the service; co-design for a servicespecific technology-enabled solution; technology configuration and customization arising from the co-design process, including co-design and co-development of service-specific content, as well as integration with service information and communications technology systems; iterative user experience and user acceptance testing; iterative evaluative processes related to technology and implementation process, highlighting adaptability of the technology; provision of ongoing education and training and technical assistance Design and usability of the technology-enabled solution [20,21,24]; adaptability/flexibility of the technology-enabled solution [17,18,22]; compatibility/fit of technology-enabled solution with service mission [17,18]; user resistance to the technology-enabled solution [22]; noninteroperability (or limited) with other information and communications technology systems [17,20]; fidelity of implementation [17,22]; availability of and user familiarity with required equipment to use the technology [21,24] Phase 3: Implementation Phases 1 and 2 are seen as critical to successful implementation, as they foster buy-in and commitment from stakeholders to the principles of co-design and quality improvement, and they should lead to a point where all stakeholders feel heard and are able to participate in the change process. As far as possible, all known barriers and facilitators to implementation, as they apply to the local context, have been identified at this stage, with a mitigation strategy identified for each. Despite this, new barriers and facilitators may be uncovered, which were previously unknown or underestimated; thus, having clearly established processes and identified persons to support the service through the change is key to address issues as they arise. Providing on-the-ground support to service users and staff is valuable in the early weeks or months of implementation. Furthermore, gathering feedback from all user groups, as they use the technology-enabled solution in practice, refines the technology and service model, but more importantly, the workforce and structural changes required to improve service quality ( Figure  2). As illustrated in Figure 1, ongoing feedback from service staff (eg, health professionals, administrators, and service managers) is collected via Web-based surveys, semistructured interviews, and workshops to evaluate and monitor the impact of embedding the technology-enabled solution in the service, including (1) service-level changes in outcomes and processes, as well (2) as the digital readiness and competence of service staff, (3) quality, usability, and acceptability of the solution, and (4) social return on investment of embedding the solution in the service. In addition, feedback from both staff and consumers about existing and newly designed functionality is captured through quarterly user testing sessions. The fortnightly Implementation Officer Logs are also used to gather data from service staff, including anonymous commentary and feedback provided by individuals with lived experience, including consumers and their families or supportive others, who are engaging with the solution as part of care, which are then fed back through to R&D processes to inform the iterative design and development of the technology-enabled solution. This feedback may include technical difficulties, as well as comments in relation to the user experience and clinical aspects of the solution. Google Analytics is also embedded within the solution to allow back-end analysis of user behavior, including details regarding the features of solution that consumers use most consistently, as well as those features with which they disengage most quickly. Of note, in the absence of a 24-hour monitoring protocol to ensure the safety of all users at this time, there is no option for consumers to provide free text feedback. Importantly, it is generally accepted in digital mental health research that the technology-enabled solution will be iteratively developed, designed, and refined during implementation. The expected outcome of Phase 3 is for the technology to be embedded and integrated within the service, such that it is seen as a vital piece of standard care, enabling and maintaining ongoing service quality improvement and reform.

Phase 4: Sustainment and Scalability
In accordance with Lennox et al [39], the primary objective of Phase 4 is the continuation and maintenance of our technology-enabled solutions and their associated outcomes within a health service, as well as the iterative process of evaluation and design, to address problems and emerging needs and demands of the service, individual populations, and the broader context [15]. To achieve this aim, prospective approaches are employed throughout the preceding phases to build relationships with and foster buy-in from key stakeholders (ie, consumers with lived experience and their families, health professionals, administrators, and service managers), as well as to iteratively design and refine the technology-enabled solution to adapt to the changing needs of the stakeholders and service [39]. Employing these processes helps to ensure the continued effectiveness of the technology-enabled solution, including improved access to care and resources to promote mental health and well-being, the integration of the technology within the service, and community ownership of the solution [40]. When taken together, this reflects the primary outcome of Phase 4, namely technology-enabled mental health services reform.
Furthermore, the objective of Phase 4 is to leave a configurable technology-enabled solution in place for ongoing and continued benefit to the service, following the replacement of a locally available Implementation Officer with a more sustainable supporting resource that is readily available remotely.

Participating Centers
At the time of this publication, the implementation science protocol had gone live in the following participating centers: headspace services (Ashfield, Camperdown, Coffs Harbour, Hurstville, Lismore, Port Macquarie, Miranda, and Tweeds Head, New South Wales, and Edinburgh North, South Australia), Butterfly's National Helpline, and Open Arms-Veterans and Families Counselling Surry Hills, New South Wales.

Sample Size
The protocol phases do not have an upper or lower limit on the number of participants, as this will vary by participating center, both in relation to the number of staff members (eg, health professionals, administrators, and service managers) and the size and diversity of their consumer base.

Data Analyses
Qualitative and quantitative data analyses will be conducted to assess the success of the implementation at the level of the consumer, health professional, and service, and, where possible, comparative analyses will be run between and within participating centers and populations, to allow for the identification of commonalities and differences in outcomes.

Ethics
Ethics approvals to conduct all aspects of the protocol are sought from the relevant governing Human Research Ethics Committees for the participating centers.

Results
At the time of this publication, all participating centers were in Phase 3. The first results from Phases 1 to 3 are expected to be submitted for publication in 2020, with Phase 4 data expected thereafter.

Discussion
The international goal of substantially improving the quality of mental health services is central to many technology-based innovation implementation efforts in mental health service delivery. As seen in a range of other industries, technology-enabled disruption brings with it significant changes to conventional practice and experience for all stakeholders. The greater the gap between the innovation and conventional practice, the greater the implementation challenge and, arguably, the greater the need for technology and service co-design with all stakeholders. This protocol incorporates the findings that affect implementation success in the rapidly evolving implementation science literature, to serve as both preemptive mitigation strategies and foci for surveillance throughout each of the implementation phases. With the aim of avoiding obsolescence of the solutions, our implementation science protocol also stresses the parallel and iterative evaluation of the effectiveness of the technology-enabled solution alongside the success, or lack thereof, of the implementation. studies to design, develop, build, and evaluate technology-enabled solutions (using the InnoWell Platform) in four populations across the lifespan, including young children and their families, young people, adults, and healthy older adults. Beyond June 2020, InnoWell will support the R&D work financially, as it represents the core work necessary to continuously and iteratively evolve the technology-enabled solution, and its role in redesigning health care aligned to key reform priorities, including person-centered care, shared decision making, staged care, and data-driven clinical decision making through routine outcome monitoring.

Authors' Contributions
IBH, TAD, and SC were integral in securing funding to support the study. All authors contributed to the development of the implementation science protocol. HML prepared the first draft of the manuscript. All authors contributed to and approved the final manuscript.

Mental Health and E-Mental Health in Canada
In 2012, more than 1 in 6 Canadians reported that they had a need for mental health care in the past 12 months [1]. However, there is a shortage of services and support. The Mental Health Commission of Canada has specifically discussed the importance of e-mental health [2] as part of a multipronged strategy to support the mental health needs of Canadians. Electronic mental health (e-mental health) is defined as "mental health services and information delivered or enhanced through the internet and related to technologies" [3].
e-Mental health could transform the current care system in Canada in 2 ways: (1) by empowering patients by providing them with health information and (2) through the use of specific technologies and electronic interventions such as Web-based therapy [2].
A number of studies have conducted an evaluation of informational websites on various mental health topics [4][5][6][7]; however, there is a lack of experience with Canadian mental health portals. Hence, it was decided to conduct an evaluation of eMentalHealth.ca using commonly accepted evaluation methods for website evaluations. To help users see that the site's information is objective and trustworthy, eMentalHealth.ca is certified as following the Health On Net code (HONcode) Principles from the Health On the Net Foundation. The HONcode certification follows a code of ethics that guides websites to provide quality, objective, and transparent medical information [10].

Rationale and Development of eMentalHealth.ca
The original eMentalHealth.ca website was created for the general public. In response to requests from different audiences using the site, 2 additional portals have been created: (1) eMentalHealth.ca/PrimaryCare for primary care providers such as family physicians, nurse practitioners, and pediatricians and (2) eMentalHealth.ca/MedicalStudents for medical students.

Case Example
Jennifer is a 20-year-old woman who is feeling stressed and tired, and she wonders if she might be having signs of depression. Her family physician is away on holiday, so Jen does what many Canadians do: she uses the internet. She finds the eMentalHealth.ca website and sees that it is a publicly funded hospital and university initiative. She fills out a screening questionnaire for depression (which uses the validated PHQ-9 scale) [11], and it shows that she may have depression. The website recommends speaking to a health care provider and provides more information and resources about depression, including local mental health services. She has not yet seen her family physician, but she is feeling more confident about what steps to take to cope with her situation.

Web Analytics
Google Analytics was used to determine the number of users and the number of sessions since the launch of each website as well as demographic information about the visitors, including the countries and cities associated with the users, the frequency and recency of the sessions, and the top site contents viewed. Google Analytics is a widely used Web analytics service offered by Google that tracks and reports website traffic [12].

Self-Administered Surveys
Web-based self-administered surveys were developed to gather data on users' characteristics and other standard questions used in website evaluations. Questionnaires were based on the Commission of the European Communities' quality criteria for health-related websites [13] and on studies by Tlach et al [4] and Kuosmanen et al [5]. The surveys were reviewed and piloted internally by hospital and university colleagues who did not have background information on the website evaluation project. A weekly draw for a Can $50 gift certificate from Amazon.ca was held for 8 weeks to encourage survey participation. The following items of the Checklist for Reporting Results of Internet E-Surveys Checklist [14] for reporting electronic survey results were also considered: design, Institutional Review Board approval and informed consent process, development and pretesting, recruitment process and description of the sample having access to the questionnaire, and survey administration and analysis.
To view a copy of the questionnaire, one can visit this link: https://www.surveymonkey.com/r/T8GSQBN. The surveys collected general demographic information such as gender, age, and type of user (general public, student or health professional, eg, family physician, nurse practitioner, psychiatrist, or other). In addition, questions assessed satisfaction with the design and content of the website, frequency of use, users' perceived trust about the information, and future intentions. All questions were Likert-type scale responses. At the end of the survey, there was 1 open-ended question for any comments about the websites, and respondents were also asked if they would like to be contacted to participate in a focus group. The responses to the open-ended question were single coded by 1 coder. The open-ended comments were used to provide overall and general insight into the users' experience. In the scope of this study, this analysis was not aimed at deeply understanding the qualitative experience of the users. Emerging themes and comments were assessed.
For the English version of the website, 53 respondents provided a viable comment (out of 207 respondents), and for the French version, 17 respondents provided a viable comment (out of 153 respondents).
The collected data on user characteristics and users' satisfaction were analyzed on SAS statistical software [15] using descriptive statistics, and the responses to the 1 open-ended question were analyzed by qualitative content analysis [16] using NVivo qualitative analysis software [17].

Ethics
An ethics application was submitted to and reviewed by the Ottawa Health Science Network Research Ethics Board. The review indicated that the project fell within the quality initiative, quality improvement, quality assurance, and/or program evaluation category, and a confirmation letter was received.

Results
Web-based surveys were posted on the main page of eMentalHealth.ca and eSantéMentale.ca, and the survey results included in this paper were collected from June to November 2017. Table 1 summarizes the Web analytics (from Google Analytics). There were more female users than male users, and the majority of users were 35 years and older. These two demographic characteristics are also reflected in the survey results.

Users' Satisfaction With the Websites
In the surveys, users were asked to rate the specific features of the websites (eMentalHealth.ca and eSantéMentale.ca). They were also asked to rate statements to measure the users' satisfaction with website design and content, perceived trust, frequency of use, and future intentions to use the website. Figure  1 demonstrates users' ratings of specific features of the websites.

Were the Website Features Helpful?
Most users of eMentalHealth.ca and eSantéMentale.ca found the specific features of the websites helpful: close to 80% of users strongly agreed or agreed that the Find Help and Info Sheets were helpful to them; and over 50% of users strongly agreed or agreed that the rest of the features (Screening Tools, Mental Health News, Mental Health Events Calendar, and Directory of Research Studies) were helpful to them. In all the surveys, very few users (less than 5%) disagreed or strongly disagreed that the specific features of the websites were helpful to them.

Were the Users Satisfied With the Website?
Most users of eMentalHealth.ca and eSantéMentale.ca reported being satisfied with the websites. For eMentalHealth.ca, more than 75% of users strongly agreed or agreed with each statement except one: just 63% of users strongly agreed or agreed that "All relevant information can be found on the front page." In addition, 88% of users would recommend the website to others and were planning to visit the website again. Similarly, for eSantéMentale.ca, more than 70% of users strongly agreed or agreed to each statement except 2 statements: 60% of users strongly agreed or agreed that "All relevant information can be found on the front page" ("Toutes les informations pertinentes se trouvent sur la page principale"), and 64% strongly agreed or agreed that "I plan to visit the website again" ("Je planifie de visiter le site web sous peu").

Overall Satisfaction With the Websites
In both surveys, 90% or more of respondents were very satisfied or somewhat satisfied with the website overall (eMentalHealth.ca and eSantéMentale.ca). Table 3 presents the overall satisfaction rating by the users of the websites.

Open-Ended Comments From Users
Survey respondents left comments at the end of the survey. Most comments were positive, stating that the website was a good resource for finding helpful mental health information and services. A few examples are shown in Table 4.
Negative comments were very helpful most of the time as they had clear suggestions on how the website could be improved. Mainly, comments were related to difficulties with setting the location of interest. Respondents also mentioned that the main page could be too busy and that some information on the website was outdated. Several examples of comments are shown in Table 5.  Table 4. Open-ended comments from users-positives.

User type Comment
Researcher "This is a great initiative because generally it is hard to find up to date information for resources in some areas..." Clinical psychologist "To help bridge the gap between assessment and treatment, I print off the info sheets to provide to parents during debrief." Caretaker (general public) "This website was surprisingly well-equipped to provide well-rounded groupings of information on mental health." Psychologist "Overall, this is an excellent resource for clients and other professionals who are researching where to find specific services." Social Worker: psychotherapy "This is a fabulous resource. Thank you!" General public "The directory is a very good and helpful resource." Counsellor/psychometrist "Very informative. Glad this tool is available for the community. It would be nice to see an info sheet and/or screening tool for compassion fatigue." Counsellor "This is the most comprehensive website for mental health resources. I use it regularly and recommend it to staff and patients." Table 5. Open-ended comments from users-negatives.

Improving the design, especially of the homepage
General public "The design is not very nice. The style itself is a little old." (translated from French) and "A lot of information at once (visually) on the main page." (translated from French) Researcher "Perhaps an even simpler design for the site, there is a lot of information presented at once. For the general public the main purpose is showing them resources, so everything to the right with the news and tips doesn't need to be in the locations where they are. The main buttons under 'find mental health help in your area' is all you need there, because sometimes maybe things are repeated and someone may not know which one to click." General public "A little confusing at first as there is a lot of information on the front page..." Psychotherapist "The website front page could use a serious design overhaul. It is too wordy...overall too much going on making it difficult to navigate."

Info sheets
General public "The info sheet for depression was simplistic." Psychiatrist "Info sheets are best part of the website but they are in general too long (try for 2 pages max)."

Improving the resource directory
General public "Little info about my hometown (Pembroke) but that may be because there is little info about counselling/help offered here. An alternative could be online counselling or resources via skype, IM offered by hospitals for this isolated and hopeless rural hell and other similar hells where people are isolated and can't get out of. There is no public transit here so accessing resources even across town is difficult for many." Psychologist "Many of the links offered are in cities other than Calgary; in fact, many of them are in Toronto." (User from Calgary.)

Improving findability of local resources
Psychiatrist "Location based search doesn't narrow down location as it should and mental health directory has too many search results and are too broad."

Principal Findings
The goals of this study were to collect information about the websites' users and to describe the user population, as well as to assess the users' satisfaction with the websites in terms of design and content, perceived trust, frequency of use, and future intentions to use the website.
First, as predicted by the Google Analytics results, the user population was, in majority, female and aged 31 years and older. Most users of the English and French websites were categorized as members of the general public, which is the targeted user population of the websites. About 90% of users of the English website had visited the website more than once. The French website had more first-time users. Google Analytics is a simple, free way to gather demographic information about users, whereas Web surveys are much more labor intensive. As shown in our surveys, the demographic results of Google Analytics were representative of what Web surveys reported. For future surveys, one could use Google Analytics for demographic data such as gender, age, language, and location of users, and the more burdensome Web survey could be shortened.
In terms of user satisfaction by language, it was noted that more English users were very satisfied than French users (59.8% of English users vs 31.7% of French users). Conversely, it was easier to make French users somewhat satisfied at 64.8% versus English users at 30.2%. It was unclear why this might be. However, the total numbers that were very satisfied and satisfied were rather similar-English users at 90.0% and French users at 96.5%.
Website survey results showed that the most highly related features were (a) the Mental Health Resource Directory listing mental health services and (b) Info Sheets about mental health conditions and topics. These are the areas that will be prioritized in the ongoing improvement of the website.
The open-ended comments from surveys showed that most respondents felt positively about the website. There were consistent themes, such as appreciating the resources directory, the information sheets, and screening tools. The project team was concerned whether entering users in a draw for a prize might create mainly positive feedback, but fortunately, this was not the case. There was a good amount of constructive negative feedback from the survey. Themes included complaints about the home page usability, findability, and searchability of the website. Negative feedback is most valuable for making changes and improvements to the website.
Top areas for improvement might include the following: 1. Improving the home page to make it less overwhelming and easier to use. The literature on home page usability recommends reducing the number of clicks [18]. Given that info sheets are popular, it might be helpful to make top info sheets clickable from the home page. 2. Improving the resource directory. Key resources should be included for major cities in Canada. 3. Further exploration of how to improve the mental health info sheets. Some users wanted briefer versions, whereas other users wanted more detailed information. Different users may have different information needs, and it might be helpful to have both brief and longer versions of info sheets.
This evaluation will help with improving the English and French websites for their users, in the ongoing effort to provide accessible, trustworthy, and credible information for those with mental health needs and for the family members and professionals who support them.

Limitations
There were a number of limitations to this study. First, small sample size of the respondents to the Web-based self-administered surveys caused by a low participation rate means the survey results may not be representative of all website users. For example, many users suggested that the home page should be improved. Hopefully, they are representative of most users and not simply a vocal few. One way to address this in a future study (after the website is updated) would be having a long Web survey period. Second, challenges remain in distinguishing expectations and preferences for the different user populations. eMentalHealth.ca targets a wide audience, including the general public, primary care providers, and medical students. Web survey results were positive for all groups of users, and there were many similarities in the types of feedback, whether users were from the general public or professionals. It may be that larger numbers of surveys would be required to see the differences between the different user groups or perhaps a different form of data collection such as focus groups or individual interviews. Third, the time frames of data collection for website surveys and Google analytics data were not same. Google analytics data were collected for January 1 to December 31, 2017, and it would have been ideal to collect Web surveys over the same time period. Unfortunately, this was not possible because of a lack of resources to gather Web surveys over that time period (lack of funds for a weekly draw for a whole year, and lack of funds for research coordinator time). For future evaluations, efforts will be made to secure more resources for a longer period of Web-based surveys. Finally, in an effort to make the Web survey questionnaire as brief as possible, there were few open-ended fields. In the end, however, the open-ended fields were extremely valuable, including negative feedback. In the next evaluations, it would be greatly helpful for continuous quality improvement of the websites to have more open-ended fields and/or to provide the option of calling survey participants for a follow-up interview to solicit more detailed comments.

Conclusions
eMentalHealth.ca and eSantéMentale.ca were viewed positively by most users. Users appreciated the mental health information, screening tools, and where to find mental health resources. However, users expressed concerns about technical issues, including homepage usability (ie, finding the homepage cluttered). Feedback from this evaluation will be used to upgrade and improve future versions of the website, which will help both the general public and the professionals that use the website.

Introduction
Background Dementia, the umbrella term for age-related disorders characterized by a decline in cognitive ability, is expected to double approximately every 20 years to affect 74.7 million people by 2030 and 131.5 million people by 2050 [1]. Efforts to improve diagnosis, treatment, and support are growing ever more important. Dementia is generally preceded by a predementia stage known as mild cognitive impairment (MCI), an intermediary stage on the continuum between age-adjusted healthy cognitive ability and dementia [2]. At this stage, daily activities can still be performed with minimal difficulty, and there is scope for intervention to impede further deterioration [3]. Any method to support the diagnosis of MCI as early as possible could therefore be of great benefit to millions of people. As such, older adults, some of whom seemingly healthy, are often referred for screening.
Neuropsychological tools for screening predementia stages are costly and not always accurate [4]. As one of the earliest clinical manifestations of cognitive impairment is topological disorientation [5], spatial navigation memory tests are used for diagnosis. These tests are generally conducted with pen-and-paper tasks, such as the Mini-Mental State Examination [6]. Computer administrated tests [7] and use of virtual reality (VR) [8] are explored more recently.
VR systems have been used as assessment tools [9,10] for physical activity [11], cognitive assessment [12], and balance assessment [13]. Studies on nonimmersive virtual environments (VEs) found those effective for assessments [8] because of being accessible and feasible while providing controlled settings for conducting cognitive sessions. Recent studies have shown that VR can facilitate information recall [14][15][16], an important consideration for measuring cognitive decline. It is therefore important to explore immersive VR as an important technology for spatial navigation because of its ability to map real-world functioning [17]. In addition, it can enhance episodic memory, that is, autobiographical memory of past temporal events, in elderly [18] and is deemed feasible as a cognitive training tool [19].
A VR platform can equip clinical neuropsychologists with a feasible assessment technology on which the setting (and therefore the assessment outcome) is generalized to real-life settings [20]. This requires that human-computer interaction (HCI) and VR technology researchers work closely with clinicians to develop new forms of interactions, such as critical dementia proposed by Lazar et al as a "lens onto the ways people with dementia are positioned and engaged by the field of HCI" [21]. Such attempt mandates a close study of the needs and requirements of the potential users to create a system with good usability. On the basis of the above, we examine the feasibility of VR technology for mediating information recall test. A number of studies provide evidence to support that choice and guide our research on the VR testing platform for older adults proposed in this paper, as outlined below.

Mental Models in Virtual Reality as a Mediator of Contextual Representation
Paper-based cognitive assessments, particularly spatial tests, are often a departure from realistic situations [5] and familiar mental models of individuals in everyday life environments. It has been long established that spatial orientation requires identifying many cues such as self-to-surrounding relationships and object-to-object spatial relations [22]. Several studies suggest that VR with realistic settings aids better information recall for both spatial and episodic memory [14,23,24]. Furthermore, head-mounted displays (HMDs) have the potential to induce a sense of presence in the VE, that is, the perception of being there.

Factors Influencing User Engagement
Several factors often impact user engagement with VR including presence, gamified designs [11,25], and natural body interactions [26,27]. Motion sickness, despite recent improvements in VR technology, still has a negative impact on user engagement [12,28,29]. Furthermore, older adults have additional requirements that need to be considered in designing the system: most commercially available HMDs are heavy [30,31], have nonintuitive controllers [32,33], can cause stress or hesitation [33], and large proportion of VR content not specifically designed for this cohort [32,33], all of which negatively influence experiences and engagement of older adult users with the system. There are constant improvements in VR devices and controllers (eg, stand-alone HMDs); however, more investigations are required to design and develop engaging content and interactions suitable for older adult users. Finally, and more relevant to our study, better engagement with the system will likely influence the memory test outcomes in VR and contribute to a better sense of well-being for the individual involved in predementia screening.

Factors Influencing Perceived Usability and Competency of the Device and Environment Relevant to User's Capabilities
A number of VR systems are designed for adult users [12,25,26,29,34] without direct consideration of their specific usability needs. Needless to say, such considerations become more significant in systems that aim to assess memory and detect cognitive impairment in older adults. In addition to impeding user engagement, poor usability in a cognitive test could impact the assessment results and consequently disrupt the validity of the outcome. Furthermore, poor usability of system controls can hinder the user's sense of presence and, subsequently, the experience of the system [35]. Most developers apply general usability principles and techniques such as the classic Nielsen's heuristic evaluations [36], recommendations on usability for VE [37], and tailoring walkthrough methods for nonimmersive VR applications [38]. Current literature has focused on 3D environments, largely covering nonimmersive VR and younger adults of these systems. However, these should be extended to include older users' competencies [39] and usability constraints of HMDs, new controllers, and motion trackers for older users. Currently, there is a noticeable lack of such evidence in usability practices for immersive VR.

Factors Influencing User's Stress and Motion Sickness
Another area of inquiry is the propensity of VR for causing motion sickness and, subsequently, stress. As cognitive disabilities (eg, dementia) often correlate with other mental health problems such as depression [40], it is important to mitigate the effect of any additional stress. Although designing VR environments for older adults and, in particular, those with memory deficit may highly benefit the screening process, it is important that the cognitive load is manageable, and the negative mental impacts on users are minimized. Researchers have found older adults tend to be reluctant to use new forms of technology [41] that can induce simulator sickness [42] and find immersive VR devices, controllers, and environments intimidating.
Providing natural interaction styles with the system, with clear instructional tutorials to familiarize users with the VR device and environment, as well as the VR task, might reduce the stress level. Furthermore, motion sickness can have a detrimental impact on assessment validity or result in discontinuation of the assessments before completion. To reduce motion sickness, users need to have control over their navigation, while avoiding sudden head movements, and exposure to unexpected changes in scenery and orientation.
This paper contributes to the design and evaluation of VR-CogAssess, a new VR platform using photorealistic imagery to assess topological cognitive impairment (ie, spatial navigation memory) as a tool for predementia diagnosis. We test VR-CogAssess with older adults to explore 3 goals. First, we investigate the compatibility of VR-CogAssess compared with a standard personal computer (SPC) setup in an experiment. We assess that based on participant's performance in landmark recall test using measures such as recall of challenging locations, test duration, and perceived presence in VE. Second, we explore the scope of usability considerations needed for VR memory assessment platforms for older adults to support their interaction. To achieve that, we examine the efficiency of system controls and participant's perceived enjoyment of using the system. Finally, we study the feasibility of using a VR platform as a memory assessment tool for spatial navigation for older adults. This was assessed based on overall test session time, alignment of the system with computer abilities of the users, and user's level of stress, which is critical for supporting their well-being. We posit that a VR memory test platform should be designed to accommodate users' abilities and computer literacy, without distracting them during the test (for instance, because of the novelty of the system) as that might result in users underperforming in their test or abandoning it before completion.
This is a very underinvestigated research area. Our study has the unique advantage of exploring the feasibility of VR using a photorealistic VE for spatial navigation assessment with older adults. Developing such a system is less complicated and more cost-effective than developing a full 3D environment. To achieve that we identify a set of propositions for designing VR systems for older adults that address their needs and then develop VR-CogAssess based on those user needs. An introduction to the propositions and the platform is presented in Methods section, along with details of a study where we examined the feasibility and the usability of the system. On the basis of the results, we discuss those propositions and further considerations for future studies.

The VR-CogAssess Platform
In this section, we describe the VR-CogAssess system and identify a set of 5 design propositions based on the literature and our own design experiences during the project to meet the unmet needs of older adults in technology use. Those propositions guided our design and development process and selection of components. Where applicable, we refer to those as design propositions followed by a number.

Architecture Overview
VR-CogAssess is built using the Unity game engine, and its library and controllers are written in Microsoft C#. Three-dimensional visuals are fetched from the Google Street View application programming interface (API) and rendered visually as 360° panoramas to the user through an Oculus Rift VR HMD. A Microsoft Band smartwatch reads physiological signals including heart rate variability (HRV) and galvanic skin response (GSR) during the assessment task. These data are then sent to an Amazon Web Services cloud service for storage through an Android mobile phone app to supplement analysis on stress. Finally, the platform supports user navigation of the VR environment through a CH Products Flightstick joystick. The setup of VR-CogAssess with main components and visual interface are shown in (Figure 1).
The VE interface is rendered through the HMD and provides a photorealistic environment with a continuous field of view around the user. Unlike SPC displays where users typically control the directional view by manually pressing left or right on a keyboard, the VR system enables the users to simply turn their head in the desired direction as the HMD monitors axis rotation against a reference point. Taking elementary usability considerations into account such as larger font and object sizes, we also designed the interface by identifying and incorporating guidelines for older users. Over multiple iterations, we then revised the system based on input and feedback from clinical neuropsychologists with extensive experience in cognitive assessment for individuals living with MCI. The simplicity of the interface is by design, for which we used a set of design propositions, as shown in Textbox 1. We then describe how those propositions were integrated into the design of VR-CogAssess. Of note, we do not aim to validate these design propositions generated from the literature; rather, we examine their feasibility to design a VR assessment system for older adults. • Design proposition 1: designing for simplicity should consider reducing interface features to the minimum necessary for the task [43].
• Design proposition 2: physical gestures to operate navigation controller should be paralleled with those of the virtual space to cater for optimum hand-eye coordination. It should also allow sensitivity calibration to control the speed of user navigation in virtual environment.
• Design proposition 3: to achieve a sense of control and avoid stressful interaction, the system should be optimized to provide users with a sense of autonomy and choice [44].
• Design proposition 4: the system should teach the user 1 skill at a time and have multiple controlled, momentary stops during the interaction so that users can seek support from the test facilitator [45].
• Design proposition 5: the platform should be customizable to accommodate new memory test protocols with personalized features.

Visual Interface
The design proposition 1 (Textbox 1) was implemented in the interface by limiting the task status cues. These include distance covered, time spent, a compass bar for assisting orientation, and a white indicator to point users to the locations they can go to. To simplify, this interface only highlights the interaction that is task related. To further improve the usability, the interface also repeatedly displays textual captions such as instructional hints or reminders about remaining time.

Navigation Controls
The navigation control in the VR-CogAssess environment is achieved using a joystick. Tilting the joystick allows the user to change the angle of view for turning corners, and pressing a large button allows forward movements in VE.
Vallejo et al [46] compared 3 different button controllers including the Razer Hydra motion sensing controller and touchpad and found that a joystick was preferred by the participants in navigation tasks. Given the reduced fine motor control and hand-eye coordination skills in older adults [43], our platform allows sensitivity calibration to control the speed of navigation in VE, based on design proposition 2 (eg, movement speed and transition to next locations).
Given that older users at risk of dementia might lack skills for operating computers or other interactive systems, it is essential to provide them with a sense of agency and volition for system's controls to support these interactions. This will contribute to their enjoyment and engagement with the system [47,48] and could potentially reduce performance anxiety. In summary, our system supports the sense of agency in navigating the environment based on design proposition 3 in a way that reduces confusion and allows a feeling of being in control of the navigation. Users are free to move in any direction with a natural tilt angle of the controller and a press of a button.

Task and Tutorial Implementation
The task involves a landmark recall test designed by author SN at the Brain and Mind Centre, The University of Sydney. Users navigate the platform environment from a starting location and are asked to identify 6 landmarks. The landmarks are scattered at different points along a designated navigation path and intentionally vary in difficulty (including 2 challenging landmarks) to locate to support performance discrimination. To avoid confounding spatial navigation ability with the user's ability to use the technology, we implemented an introductory tutorial (5-min long). The tutorial provides a computer-synthesized voiceover for delivering the instructions for completing the task. This reduces the workload of the facilitator and enforces a level of consistency in the information received across participants.
The system displays on-screen textual hints and audio instructions, one at a time aligned with design proposition 4. There is considerable scope for task customization in this platform guided by design proposition 5, as any location available in the far-reaching Google Street View API can be used [49]. The Unity controllers for managing the environment to the HMD is also written to allow other APIs. Owing to this modifiability, the platform can be repurposed to other potential spatial navigation assessment tasks. For example, although the current task is designed to assess spatial navigation ability in an unfamiliar location to the user, the location can be personalized. This can be an immediate vicinity of the user's home, neighborhood, or city, which may assist health care practitioners in evaluating or providing intervention based on how a person performs in a familiar setting.

Study
In this section, we describe the 2 conditions of the study: (1) we use VR-CogAssess for a landmark recall memory test and (2) we use an SPC setup. The VR-CogAssess platform was designed with clinical neuropsychologists to capture task variables including task duration, date, distance traveled, navigational mistake count, landmark recall count, and speed and auxiliary data such as user details, heart rate, and GSR. VR-CogAssess facilitates data collection for clinical practitioners by capturing multitudes of variables mentioned above. Data collection and processing are managed automatically in VR-CogAssess to reduce workload, cost, and save time. For example, counting navigational mistakes would require the test administrator to rigorously monitor participants for the whole duration of a task. Instead, a Unity controller is programmed to record the identification of locations visited outside a prerecorded correct path. Furthermore, the actual trail taken by the participant is visualized on task completion based on recorded sequence taken. Physiological data on HRV and GSR from a Microsoft Band is timestamped and recorded in XML format and stored on a cloud server. This captures changes in the stress level of participants, as excessive stress can be detrimental to confidence on assessment scores.
In SPC condition, users were expected to navigate the same locations in recall test as was assigned to VR users. However, there were differences in navigation controls (standard keyboard arrow keys were used in SPC), and the visual interface was a standard monitor. A 5-min long tutorial was presented to SPC users, same as in VR condition, to familiarize them with navigation controls and test environment. In summary, the 2 conditions presented the same locations in recall test, but there were differences in navigational controls, data recording method (administered by researcher), and the visual interface.
We investigated the following research questions (RQs) aligned with our initial aims of evaluating immersive VR platform for spatial navigation.
• RQ1: Are there any differences in assessment outcomes between the VR and SPC conditions?
• RQ2: Are there any usability differences between the VR and SPC conditions?
• RQ3: Is it feasible to use VR-CogAssess for older adults to complete the memory test assessment with minimal stress level, given their computer skills and competency?

Participants
Participants were recruited at a community center in Sydney, Australia. Participants were healthy older adults (N=42; mean age 73.22 years, SD 9.26). They all gave informed consent for this study approved by the Human Ethics Committee at authors' university (protocol #2016/629). Participants were randomly allocated to 1 of the 2 study conditions, whereby they completed a landmark recall test using either the VR-CogAssess platform with the joystick control (VR, n=22) or a standard computer setup running Google Street View (SPC, n=20). Both conditions shared a similar starting point, environment location, and the recall assessment task.

Study Introduction
At the start of the session, participants were introduced to the study and gave informed consent. Participants then proceeded with the next step to complete the pretest questionnaire.

Pretest Questionnaire
In a pretest questionnaire, participants were asked to provide information about age, highest level of education, and basic computer skills proficiency using a range of options (see Table  1). Participants then proceeded with learning about the platform.

Platform Learning
This step provided participants with essential skills to complete the upcoming landmark recall test and become familiar with the relevant platform. Participants completed 1 of the 2 tutorials depending on the study condition they were randomly assigned to. The tutorials shared the same location (Sydney Harbour Bridge; Figure 2) and involved learning the same basic navigation actions (move forward or back and turn left or right). In SPC condition, the standard keyboard up, left, and right arrow keys were used as navigation controls, whereas in VR condition, a joystick was used. Moving forward in VR was achieved by pressing a front trigger with the thumb and turning involved tilting the joystick left or right. Participants learned these navigation controls over a series of instructions that involved moving toward a bridge, turning to identify another landmark, and then moving back to the original location. This step was verbally administered by the researcher in the SPC condition, whereas VR condition had built-in tutorial with synthesized voice for guiding participants. All participants were asked to wear a Microsoft Band 2 to track physiological signals before and during recall part of the test. This allowed the researchers to record baseline signals before the test phase.

Landmark and Navigation Memorization
This step required participants to memorize the appearance of and navigation path to 6 landmarks. All landmarks were in Cambridge, United Kingdom, and were selected because they would be interesting but in a foreign country (therefore less likely that participants would know the place well). These landmarks (highlighted in Figure 2) Figure  2).
We first explained the 2D map: starting location, path, and the 6 landmarks. The landmarks were located on different points along a path and were intentionally chosen to create various difficulty levels. Large images of the 6 landmarks were also printed and made available to the participants. We informed the participants that landmark name and number were not required for information recall test. Participants could go back at any time if they missed a landmark during the task. Although a map was available for the memorization phase, individual landmark pictures were shown to participants one by one for only 10 seconds at a time. On presentation of each landmark, its location was also indicated on the map. This was to ensure that the participant relied on their memory the landmark location (rather than its appearance) in the main recall phase. The process was repeated twice to provide participants with sufficient opportunity to memorize the location and appearance of landmarks.

Mini Quiz
Participants were then asked to complete a paper-based mini quiz to identify the landmarks learnt in the previous step. Participants who could not identify at least one landmark correctly would be eliminated. This quiz included a combination of 2 correct and 2 incorrect landmarks. As this step involved further opportunities for viewing some of the correct landmarks, different copies of quiz ( Figure 3) were used with various permutations of correct and incorrect landmarks. Participants were not informed of their performance on the quiz to not pronounce their landmarks learning at this stage.

Landmark Recall Test
Participants were allocated to a condition matching the learning phase and were tasked with identifying 6 landmarks. A brief reminder instructed participants to identify the 6 learnt landmarks within 15 min. Participants could ask any question they might have before the test begins. Physiological signals were recorded for the entire session via an Android App. We also recorded the number of correct landmark recalls, the specific landmarks identified, and number of time navigational mistakes were made (wrong turns made).

Posttest Questionnaire
At the end of each session, participants completed a questionnaire to provide an assessment of their perceived competence, presence, intuitive controls, enjoyment or interest, and pressure or tension during the test. These scales are based on a standard questionnaire, Players Experience of Need Satisfaction (PENS) [50] and Intrinsic Motivation Inventory (IMI) [51]. We added an additional open-ended question, "Do you have any other comments, feedback and recommendations?"

Assessment Ability and Outcome
The 2 conditions were compared to establish the extent to which navigation performance and assessment scores are comparable in the 2 conditions. This pertains to RQ1 and is based on recorded landmark recall test information, physiological data, and posttest questionnaire.

Correct Landmark Recall and Navigation Mistake Count
The number of landmark recalls and navigation mistakes were recorded for participants who were randomly assigned to 1 of the 2 conditions (SPC or VR). We define higher spatial navigation ability based on higher landmark scores and lower mistake counts. Although participants in both conditions had similar cognitive status (did not declare any cognitive impairment before the test), on average, those in the VR condition identified more landmarks correctly and made less navigational mistakes. A t test suggests there is a significant difference between the 2 conditions for correct landmark recalls (t 40 =−3.02; P=.004) and navigational mistakes (t 40 =2.11; P=.04).
Cohen d suggests a notably large effect size for landmarks recall count (d=0.94) and medium effect for navigational mistakes (d=0.65). Table 2 summarizes detailed results.

Identifying Challenging Landmarks
It was hypothesized that challenging locations might have different recalls in different conditions. Challenging locations were intentionally outside the immediate field of view of the participants with Bath House being particularly challenging, as it required the participant to navigate through multiple streets. We found significant differences ( Table 2) for 1 of the 2 challenging landmarks, the Bath House; (t 40 =−3.03; P=.004) with very large effect size (d=0.95). VR-CogAssess closeness to real-world spatial information was also approved as a challenging landmark that involved multiple turns was identified more times in VR condition than SPC.

Task Duration
We recorded the task duration in each condition as a measure for the platform-specific assessment ability. We expected participants in VR condition to complete the recall task faster, as the CogAssess platform enables better spatial navigation. However, results (in Table 2) suggest no significant differences between the 2 conditions. This measure, however, may not be very accurate because of certain design decisions made in relation to VR-CogAssess. For example, we intentionally designed a slow joystick turning speed that adds to task duration for maneuvering. Therefore, the number of correct landmark recall and navigational mistakes remain the major measure for assessment ability.

Usability
In this section, we examine the result from the posttest questionnaire in relation to RQ2: "To what degree is the VR condition usable and enjoyable to the participant?" In the pretest questionnaire, when asked about technology competencies in SPC condition, 65% (13/20) participants reported beginner level and a need for assistance with Web browsing, emails, and use of keyboard and mouse, whereas 20% (4/20) participants in SPC condition self-reported competent in basic computer skills. In the VR condition, 36% (8/22) participants required assistance, and 59% (13/22) self-reported being competent in computer skills.

Intuitive Control
The measures of intuitive control from PENS questionnaire indicate a perception of usability: (1) "learning the task controls was easy," (2) "the task controls are intuitive," (3) "when I wanted to do something in the task, it was easy to remember the corresponding control." On the basis of the results for those questions, we found no significant differences between the 2 conditions. Usability of controls was critical for users to perform landmark recall test quickly and with minimal frustration following a basic pretest tutorial.
After the memory recall test, the average rating reported for intuitiveness of the controls used was lower in the VR condition (mean 4.45, SD 1.59). A cross-tabulation suggests most participants, regardless of the condition they were assigned to, reported midrange scores (SPC=10 and VR=12). Nonetheless, relatively higher number of participants reported high perceived intuitive controls (SPC=8 and VR=7) than low perceived intuitive controls (SPC=2 and VR=3). This suggests almost equal number of participants struggled with controls in both conditions. Only 9% (2/22) participants reported slight motion sickness toward the end of the session after approximately 10 min in VR condition.

Enjoyment
Results relevant to enjoyment are summarized in Table 2. On average, aggregate ratings of enjoyment were higher in VR (mean 5.64, SD 1.22) compared with SPC (mean 4.25, SD 1.65), and the difference was significant (t 40 =−3.12, P=.003, and d=0.95).

Observations
Observations during the study session and poststudy feedback gave us further insights into participants' interactions with 2 conditions. Some participants in both SPC and VR condition appeared to struggle initially with controls before reaching to a comfortable level. One participant in VR condition reported slow joystick movements, whereas another said it was not user friendly. The slow joystick reaction was our design decision to mitigate possible motion sickness as a consequence of sudden movements in immersive VR. Slow controller movements aimed to provide control and agency to the older adults. Participants with advanced computer skills noted the slow joystick turns hindered their experience. A walking pad or treadmill was suggested as a replacement to joystick by 1 VR participant. This feedback suggests the value of personalized settings of controllers for users with different competency levels. Being flexible platform, VR-CogAssess is easily modifiable to provide natural interactions; however, feasibility of other controllers needs appropriate evaluation for this cohort. Users' observed controller interactions in both conditions emphasize the need to extend the initial tutorial for longer period. The tutorial should however provide 1 challenge or teach 1 skill at a time.

Feasibility and Perceived Competence
A VR platform is feasible (RQ3) when it allows users to complete the assessment session in a reasonable time window, with less stress and matching to their perceived and actual computer competencies. In the VR condition, the recall test took on average 10.45 min (SD 3.73) where the entire session lasted on average 33.64 min (SD 8.5). Stress level during the assessment session was another concern and defining criterion for the feasibility of VR platform. Participants reported the perceived stress they experienced during the assessment task, using the IMI measures in the posttest questionnaire. A t test ( Table 2) on perceived stress found no significant differences between SPC (mean 2.80, SD 1.51) and VR (mean 2.73, SD 1.39). Notably, participants in both conditions reported mild aggregated ratings of stress (mean<2.80). In addition, we used a wristband monitoring device to record the mean HRV data during the recall test. Similar to perceived stress, we found no significant differences in mean HRV (t 40 =0.58; P=.57) between the 2 conditions. However, GSR was excluded from analysis because of the presence of noise in the collected data.
Another important factor for feasibility of using the VR condition was the level of computer skills required for cognitive assessment relevant to perceived competencies of the user. Participants self-reported their perceived competence using 3 rating subscales on PENS questionnaire (recently validated for gaming environments [52]). We averaged those ratings and performed a t test to examine the differences. There was no significant difference between ratings received from participants in SPC (mean 3.95, SD 1.64) and VR (mean 4.27, SD 1.78). This measure was important to investigate older adults' capabilities to complete the designed assessment task. The competence questions capture the participant's perception of their own ability to perform the test on navigating to 6 landmarks. Large range of scores (SPC: SD 1.64; VR: SD 1.78) were observed in competence rating, a cross-tabulation assists to further investigate ratings on individual scale. Participants ratings were separated for low (1-2), mid (3)(4)(5), and high (6-7) ranges. However, more VR participants (n=9) self-reported high competence than SPC condition (n=4), where low competence was reported by equal number of participants in both conditions (n=6).
Furthermore, we examined the difference between the landmark recall performance in participants based on their actual computer skills (self-reported in pretest questionnaire). Those who reported computer proficiency as never used, beginner, and competent were compared. We found significant differences (t 18 =−2.60, P=.02) in landmark count during recall phase between SPC (mean 3.25, SD 1.29) and VR (mean 4.63, SD 0.92) participants when differentiated as beginner based on computer skills. However, these results represent a small sample, as groups were further divided into subcategories, and there could be subjective differences in computer proficiency reported between participants.

Principal Findings
Our results show that VR-CogAssess is a feasible platform for spatial memory assessment. The results extend previous research on feasibility studies for less-immersive VR setups [17][18][19] in cognitive assessment and introduce a fully immersive platform to conduct memory recall tests using VR HMD. Our findings reveal that using VR-CogAssess with HMD, when compared with an SPC setup using the same locations on Google Street View, results in better assessment outcomes, probably because of better alignment of the VE with users' mental models. We base this on the better results we received from participants in the VR group with respect to their correct landmark recall, less navigation mistakes, successful identification of challenging landmarks, and better perceived presence. Furthermore, VR-CogAssess achieves better assessment feasibility compared with the SPC setup, as users in the VR group perceived slightly less stress (although not significantly different to SPC), and their performance competency was comparable with SPC group. Stress-free interactions with novel VR technology are imperative for self-efficacy of older adults, raising their confidence in the task performance [33] and assessment's acceptability. Experiencing enjoyment and lower levels of stress is highly important in this type of situation, as it may increase the likelihood of people participating in memory screening, particularly in clinical settings, which are often perceived as stressful.
Designing immersive VR experiences for older adults involves several challenges such as limited physical, cognitive, and technical competencies. Design of immersive VR experiences for older adults in general and those who are at risk of cognitive decline specifically requires several iterations. We were able to demonstrate a comparable level of perceived usability between the VR and SPC conditions, despite the relative novelty of VR technology for our participants. This can be attributed to a set of 5 evidence-based design propositions that guided the design of VR-CogAssess platform, such as the need for easy-to-use controls and natural interactions. Initial training plays an important role in giving the users an opportunity to familiarize themselves with VR setup, learn the controls, and understand the basic actions. These interactions should recognize user's agency during the experience and provide them with choices. User should be given effective and easy tools for exploring the VE aligned with the main goal of spatial navigation. Fulfilling this basic psychological need of users will contribute toward their enjoyment, satisfaction [48], and ultimately to their sense of well-being. Future studies on VR environments for cognitive assessment will benefit from considering the 5 propositions suggested in this paper and from working in interdisciplinary groups.
Diagnostic assessments are generally done in the hospital environment, where individuals undergo a series of time-consuming tests, and the health professionals experience time pressure. Therefore, memory assessment such as landmark recall test needs to be completed in a reasonable time frame without imposing additional distractions for users when they engage with an immersive (and potentially novel) VR environments. For instance, when the VR environment for memory assessment presents virtual objects and stimuli (eg, trees, people, and cars), design should aim for simple interactions with minimal distractions to keep users focused on the main memory recall task. This may pose a dilemma for the designer, as we believe the very same stimuli in VR environments (eg, environmental objects and textures) provide better alignment to the users' mental models of the real world and therefore increase the validity of the assessment outcome. VR cognitive assessments have the advantage of higher levels of presence and enjoyment (as demonstrated in our results) to engage users in even long assessment sessions, where stress levels needs to be closely monitored. Any attempt to improve presence and enjoyment in VR experiences can therefore assist to keep stress levels in check.
One of the limitations of this study is that feasibility and usability of our platform was tested with an Australian cohort of healthy older adults with no memory issues reported. Although this is the population at risk of dementia that is often referred for screening, it is important to conduct further research in clinical setting and include participants who are readily identified as exhibiting symptoms of memory impairment. We note however that some of our design considerations such as the joystick controller's speed and interaction with VR environment were made slow to match common competencies of the aged participants; these may have to be adjusted for visitors in clinics, particularly those with musculoskeletal complaints or neurodegenerative diseases that limit movement or are characterized by tremor such as Parkinson disease. Simultaneously, it is possible that users with better computer skills may find the design not matching their competency level and experience disinterest.
In the future, we intend to further investigate VR-CogAssess in clinical setting and include a comparison of diagnostic accuracy and usability between healthy older adults and individuals with MCI. Another worthwhile endeavor would be to conduct a longitudinal study to study whether VR-CogAssess can monitor a participant's memory decline or recovery; this may require configuring the task to use a familiar location and monitoring performance changes over time. A strength of using Google Street View is that the platform can easily be modified to use nearly any location. This ability to contextualize the test allows user personalization and might be valuable to study.

Conclusions
Dementia is a complicated disease that can be detected using novel assessment tools and technologies developed through multidisciplinary efforts of HCI researchers and clinical neuropsychologists. In this paper, we introduced VR-CogAssess, a new platform for assessing spatial navigation memory in older adults. The evaluation compared the VR platform with SPC, a desktop setup, and involved healthy older adults. The VR participants achieved higher landmark recall scores, reported higher levels of presence, and enjoyed the task more when compared with SPC participants. The VR participants also perceived slightly less stress, suggesting better accommodation of mental health needs of older adults when memory assessment is administered through VR technology. These findings are promising, showing the feasibility of our immersive VR platform as a potential tool for cognitive assessment based on spatial navigation memory.
This study focused on the design and evaluation of VR-CogAssess, proposing a set of 5 design propositions for maintaining a reasonable level of usability for older adults compared with SPC setups. These propositions encourage VR systems' design that consider aging population needs and contribute to their well-being. In a future iteration of VR-CogAssess, we plan to allow customized controller speed based on individual needs and skills.