What’s in it for me? Qualitative evaluation of the QoL- ME, a visual and personalized quality of life assessment App for people with severe mental health problems

Background: The QoL-ME is a digital, visual and personalized QoL assessment App for people with severe mental health problems. Research reveals that e-mental health Apps such as the QoL-ME frequently fall short of expectations regarding their impact on daily practice. Studies often indicate that e-mental health Apps ought to respect the needs and preferences of end-users to achieve optimal user-engagement Objective: The degree to which the QoL-ME matches the needs and preferences of end-users was investigated in this study. Special attention is paid to whether the QoL-ME is actionable and beneficial for its users. Methods: Eight end-users who gained experience using the QoL-ME contributed to semi-structured interviews. An interview guide was used to direct the interviews. All interviews were audio recorded and transcribed verbatim. Transcriptions were analysed and coded thematically. Results: Analysis revealed three main themes 1) Obtained benefit, 2) Actionability and 3) Characteristics of the QoL-ME. Conclusions: The QoL-ME can be beneficial to users as it provides them with insight into their QoL and elicits reflection. Incorporating more functionalities that facilitate self-management, such as advice and strategies for improving lacking areas will likely make the App more actionable. Most of the additional characteristics of the QoL-ME, including its usability, design and content, match the needs and preferences of users. (JMIR Preprints 24/04/2020:19593) DOI: https://doi.org/10.2196/preprints.19593 Preprint Settings 1) Would you like to publish your submitted manuscript as preprint? Please make my preprint PDF available to anyone at any time (recommended). Please make my preprint PDF available only to logged-in users; I understand that my title and abstract will remain visible to all users. Only make the preprint title and abstract visible. No, I do not wish to publish my submitted manuscript as a preprint. 2) If accepted for publication in a JMIR journal, would you like the PDF to be visible to the public? Yes, please make my accepted manuscript PDF available to anyone at any time (Recommended). Yes, but please make my accepted manuscript PDF available only to logged-in users; I understand that the title and abstract will remain visible to all users (see Important note, above). I also understand that if I later pay to participate in <a href="https://jmir.zendesk.com/hc/en-us/articles/360008899632-What-is-the-PubMed-Now-ahead-of-print-option-when-I-pay-the-APF-" target="_blank">JMIR’s PubMed Now! service</a> service, my accepted manuscript PDF will automatically be made openly available. Yes, but only make the title and abstract visible (see Important note, above). I understand that if I later pay to participate in <a href="https://jmir.zendesk.com/hc/en-us/articles/360008899632-What-is-the-PubMed-Now-ahead-of-print-option-when-I-pay-the-APF-" target="_blank">JMIR’s PubMed Now! service</a> service, my accepted manuscript PDF will automatically be made openly available. https://preprints.jmir.org/preprint/19593 [unpublished, non-peer-reviewed preprint] JMIR Preprints Buitenweg et al


Introduction
Quality of Life (QoL) assessment in people with severe mental health problems faces several challenges. First, respondents may not have had the opportunity to develop the abilities necessary to engage in traditional, language-based QoL assessment [1][2][3]. Alternatively, comorbid intellectual disabilities [3][4][5] or psychopathology [6][7][8] may compromise the validity of QoL results. Second, in mental health QoL is understood as an inherently subjective concept that is shaped by individuals' values and preferences [9][10][11].
Third, QoL assessment instruments may promote patient empowerment by providing patients with insight in their QoL scores, which is an important prerequisite for shared-decision making [17][18][19]. Both patient empowerment and shared-decision making have become important goals in mental health services [15,16].To meet these three challenges, an innovative, personalized and visual QoL assessment App was developed: the QoL-ME [15]. The QoL-ME consists of a core version that can be supplemented with additional modules. The core version involves a mandatory set of three universal QoL domains. In addition, respondents can choose from eight additional modules. Every module involves a domain of QoL that respondents may select if it is important for their QoL. Respondents only answer questions on their selection of additional modules. After filling out the QoL-ME, respondents receive direct feedback from the App in the form of an overview of their answers. The QoL-ME was developed co-creatively in close collaboration with patients, family members and care professionals [15,16].
Both research and practice reveal that e-mental health Apps such as the QoL-ME frequently suffer from low engagement and fall short of expectations regarding their impact on the daily lives of patients. [17][18][19][20][21]. Researchers have therefore investigated what factors enable e-mental health Apps to bridge the gap from development to high engagement and practical use by patients [20,22,23]. Generally, these studies often indicate that e-mental health Apps such as the QoL-ME ought to respect the needs and preferences of patients to achieve optimal user-engagement [20,[22][23][24]. Regarding the QoL-ME, two specific factors are of special importance. First, it is essential that the QoL-ME is actionable. The QoL-ME is actionable if provides a useful base for practical action for patients [25]. Examples of practical action include patients altering their sleep schedule after using an app that has sleep tracking functionality [25] or opting not to engage in a romantic relationship based on the results of a self-management App [26]. Second, use of the QoL-ME ought to be beneficial to patients. The QoL-ME should effectively address an issue patients care about so that they derive a tangible benefit from utilizing the App [27].
End users played a vital role in the development of the QoL-ME. In the context of this development, participants rated the usability of the App as "very high" [15]. It is unknown, however, whether the intensive user-involvement and positive rating of the usability of the QoL-ME translates to an instrument that is of use for patients in a real-life setting.
In light of the discrepancy between the potential of e-mental health Application and their lack of impact on patients' daily lives, it is crucial to investigate the experiences of patients who used the QoL-ME. In addition, it is of special importance to examine to what degree the QoL-ME is actionable and beneficial to its users. The aim of this study was to explore the experiences of users regarding the usability and functionality of the QoL-ME and whether the App is actionable and beneficial for patients. To this end, participants who had used the QoL-ME were interviewed about their experiences with several aspects of the QoL-ME including its usability, functionality and beneficiality.

Participants
This study targeted three specific populations of people with severe mental health problems: 1) people with psychiatric problems, 2) people treated in forensic psychiatry and 3) people who are homeless.
Homeless individuals were included in this study because of the high prevalence of severe mental health problems in this group [3,28,29]. These groups may have difficulties with traditional language-based QoL assessment due to experiencing fewer educational opportunities [1][2][3], co-occurring intellectual disabilities [3][4][5] and compromising psychopathology [6,7]. A consortium consisting of six societal institutions was formed to facilitate this study and the broader research project. These institutions include a multimodal day treatment centre for multi-problem young adults, a hospital for forensic psychiatry, a mental health institution, a day centre for people who are homeless and two research institutions focusing on lifestyle, homelessness and addiction. Participants were recruited with the help of the consortium partners.
The research population consisted of individuals who had gained experience with the QoL-ME in the context of a psychometric evaluation of the App. In this psychometric evaluation, respondents were invited to use the QoL-ME monthly for a period of six months. A specific inclusion criterion of at least five uses of the QoL-ME was employed. This criterion ensures that patients had sufficient experience with the QoL-ME to be able to contribute valuable information. The aim was to include enough participants to reach saturation in the sample, defined as a lack of new information in the final two interviews [30]. The QoL-ME six iterations divided over three stages. In the first stage, patients were invited to share their ideas regarding the design and functionality of the QoL-ME. In the second stage, initial designs and wireframes were developed into a fully functioning prototype of the QoL-ME. This process was guided by the feedback and ideas of patients. The prototype was subjected to a usability evaluation in the final stage [15].
The QoL-ME encompasses two separate core versions. The first core version targets people with psychiatric problems and people treated in forensic psychiatry and includes three domains of the Lancashire Quality of Life Profile (LQoLP; [11]): 'safety', 'living situation' and 'finances'. A recent study indicates that these three LQoLP domains are universal [12]. The LQoLP uses a 7-point Likert scale, ranging from 'cannot be worse' (1) to 'cannot be better' (7). The second core version is tailored to people who are homeless and comprises the Dutch version of the Meaning in Life Questionnaire (MLQ), a 10-item measure that assesses both the presence of meaning in one's life, and the search for meaning in life [31]. Research indicates that having meaning in life is especially important for people who are homeless [32,33]. The MLQ also uses a seven-point Likert scale, ranging from 'completely disagree' (1) to 'completely agree' (7).

Approach
A qualitative research approach was employed to explore the participants' experiences with the QoL-ME.
Specifically, individual semi-structured interviews were utilized as they allowed participants to elaborate on their experiences, and allowed the researcher to clarify on any confusing or unclear questions when necessary. In addition, the context of individual interviews enabled referring back to the QoL-ME to make questions more tangible. The use of semi-structured interviews combined a guiding structure with providing participants freedom to expand on their answers.

Content of the interview
An interview guide was used to guide the interviews conducted in this study. Four sources of information were consulted to inform this interview guide (See Table 1). First, insights regarding patients' needs and preferences concerning the QoL-ME gained during the development of the QoL-ME were fed back into the interview guide. Second, the Health Information Technology Acceptance Model (HITAM) was consulted [34].
The HITAM describes consumers' behavioral intentions towards the use of health technology. Third, relevant information was extracted from two questionnaires designed to evaluate mobile (health) Apps 1) the Mobile App Rating Scale (MARS; [35]) and 2) the App Chronic Disease Checklist (ACDC; [36]). Fourth, the scientific literature was examined and information regarding patients' needs and preferences regarding mobile mental health Apps was extracted [20,27,37,38]. The 18 identified topics were grouped into five overarching themes (see Table 1) and each theme was introduced using a short prime.

Data analysis
A deductive, or theoretical [39], analysis approach was employed, starting from a specific predefined research question. All interviews were audio recorded. The recordings were transcribed verbatim and transcripts were coded thematically utilizing the six-step method as outlined by Braun and Clarke [39] in order to capture user experience themes. Initial themes were continuously refined and reflected on using a deductive approach. In the First Step, the researchers familiarized themselves with the data through checking and verifying the accuracy of the transcripts. The Second Step involved the selection of an initial set of codes and themes based on the first three interviews. Codes are used to label and organize qualitative data. Codes with similar content are clustered into overarching themes. The coding was performed using ATLAS ti version 8. The two researchers compared their initial codes to ensure consistency throughout the coding process.
Once the initial set of codes was confirmed, the researchers independently coded all of the interviews using the initial set. This set was modified or added to if necessary. Once all the interviews had been coded and the researchers reached consensus regarding the coding of the transcribed interviews, the Third Step involved clustering of the codes into overarching themes. Themes were identified based on recurring codes. In Step Four, the researchers discussed the themes and modified them when required to reach consensus on their content and labelling.
Step Five encompassed the interpretation and naming of the emerging themes. The results of the six-step analysis method were reported in Step Six [39].

Procedure
Participants who contributed to the quantitative evaluation of the QoL-ME [40] were invited to participate in the interview. Participants who met the inclusion criteria were contacted via e-mail, via care professionals at the consortium institutions, or via telephone if possible. Participants, who expressed interest in contributing, were provided with additional information on the qualitative study. Once a participant agreed to contribute, the researcher (DB) and participant scheduled an appointment for an interview. Interviews were held at the institution that supported the participant, or at a neutral location such as a café. Prior to the interview, the researcher provided a detailed explanation of the study and of what was expected of the participant.
Moreover, the researcher explained that there were no right or wrong answers and that it was important that participants freely shared their opinions. Next, the researcher and participant went through the QoL-ME together to ensure that all participants had a refreshed understanding of the QoL-ME. The interview guide as displayed in Table 1 steered the interview, while the interviewer elaborated on topics when necessary. Upon completing the interview, the interviewer explained how the gathered data would be analyzed and how this aided the study. Participants were given room to ask any further questions. The interview ended when all questions were addressed whereupon the participant received a gift voucher. The duration of the interviews varied between 17 and 42 minutes and the average duration was 31 minutes. Of the ten patients who initially agreed, one patient could not be reached anymore and another was too busy to schedule an appointment. Therefore, eight individuals with severe mental health problems participated in this study. The aim of continuing inclusion until saturation was not entirely met, because the number of experienced users that agreed to participate in the interviews was relatively low. Participants' demographic characteristics are provided in Table 2. Five participants were male, the mean age of participants was 34 (SD = 12) and five of the eight participants had a Dutch cultural background. All participants had gained experience using the QoL-ME by contributing to the psychometric evaluation of the QoL-ME [40]. On average, participants had filled out the QoL-ME 6 times (range: 5-7) over a period of between four and six months. Six of eight participants reported using the QoL-ME on their personal smartphone device and the remaining two participants used their personal computer. Participants primarily used the QoL-ME at home, whilst some reported using the QoL-ME at their care institution.

Main findings
The following three themes were identified based on analysis of the interviews: 1) Obtained benefit, 2) Actionability and 3) Characteristics of the QoL-ME. An overview of the codes and themes is provided in Multimedia Appendix 1. This appendix includes both an overview in table-form and a graphic depiction of the network of codes and themes. As the first two themes pertain to the two concepts (beneficial and actionable) that were of special interest in this study, these themes will be discussed in more detail.

Obtained benefit
According to six of the eight interviewees, using the QoL-ME was beneficial to them. All six of these participants mentioned that using the QoL-ME made them more aware of their level of satisfaction on the life domains incorporated in the QoL-ME.  For other participants, the QoL-ME facilitated the realization that they were happier than they thought they were.

[Ehmm. I started to think more consciously about how happy I actually was. And I turned out to be happier than I actually thought.] -Participant 8
The two participants for whom the QoL-ME was not beneficial mentioned having sufficient insight already into how satisfied they were with their lives as the main reason for this lack of benefit: [No, no the questions that were asked, I already had some kind of insight in them. In those areas. So no I

didn't really get anything out of it.] -Participant 5
Both participants did feel that the QoL-ME would be more beneficial to them if they lacked this insight:

Actionability
For three participants, the QoL-ME proved to provide a useful base for taking actions in their daily lives. One participant mentioned that using the QoL-ME assisted her in the maintaining of social relationships. Regarding the potential negative effects of confronting users of the QoL-ME with a decrease in their QoL scores in the absence of care professionals, none of the participants expected this to be a problem.
[Yes for some people you wouldn't want to see that of course. But I feel like ... it's how you feel at the time.
The situation may still be the same, but the way you deal with it may be different. You can feel different every

day.] -Participant 3
Some participants provided tips for improving the clarity of the results section, which would also improve the actionability of the QoL-ME but is discussed under the third theme. One participant recommended including advice for how to improve low scores in the QoL-ME to improve its actionability. He used a food diary App as an example. Users register what they eat on a daily basis and the App generates an advice based on user input.
[D: Yes, okay, so it really is for you… yes maybe you can generate an advice at the end of such a test. We see from your answers that you score negative on these topics and maybe you can think about that. Something like that. I: Is that also a way to get more benefit from it? D: Sure, I think so. That is ultimately what you want, a system that thinks along with you. I have an example, a silly example maybe, but I have an App from the nutrition center. This keeps track of exactly what you eat, and there is also advice. We see that you eat too much salt and too many unhealthy products. And then you are really triggered like I have to fall within the margins of that App. Or something like a pedometer, things like that.] -Participant 4

Characteristics of the QoL-ME
Overall, participants welcomed the opportunity to view their results upon completing the QoL-ME. Three participants provided specific advice for improving the clarity of the results section and thereby increasing the actionability of the QoL-ME: Seven participants appreciated the possibility to personalize the content of the QoL-ME. The one participant who disagreed, indicated that he found all domains important and therefore preferred a version in which no choices had to be made. Participants were divided regarding the option to personalize the appearance of the QoL-ME. Four participants welcomed this functionality, but the other participants thought it added too little value.
Several participants commented on the content of the QoL-ME. One participant thought that the items on the financial situation of respondents were too direct and advised an alternative formulation. Four participants commented on the images used in the additional modules of the QoL-ME. One participant recommended using pictures that are more diverse. Three participants reported that some of the images used were unclear to them. They advised including a written description of the content of the item using a word or a short sentence for clarification.
None of the participants had trouble with the duration of filling out and the number of questions.
Three participants did miss a clear ending message and they advised including this. One participant had issues with the low contrast between foreground and background elements due to her visual handicap.
Seven participants thought the QoL-ME looked professional, primarily due to the uncluttered and simple layout.
No participant reported having insight in which persons and parties had access to their data. Still, six participants trusted the security of their data. The inclusion of a disclaimer containing information regarding data access and use was a welcome addition for seven participants.
In general, all participants were very positive regarding the design and usability of the QoL-ME. Participants appreciated the clear structure of the App and favored the navigational system.

Discussion
This study explored the experiences of users regarding the usability and functionality of the QoL-ME and whether the App is actionable and beneficial for patients. As it is important that an e-mental health tool such as the QoL-ME is both beneficial and actionable to its users, special attention was paid to these concepts.
The interviews revealed that using the QoL-ME is beneficial to most users, primarily by pushing them to consider their satisfaction with various life domains. The QoL-ME did not prove to be actionable for most respondents. In addition, respondents were positive about the design and usability of the QoL-ME, but also had some tangible tips and advice for improvement.
The main way in which the QoL-ME was beneficial to users was through providing insight and facilitating reflection. Some respondents indicated that their use of the QoL-ME made them realize that they were more satisfied with their lives than expected. This result echoes findings by Morton and colleagues [26] in their evaluation of a QoL self-monitoring tool for people with bipolar disorder. Respondents also indicated that they were sometimes surprised by how high their scores were, which lead to the insight that 'things were not so bad'. Two participants indicated that they already had sufficient insight into their own QoL and therefore derived no extra benefit from using the QoL-ME. This finding echoes results found by Berry and colleagues [41], who investigated views on using digital self-management tools among people with severe mental health problems. A number of participants who contributed to this qualitative interview study indicated that they were sufficiently self-aware already and expected little benefit from using digital selfmanagement tools [41].
Participants provided three useful suggestions for making the QoL-ME more actionable. First, half of the participants proposed to include numerical indicators for users' satisfaction scores for every item or domain. The results section of the current version of the QoL-ME does not include numbers but only shows a bar that is partly filled based on underlying scores. The Personal Health Information Self-Quantification System model [42] outlines how self-quantification is of vital importance for the self-management of health.
In the model, self-quantification is described as the step in which an individuals' goal (having a good QoL) is transformed into objectively measured units [42]. Results by Morton and colleagues [26] confirm the importance of quantification, as respondents indicated that it was the quantification of their QoL that enabled self-management. A second important suggestion to make the QoL-ME more actionable, raised by two participants, was to incorporate practical advice for improving users' satisfaction on certain life domains.
The tool evaluated by Morton and colleagues [26] was integrated in a larger digital self-management platform that included practical advice and strategies for self-management. The results section of the tool provided direct links to these strategies, a feature that participants were very enthusiastic about [26].
Expanding the QoL-ME to include similar functionality will likely make the App more actionable for users. The third suggestion pertains to enable users to consult the development of their QoL scores over time. Every participant saw this as a welcome addition. This finding is in accordance with findings by Morton and colleagues [26] and by Berry and colleagues [41]. These three suggestions may be used to strongly improve how beneficial and actionable an assessment tool such as the QoL-ME is to patients.
Several participants acknowledged the possibility of discussing the results of the QoL-ME with other individuals such as a family member or professional caregiver. The fact that none of them did so may be an indication of social isolation, which has frequently been reported in this population [1][2][3]. Moving towards self-management, future versions of the QoL-ME may actively encourage users to share their results and include practical suggestions for decreasing social isolation.
Participants were unanimously positive regarding the QoL-ME's usability. They found the application easy to use, appreciated its linear structure and prized the calm and clean layout. These results confirm what was found during the usability evaluation that made up the last part of the development of the QoL-ME [15] and serve as additional corroboration of the design recommendations [37,43] consulted during the Apps' development. Several respondents preferred combining the visual material used in the additional modules with a word or short sentence to denote the content of its item. Comparable pictorial assessment instruments, such as the pictorial version of the Aachen Quality of Life Interview [44] and the pictorial motivation scale in physical activity [45] also combine both visual and verbal content. Respondents had very limited insight into which persons and parties had access to their data. This did not deter them from engaging with the QoL-ME. This may be because respondents used the QoL-ME in the context of a scientific study, or because participating did not require respondents to share any personal information.
The results draw attention to several ways in which the QoL-ME may be modified so that it is more beneficial for patients. Future research may further investigate what images used in the QoL-ME are unclear and identify alternative images. Moreover, the results section may be updated to display the development of results over time. In addition, following the example by Morton and colleagues [31], the QoL-ME may be integrated into a larger self-management platform for people with severe mental health problems.

Strengths and limitations
This study provides an important contribution to the field of e-mental health App development. The qualitative methodology provided patients with the opportunity to share their opinions regarding the usability and functionality of the QoL-ME and to what degree the App is beneficial and actionable to them.
The results draw attention to the fact that patients require functionalities that target their needs before an App to become beneficial to them. Specifically, patients require functionality targeting self-management. In addition, the content of the interview was partially derived from existing frameworks that have proven to be effective for evaluating health Apps [46]. Still, the results do need to be interpreted in light of three limitations involving the sample of participants who contributed to this study. The first limitation pertains to the size of the convenience sample used in this study. The eligible research population, based on the criterion of having completed at least five measurements, was small. Still, the results provide important insights into user experiences and in the extent to which the QoL-ME is beneficial and actionable for users. Once a larger group of patients starts using the QoL-ME, additional research will have to reveal whether the current results hold up in this larger population.
Analyses revealed that saturation, defined as a lack of new information in the final interviews, was not fully attained in the sample. The final two interviews did contain new information, but these were not substantial insights and no changes to the codes or themes were made based on these interviews.
The context in which participants gained experience with the QoL-ME forms a second limitation.
Participants were aware that they used the QoL-ME in the context of a scientific study in which the psychometric quality of the QoL-ME was evaluated. Moreover, participants were incentivized to use the QoL-ME and to participate in the interviews. Therefore, their use of the QoL-ME may not represent use in a reallife setting and their responses in the interviews may have been biased. To counter possible bias due to the incentives, the researcher indicated that respondents were allowed to freely give their opinions before the interviews started. Future research may investigate to what degree the current results hold up when patients' who used the QoL-ME on their own accord are interviewed.
The third limitation pertains to the absence of data on participants' medical background, such as psychiatric diagnoses or symptom severity. Still, all participants received care from the consortium institutions and we can therefore be certain that they are part of the QoL-MEs' target population. Future research may investigate whether individuals with specific symptoms or diagnoses experience have differing experiences using the QoL-ME.

Conclusions
The QoL-ME can be beneficial to users as it provides them with helpful insight into their QoL. Including added functionality in support of self-management, such as advice and potential strategies for improving QoL domains App users are dissatisfied with will likely make the QoL-ME app more actionable. Overall, the interviewed patients positively regarded the usability, functionality and contents of the QoL-ME.

Figures
Four screenshots depicting the additional modules of the QoL-ME. The top-left panel displays how respondents select additional modules. Respondents are invited to drag eight icons, corresponding to the eight modules, to a circle that says 'important' or a circle that says 'not important'. The top-right panel shows how respondents are asked to confirm their choice of additional modules. The two bottom panels provide examples of items of the additional modules.