The Association Between Increased Levels of Patient Engagement With an Internet Support Group and Improved Mental Health Outcomes at 6-Month Follow-Up: Post-Hoc Analyses From a Randomized Controlled Trial

Introduction

Background

Internet support groups (ISGs) are specialized social media websites that connect individuals with common health conditions and provide a forum for peers to exchange information, resources, and support [1,2]. While ISGs for mental health conditions have become increasingly common [3], randomized trials [4,5] and systematic reviews [6-8] find they have mixed benefits for reducing psychologic distress. In a recent randomized controlled trial, we reported that providing depressed and anxious primary care patients with access to a moderated ISG in addition to a computerized cognitive behavioral therapy (cCBT) program provided no additional intent-to-treat benefit in patients’ health-related quality of life (HRQoL) or mood and anxiety symptoms over the cCBT program alone at 6-month follow-up, although cCBT was more effective than primary care physicians’ (PCPs) usual care [9] (NCT01482806). These null findings raise questions about whether any subgroups of ISG members may have benefitted differentially from the ISG based on their level of engagement.

One approach to classify engagement with an online community is the 1% rule [10,11]. Adapted from the digital marketing literature, the 1% rule posits that 1% of online community members (superusers) create approximately 90% of user-generated content, approximately 10% of members (contributors) create less than 10% of the remaining content, and 90% of members (observers) rarely contribute but mainly observe activity. A recent observational study replicated the 1% rule in 4 large ISGs for individuals with addiction and mood disorders [12] and found that participants’ demographic and disease-specific characteristics were not associated with their level of engagement with these online communities [13].

Goal of This Study

Very little work has been done to investigate the relationship between level of ISG engagement and clinical outcomes for treating depression and anxiety or any other mental health condition in primary care [5,14]. Therefore, to classify the patients randomly assigned to our trial’s ISG arm by their level of engagement, we applied the 1% rule based on the number of posts they created on the ISG. We then conducted post hoc analyses to compare these engagement level subgroups with patients randomly assigned to the ISG who never logged in to examine whether any patient subgroup benefitted from participating in our online community.

Methods

Study Setting, Patient Eligibility, and Randomization and Experimental Conditions

The protocol for the Online Treatment for Mood and Anxiety Disorders Trial was approved by the University of Pittsburgh’s Institutional Review Board and detailed in the trial’s primary outcomes report [9]. Briefly, PCPs in 26 southwestern Pennsylvania practices referred patients with a Generalized Anxiety Disorder scale (GAD-7) [15] or Patient Health Questionnaire (PHQ-9) [16] score ≥10, indicating moderately severe anxiety or depression symptoms, between August 2012 and September 2014. We randomized 704 protocol-eligible participants to either (1) care manager–guided access to the 8-session “Beating the Blues” cCBT program designed to provide users with basic CBT skills [17] (cCBT-only; 301/704), (2) cCBT plus additional access to our password-protected and moderated ISG (ISG+cCBT; 302/704), or (3) their PCP’s usual care (101/704). All study arms had similar baseline sociodemographic and clinical characteristics [9]. Analyses in this report focus solely on the 302 participants assigned to the ISG+cCBT arm.

Care Manager Support

Following randomization, the care manager exclusively assigned to the ISG+cCBT arm contacted each participant via telephone to provide basic psychoeducation and encourage them to start the cCBT program and log in to the ISG. Later, he contacted participants (we define these as care manager contacts) via email, text, and telephone to promote adherence with the cCBT program and treatment recommendations, including suggestions to access various resources on the ISG. The care manager presented each participant’s progress to the study PCP, psychiatrist, and psychologist at a weekly case review meeting [18].

Internet Support Group

We used WordPress (Automattic Inc) software to create our password-protected ISG that was accessible via computer or smartphone (Figure 1). The ISG featured moderated discussion boards created by the care manager–ISG moderator and study participants. The study team used an iterative process to decide on the initial discussion board topics, with a focus on common challenges faced by patients with depression and anxiety (eg, managing symptoms, discussing mental health issues with friends, common triggers). The ISG also curated links to external resources including local $4 generic pharmacy programs; find-a-therapist; crisis hotlines; brief YouTube videos on insomnia, nutrition, exercise, and other topics; our electronic medical record system’s patient portal; and the cCBT program (Multimedia Appendix 1).

To preserve confidentiality, we assigned members usernames and regularly reminded them not to post any self-identifying information or photographs. Additionally, a study investigator logged in to the ISG daily to review new posts for suicidal thoughts and other potentially inappropriate content. Participants were also able to flag comments for review by the ISG moderator and possible removal.

‎View this figure

Engagement with the Internet Support Group

We provided participants with password-protected access to the ISG approximately 3 months after the start of subject enrollment once the first 25 patients were randomized to the ISG arm to promote user-generated activity. Afterward, we provided participants with ISG access shortly after randomization.

Participants created content on the ISG discussion boards by either initiating a new discussion thread or commenting on an ongoing thread (posts). On most weeks, the care manager–ISG moderator also initiated new discussion threads on such topics as coping with mental health symptoms, talking about depression and anxiety with friends, stressors (eg, holidays, work-life balance), and lifestyle challenges (eg, healthy diet, losing weight, exercise).

Although we encouraged participants to log in and post on the ISG throughout their 6-month intervention phase, we did not require them to do so. Still, we took several measures to encourage participants to log in and post by featuring status indicators on their profiles and posts (eg, stars and likes), emailing notifications of new ISG activities and posts, highlighting new posts on their homepage based on their past ISG activity, inviting participants to serve as guest moderators, and holding various contests that promoted logging in and posting.

Assessments

Following confirmation of protocol eligibility and consent, a study assessor collected sociodemographic and clinical information from our study practices’ electronic medical record system and from the participant, ascertained self-identified race, and administered the Primary Care Evaluation of Mental Disorders (PRIME-MD) Anxiety and Mood Modules to establish a psychiatric diagnosis [19], the 12-Item Short Form Health Survey Mental Components Score (SF-12 MCS) to measure HRQoL [20], and the Patient-Reported Outcomes Measurement Information System (PROMIS) Depression and Anxiety short forms to measure depression and anxiety symptom levels [21]. Later, an assessor who was blinded to participants’ randomization assignment telephoned participants to readminister the PROMIS and SF-12 MCS at 3 months and at the 6-month primary outcome time point.

We obtained counts of unique patient log-ins and posts from the logs of the server that hosted the ISG. We defined a post as an entry that initiated a new discussion thread or added an entry to an existing discussion thread, and we summarized the number of posts each participant made to arrive at a total.

Classification of Internet Support Group Engagement

Using the 1% rule as our starting point, we classified participants into subgroups by level of engagement as measured by the total number of posts each created during the first 6 months after randomization (top 1% of posters, next 9%, and remaining 90%) [10,11]. Given our interest in identifying the gradient of participant engagement, we further classified participants into the following subgroups: superusers (top 1%), top contributors (next 9%), contributors (made at least 1 post), observers (logged in at least once but never posted), and those who never logged in. Since several participants between the 9th and 11th percentiles made the same number of posts, we reclassified our top contributors as the next highest 10% of posters after superusers, rather than the next 9%.

Statistical Analysis

We calculated the baseline sociodemographic and clinical characteristics across the 5 ISG engagement groups using percentages, means and standard deviations, and medians and interquartile ranges (IQR), and we made group comparisons using analysis of variance and chi-square tests. As we had only 3 superusers, we grouped them with the 30 top contributors for all analyses to conduct more meaningful comparisons.

We used linear mixed models for each of the clinical outcomes (SF-12 MCS, PROMIS Depression, PROMIS Anxiety) that included fixed effects for engagement subgroup, time, group-by-time interaction, education, self-identified race, gender, and random effects for participants. We also compared the 6-month change in HRQoL and depression and anxiety symptoms between participants who were assigned to the ISG arm (ISG+cCBT) but never logged in and participants in our combined superuser + top contributor subgroup. All analyses were conducted using SAS version 9.4 (SAS Institute Inc).

Results

Baseline Sociodemographic and Clinical Characteristics

At baseline (Table 1), the 302 participants randomized to the ISG+cCBT arm reported moderately severe depression (PHQ-9 mean 13.4, SD 4.7) and anxiety symptoms (GAD-7 mean 12.6, SD 4.5) and low HRQoL (SF-12 MCS mean 31.7, SD 9.4). They had a mean age of 42.6 years, 81.1% (245/302) were female, and 47.7% (144/302) had at least a college education.

While each engagement subgroup was predominately female, white, and had comorbid depression and anxiety, reflecting the overall composition of our study cohort, participants who were female, white, and college educated were more likely to be in the superusers + top contributors subgroup (eg, ≥4-year college education: 70%, 23/33 superusers + top contributors vs 36%, 27/74 of the never log-ins; Table 1).

Distribution of Engagement

Seventy-five percent of participants (228/302) logged in to the ISG at least once during their 6-month intervention phase, for a total of 2041 log-ins. Of those, the median number of log-ins per participant was 4 (IQR 2-9.5; range 1-214). Participants created 1488 posts over the 6-month intervention phase, and 61.8% (141/228) made at least 1 post (median posts per participant: 1, IQR 0-5).

As expected, the mean number of log-ins and posts differed widely across engagement subgroups (P<.001, Tables 2 and 3). However, the distribution of posts in our sample was less skewed than predicted by the 1% rule, with superusers making 42.3% (630/1488) of posts (median 246, IQR 78-306), top contributors 34.6% (515/1488, median 11, IQR 10-18) and contributors 23.1% (343/1488, median 3, IQR 1-5). Moreover, only 28.8% (87/302) of participants in the ISG were classified as observers (ie, they logged in to the site at least once but never posted).

Process Measures of Care

Overall, the mean number of cCBT sessions completed was 5.5 (SD 2.7), and 35.8% (108/302) completed all 8 cCBT sessions. Across engagement subgroups, participants who created more posts also completed more cCBT sessions (P<.001) and had more care manager contacts (P<.001; Tables 2 and 3).

Mental Health Outcomes at 6 Months

After adjusting for gender, race, and education level, all engagement subgroups reported similar improvements in symptoms at 6-month follow-up regardless of level of engagement with the ISG (Tables 4 and 5). Furthermore, compared to participants who never logged in to the ISG, the combined superusers + top contributors subgroup reported a greater improvement in HRQoL (mean ∆ SF-12 MCS: 16.1, SE 1.9 vs 10.1, SE 1.3, P=.01) and anxiety symptoms (mean ∆ PROMIS T-score: –11.6, SE 1.5 vs –7.8, SE 1.0, P=.04); we did not observe a similar improvement in depression symptoms.

Discussion

Principal Findings

To the best of our knowledge, this is the first report to demonstrate that high levels of patient engagement with a moderated ISG, compared to no engagement with the ISG, are associated with improved anxiety symptoms and HRQoL in primary care. Our findings also provide further empirical evidence to support the participation inequality suggested by the 1% rule, although we observed a broader distribution of posting than posited by the 1% rule.

Our work confirms that depressed and anxious primary care patients are willing to engage in an ISG even when not required by study protocol to do so. Indeed, the sizable majority of our study subjects logged in to the ISG at least once, which is consistent with log-in rates reported in other studies of ISGs for depression [4,22]. Furthermore, among those who logged in to the ISG, participation inequality was less extreme than expected based on the 1% rule, as our top 1% and 10% of posters together generated 78% of all user-created content on our site, not 99% as the 1% rule predicts. Still, challenges remain in developing even more equitably engaged online communities to improve health and HRQoL.

Prior work on the impact of ISG engagement has been limited largely to comparing psychosocial outcomes between posters (defined as individuals who made at least 1 post) and observers in ISGs for women with breast cancer [23,24]. Findings from this work were mixed: while a moderate sized cross-sectional study showed more benefits in perceived social support in posters than observers [23], a large prospective study showed higher perceived functional well-being and fewer mood symptoms in observers than posters at 3-month follow-up [24]. To our knowledge, the only other study to explore the impact of engagement on mood symptoms in ISGs for mental health measured engagement by time spent on the ISG, showing that members who spent more than 5 hours on the ISG over a 2-week period were more likely to have resolution of depression at 6 months than members who spent less time [4].

Our finding that the participants who were highly engaged with the ISG reported improved anxiety symptoms and HRQoL at 6 months compared to individuals who never logged in identifies a subgroup that may benefit from participating in an ISG. Interestingly, this group did not report similar benefit for depression symptoms compared to the group that never logged in. On average, this highly engaged subgroup posted 5.8 times per month, which averaged approximately 1 post per log-in. Demographically, this subgroup had higher proportions of participants who were female, white, and college-educated than the group that never logged in, but both groups had similar levels of baseline depression and anxiety. This finding offers encouragement about the potential for ISGs to improve clinical outcomes in individuals who engage highly with an ISG. Still, more work is needed to confirm our findings in a randomized trial and identify the critical threshold of engagement needed to demonstrate clinically meaningful improvements in health.

Our work motivates further study into how to most accurately measure engagement with an ISG. We quantified engagement using the relatively simple metric of number of posts, and we assigned each post an equivalent weight. However, other quantitative metrics such as time spent on the ISG and number of pages viewed may offer a different perspective. Moreover, qualitative metrics that analyze post content may also be an important dimension of engagement, particularly considering evidence from breast cancer ISGs suggesting that a subset of members derive psychological benefit from creating posts that provide “insightful disclosure” [25].

Limitations

Our study has several limitations. First, our finding that high levels of engagement improved clinical outcomes reflects a post hoc analysis that we undertook to identify a subgroup that may have benefitted from the ISG. Second, the limited size of the ISG precluded further subgroup analyses and required us to combine the superuser and top contributor groups for all outcome analyses. Third, we quantified engagement using a simple measure of post counts, and we used this measure to stratify the sample into engagement levels based on the 1% rule rather than statistical methods that avoid specifying an a priori hypothesis about engagement distribution. Fourth, we were not able to include a content analysis of posts or examine the impact of post content on outcomes. Finally, since all participants had access to the cCBT program, we cannot exclude that the overall improvements we observed could be attributed to the cCBT program given its demonstrated efficacy [9,26].

Implications

Our work shows that primary care patient engagement in an online community for depression and anxiety may contribute to improved mental health at 6 months after enrollment but only at the highest levels of engagement. We strongly encourage researchers, clinicians, and health care delivery systems considering deployment of a similar ISG to first develop plans to encourage and sustain high and broad levels of user engagement. Future work is needed to (1) confirm our findings with mental health and other conditions, (2) establish the threshold of patient engagement required to benefit from an ISG, and (3) perform content and other qualitative analyses of discussion board posts to explore the influence of this content with patient engagement and other outcomes of interest.

Conclusions

In summary, we demonstrated that patient engagement with our moderated ISG for depressed and anxious primary care patients generally approximates the 1% rule. Although we observed broader engagement levels with the ISG than predicted by the 1% rule, only ISG members who engaged at the highest levels of engagement reported measurable improvement in symptoms and quality of life at 6-month follow-up.