Web-Based Self-Management for Patients With Lymphoma: Assessment of the Reach of Intervention of a Randomized Controlled Trial

Introduction

Randomized controlled trials (RCTs) are widely considered to be the gold standard for evaluating the effects of an intervention in behavioral and psycho-oncological research [1,2]. In contrast to the effects of interventions that are most often examined extensively in RCTs [3], much less attention has been paid to the proportion of patients who participate in these interventions and whether those who choose to participate are representative of the target population in terms of sociodemographic and clinical characteristics [2,4]. Thus, RCTs often provide accurate estimates of the internal validity (ie, effect of an intervention for the sample enrolled in the RCT), but do not typically provide information about the external validity or generalizability (ie, effect of an intervention in the target population) [5-7].

The reach of an RCT provides information on the absolute number, proportion, and representativeness of the sample that participates in the trial [8]. The absolute number and proportion of RCT participants are relatively easy to assess and are therefore most often reported. However, few studies report the representativeness of the sample enrolled in an RCT, which is a much more challenging metric to assess [8,9] since it requires sociodemographic information, and preferable psychosocial, clinical, or case mix information, on RCT participants as well as nonparticipants. It is particularly challenging to collect information on nonparticipants who typically do not consent to be included in the research [8].

Interventions with promising effects in RCTs have been implemented in daily practice without specific knowledge of the generalizability of the results. Therefore, more attention should be paid to providing information related to the representativeness of the sample enrolled in an RCT. Lack of representativeness may occur as a result of inadequate selection procedures (ie, sampling bias) or when the probability of nonparticipation in the study is related to the object of research (ie, nonresponse bias) [10,11].

To fill this gap, the aim of the current study was to address the reach of a web-based self-management intervention within the context of the Lymphoma InterVEntion (LIVE) trial (Netherlands Trial Register NTR5953), whose objectives have been described elsewhere [12]. For the LIVE trial, patients were selected from the population-based Netherlands Cancer Registry (NCR) that routinely collects data on sociodemographic and clinical characteristics. This can provide unique insight into the characteristic differences between RCT participants and nonparticipants to estimate the reach of this intervention. The primary objectives were to (1) describe the proportion of RCT participants compared to all patients invited to participate, and (2) compare sociodemographic and clinical characteristics of RCT participants with those of all respondents (ie, patients who completed a web-based or paper questionnaire), all patients invited to participate, and all patients selected from the NCR. In addition, as patients had the option of completing a web-based questionnaire (ie, enrollment in the RCT) or a paper-based questionnaire (ie, no enrollment in the RCT; observational cohort), a secondary objective was to assess baseline differences in psychological distress, self-management skills, and satisfaction with information provision (ie, RCT outcome variables) between the two groups.

Methods

Study Design

Baseline data were collected from an RCT embedded in a population-based registry [13] as an observational cross-sectional dataset without information on the effectiveness of the intervention. In short, the LIVE trial examines the effectiveness of feedback on patient-reported outcomes and a web-based self-management intervention on self-management skills, satisfaction with information provision, and psychological distress among patients with lymphoma [12].

Participants and Recruitment Procedure

From October 2016 to February 2019, patients who were diagnosed with lymphoma, including Hodgkin lymphoma, nonHodgkin lymphoma, or chronic lymphocytic leukemia (CLL) as defined by the International Classification of Diseases for Oncology-3 codes [14], from 13 hospitals in the Netherlands were selected for participation via the NCR. The NCR registers all patients newly diagnosed with cancer in the Netherlands within the first year after diagnosis and routinely collects detailed data on sociodemographic and clinical characteristics (eg, patient age and sex, date of cancer diagnosis, cancer type, and primary treatment). Patients had to be 18 years or older at the time of study invitation. Treating hematologists were asked to verify the patients’ eligibility for the study and to exclude patients who were deceased, had severe psychopathology, were too ill, were not able to complete a questionnaire in Dutch, or had severe cognitive impairment. All eligible patients were invited by mail to participate by their own hematologist. Patients had the option to complete a web-based or paper-based questionnaire. Patients were informed that completion of the web-based questionnaire automatically resulted in enrollment in the RCT with randomization to one of the study arms, whereas completion of a paper questionnaire resulted in participation in the observational Patient Reported Outcomes Following Initial treatment and Long term Evaluation of Survivorship (PROFILES) registry [13] but not enrollment in the RCT. To address the primary objective of the current study—describing the proportion of RCT participants compared to all patients invited to participate—paper respondents were assessed as nonparticipants as they did not participate in the RCT.

Measures

Sociodemographic and Clinical Measures

Sociodemographic characteristics (age and sex) and detailed clinical information (date of diagnosis, cancer type, primary treatment) were available from the NCR. Information on education level and marital status was assessed from the questionnaire (data only available for respondents).

Comorbidities at the time of questionnaire completion were assessed using an adapted version of the self-administered comorbidity questionnaire [15]. Patients were asked to identify comorbidities present within the past 12 months, including heart disease, hypertension, arthritis, stroke, lung disease, diabetes, stomach disease, kidney disease, liver disease, anemia, thyroid disease, and rheumatoid arthritis. Positive responses were summed to a total score ranging from 0 to 12 (data only available for respondents).

Personality Traits

Personality traits were assessed with the Big Five Inventory, a 44-item inventory designed to measure the Big Five dimensions of personality: extraversion, neuroticism, conscientiousness, agreeableness, and openness to experience [16]. Each item was scored on a 5-point scale. Scale scores were obtained by averaging all items for each domain ranging from 0 to 5. Each trait is assumed to represent a continuum from high to low on the specific attribute and is partnered with a trait on the opposite pole of the spectrum [17,18].

Information Preferences

One question from an adapted version of the Information Satisfaction Questionnaire was used to measure information preferences [19]. Patients had to categorize themselves into one of three groups: those who would like (1) all available information, (2) only positive information about the illness, and (3) only limited information. Patients were further asked whether they use the internet (yes/no).

Psychological Distress, Self-Management Skills, and Satisfaction With Information Provision

Primary outcomes to assess the effectiveness of the intervention were psychological distress, self-management skills, and satisfaction with information provision.

Psychological distress was assessed with the 14-item Hospital Anxiety and Depression Scale [20]. Each item is rated on a 4-point scale from 0 to 3. The total score was obtained by adding all item scores and ranged from 0 to 42, in which higher scores indicate higher levels of psychological distress [21]. Patients with a Hospital Anxiety and Depression Scale sum score ≥13 were categorized as “psychologically distressed” [22].

Self-management skills were assessed with the Health Education Impact Questionnaire (heiQ) that contains 40 items across eight scales: positive and active engagement in life, health-directed activities, skill and technique acquisition, constructive attitudes and approaches, self-monitoring and insight, health service navigation, social integration and support, and emotional distress [23]. Each item is scored on a 4-point scale. Scale scores were obtained by averaging all items for each domain and ranged from 1 to 4. Higher scores indicate better status or self-management, except for emotional distress in which higher scores indicate greater distress [23].

Satisfaction with overall information provision was assessed with one item from an adapted version of the Information Satisfaction Questionnaire [19]. Patients were asked to rate their level of satisfaction for overall information provision on a scale from 1 (“very unsatisfied”) to 5 (“very satisfied”).

Statistical Analyses

The proportion of RCT participants (ie, participation rate) was calculated by dividing the number of patients who were enrolled in the RCT by the total number of eligible patients who were invited to participate. Sociodemographic and clinical characteristics of RCT participants were compared with those of all respondents (ie, patients who completed a web-based or paper questionnaire), all patients invited to participate, and all patients selected from the population-based NCR. In addition, personality traits and information preferences of RCT participants were compared with those of all respondents. Differences on sociodemographic and clinical characteristics between RCT participants and nonparticipants were compared using analysis of variance for continuous variables and Chi-square tests for categorical variables.

Differences in baseline psychological distress, self-management skills, and satisfaction with information provision (ie, RCT outcome variables) between RCT participants and paper respondents were compared using analysis of variance for continuous variables and Chi-square tests for categorical variables. All statistical analyses were performed with SAS version 9.4 (Cary, NC, USA). P≤.05 indicated statistically significant differences.

Results

Patients Selected From the Netherlands Cancer Registry

As shown in Figure 1, a total of 1193 patients with lymphoma or CLL who were ≥18 years old at diagnosis from 13 hospitals were selected from the population-based NCR. The basic characteristics of the invited patients are summarized in Table 1. The majority of patients were men, were diagnosed with high-grade nonHodgkin lymphoma, and were actively being treated, with chemotherapy as the most common treatment.

‎View this figure

Patients Invited to Participate

A flowchart of the patient selection process is shown in Figure 1. Of all selected patients, 24.81% (296/1193) were excluded after verifying eligibility by their treating hematologists for the following reasons: deceased, severe psychopathology, too ill, insufficient proficiency of the Dutch language, and cognitive impairment. In addition, 156 patients were not eligible for other reasons, including 36 patients who received treatment or follow up in another hospital and 20 patients for whom the ultimate diagnosis did not meet our inclusion criteria (eg, myelodysplastic syndrome, acute lymphoblastic leukemia). The remaining 100 patients were excluded by the hematologists for unknown reasons. Furthermore, 5 patients were excluded as they declined participation in previous studies within the PROFILES registry. After exclusion of these patients, 892 patients (74.77%) were invited to participate and completed a questionnaire. Patients invited to participate did not significantly differ from all patients selected from the NCR in terms of age (P=.38) and sex (P=.07) (Table 1).

Respondents

Among the 892 invited patients, 456 patients (51.1%) responded and completed either a web-based or paper questionnaire. The mean age of all respondents (Table 1) was comparable with that of nonrespondents (63.8 years, P=.43), and the majority of the respondents were also men. Respondents did not differ from all patients selected from the NCR in terms of age (P=.81) and sex (P=.26). Respondents were more often actively treated than nonrespondents (75% vs 66%, P=.01). Half of the respondents (229/456, 50.2%) completed a paper questionnaire, whereas the other half (227/456, 49.8%) completed a web-based questionnaire and were enrolled in the RCT. Nearly half of the respondents stated that they would like to receive all available information, with a lower proportion preferring limited information, and even less indicating that they would like to receive only positive information about the illness (Table 1). Approximately 82% of all respondents reported using the internet.

Randomized Controlled Trial Participants

A quarter of all invited patients (227/892) participated by completing a web-based questionnaire, which resulted in a participation rate of the RCT of 25.4%. The mean age of RCT participants was slightly lower than that of nonparticipants (ie, nonrespondents and paper respondents (n=665, 65.3 years, P<.001) with a slightly higher mean time since diagnosis, and comprised a higher proportion of men (70.9%, 161/227 vs 57.0%, 379/665, P<.001). The proportion of patients who were actively treated were comparable between RCT participants and nonparticipants (75.8%, 172/227 vs 69.0%, 459/665, P=.13). In addition, RCT participants were significantly younger than respondents, patients invited to participate, and patients selected from the NCR (all P<.001). Furthermore, there was a higher proportion of men among RCT participants compared with all patients invited to participate and all patients selected from the NCR (both P<.001).

Randomized Controlled Trial Participants Versus Paper Participants

RCT participants were younger than paper respondents (60.7 vs 68.3 years, P<.001). In addition, RCT participants were more often male (71% vs 57%, P=.002), more highly educated (50% vs 20%, P<.001), and more often had a partner (84% vs 75%, P=.02). No significant differences were found between RCT and paper respondents regarding cancer type or primary treatment (P=.54 and P=.06, respectively). RCT participants also reported fewer comorbidities than paper respondents (1.1 vs 1.4, P=.02).

Concerning personality traits, RCT participants had lower scores on neuroticism (2.4 vs 2.6, P=.003) and higher scores on openness to experience (3.5 vs 3.4, P=.002) than paper respondents, although effect sizes were small (Cohen d=0.29 and 0.28, respectively). With respect to information preferences, the majority of RCT participants stated a preference for receiving all available information, whereas only 38.9% (89/229) of paper respondents indicated this preference (P=.001). Conversely, paper respondents more often preferred receiving limited information (41.9%, 96/227 vs 34.4%, 78/227; P=.001). Furthermore, RCT participants more often used the internet (96.9%, 220/227 vs 66.4%, 152/229; P<.001) (Table 1).

Emotional distress, as measured with the heiQ, was significantly lower among RCT participants compared with the score for paper respondents (Table 2), although the effect size was small (Cohen d=0.25). No significant differences were observed regarding other self-management skills between the RCT and paper groups. In addition, no significant differences were observed in the proportion of patients with psychological distress between RCT participants and paper respondents, although paper participants seemed to have higher mean scores. Furthermore, no differences were observed between RCT participants and paper respondents regarding satisfaction with overall information provision (Table 2).

Discussion

Principal Findings

This reach analysis among RCT participants within a population-based sample showed a selective reach with an underrepresentation of older patients, women, and those with a medium to low level of education. In addition, our RCT participants may represent individuals with relatively better psychological well-being as scores for emotional distress were lower in this group.

Approximately a quarter of the population-based sample of patients with lymphoma and CLL were assessed to be not eligible for the study for various reasons (eg, deceased, severe psychopathology or cognitive impairment, a different diagnosis was ultimately made). Among the eligible patients who were invited to participate, 51% responded and completed a questionnaire, half of whom completed the web-based questionnaire and were enrolled in the RCT, resulting in a participation rate of 25%. This means that only one in four of all eligible patients actually participated in the RCT. This participation rate was lower compared with that of an RCT on the fully automated electronic health (eHealth) application Oncokompas that supports cancer survivors in their self-management (48%) [24]. Similar to the patients in our RCT, patients in the Oncokompas RCT were selected from the population-based NCR. However, cancer survivors in the Oncokompas RCT were first invited in an online survey study on supportive care and eHealth to assess internet use. Their participation rate was calculated as the number of RCT participants divided by the number of eligible respondents of the survey (access to the internet and email address). Thus, their group of eligible respondents was more selective compared with our sample. In our sample, only 82% of all respondents used the internet, and this percentage may be even lower among all patients invited to participate.

The results of the current study demonstrate that the RCT participants were younger, more often men, and more often actively treated compared to nonparticipants. Thus, the sample of RCT participants may not be representative of the target population. Therefore, even though the sample size reached the required number of patients [12], this sample may not be reliable for drawing conclusions about the target population. Furthermore, the effects of the intervention on the target population may be different from the effects that were found in the RCT sample [10,25].

These results also provide information about the response rate of observational research, which was 51%. This is comparable with response rates from other population-based studies on quality of life among lymphoma survivors in Germany (54.7%) [26] and the United States (54.8%) [27]. However, the current response rate is lower compared with that reported from earlier observational research within our study group at approximately 80%, despite similar patients and recruitment procedures [28]. This might be explained by the knowledge that the more information that is disclosed about the study—which is inherently more for an RCT than for observational research—the higher the proportion of nonrespondents [29]. Patients received abundant information along with the invitation, especially about participation in an RCT and randomization. The amount of information, as well as the knowledge of being randomized when completing a web-based questionnaire, may have deterred patients from participating. In addition, the type of intervention may have influenced the participation rate, as the majority of patients did not have problems with emotional adjustment to having cancer and therefore may have been less interested in a self-management intervention. Another explanation may be related to the fact that participation and response rates for health-related research have been declining over the past several years [30,31], and potential participants are faced with an increasing number of requests to participate in studies. This may result in patients refusing to participate in all studies [32].

We further compared the characteristics of RCT participants with those of paper respondents. RCT participants, who completed the web-based questionnaire, were younger, more often men, and more highly educated than the paper respondents, which are similar to the characteristics from previous observational studies within our study group [33,34]. Highly educated patients more often display prosocial behavior than patients with lower levels of education, and therefore the former group may be more likely to participate in an RCT for altruistic reasons [35]. In addition to differences in sociodemographic characteristics, RCT participants reported lower scores related to neuroticism and higher scores related to openness than paper respondents. In addition, information preferences slightly differed between RCT and paper respondents, as RCT participants more often wanted to receive all available information on their disease. RCT participants also more often reported using the internet. To complete a web-based questionnaire, and subsequently be enrolled in a web-based self-management intervention RCT, patients must not only be able to use a computer but also be sufficiently skilled in browsing the internet [33]. Although there seems to be a trend of older individuals becoming more active online [33,36], there is still a subgroup of patients who do not use the internet and thus have no access to a web-based questionnaire or internet-based intervention.

Despite these various differences between RCT participants and paper participants, baseline scores on self-management skills, satisfaction with information provision, and psychological distress appeared to be comparable between these groups, although scores for emotional distress were slightly lower among RCT participants.

Strengths

The strengths of this study include its unique setting. As patients were recruited from the population-based NCR, we had information on sociodemographic and clinical characteristics of both RCT participants and nonparticipants. In addition, as the RCT was embedded in the PROFILES registry, we were able to assess differences between RCT participants and paper respondents on sociodemographic and clinical characteristics, in addition to personality traits, information preferences, and baseline psychological distress, self-management skills, and satisfaction with information provision (ie, RCT outcome variables). This information provided the opportunity to determine both the reach and generalizability of the RCT sample.

Limitations

The current study has some limitations. Although information regarding sociodemographic and clinical characteristics of nonparticipants was available, we did not have information about nonparticipants’ reasons for declining participation or their physical and psychological health. Therefore, it remains unclear whether the physical and psychological health of RCT participants is similar to that of nonparticipants. In a previous study that assessed the generalizability of the results of observational research among cancer survivors by comparing characteristics of participants and nonparticipants, sensitivity analysis demonstrated that quality of life might be lower among nonparticipants [34]. As RCT participants may have a systematically higher quality of life or report fewer symptoms compared with nonparticipants, observed outcomes may represent a group of healthier patients with better outcomes. This may lead to circumspection in generalizing the results of an RCT to the target population. It is important to keep this in mind when interpreting RCT results that may only represent a minority of the target population.

Conclusions

The participation rate in the RCT was 25%. RCT participants may be not representative of the target population owing to different sociodemographic and clinical characteristics. RCT results should be considered with caution, as RCT participants represent a minority of the target population, and may actually be those least in need of the intervention.