Testing the Effectiveness of an Animated Decision Aid to Improve Recruitment of Control Participants in a Case-Control Study: Web-Based Experiment

Introduction

One of the most effective methods to test for exposure in epidemiological research is to conduct a case-control study in which people who have an illness are compared retrospectively with a matching population without the outcome [1]. Although it is a reliable methodology to test for associations, poor recruitment of population representative controls often undermines such studies [2].

Previous research reports the most common methods of recruiting control participants for case-control studies as follows: door-to-door recruitment, postal invitation, and random digit dialing [1,2]. More recently, with greater access to the internet, many cohort studies have moved their participant management and recruitment online (using unique websites), providing new opportunities to recruit participants, potentially improving diversity and ease of data collection [3,4]. Despite several advantages, however, caution needs to be exercised with the opportunistic recruitment of participants to web-based studies. For example, a recent study reported less than 4% of participants who visited a study recruitment website, after clicking on a targeted social media advertisement campaign, went on to sign up to the research [5]. This indicates that while individuals may be forming some interest to take part in research studies by clicking on a recruitment advertisement, their intention does not always translate to survey completion after they land on the research website.

Evidence on the barriers and facilitators of web-based survey completion primarily relates to the completion of stand-alone web-based surveys, rather than the use of unique websites to recruit participants to case-control studies [6-11]. These studies suggest that individuals’ trust in the organization carrying out the research, whether they are early adopters of technology and high in literacy, and whether the research is in line with the individuals’ values and beliefs are positive predictors of individual participation. Recommendations to achieve better outcomes include clear communication of the research goals, transparency about how data will be used, and shorter survey length.

Clinical trials have attempted to address some of the above (eg, transparency about how data will be used), using audio-visual decision aids to supplement the process of obtaining informed consent [12]. Communicating information via these mediums (enabled through web-based recruitment strategies), have the potential to reduce the associated cognitive load, facilitate further engagement with the research aims, generate positive attitudes toward the targeted behavior, and subsequently motivate engagement in the behavior itself [13-16]. To our knowledge, the potential impact of animated decision aids on intentions to take part in a web-based case-control study has not previously been investigated. This study aims to measure the effectiveness of an animated decision aid as a supplementary tool on a simulated website of a case-control study to encourage participation.

Methods

Setting

This study comprised a randomized web-based experiment, which assessed the effectiveness of adding an animated decision aid to a simulated website. The simulated website the animation was designed for (or added to) was the recruitment website for the case-control study: Cancer Loyalty Card Study (CLOCS) [17].

Cancer Loyalty Card Study

CLOCS is an observational case-control study that aims to investigate the self-care behaviors of patients with ovarian cancer prior to their cancer diagnosis. It seeks to do this by investigating differences in transactional data (such as medication purchasing) between women with and without ovarian cancer (the transactional data are collected through the loyalty cards of 2 UK-based high street retailers). Cases (ie, women with ovarian cancer) are recruited through participating National Health Service sites, while controls are recruited through the study website. Full details for CLOCS have previously been reported in the study protocol [17].

Animated Decision Aid

A key challenge for CLOCS recruitment has been communicating the research aims clearly, and our previous research highlights that the public often needs further explanations for how individual transactional data can be used in health research [18]. To improve public understanding and engagement with the aims of CLOCS, an animated decision aid was jointly prepared by Science Animated Limited, 2 patient representatives, and the CLOCS research team prior to the initiation of this web-based experiment. It aimed to convey key facts on ovarian cancer, the potential contribution of CLOCS in informing earlier diagnosis, as well as how women in the general population can play a role to aid its efforts (based on the participant information sheet tailored and approved by a National Health Service Research Ethics Committee [19/NW/0427-SA1], included in Multimedia Appendix 1). However, it should be noted that the animation was designed as a supplement to the main study materials, not as a key participant communication material to prompt informed decision-making. The animation features English subtitles and is 123 seconds (2 minutes and 3 seconds) long [19].

Procedure

The randomized web-based experiment was programmed in Survey Monkey. In July 2020, women who were eligible to take part in CLOCS as a control(ie, between the ages of 18 and 70, living in the United Kingdom, and without an ovarian cancer diagnosis) were recruited through a survey vendor (Dynata Limited). For those who were interested in taking part, information about the experiment, including a brief description of CLOCS, was presented and followed by the completion of the study consent form. If participants consented and were eligible, they were randomized (in a 1:1 ratio) to one of the following two experimental conditions: the simulated CLOCS website without the animation (control), or the simulated CLOCS website with the animation (animation) (Multimedia Appendix 2 and 3).

Once everyone viewed the simulated website, they were asked to complete the survey, where they were required to indicate their intention to take part in CLOCS using a 4-point Likert scale (definitely yes, probably yes, probably not, and definitely not) adapted from previous research [20-22]. The study participants were then asked to indicate their loyalty card use by selecting from a list of high street retailers’ loyalty cards and whether they would be willing to share their loyalty card data for research purposes. The latter question was adapted from research on willingness to share electronic health data, which uses 4 commonly used models of consent for the use of data [23].

In the next step, the study participants were asked about their educational level, annual household income, and health literacy. For the latter, the eHealth Literacy Scale was used. This scale assesses individuals’ retrieval and judgement of health information on the internet [24]. It consists of 8 items rated on a 5-point scale, ranging from 1 to 5, and has demonstrated considerable reliability and validity [24]. The participants’ scores across the 8 items on the scale were summed and calculated for a sample mean. Individual scores below and above (or equal to) this sample mean were defined as low and high health literacy, respectively [25,26].

The final survey item was included as a behavior proxy. The participants were asked whether they want to be redirected to the actual CLOCS website for more information on how to take part. Those who responded that they would like to visit the website were provided with a link to the CLOCS website at the final page of the survey [22]. The website opened in a new tab for participants who clicked on the link. No data were collected from the study participants for their direct participation in CLOCS associated with the experiment.

Ethics Approval

Ethics approval for this study was granted by the UCL Research Ethics Committee (Project ID: 17823/001).

Data Analysis

A pilot study has been conducted beforehand for the purpose of sample size calculations. Based on the findings from the initial sample of 359 participants, with a 10% difference in intention to take part (definitely yes or probably yes versus definitely no or probably no), we determined the number of participants needed to achieve 95% certainty and 80% power was 650 per trial arm. Data from participants in both the pilot and final sample were combined for analysis.

Sample characteristics were assessed using descriptive statistics (Table S1 in Multimedia Appendix 4). To aid interpretation, the participants’ income and educational levels were dichotomized in inferential analyses. For income, we used £30,000 (US $37,000) as the cut-off point, based on the average household income in the United Kingdom (reported by the Office for National Statistics, 2020) [27]. For education, we categorized participants into those with General Certificate of Secondary Education or A Levels and those with a university degree [20-22].

Differences in intentions to take part in CLOCS were assessed using univariate and multivariate ordinal logistic regression. Willingness to visit the actual website and willingness to share loyalty card data for research purposes, between groups, were assessed using univariate and multivariate binary logistic regressions. Odds ratios (ORs), 95% confidence intervals, and P values are presented in the results, with P values below .05 regarded as statistically significant. Participants who spent a short amount of time completing the survey (ie, survey speeders) were excluded from the analysis (Multimedia Appendices 5-8, based on the 50% cut-off points provided from previous research using median values) [28]. We report the analysis for the whole sample in Table S2 (Multimedia Appendix 9) and the distribution of the time spent on the survey before and after the exclusion of the speeders in Multimedia Appendices 5-8 [29]. Additional analyses for interaction between the intervention and health literacy led to null results and were not reported due to the unbalanced proportion of those with low health literacy and high literacy in the study population.

Results

Study Sample

Figure 1 demonstrates the flow of participants through the study. In total, 6034 women were invited to participate, of which 4609 (76.4%) were excluded as they were not eligible, dropped out, or discontinued the initial screening. The remaining 1425 (23.6%) were then randomized to one of the following two experimental conditions: 732 (51.4%) were randomly allocated to the control condition, and 693 (48.6%) to the animation condition. Across conditions, 131 (9.2%) did not finish the survey after randomization. Furthermore, 137 (9.6%) were excluded as they spent less than 50% of the median time. The analytical sample consisted of 1157 women—610 (52.7%) participants in the control condition, and 547 (47.3%) in the animation condition.

Most women in the analytical sample were aged between 55 and 70 years (n=417, 36.9%), did not have a university degree (n=686, 59.3%), and had an annual household income of less than £30,000 (US $37,000; n=622, 53.8%). The mean eHealth Literacy Scale score for the participants was 30.3 out of 40; thus, those who scored below this mean were classified as having low health literacy [20,21]. Post hoc comparisons revealed that sociodemographic characteristics were comparable between the two experimental conditions (Table S1 in Multimedia Appendix 4).

‎View this figure

Intentions to Participate in CLOCS

Intentions to participate in CLOCS were generally very high, with 69.7% (n=807) of women stating that they would probably or definitely participate. Figure 2 shows the distribution of the intentions to participate in CLOCS after exposure to the simulated website. The ordered logistic regressions in Table 1 show that the inclusion of the animation did not affect participation intentions (OR 1.12; 95% CI 0.90-1.39 and AOR 1.09; 95% CI 0.88-1.35). The regression further shows that older women aged 55-70 years stated lower intentions to participate (OR 0.56; 95% CI 0.42-0.74), while those with an income above average (OR 1.36; 95% CI 1.06-1.63), one or more existing loyalty cards (AOR 2.05; 95% CI 1.16-3.62), and low health literacy (AOR 1.26; 95% CI 1.01- 1.58) had higher intentions to take part in CLOCS.

‎View this figure

Willingness to Share Loyalty Card Data for Research

Most study participants stated that their data should be used but that they should have the option of saying no (control condition: n=246, 40.3%; and animation condition: n=181, 33.1%; Figure 3). However, significantly more participants in the animation condition, compared to control, indicated that they would be willing to provide the data if they were needed—as shown in the binary logistic regressions in Table 2 (control: n=109, 17.9% vs animation: n=142, 26.0%; OR 1.61; 95% CI 1.22-2.14 and AOR 1.64; 95% CI 1.23-2.18). Similarly, as with the intentions to participate in CLOCS, women aged 55-70 years were again less likely to state that their data should be used if needed compared to those aged 18-34 years (n=71, 17% vs n=72, 24.8%; AOR 0.62; 95% CI 0.43-0.89).

‎View this figure

Willingness to Visit the CLOCS Website After the Survey

A slight majority of the study participants (n=596, 51.5%) indicated that they would like to visit the CLOCS website after the end of the survey (Multimedia Appendix 10). Table 3 shows that there was no difference between the two experimental conditions (control: n=308, 50.5% vs animation: n=288, 52.6%; OR 1.09; 95% CI 0.87-1.37 and AOR 1.08; 95% CI 0.85-1.37). Women with low health literacy, in comparison to those with high health literacy (n=111, 60% vs n=485, 49.9%; AOR 1.43; 95% CI 1.03-1.98), and women aged 35-44 years, in comparison to those aged 18-34 years, were more interested in visiting the study website (n=145, 60.4% vs n=148, 51%; AOR 1.49; 95% CI 1.05-2.12).

Discussion

Key Findings

This randomized web-based experiment examined the effectiveness of an animated decision aid to increase the willingness to participate in a case-control study. The results show that the animation did not increase intentions to participate in a real-world case-control study (CLOCS), or willingness to visit the real study website after the survey. However, the animation increased the participants’ willingness to share data from their loyalty cards for research. Interestingly, immediately after the completion of this web-based experiment, there was a spike in activity within the case-control study, with over 100 people signing up to participate in CLOCS.

Comparison With Previous Literature

Our findings reflect the mixed evidence currently available in the literature. While there has been some support for the effectiveness of animated decision aids in the context of health behavior research, many studies have focused on how the interventions can improve participants’ knowledge of the health behavior concerned, as their main outcome. When assessing participant intention to engage in the behavior (or an objective measurement of the behavior itself), the findings have been more inconsistent [30,31].

An interesting finding in our study is that, among all participants, those with lower health literacy scores were more interested to find out about the study compared with those with high health literacy scores. Our sample size calculations were based solely on the primary outcome; thus, we might have been underpowered to detect the interaction effects of health literacy and outcomes on the behavior proxy. However, the previous studies indicate that multimedia interventions are not always significant in individuals with lower educational levels [14] and low health literacy [15,16]. It has been previously shown in the cancer screening literature that gist-based supplementary materials could be used to enhance engagement with the main literature among people with low numeracy [32], and perhaps using animation and other easy-to-read materials could enhance participant recruitment in health research [33]. Furthermore, the positive association between low health literacy and willingness to visit the website may be explained by factors such as wanting to find more information, the salience of the research topic, and other factors that were not included in this web-based experiment. As such, future studies focusing on the comprehension of the materials among people with low health literacy using a think-aloud methodology could explain this outcome.

Strengths and Limitations

This study has some important limitations, which call for follow-up research. First, we did not include a comprehension assessment to check if participants fully understood or watched the animation. Second, we did not measure attitudes toward the simulated website and CLOCS. A recent systematic review on participant comprehension and informed consent in health research further highlighted that while there are efforts to improve participation rates using various methods, there is a lack of assessment of participant readability, literacy, and standardization of recruitment methods in health research for informed consent procedures [33]. While all the participant-facing CLOCS research materials have been reviewed by patient and public representatives, further assessment of participant comprehension prior to the animation experiment would have strengthened our methodology. However, the rationale for the exclusion of these measures was based on the assumptions that people often form immediate decisions about whether something is relevant to them using heuristic decision-making before establishing deliberative decisions [34]. As such, by excluding cognitive measures in our assessment to minimize judgement and bias, we tried to capture individuals’ potential reactions to the website as close to their reaction in real life. In this context, further studies using eye-tracking experiments on the simulated website will be highly informative to build a better understanding of the interaction with the website and the contents [35].

On the other hand, exposure to the animation increased the intentions to share loyalty card data for research in general, but not for intentions to participate in the CLOCS study; this suggests that there are study-specific characteristics that did not appeal to individuals (eg, actively signing up to provide information) or that the study participants were not eligible. However, a recent study also shows that only half of the population is willing to share shopping data for health research, highlighting the differences in sociodemographic characteristics of people who are willing to share their data for health research [36]. The characteristics of the participants who are willing to take part in CLOCS in this study mirror the results of this age-stratified survey employed in England, with older women less willing to take part. While self-sampling bias will continue to be a concern of case-control studies based on the differences in characteristics of the people who are willing to take part, recruitment strategies could be stratified and tailored to engage different populations who are less willing to take part in health research based on this evidence and the validation of public acceptability. Our results support this evidence further using experimental design with greater internal validity for potential barriers in recruiting participants to a case-control study.

Implications for Policy and Future Research

While such questions of generalizability are warranted, this study still poses important relevance and implications to current research contexts. Due to the COVID-19 pandemic, many research studies have been forced to consider the possibility of being adapted online. This may require researchers to derive additional strategies to reach web-based samples with different characteristics. Underrepresentation in health research is already an issue for minority populations, people with low literacy, and those with greater deprivation using traditional methods of recruitment, unless they are specifically targeted [37,38]; thus, there is a further need to ensure that web-based strategies can provide means for researchers to attract representative samples.

Future studies should therefore continue exploring web-based methods to facilitate complex decision-making processes for potential participants of health research. The CLOCS animation has not been actively disseminated for participant recruitment following this dearth of evidence; however, unique findings might be obtained for research of a different nature. Other multimedia formats or mediums such as social media can be further explored in future studies, along with the consideration of potentially important variables such as participants’ willingness to share data for research purposes.

Conclusion

The results of this study indicate that the animated decision aid did not influence the participants’ intention to take part in CLOCS or visit the study website. The animation, however, increased the probability of individuals stating that they would share their loyalty card data for research. Future research should continue exploring methods that can effectively engage participants with low health literacy to participate in complex health research.