This paper is in the following e-collection/theme issue:

Recruitment of Research Participants (296) Innovations in Clinical Trials and Research Data Management (83) Digital Health Reviews (1289)

Published on in Vol 22, No 11 (2020): November

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/22179, first published .
Online Patient Recruitment in Clinical Trials: Systematic Review and Meta-Analysis

Online Patient Recruitment in Clinical Trials: Systematic Review and Meta-Analysis

Online Patient Recruitment in Clinical Trials: Systematic Review and Meta-Analysis

Authors of this article:

Mette Brøgger-Mikkelsen1, 2, 3 Author Orcid Image ;   Zarqa Ali1, 2 Author Orcid Image ;   John R Zibert3 Author Orcid Image ;   Anders Daniel Andersen3 Author Orcid Image ;   Simon Francis Thomsen1, 2 Author Orcid Image
Article Authors Cited by (69) Tweetations (12) Metrics

    Review

    1Department of Dermatology, Bispebjerg Hospital, Copenhagen, Denmark

    2Department of Biomedical Sciences, University of Copenhagen, Copenhagen, Denmark

    3Studies&Me A/S, LEO Innovation Lab, Copenhagen, Denmark

    Corresponding Author:

    Mette Brøgger-Mikkelsen, MA

    Department of Dermatology

    Bispebjerg Hospital

    Bispebjerg Bakke 23

    Copenhagen, 2400

    Denmark

    Phone: 45 28901012

    Email: mette.mikkelsen@studiesandme.com


    Background: Recruitment for clinical trials continues to be a challenge, as patient recruitment is the single biggest cause of trial delays. Around 80% of trials fail to meet the initial enrollment target and timeline, and these delays can result in lost revenue of as much as US $8 million per day for drug developing companies.

    Objective: This study aimed to conduct a systematic review and meta-analysis examining the effectiveness of online recruitment of participants for clinical trials compared with traditional in-clinic/offline recruitment methods.

    Methods: Data on recruitment rates (the average number of patients enrolled in the study per month and per day of active recruitment) and conversion rates (the percentage of participants screened who proceed to enroll into the clinical trial), as well as study characteristics and patient demographics were collected from the included studies. Differences in online and offline recruitment rates and conversion rates were examined using random effects models. Further, a nonparametric paired Wilcoxon test was used for additional analysis on the cost-effectiveness of online patient recruitment. All data analyses were conducted in R language, and P<.05 was considered significant.

    Results: In total, 3861 articles were screened for inclusion. Of these, 61 studies were included in the review, and 23 of these were further included in the meta-analysis. We found online recruitment to be significantly more effective with respect to the recruitment rate for active days of recruitment, where 100% (7/7) of the studies included had a better online recruitment rate compared with offline recruitment (incidence rate ratio [IRR] 4.17, P=.04). When examining the entire recruitment period in months we found that 52% (12/23) of the studies had a better online recruitment rate compared with the offline recruitment rate (IRR 1.11, P=.71). For cost-effectiveness, we found that online recruitment had a significantly lower cost per enrollee compared with offline recruitment (US $72 vs US $199, P=.04). Finally, we found that 69% (9/13) of studies had significantly better offline conversion rates compared with online conversion rates (risk ratio 0.8, P=.02).

    Conclusions: Targeting potential participants using online remedies is an effective approach for patient recruitment for clinical research. Online recruitment was both superior in regard to time efficiency and cost-effectiveness compared with offline recruitment. In contrast, offline recruitment outperformed online recruitment with respect to conversion rate.

    J Med Internet Res 2020;22(11):e22179

    doi:10.2196/22179

    Keywords

    online clinical trial; web-based clinical trial; hybrid clinical trial; online recruitment; remote recruitment; recruitment; clinical trial; conversion rate 


    Historically, recruitment of participants for clinical trials has been critically dependent upon physician referrals and overall site performance. Increasing needs for more effective recruitment methods have led to “trial and error” models, where a number of different recruitment strategies are utilized and modified according to observed effects on recruitment [1]. Such traditional recruitment strategies include, but are not limited to, soliciting subjects through mail and telephone using health records and registers, media campaigns, newspaper advertisements, and input during radio and television talks [2].

    Currently, recruitment for clinical trials continues to be a challenge, as patient recruitment is the single biggest cause of trial delays. Around 80% of trials fail to meet the initial enrollment target and timeline, and these delays can result in lost revenue of as much as US $8 million per day for drug developing companies [3]. As pointed out by Gul and Ali [4], slow and inefficient recruitment may have scientific, economic, and ethical consequences. The costs associated with recruitment are wasted, and data quality is hampered by a reduction in statistical power due to underrecruitment. Recruiting appropriate participants in a sufficient number to fulfill sample size requirements is critical for the validity of the research findings, and failure may lead to invalid or inconclusive results. Further, for traditional offline recruitment strategies, location of trial sites quickly becomes the bottleneck for participant diversity in clinical research, as sites only succeed in recruiting patients within a relatively short radius. Potentially, this results in clinical research that lacks generalizability and makes it difficult for clinical trials to be a cornerstone for providing scientific evidence on the safety and efficacy of novel pharmaceutical compounds.

    Using online recruitment strategies, such as social media advertisements, Google search engine advertisements, and other website campaigns, may enable researchers to target specific study populations by demographic characteristics, location, and keywords previously used in potential participants’ user profiles. In 2018, Akers et al [5] found that Facebook advertisements gave “flexibility to monitor and modify advertisement tactics based on feedback,” and Shere et al [6] argued that social media recruitment should be redefined as an active recruitment tool rather than a low-budget passive tool, as targeting specific populations effectively yielded high recruitment rates. Similar arguments were made by Watson et al [7], Jones et al [8], and Carter-Harris et al [9] who all reported social media advertisements to be a “viable tool for more efficient and cost-effective recruitment.” The potential reach of online recruitment by far exceeds the reach that traditional recruitment methods are able to generate, but whether online recruitment strategies outperform traditional offline recruitment strategies still remains unclear in the literature. Herein, traditional in-clinic recruitment methods are referred to as offline recruitment.

    The aim of this study was to evaluate the effectiveness of online patient recruitment by systematically reviewing studies that utilize online strategies for patient recruitment and by conducting meta-analyses comparing online and offline recruitment strategies on the following two recruitment metrics: recruitment rate (the number of patients enrolled in the study on average per month and per day of active recruitment) and conversion rate (the percentage of participants screened who proceed to enroll into the clinical trial). Further, this study investigated the cost-effectiveness of online recruitment compared with offline recruitment in clinical research. In the study, we investigated the following three hypotheses: (1) The recruitment rate is higher in online recruitment compared with offline recruitment; (2) The conversion rate is higher in offline recruitment compared with online recruitment; and (3) The cost per enrolled subject is lower in online recruitment compared with offline recruitment.


    Literature Search

    We conducted a systematic review and meta-analysis in accordance with the PRISMA statement [10] searching the following databases: PubMed, EMBASE, Cochrane Library, Web of Science, and Google Scholar. The search was conducted between February and May 2020, and was carried out by combining keywords within the following three topic domains: “online/remote/web-based,” “patient/participant/subject recruitment,” and “clinical trial/study.” Two reviewers (MBM and ZA) independently screened all titles and abstracts, and any discrepancies were resolved through discussion. Apart from duplicates, studies were excluded based on the content of the abstract if there was no clear indication that they investigated the feasibility of online patient recruitment.

    Screening and Study Selection

    Studies were included in the systematic review provided that they fulfilled the following inclusion criteria: (1) Clinical studies using online recruitment and/or prescreening of patients for randomized controlled trials (RCTs), observational studies, and online surveys relevant to the focus of this study and (2) Clinical studies using a fully virtual approach from screening to data collection. For the meta-analysis, only studies that compared online patient recruitment with offline patient recruitment were included. Studies were excluded if they fulfilled one or more of the following exclusion criteria: (1) non-English papers; (2) systematic reviews; and (3) other (nonrelevant online programs, eg, parenting training programs).

    Data Extraction

    For data collection, the first reviewer (MBM) used a structured form to extract the following qualitative and quantitative data: (1) study design and year; (2) study location by country; (3) total number of participants enrolled in the study; (4) online and offline recruitment metrics (full recruitment period in months, number of days with active recruitment, number of patients completing prescreening, number of patients enrolled by recruitment method, and costs); and (5) age and gender of the participants in the study. For studies where stated data collection was inadequate, the study authors were contacted and necessary data were obtained when possible.

    Outcomes Assessed

    The primary outcome variable for analysis was an aggregate measure of recruitment effectiveness defined by the recruitment rate and conversion rate. Two analyses were carried out. First, the recruitment rate was defined as the number of patients recruited per month for the entire recruitment period, and second, the recruitment rate was defined as the number of patients recruited per day of active recruitment days. As online campaigns are mostly run in shorter periods with active online advertisements, removing the days in between advertisements was expected to provide a more realistic understanding of the recruitment. For offline recruitment, days in between nonactive recruitment were removed in the second analysis. The conversion rate was defined as the percentage of patients screened who proceeded to enroll into the clinical trial. Prescreening is either an online prescreening questionnaire, or on-ground screening or telephone call for assessing primary eligibility. If necessary, we recalculated the metrics needed for the analyses where possible.

    The secondary outcome of interest was the cost-effectiveness of online recruitment compared with offline recruitment. To standardize cost data, we adjusted all costs to US$ using XE Live Exchange Rate 2020.

    Statistical Methods

    We pooled effect sizes based on the Mantel-Haenszel method for both recruitment rate (incidence rate ratio [IRR]) and conversion rate (risk ratio [RR]). A random effects DerSimonian and Laird meta-analysis [11] was used to report both incidence rates and relative risks. Further, we calculated heterogeneity (DerSimonian and Laird estimator), which was examined using the I2 statistics, and 95% prediction intervals were calculated. The Hartung-Knapp adjustment for a random effects model was used to calculate 95% CIs, reflecting the uncertainty in heterogeneity. All data analyses were conducted in R (R Foundation for Statistical Computing) [12] (packages included devtools, meta, dmetar, and pbkrtest), and P<.05 was considered significant.

    Meta-Analysis

    The only criterion for carrying out the meta-analysis was the availability of sufficient outcomes. We considered that any amount of statistical heterogeneity would be acceptable, as studies included in this paper recruited for a wide range of therapeutic areas and trial interventions. Hence, we expected high heterogeneity among the included studies. However, the recruitment strategy was far more homogeneous between studies, and therefore, we considered the findings worth reporting. We performed two separate meta-analyses to determine the effectiveness of online patient recruitment.

    Recruitment Rate

    The effect size for recruitment rate was calculated as an incidence rate (IR), as this rate signifies how many events occur within a standardized timeframe. Effect sizes were pooled to generate the IRR, examining the relation between the incidence rate in the online recruitment group (IRonline) and the one in the offline recruitment group (IRoffline). The pooled data for recruitment rate was presented in a forest plot showcasing whether the recruitment rate was in favor of online or offline recruitment.

    Conversion Rate

    The conversion rate was calculated as an event rate with relative risk as the effect size, since event rate data deal with the number of persons experiencing an event in each group and the total sample size in each group. Effect sizes were pooled to generate RRs as the summary measure. As for recruitment rate, the pooled data were presented in a forest plot showcasing whether the conversion rate was in favor of online or offline recruitment.

    Additional Analysis

    This study presents a cost-effectiveness analysis for studies included in the meta-analyses defined as cost per enrollee. As the distribution in our two paired data sets was nonnormal, cost-effectiveness was examined by a nonparametric paired sample Wilcoxon test that does not assume a specific underlying distribution of data.

    Risk of Bias

    The risk of bias was assessed using the Cochrane Risk of Bias Tool for randomized studies and the ROBINS-I tool for nonrandomized studies. The studies were assessed for risk of bias in relation to our review question and not the study authors’ research question.


    Literature Search

    A total of 3861 articles, including references from articles, were identified for possible inclusion based on their titles, and of these, 395 were selected for abstract screening. Of these, 135 studies were selected for full-article review, and 61 studies that reported the use of online patient recruitment without meeting any of the exclusion criteria were included in the systematic review. Of the selected studies, 23 studies investigated the feasibility of online patient recruitment compared with offline patient recruitment and were included in the meta-analysis after removing duplicates (Figure 1).

    ‎
    Figure 1. Flow diagram of the literature search.
    View this figure

    Characteristics of All Included Studies

    Multimedia Appendix 1 shows the detailed characteristics of all studies included in this review. Most were conducted in the United States or Australia, and together, the studies covered a wide range of therapeutic areas and trial interventions, with the majority of studies recruiting either adult smokers [7,9,13-24], men who have sex with men [8,25-33], or pregnant and postpartum women [6,34-39]. Of all the studies included, 15 reported a full clinical trial utilizing online recruitment itself (eg, [40-44]), while 46 of the studies described the recruitment strategy utilized in a clinical trial reported in a separate paper. In total, 39 studies covered the recruitment strategy for RCTs, 14 studies covered the recruitment strategy for observational research trials, and eight studies covered the recruitment strategy for online surveys.

    Recruitment Strategy

    Of the 61 studies included in this review, 42 studies concluded that targeting potential study participants online was an effective tool for recruitment. Only four studies reported that online recruitment was not effective [17,45-47], which might reflect the timing and time period of the online strategy in these four studies. The remaining studies did not conclude on effectiveness. A total of 55 studies used paid advertisements in their online recruitment strategy, of which 42 studies specifically used Facebook. In 2016, Adam et al [36] used Facebook advertisements to target pregnant women for an RCT and found that online strategies compared with offline strategies recruited a representative population and that recruitment rates had been “dramatically improved.” Similar findings were reported by Cowie et al [48], who found that Facebook yielded highly efficient and cost-effective results when targeting people aged 60 years or older. In contrast, Rait et al [17] found that Facebook advertisements expanded reach when recruiting young adults for a smoking cessation trial; however, only a small proportion was eligible for the study, and offline methods were therefore superior to online methods both in regard to cost and time-efficiency. In total, 26 studies further used websites relevant for the specific trial, for example, popular drug control websites when recruiting for a drug abuse prevention study [34,49,50].

    Demographics

    On examining the overall patient demographics of the included studies, most papers recruited participants aged 18 years or above, and the vast majority of studies recruited both men and women. Furthermore, 46 of the studies targeted a so called hard-to-reach population, reaching a sample that has previously been shown to be difficult to recruit owing to either stigmatization, such as smoking cessation research (eg, [7,21]) and HIV prevention research (eg, [29,30,51-53]), or underrepresentation and low prevalence, such as research involving children with fetal alcohol spectrum disease [54] and some psychiatric conditions [55-59]. In 2014, Shere et al [6] successfully utilized social media to recruit a hard-to-reach population of women in the periconceptional period for an RCT, and among others, Morgan et al [60] effectively recruited a sample for an RCT evaluating a depression intervention through several online strategies [61].

    Characteristics of the Studies Included in the Meta-Analyses

    For the meta-analyses, we included 23 studies, and all reported data on comparing the effectiveness of online and offline recruitment strategies. In total, 14 of the studies included in the meta-analyses recruited for RCTs reported in a separate paper [6-8,19-21,27,30,36,45,46,62-64]. Of the 23 studies, approximately 25% began online recruitment at a later time point than offline recruitment owing to low recruitment rates through offline recruitment approaches [6,28,36,45,62]. Data extracted from the 23 studies were used to calculate the recruitment rate, conversion rate, and cost per enrollee. Multimedia Appendix 2 shows the aggregated characteristics of the studies included in all the meta-analyses.

    Effectiveness of Online Patient Recruitment

    Principal Findings

    This study found that online recruitment strategies are superior to offline recruitment initiatives when measuring recruitment effectiveness by recruitment rate and cost-effectiveness. With online strategies, participants are recruited faster and more cost-effectively. However, this study found that offline recruitment outperforms online recruitment when converting potential participants to actual enrollees, which altogether is in line with our study hypotheses.

    For recruitment rate, we found online recruitment to be significantly more effective when examining the recruitment rate for active days of recruitment, where 100% (7/7) of the studies included had a better online recruitment rate compared with the offline recruitment rate (P=.04). When examining the entire recruitment period in months, we found that 12 of 23 studies (52%) had a better online recruitment rate compared with the offline recruitment rate. For the conversion rate, we found that only 4 out of the 13 studies (31%) had a better online conversion rate compared with the offline conversion rate.

    Meta-Analyses
    Recruitment Rate

    For the seven studies included in our first meta-analysis, we found that the recruitment rate for online recruitment was superior to that for traditional offline recruitment when comparing the number of active recruitment days presented as an IRR. For online recruitment, this corresponded to the number of days the advertisements were active on social media or other relevant websites. For offline recruitment, this was the number of days with active on-ground recruitment and active days of advertisements in newspapers, on busses, etc. Pooling the data from the seven studies, we found that online recruitment strategies recruited subjects significantly faster than offline recruitment strategies (IRR 4.17, 95% CI 1.12-15.59, P=.04). Hereby, online recruitment strategies yielded 4.17 times more participants per day of active recruitment compared with offline recruitment strategies. As expected, we found high heterogeneity between studies (I2=100%) (Figure 2).

    ‎
    Figure 2. Recruitment rate for active days of enrollment for online and offline recruitment. Value <1, in favor of offline recruitment; value >1, in favor of online recruitment.
    View this figure

    For our second meta-analysis, all 23 articles were included. Here, we compared the full period of recruitment for both online and offline strategies, looking at the number of months of recruitment from the start of the first advertisement or campaign to the end of the last advertisement or campaign. Pooling the data from the 23 included studies, we found that online recruitment was similar to offline recruitment with respect to effectiveness when nonactive days within the recruitment period were also included in the analysis (IRR 1.11, 95% CI 0.62-1.97, P=0.7). Again, we found high heterogeneity between studies (I2=99%) (Figure 3).

    ‎
    Figure 3. Recruitment rate for the entire period of recruitment online and offline in months. Value <1, in favor of offline recruitment; value >1, in favor of online recruitment.
    View this figure
    Conversion Rate

    For our meta-analysis on conversion rate, we included 13 of the 23 articles. We found that traditional offline recruitment strategies were superior to online recruitment strategies when comparing the percentage of participants screened who proceed to enroll into the clinical trial presented as a RR. For online recruitment, potential participants were screened through online questionnaires when clicking on online advertisements. For offline recruitment, potential participants who had shown interest in participating in the clinical trial were screened on ground or through telephone calls. Pooling the data from the 13 included studies, we found that online recruitment strategies converted significantly fewer potential participants into enrolled subjects compared with offline recruitment strategies (RR 0.8, 95% CI 0.67-0.96, P=.02). Hereby, offline recruitment strategies are more effective in converting participants who are screened into enrolled subjects in clinical trials. As for the recruitment rate, we found high heterogeneity between studies (I2=96%) (Figure 4).

    ‎
    Figure 4. Conversion rate for online and offline recruitment. Value <1, in favor of offline recruitment; value >1, in favor of online recruitment.
    View this figure
    Additional Analysis
    Cost-Effectiveness of Online Recruitment

    Of the 23 articles included for the meta-analyses, 13 studies reported data on the cost per enrolled participant [7,8,17,19-21,24,28,46,47,57,62,63]. The median cost per enrollee for online recruitment strategies was US $72 (range US $3.9-251.2), while the median cost per enrollee for offline recruitment strategies was US $199 (range US $19.1-839.0). However, the average cost per enrolled participant varied between the different studies, and in total, 4 out of the 13 included studies reported online recruitment to be less cost-effective compared with offline recruitment. We found a significant difference between online and offline cost-effectiveness (P=.048), with a V value of 17, meaning that there was a large difference between the two groups. Multimedia Appendix 3 presents cost per enrollee for all studies included in this analysis.

    Risk of Bias for Studies Included in the Meta-Analyses

    For the 14 randomized studies, none of the studies were deemed to be at risk of bias owing to allocation concealment, blinding, and missing outcome data when assessing the articles in relation to the focus of this review. Hence, no performance bias, detection bias, or attrition bias was identified. For the nine nonrandomized studies, no bias related to confounding, selection of participants, and classification of interventions was identified. However, for the studies included, the representativeness of the recruited samples was often discussed. In total, 13 of the studies included tested differences in representativeness relative to samples obtained through offline recruitment [6-8,19-21,24,28-31,36,47]. Of these, only 31% (4/13) reported that samples recruited online were similarly representative to samples recruited offline, whereas 69% (9/13) found relevant differences among the two groups. Characteristics that were most frequently reported to be imbalanced included gender (no consistent trend found), age (no consistent trend found), and education (higher education overrepresented for offline recruitment). A summary of the risk of bias can be found in Multimedia Appendix 2.


    Overall Findings

    This study found that targeting potential participants using online remedies is an effective approach for patient recruitment for clinical research. Online recruitment strategies were superior in regard to time efficiency and cost-effectiveness compared with offline recruitment strategies. In contrast, offline recruitment strategies outperformed online recruitment strategies when examining conversion rate. To our knowledge, this is the first time a meta-analysis has been performed on the effectiveness of online patient recruitment. Our findings are consistent with findings from previous reviews on online patient recruitment [65].

    Quantitative Analysis

    The recruitment rate reported in this study was only significantly better for online recruitment when days in between active recruitment were removed from the analysis (P=.04). This emphasizes that advertisements and campaigns on social media are efficient in a relatively short time period, after which the recruitment effectiveness starts dropping. Such findings were also reported by Juraschek et al [45], who concluded that offline recruitment was superior to online recruitment in a randomized trial recruiting cancer survivors. Therefore, online strategies should be run in intermittent campaigns to maximize full recruitment capacity, and they could also be of value when considering a short-term high-output recruitment solution. As pointed out previously, a large proportion of the studies that compared online and offline methods utilized online recruitment strategies only after realizing that offline strategies did not provide enough participants [6,7,28,36,46,62]. This could have an impact on the results for online recruitment, because the online solution here is at risk of being a short-sighted sticking-plaster solution. If online targeting had been the primary strategy of recruitment, the results might have been in the favor of online recruitment in this case [66].

    Offline conversion rates were found to be significantly higher than online conversion rates (P=.02), as originally hypothesized. This could be due to sites having existing health records of suited patients for specific studies. For prescreening in an offline setting, it is only patients who have already shown interest in participating or referrals who already have prequalified for enrollment that are actually screened by inclusion and exclusion criteria. Further, the reach of traditional offline recruitment strategies, such as measurements of how many people read newspapers or how many people listen to campaigns on radio, cannot be quantified to the same extent as the reach for online advertisements. Hence, we do not know the actual conversion rates for specific initiatives, and therefore, the results on the conversion rate in this paper could be overestimated for offline recruitment strategies.

    Our cost-effectiveness analysis showed that online recruitment is more cost-effective compared with offline recruitment. For online strategies, organic reach, such as social sharing generated online, may contribute to high cost-effectiveness, as it gives an exponential increase in message exposure, and Shere et al [6] referred to online sampling as “snow-ball recruitment,” which involves letting social media automatically expand the reach of a similar population. Even so, the cost differences arguably can be even larger, since the costs associated with offline recruitment strategies may have been underestimated. This is because for offline recruitment, the costs of personnel time were only included in very few of the cost measurements in the articles. On the other hand, setting up online advertisements and tracking and monitoring them demands personnel hours, especially in cases where the staff is inexperienced in online advertisements and campaigns and needs thorough training. As alluded earlier, to obtain the best out of online recruitment efforts, trained personnel specifically dedicated to plan proper recruitment strategies and mitigations are required. As Facebook and other online sites are in a constant state of flux, best practices for recruitment strategies here are hard to develop since no fit-for-all model works [5].

    Qualitative Analysis

    This review found that the majority of published studies that used online patient recruitment were recruiting a hard-to-reach population and utilizing the potential for online recruitment tools to specifically target an underrepresented population. For this finding, it is worth noting that online remedies also have the advantage of no in-person contact, as these hard-to-reach populations often are rather stigmatized (eg, drug abusers). Further, studies included in this review targeted men and women aged 18 years or above, with no studies reporting age as a critical bottleneck for online recruitment. Overall, this study found that online recruitment was favorable compared with offline recruitment. For clinical researchers, these findings could be of great value when designing future research studies.

    Shortcomings of Online Patient Recruitment

    Online methods have recruited samples with atypical demographic characteristics compared with offline methods, suggesting that the internet may reach a different population of subjects compared with samples recruited offline and hereby introduce a risk of recruitment bias. For instance, one study recruited smokers who were more likely to be nondaily smokers, exhibit high motivation to quit, and use alcohol than reported in other studies [13,67], and Bull et al [27] recruited a sample through Facebook, where the vast majority of individuals were Caucasian. On the other hand, this could introduce higher diversity in the trial, reflecting real-world demographics. Therefore, some studies argue that a combination of online and offline strategies to reach sample targets yields the most representative and unbiased samples [47]. However, during recent years, being on the internet has become the norm for almost everyone, and differences in populations may therefore be declining. Furthermore, design of advertisements is also likely an important factor, as different designs and persona character types may influence the audience signing up for a study.

    Strengths and Limitations

    The vast majority of studies included in both the review and meta-analyses were RCTs, which are considered the “gold standard” for clinical trials. Further, online recruitment can be easily monitored, and data metrics from online sites are detailed and easy to obtain. However, although online recruitment metrics are widely available, many studies included in our meta-analysis lacked recruitment data for both online and offline recruitment methods. As such, the data were not complete for all of the 23 studies included in the meta-analysis. In addition, our 95% prediction intervals should be interpreted with caution because prediction intervals have been reported to be less reliable in meta-analyses with unbalanced study sizes [68]. Although speculative, more complete data could potentially have reduced any ambiguity in the results. Finally, what we have considered for our review may not be exhaustive, as there could be underlying factors that have not been investigated. We collected a limited amount of demographic data from the articles, and only touched upon the discussion related to demographics and the two recruitment methods.

    Implications for Clinical Trials

    Inability to meet recruitment targets in clinical research is the biggest cause of trial delays. Our findings suggest that online strategies for patient recruitment for clinical trials can speed up recruitment and potentially reduce trial delays, as this paper substantiates that online recruitment is both more time-efficient and cost-effective compared with offline recruitment. However, this paper also suggests that to obtain the best possible results, both effort and money should be invested in online recruitment campaigns. To maximize the recruitment rate, online strategies should not be seen as an add-on to offline recruitment, but should be a primary recruitment strategy itself. Nevertheless, despite our recommendation, dealing with online recruitment methods requires engaging patients fast in the recruitment process and making sure that subjects who are transferred from the digital platform are contacted and scheduled instantly for a screening visit, as the online recruitment method may be inefficient if this is not happening [69].

    Unanswered Questions and Future Research

    For future trials, researchers need to be able to adapt from an offline setting to an online setting, including online remedies in clinical trials, as hybrid and fully virtual trials are already emerging [70-72]. More research on online and offline recruitment strategies and what methods within the two recruitment strategies are most effective is therefore needed.

    Conflicts of Interest

    LEO Innovation Lab, an independently established unit of LEO Pharma A/S, employs some of the authors of this paper. This does not alter our adherence to the journal’s policies.

    ‎
    Multimedia Appendix 1

    Individual characteristics of all included studies.

    PDF File (Adobe PDF File), 1627 KB
    ‎
    Multimedia Appendix 2

    Individual characteristics of studies included in the meta-analyses.

    PDF File (Adobe PDF File), 53 KB
    ‎
    Multimedia Appendix 3

    Cost per enrollee for online and offline recruitment.

    PDF File (Adobe PDF File), 47 KB
    1. Caldwell PH, Hamilton S, Tan A, Craig JC. Strategies for increasing recruitment to randomised controlled trials: systematic review. PLoS Med 2010 Nov 09;7(11):e1000368 [FREE Full text] [CrossRef] [Medline]
    2. Weiss SM. Proceedings of the National Heart and Lung Institute Working Conference on Health Behavior: Basye, Virginia, May 12-15, 1975. Bethesda, MD: National Institutes of Health; 1975.
    3. Johnson O. An evidence-based approach to conducting clinical trial feasibility assessments. Clinical Investigation 2015 May;5(5):491-499 [FREE Full text] [CrossRef]
    4. Gul RB, Ali PA. Clinical trials: the challenge of recruitment and retention of participants. J Clin Nurs 2010 Jan;19(1-2):227-233. [CrossRef] [Medline]
    5. Akers L, Gordon JS. Using Facebook for Large-Scale Online Randomized Clinical Trial Recruitment: Effective Advertising Strategies. J Med Internet Res 2018 Nov 08;20(11):e290 [FREE Full text] [CrossRef] [Medline]
    6. Shere M, Zhao XY, Koren G. The role of social media in recruiting for clinical trials in pregnancy. PLoS One 2014;9(3):e92744 [FREE Full text] [CrossRef] [Medline]
    7. Watson NL, Mull KE, Heffner JL, McClure JB, Bricker JB. Participant Recruitment and Retention in Remote eHealth Intervention Trials: Methods and Lessons Learned From a Large Randomized Controlled Trial of Two Web-Based Smoking Interventions. J Med Internet Res 2018 Aug 24;20(8):e10351 [FREE Full text] [CrossRef] [Medline]
    8. Jones R, Lacroix LJ, Porcher E. Facebook Advertising to Recruit Young, Urban Women into an HIV Prevention Clinical Trial. AIDS Behav 2017 Nov;21(11):3141-3153 [FREE Full text] [CrossRef] [Medline]
    9. Carter-Harris L, Bartlett Ellis R, Warrick A, Rawl S. Beyond Traditional Newspaper Advertisement: Leveraging Facebook-Targeted Advertisement to Recruit Long-Term Smokers for Research. J Med Internet Res 2016 Jun 15;18(6):e117 [FREE Full text] [CrossRef] [Medline]
    10. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009 Jul 21;6(7):e1000097 [FREE Full text] [CrossRef] [Medline]
    11. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986 Sep;7(3):177-188. [CrossRef] [Medline]
    12. R: A language and environment for statistical computing. ScienceOpen.   URL: https://www.scienceopen.com/document?vid=300a2dc0-3207-4383-818c-51eb0f49f561 [accessed 2020-05-22]
    13. Buller DB, Meenan R, Severson H, Halperin A, Edwards E, Magnusson B. Comparison of 4 recruiting strategies in a smoking cessation trial. Am J Health Behav 2012 Sep;36(5):577-588 [FREE Full text] [CrossRef] [Medline]
    14. Ramo DE, Rodriguez TM, Chavez K, Sommer MJ, Prochaska JJ. Facebook Recruitment of Young Adult Smokers for a Cessation Trial: Methods, Metrics, and Lessons Learned. Internet Interv 2014 Apr;1(2):58-64 [FREE Full text] [CrossRef] [Medline]
    15. Bricker J, Wyszynski C, Comstock B, Heffner JL. Pilot randomized controlled trial of web-based acceptance and commitment therapy for smoking cessation. Nicotine Tob Res 2013 Oct;15(10):1756-1764 [FREE Full text] [CrossRef] [Medline]
    16. Cobb NK, Jacobs MA, Wileyto P, Valente T, Graham AL. Diffusion of an Evidence-Based Smoking Cessation Intervention Through Facebook: A Randomized Controlled Trial. Am J Public Health 2016 Jun;106(6):1130-1135. [CrossRef] [Medline]
    17. Rait MA, Prochaska JJ, Rubinstein ML. Recruitment of adolescents for a smoking study: use of traditional strategies and social media. Transl Behav Med 2015 Sep;5(3):254-259 [FREE Full text] [CrossRef] [Medline]
    18. Muñoz RF, Aguilera A, Schueller SM, Leykin Y, Pérez-Stable EJ. From online randomized controlled trials to participant preference studies: morphing the San Francisco Stop Smoking site into a worldwide smoking cessation resource. J Med Internet Res 2012 Jun 27;14(3):e64 [FREE Full text] [CrossRef] [Medline]
    19. Gordon JS, Akers L, Severson HH, Danaher BG, Boles SM. Successful participant recruitment strategies for an online smokeless tobacco cessation program. Nicotine Tob Res 2006 Dec;8 Suppl 1:S35-S41. [CrossRef] [Medline]
    20. Brodar KE, Hall MG, Butler EN, Parada H, Stein-Seroussi A, Hanley S, et al. Recruiting Diverse Smokers: Enrollment Yields and Cost. Int J Environ Res Public Health 2016 Dec 16;13(12) [FREE Full text] [CrossRef] [Medline]
    21. Heffner JL, Wyszynski CM, Comstock B, Mercer LD, Bricker J. Overcoming recruitment challenges of web-based interventions for tobacco use: the case of web-based acceptance and commitment therapy for smoking cessation. Addict Behav 2013 Oct;38(10):2473-2476 [FREE Full text] [CrossRef] [Medline]
    22. Graham AL, Milner P, Saul JE, Pfaff L. Online advertising as a public health and recruitment tool: comparison of different media campaigns to increase demand for smoking cessation interventions. J Med Internet Res 2008 Dec 15;10(5):e50 [FREE Full text] [CrossRef] [Medline]
    23. Graham AL, Fang Y, Moreno JL, Streiff SL, Villegas J, Muñoz RF, et al. Online advertising to reach and recruit Latino smokers to an internet cessation program: impact and costs. J Med Internet Res 2012 Aug 27;14(4):e116 [FREE Full text] [CrossRef] [Medline]
    24. Byaruhanga J, Tzelepis F, Paul C, Wiggers J, Byrnes E, Lecathelinais C. Cost Per Participant Recruited From Rural and Remote Areas Into a Smoking Cessation Trial Via Online or Traditional Strategies: Observational Study. J Med Internet Res 2019 Nov 12;21(11):e14911 [FREE Full text] [CrossRef] [Medline]
    25. Witzel TC, Gabriel MM, McCabe L, Weatherburn P, Gafos M, Speakman A, et al. Pilot phase of an internet-based RCT of HIVST targeting MSM and transgender people in England and Wales: advertising strategies and acceptability of the intervention. BMC Infect Dis 2019 Aug 08;19(1):699 [FREE Full text] [CrossRef] [Medline]
    26. Bowen A, Williams M, Horvath K. Using the Internet to Recruit Rural MSM for HIV Risk Assessment: Sampling Issues. AIDS Behav 2004 Sep;8(3):311-319 [FREE Full text] [CrossRef] [Medline]
    27. Bull SS, Lloyd L, Rietmeijer C, McFarlane M. Recruitment and retention of an online sample for an HIV prevention intervention targeting men who have sex with men: the Smart Sex Quest Project. AIDS Care 2004 Nov;16(8):931-943. [CrossRef] [Medline]
    28. Hernandez-Romieu AC, Sullivan PS, Sanchez TH, Kelley CF, Peterson JL, Del Rio C, et al. The comparability of men who have sex with men recruited from venue-time-space sampling and facebook: a cohort study. JMIR Res Protoc 2014 Jul 17;3(3):e37 [FREE Full text] [CrossRef] [Medline]
    29. Buckingham L, Becher J, Voytek CD, Fiore D, Dunbar D, Davis-Vogel A, et al. Going social: Success in online recruitment of men who have sex with men for prevention HIV vaccine research. Vaccine 2017 Jun 14;35(27):3498-3505 [FREE Full text] [CrossRef] [Medline]
    30. Raymond HF, Rebchook G, Curotto A, Vaudrey J, Amsden M, Levine D, et al. Comparing internet-based and venue-based methods to sample MSM in the San Francisco Bay Area. AIDS Behav 2010 Feb;14(1):218-224. [CrossRef] [Medline]
    31. Parsons JT, Vial AC, Starks TJ, Golub SA. Recruiting drug using men who have sex with men in behavioral intervention trials: a comparison of internet and field-based strategies. AIDS Behav 2013 Feb;17(2):688-699 [FREE Full text] [CrossRef] [Medline]
    32. Yuan P, Bare MG, Johnson MO, Saberi P. Using online social media for recruitment of human immunodeficiency virus-positive participants: a cross-sectional survey. J Med Internet Res 2014 May 01;16(5):e117 [FREE Full text] [CrossRef] [Medline]
    33. Prescott TL, Phillips Ii G, DuBois LZ, Bull SS, Mustanski B, Ybarra ML. Reaching Adolescent Gay, Bisexual, and Queer Men Online: Development and Refinement of a National Recruitment Strategy. J Med Internet Res 2016 Aug 04;18(8):e200 [FREE Full text] [CrossRef] [Medline]
    34. Christensen T, Riis AH, Hatch EE, Wise LA, Nielsen MG, Rothman KJ, et al. Costs and Efficiency of Online and Offline Recruitment Methods: A Web-Based Cohort Study. J Med Internet Res 2017 Mar 01;19(3):e58 [FREE Full text] [CrossRef] [Medline]
    35. Maloni JA, Przeworski A, Damato EG. Web recruitment and internet use and preferences reported by women with postpartum depression after pregnancy complications. Arch Psychiatr Nurs 2013 Apr;27(2):90-95. [CrossRef] [Medline]
    36. Adam LM, Manca DP, Bell RC. Can Facebook Be Used for Research? Experiences Using Facebook to Recruit Pregnant Women for a Randomized Controlled Trial. J Med Internet Res 2016 Sep 21;18(9):e250 [FREE Full text] [CrossRef] [Medline]
    37. Ingersoll K, Frederick C, MacDonnell K, Ritterband L, Lord H, Jones B, et al. A Pilot RCT of an Internet Intervention to Reduce the Risk of Alcohol-Exposed Pregnancy. Alcohol Clin Exp Res 2018 Jun;42(6):1132-1144 [FREE Full text] [CrossRef] [Medline]
    38. MacDonnell K, Cowen E, Cunningham DJ, Ritterband L, Ingersoll K. Online recruitment of a non-help-seeking sample for an internet intervention: Lessons learned in an alcohol-exposed pregnancy risk reduction study. Internet Interv 2019 Sep;17:100240 [FREE Full text] [CrossRef] [Medline]
    39. Guthrie KA, Caan B, Diem S, Ensrud KE, Greaves SR, Larson JC, et al. Facebook advertising for recruitment of midlife women with bothersome vaginal symptoms: A pilot study. Clin Trials 2019 Oct 06;16(5):476-480. [CrossRef] [Medline]
    40. Heckman CJ, Handorf EA, Darlow SD, Ritterband LM, Manne SL. An online skin cancer risk-reduction intervention for young adults: Mechanisms of effects. Health Psychol 2017 Mar;36(3):215-225 [FREE Full text] [CrossRef] [Medline]
    41. van Lettow B, de Vries H, Burdorf A, Boon B, van Empelen P. Drinker prototype alteration and cue reminders as strategies in a tailored web-based intervention reducing adults' alcohol consumption: randomized controlled trial. J Med Internet Res 2015 Feb 04;17(2):e35 [FREE Full text] [CrossRef] [Medline]
    42. Irvine AB, Gelatt VA, Seeley JR, Macfarlane P, Gau JM. Web-based intervention to promote physical activity by sedentary older adults: randomized controlled trial. J Med Internet Res 2013 Feb 05;15(2):e19 [FREE Full text] [CrossRef] [Medline]
    43. Kuhn E, Kanuri N, Hoffman JE, Garvert DW, Ruzek JI, Taylor CB. A randomized controlled trial of a smartphone app for posttraumatic stress disorder symptoms. J Consult Clin Psychol 2017 Mar;85(3):267-273. [CrossRef] [Medline]
    44. Jonas B, Leuschner F, Tossmann P. Efficacy of an internet-based intervention for burnout: a randomized controlled trial in the German working population. Anxiety Stress Coping 2017 Mar;30(2):133-144. [CrossRef] [Medline]
    45. Juraschek SP, Plante TB, Charleston J, Miller ER, Yeh H, Appel LJ, et al. Use of online recruitment strategies in a randomized trial of cancer survivors. Clin Trials 2018 Apr;15(2):130-138 [FREE Full text] [CrossRef] [Medline]
    46. Bracken K, Hague W, Keech A, Conway A, Handelsman DJ, Grossmann M, et al. Recruitment of men to a multi-centre diabetes prevention trial: an evaluation of traditional and online promotional strategies. Trials 2019 Jun 19;20(1):366 [FREE Full text] [CrossRef] [Medline]
    47. Frandsen M, Thow M, Ferguson SG. The Effectiveness Of Social Media (Facebook) Compared With More Traditional Advertising Methods for Recruiting Eligible Participants To Health Research Studies: A Randomized, Controlled Clinical Trial. JMIR Res Protoc 2016 Aug 10;5(3):e161 [FREE Full text] [CrossRef] [Medline]
    48. Cowie JM, Gurney ME. The Use of Facebook Advertising to Recruit Healthy Elderly People for a Clinical Trial: Baseline Metrics. JMIR Res Protoc 2018 Jan 24;7(1):e20 [FREE Full text] [CrossRef] [Medline]
    49. Schaub M, Sullivan R, Haug S, Stark L. Web-based cognitive behavioral self-help intervention to reduce cocaine consumption in problematic cocaine users: randomized controlled trial. J Med Internet Res 2012 Nov 28;14(6):e166 [FREE Full text] [CrossRef] [Medline]
    50. Schwinn T, Hopkins J, Schinke SP, Liu X. Using Facebook ads with traditional paper mailings to recruit adolescent girls for a clinical trial. Addict Behav 2017 Feb;65:207-213 [FREE Full text] [CrossRef] [Medline]
    51. Raviotta JM, Nowalk MP, Lin CJ, Huang H, Zimmerman RK. Using Facebook™ to Recruit College-Age Men for a Human Papillomavirus Vaccine Trial. Am J Mens Health 2016 Mar 11;10(2):110-119. [CrossRef] [Medline]
    52. Bull SS, Vallejos D, Levine D, Ortiz C. Improving recruitment and retention for an online randomized controlled trial: experience from the Youthnet study. AIDS Care 2008 Sep;20(8):887-893. [CrossRef] [Medline]
    53. Das R, Machalek DA, Molesworth EG, Garland SM. Using Facebook to Recruit Young Australian Men Into a Cross-Sectional Human Papillomavirus Study. J Med Internet Res 2017 Nov 17;19(11):e389 [FREE Full text] [CrossRef] [Medline]
    54. Wozney L, Turner K, Rose-Davis B, McGrath PJ. Facebook ads to the rescue? Recruiting a hard to reach population into an Internet-based behavioral health intervention trial. Internet Interv 2019 Sep;17:100246 [FREE Full text] [CrossRef] [Medline]
    55. Carmi L, Zohar J. A comparison between print vs. internet methods for a clinical trial recruitment--a pan European OCD study. Eur Neuropsychopharmacol 2014 Jun;24(6):874-878. [CrossRef] [Medline]
    56. Bailey JV, Pavlou M, Copas A, McCarthy O, Carswell K, Rait G, et al. The Sexunzipped trial: optimizing the design of online randomized controlled trials. J Med Internet Res 2013 Dec 11;15(12):e278 [FREE Full text] [CrossRef] [Medline]
    57. Williams MT, Proetto D, Casiano D, Franklin ME. Recruitment of a hidden population: African Americans with obsessive-compulsive disorder. Contemp Clin Trials 2012 Jan;33(1):67-75 [FREE Full text] [CrossRef] [Medline]
    58. Clarke J, Sanatkar S, Baldwin PA, Fletcher S, Gunn J, Wilhelm K, et al. A Web-Based Cognitive Behavior Therapy Intervention to Improve Social and Occupational Functioning in Adults With Type 2 Diabetes (The SpringboarD Trial): Randomized Controlled Trial. J Med Internet Res 2019 May 21;21(5):e12246 [FREE Full text] [CrossRef] [Medline]
    59. Pedersen ER, Helmuth ED, Marshall GN, Schell TL, PunKay M, Kurz J. Using facebook to recruit young adult veterans: online mental health research. JMIR Res Protoc 2015 Jun 01;4(2):e63 [FREE Full text] [CrossRef] [Medline]
    60. Morgan AJ, Jorm AF, Mackinnon AJ. Internet-based recruitment to a depression prevention intervention: lessons from the Mood Memos study. J Med Internet Res 2013 Feb 12;15(2):e31 [FREE Full text] [CrossRef] [Medline]
    61. Pratap A, Renn BN, Volponi J, Mooney SD, Gazzaley A, Arean PA, et al. Using Mobile Apps to Assess and Treat Depression in Hispanic and Latino Populations: Fully Remote Randomized Clinical Trial. J Med Internet Res 2018 Aug 09;20(8):e10130 [FREE Full text] [CrossRef] [Medline]
    62. Partridge SR, Balestracci K, Wong AT, Hebden L, McGeechan K, Denney-Wilson E, et al. Effective Strategies to Recruit Young Adults Into the TXT2BFiT mHealth Randomized Controlled Trial for Weight Gain Prevention. JMIR Res Protoc 2015 Jun 05;4(2):e66 [FREE Full text] [CrossRef] [Medline]
    63. Moreno MA, Waite A, Pumper M, Colburn T, Holm M, Mendoza J. Recruiting Adolescent Research Participants: In-Person Compared to Social Media Approaches. Cyberpsychol Behav Soc Netw 2017 Jan;20(1):64-67. [CrossRef] [Medline]
    64. Anguera JA, Jordan JT, Castaneda D, Gazzaley A, Areán PA. Conducting a fully mobile and randomised clinical trial for depression: access, engagement and expense. BMJ Innov 2016 Jan;2(1):14-21 [FREE Full text] [CrossRef] [Medline]
    65. Thornton L, Batterham PJ, Fassnacht DB, Kay-Lambkin F, Calear AL, Hunt S. Recruiting for health, medical or psychosocial research using Facebook: Systematic review. Internet Interv 2016 May;4:72-81 [FREE Full text] [CrossRef] [Medline]
    66. Choi I, Milne DN, Glozier N, Peters D, Harvey SB, Calvo RA. Using different Facebook advertisements to recruit men for an online mental health study: Engagement and selection bias. Internet Interv 2017 Jun;8:27-34 [FREE Full text] [CrossRef] [Medline]
    67. Ramo DE, Hall SM, Prochaska JJ. Reaching young adult smokers through the internet: comparison of three recruitment mechanisms. Nicotine Tob Res 2010 Jul;12(7):768-775 [FREE Full text] [CrossRef] [Medline]
    68. Partlett C, Riley RD. Random effects meta-analysis: Coverage performance of 95% confidence and prediction intervals following REML estimation. Stat Med 2017 Jan 30;36(2):301-317 [FREE Full text] [CrossRef] [Medline]
    69. Katz B, Eiken A, Misev V, Zibert J. Optimize clinical trial recruitment with digital platforms. Dermatology Times. 2019.   URL: https://www.dermatologytimes.com/view/optimize-clinical-trial-recruitment-digital-platforms [accessed 2020-02-09]
    70. Clinical Trial in Subjects With Mild to Moderate Acne Vulgaris. ClinicalTrials.gov.   URL: https://clinicaltrials.gov/ct2/show/NCT02832063 [accessed 2018-06-27]
    71. A Study to Evaluate the Efficacy and Safety of Rituximab Versus Mycophenolate Mofetil (MMF) in Participants With Pemphigus Vulgaris (PV). ClinicalTrials.gov.   URL: https://clinicaltrials.gov/ct2/show/NCT02383589 [accessed 2019-10-25]
    72. Clinical Trial of Patidegib Gel 2%, 4%, and Vehicle Applied Once or Twice Daily to Decrease the GLI1 Biomarker in Sporadic Nodular Basal Cell Carcinomas (BCC). ClinicalTrials.gov.   URL: https://clinicaltrials.gov/ct2/show/NCT02828111 [accessed 2019-10-30]

    ‎
    IR: incidence rate
    IRR: incidence rate ratio
    RCT: randomized controlled trial
    RR: risk ratio

    Edited by G Eysenbach; submitted 07.07.20; peer-reviewed by A Godinho, D Hansen; comments to author 12.08.20; revised version received 19.08.20; accepted 15.09.20; published 04.11.20

    Copyright

    ©Mette Brøgger-Mikkelsen, Zarqa Ali, John R Zibert, Anders Daniel Andersen, Simon Francis Thomsen. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 04.11.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.