Evaluation of Different Recruitment Methods: Longitudinal, Web-Based, Pan-European Physical Activity Through Sustainable Transport Approaches (PASTA) Project

Introduction

Recruiting participants has become increasingly challenging in the face of privacy concerns, an increasing number of surveys, expectations of rewards for survey completion, and necessary effort to achieve an unbiased representative sample [1]. Sufficient sample sizes and minimal sample bias are core requirements for empirical data collection in order to properly obtain answers to research questions [2]. Meeting these requirements is especially challenging for surveys with high response burdens such as longitudinal studies [3]. Traditional recruitment methods, such as mailed invitations based on samples drawn from population registries, are costly and increasingly yield sample biases due to declining response rates and selectivity effects [4-7], for example, because of an increase in the exclusive use of mobile phones or email rather than traditional mail or landline phone. Opportunistic approaches, such as recruitment through social media, promise cost savings [4,7-12] and a better coverage of person groups that are hard to reach with the traditional recruitment methods [13], like parents of adolescents [8], adolescents themselves [10], people with special conditions [14,15], smokers [16], low-income people [17], or people with a disproportionate risk for poor health outcomes [18]. Social media, like Facebook and Twitter, can potentially have a strong snowballing effect [19], given their intensive use and continuing growth (around 1.94 billion monthly active Facebook users worldwide [20]) and their features that allow information to be shared very easily among networks [19]; therefore, they are able to reach a large number of people in a very short time. Combining opportunistic recruitment with a Web-based survey and data-collection platform comprises additional benefits such as real-time monitoring of recruitment progress and enabling ongoing optimization of recruitment activities [4]. Poor response rates can further be improved by including rewards for participation such as financial compensations [21]. One major drawback of the opportunistic recruitment methods is the concern of sample bias, as the population sampled does not necessarily represent each group of the total population equally well [10]. Specific types of social media might, for example, be preferably used by younger people [3,7,11] and not by the elderly. In this case, the elderly have little chance to be sampled at all. Other, more traditional methods, such as flyers, involve problems of respondents not having a direct and convenient access to the survey and may result in a smaller recruitment rate [22].

The PASTA (Physical Activity through Sustainable Transport Approaches) Project [23,24] used a combination of different opportunistic recruitment methods to utilize strengths and minimize weaknesses. The project study collected data in a longitudinal, Web-based survey with a cohort design to study the effects of active mobility (like walking and cycling) on the overall physical activity and health, crash risks, and exposure to traffic-related air pollution. Data collection was performed in seven European cities: Antwerp, Barcelona, London, Örebro, Rome, Vienna, and Zurich. The target population for the project survey was the entire adult population in each of the seven case study cities, with the aim of oversampling participants who use a bicycle for their daily trips.

The objective of this paper was to report the success of these different methods with regard to obtaining participants’ characteristics, participation behavior, recruitment rates, and representativeness of the sample. More specifically, we aimed to (1) describe participant characteristics in the seven European cities in terms of the number of recruited people, gender, age, education level, and employment status; (2) show how participants found out about the survey; (3) illustrate participation behavior by reporting on the number of filled-in questionnaires, attrition, and withdrawal rate; (4) present the effectiveness (the number of predicted participants) and time efficiency of different recruitment approaches; and (5) compare our sample with the general population. Finally, we present our conclusions for the use of recruitment studies in future research on comparable topics.

Methods

Approach

The PASTA project used a longitudinal, Web-based survey between November 2014 and December 2016. After the first questionnaire, which collected baseline information, participants received a follow-up questionnaire every 13 days to collect prospective data on travel behavior, levels of physical activity, and traffic safety incidents [23]. During this period, participants in the seven European cities were recruited on a rolling basis by using different opportunistic approaches. To reach a sufficient number of adult participants, a standardized guide on recruitment strategy was developed for all cities. A common recruitment strategy was important to achieve a similar number of participants and ensure more evenly distributed participation across transport modes and social groups in each city. The strategy for all cities included press releases and editorials; common promotional materials following the same visual identity guidelines; direct targeting of local stakeholders and community groups to distribute survey information through their communication channels (like newsletters, intranet, and webpages); extensive use of social media (each city had its own Facebook and Twitter pages where the link to the survey was regularly posted; Figure 1); and incentivizing for participation (ie, participants entered into a prize draw if they completed a questionnaire). The chance of participants winning increased with each additional completed questionnaire, except for participants in Sweden (Örebro) where the lottery was not allowed by law.

A professional designer created a visual identity to enable clear and easy recognition of the project. All recruitment materials produced under the project were designed in line with the visual identity, which resulted in the creation of a strong brand (Figure 2). The survey as well as different dissemination materials (eg, flyers) were developed first in English and then translated into the local languages (ie, Dutch, Spanish, Catalan, Swedish, Italian, Swizz German, and Austrian German). This guaranteed that the same recruitment materials with the same contents were used in each city. Within this framework, there was flexibility to enable local initiatives and targeted city-specific recruitment, such as promoting recruitment for the project at social and cultural events. Furthermore, the city of Örebro applied an additional random sampling approach by contacting people aged 18-74 years through mail or phone.

To ensure high-data quality, several measures were put in place to reduce attrition rates, such as a user-friendly and custom-made survey platform and the automatic sending of reminder emails. Furthermore, participants were able to log into the platform at any time. There, they received an overview of their personal completed and open questionnaires and were able to complete unfinished questionnaires. Furthermore, they were given the opportunity to withdraw themselves actively from the survey if they did not want to participate any longer. In addition to the participants’ user interface, the platform also featured a researchers’ user interface and dashboard for real-time monitoring of recruitment and survey data collection. A user engagement strategy was also developed, including regular contact with the respondents, project branding, regular posting on social media, and keeping the project website up-to-date.

Participants were asked in the baseline questionnaire how they found out about the survey. They were given a choice between several different options, ranging from word of mouth to large-scale advertising campaigns. This question was answered by all participants who finalized the baseline questionnaire. At the same time, all city partners kept records on their local recruitment activities to measure invested efforts, including date, category, description, and invested time for each applied recruitment activity. Different categories were classified as follows: collaboration with local administration or organization (eg, survey link on webpages, newsletters, and intranet); handout of flyers at specific locations or specific events; display of posters at specific locations; use of mailing lists; advertisement in Web-based media or print media; papers in Web-based media, print media, or magazines; oral presentations for recruitment purposes; radio or television spots; Facebook; Twitter; street recruitment; and use of random sampling.

Ethics approval was obtained from the local ethics committees in the countries where the work was conducted and sent to the European Commission before the start of the survey.

Statistical Analysis

Standard descriptive statistics outlined overall participant characteristics and were stratified by city, gender, age, education level, and employment status. To assess participation behavior, we tested the number of filled-in questionnaires by different sociodemographic characteristics using the nonparametric Kruskal-Wallis rank-sum test. Each significant result (P<.05) was followed by a Dunn test to account for significant differences within a variable. The sociodemographic characteristics of the sample (age and gender) were compared with each city’s census data by applying the Pearson chi-square test and size effect calculations.

‎View this figure

‎View this figure

To estimate the effectiveness (number of participants who started the baseline survey) of different recruitment approaches, we developed a recruit-prediction model in the form of a nonlinear least squares model in cities that provided the most comprehensive and detailed information on their local recruitment activities (Antwerp, Barcelona, and Vienna). The prediction model is based on the assumption that each recruitment activity generates an effect on the number of participants, which resembles a density function of a log-normal distribution, that is, a steep increase and a flat decrease:

Where r_i denotes the predicted number of responses on day i; the index c=1, 2... C refers to different categories of recruitment activities (such as Facebook, Twitter, and flyer); the index a=1, 2... A refers to particular activities of category C; e_c is the intrinsic effectiveness of an activity of category c; i_ca is the intensity of a particular activity a of category c; d is the number of days elapsed since the start of the activity; μ and σ are the location and dispersion parameter of the log-normal density function, respectively. The curve characteristics of the density function were assumed to be the same for all recruitment categories: Only one set of parameters μ and σ was estimated for all categories. The intensity was assumed to vary as follows: each category has its intrinsic (baseline) effectiveness e_c; within a given category, the intensity varies according to the invested effort i_ca (indicated by the reported number of working hours); and in some cases, with strong peaks of recruited people, which could unambiguously be assigned to particular recruitment activities with exceptional success, the intensity parameter was manually increased to capture the success of this activity adequately.

To estimate the time efficiency of different recruitment categories, we divided the number of predicted participants by the number of invested hours for each recruitment category. This was possible because the hours of work for project members to apply each activity (except for Facebook and Twitter posts) was recorded in detail. For social media activities, an average time value of 5 minutes per post was used. All statistical analyses were performed using R (version 3.4.0; The R Foundation for Statistical Computing, Vienna, Austria). Values given throughout the text are mean (SE).

Results

Sociodemographic Characteristics of Participants

A total of 10691 participants were recruited over a period of 27 months in the seven European cities, ranging from 1844 (17.25%) individuals in Rome to 1356 (12.68%) in Zurich. In all cities, except Rome, more women than men were recruited, with an average age of 41.9 (SE 0.19) years for men and 40.0 (SE 0.17) years for women. Most participants who stated their educational level were highly educated, with 6180 participants (of 8525, 72.49%) possessing a university degree and 2217 participants (of 8525, 26.01%) possessing a secondary education. In addition, 5993 participants (of 9892, 60.58%) were full-time employees, followed by 1662 (of 9892, 16.80%) part-time employees and 1361 (of 9892 13.76%) students; furthermore, 876 (of 9892, 8.86%) participants had home duties, were retired, or were unemployed (Table 1). Regarding nationality, 3560 participants (of 8586, 41.46%) were from Western Europe (predominantly in Antwerp, Vienna, and Zurich), 2840 (of 8586, 33.08%) were from Southern Europe (Barcelona and Rome), and 1858 (of 8586, 21.64%) were from Northern Europe (Örebro and London; according to the Geographic Regions listed by the United Nations).

How Participants Found Out About the Survey

Table 2 shows that for all 10,691 participants, the three main sources of finding out about the survey were workplaces or employers (2300, 21.51%), outreach promotion (2219, 20.76%) such as presenting the project at different events or street recruitment, and social media (1859, 17.39%). The results varied across cities. In Antwerp, London, and Zurich, the highest share of respondents were reached through workplaces, whereas respondents in Rome were primarily reached through social media announcements and those in Barcelona, Örebro, and Vienna were reached through outreach promotion. In terms of gender, men were more likely to be recruited through outreach promotion (976/4922, 19.83%) or social media (942/4922, 19.14%), while women were most likely to be recruited through their workplace (1399/5769, 24.25%). Although participants aged 30-60 years could be best reached through their workplace, those aged 20-29 and >60 years were reached most often through outreach activities. In addition, students and participants without employment could best be reached through outreach activities. In case of respondents without a school-leaving qualification, recruitment through workplace or outreach activities was also the most successful approach.

Participation Rates and Behavior

A total of 12,825 people registered for the survey; however, 2134 never started the baseline questionnaire (attrition rate 16.64%). From the remaining 10,691 participants who started the baseline questionnaire, 8567 finalized it (additional attrition rate 19.87%). The attrition rates between people who registered, started, and finalized the baseline questionnaire varied across cities, with the lowest rates in Antwerp and the highest rates in Örebro and London (Figure 3).

The number of filled-in questionnaires per participant varied significantly across the cities (P<.001), with the highest number in Zurich (11.0, SE 0.33) and the lowest in Örebro (4.8, SE 0.17). In almost all cities, women filled in fewer questionnaires than men (7.7 [SE 0.1] vs 8.6 [SE 0.2]); furthermore, younger people or students tended to fill in fewer questionnaires than people aged 30-80 years or employees and people with home duties (P<.001). In addition, the way people were informed about the survey had a significant impact on the number of filled-in questionnaires (P<.001). Most questionnaires were filled in when people found out about the survey through other organizations (9.6, SE 0.3) or the news (9.5, SE 0.4; Table 3). In total, 12.17% (1301/10,691) participants withdrew from the survey, with the highest share in Örebro (311/1401, 22.20%) and the lowest share in Rome (82/1844, 4.45%).

‎View this figure

Effectiveness and Efficiency of Different Recruitment Approaches

Table 4 gives an overview of only effective recruitment categories, that is, only activities that were able to recruit participants according to the model. The categories in different cities were as follows: (1) Antwerp: Facebook, mailing lists, collaboration with local administrations and organization, use of flyers and posters, radio spots, and Web-based advertisement; (2) Barcelona: Facebook, mailing lists, street recruitment, and print media; and (3) Vienna: Facebook, Web-based media, mailing lists, collaboration with local administrations and organizations, use of flyers and posters, and street recruitment. One of the most effective approaches in all 3 cities was Facebook, with >400 predicted participants in Antwerp and Barcelona and >500 predicted participants in Vienna (P<.001;Table 4). In Antwerp, most people could be reached through different mailing lists (>900 participants) and in Barcelona, through a range of street recruitment activities (>1000 participants). In Vienna, especially, collaborations with local organizations (like the local bike sharing provider) were very effective in reaching a high share of predicted participants (eg, peak in Figure 4, the bottom graph). Considering the invested working hours, one of the most time-efficient categories in all 3 cities was Facebook, with >30 participants per invested working hour. Although mailing lists or Web-based advertisements were also effective in Antwerp, reaching approximately 60 participants per invested working hour, only 1 participant could be reached per working hour in Barcelona through street recruitment. In Figure 4, lines represent the sum of different recruitment activities. The nonlinear model does not provide an R², because R² is not defined for nonlinear models. We defined a pseudo R² value, because the model is reasonably close to a linear model and the sample size is sufficiently large.

Representativeness of the Sample

Compared with the cities’ census data, study participants in all cities, except Rome, were broadly representative in terms of gender distribution (P>.05). This was mainly the case if participants were informed about the survey by news, through word of mouth (friends, neighbors, or relatives) or social media (Multimedia Appendix 1). The main difference was that our recruited sample was on an average younger than the general population (high deviation from census data within the age class >60 years; Multimedia Appendix 2).

‎View this figure

Discussion

The main finding of this study was that Facebook was one of the most effective approaches in reaching a high share of participants. Considering the invested working hours, it was also one of the most time-efficient recruitment methods. Recruited participants were representative of the population in terms of gender but younger than the general population.

Principal Results and Participation Behavior

The main source of information about the survey was workplaces or employers. Collaborations with different organizations who forwarded the survey information to their employees (eg, through their intranet or regular newsletters) was fundamental for raising awareness to drive recruitment. Outreach promotion by project members (eg, direct face-to-face recruitment at different events or on public places) and the use of social media channels (Facebook and Twitter) were the next most informative activities. This was especially the case in Rome, where social media was based on an account of the city council with a lot of followers.

One-third of people who registered for the survey did not complete the baseline questionnaire, with the highest attrition rate in Örebro and the lowest rate in Antwerp. In addition, Örebro had the lowest number of filled-in questionnaires per participant and the highest share of people who actively deregistered from the survey. One explanation may be the different approach adopted in Örebro, which also recruited participants through random sampling. The findings suggest that people who are more interested in the topic (in this case, active mobility research) are more willing to participate in a survey and more likely to stay in the study than those who are selected randomly and may not have a strong connection to the research topic [4-7]. Therefore, a (costly) random selection may eventually still lead to a biased sample that was to be avoid. However, poor response rates can also be improved by including rewards for participation [21]. The high attrition and deregistration rate in Örebro may also be (partly) caused by the fact that participation in Örebro was not rewarded.

Comparative Effectiveness and Representativeness of Recruitment Methods

Although direct face-to-face contacts (eg, street recruitment) were very effective in terms of the number of recruited participants, recruiting people through social media (mainly Facebook) was not only effective but also very time efficient. Similar results were observed by others [4,10,11], who applied Web-based sampling in their research. By using the same medium (ie, internet), such sampling reduces the burden of participants because they can easily reach the survey by only clicking on a provided link. Regarding the near ubiquity of the internet, it has become easier for people to engage in surveys [25], as it can overcome barriers such as physical distance, transportation, and limited time [14]. Nevertheless, the effectiveness and time efficiency must be balanced with how representative the resulting sample is of the target population. Compared with the general population, study participants in almost all cities were broadly representative in terms of gender distribution. In particular, reaching people through news, word of mouth, or social media were the most successful options in recruiting a gender-balanced sample that represented a city’s population. There was, however, an age bias among the applied strategies compared with the city’s census data. Although the cities applied different strategies to include older people (eg, by visiting seniors who recently completed computer courses), most strategies attracted a higher proportion of younger people. Several studies, however, report successfully recruiting people who reflect the demographic spread of the general population using opportunistic sampling approaches [9,11,19,26]. People in our sample were further highly educated (6180/8525, 72.49% participants possessed a university degree), which is a common occurrence for survey research [9,27,28]. Nevertheless, this study found that targeting people without a school-leaving qualification through their workplaces or outreach activities was most promising; as such, some recruitment activities are better suited than others to attract hard-to-reach groups.

Limitations and Strengths

Although this study represents a comprehensive examination of different recruitment approaches in a longitudinal, Europe-wide, Web-based survey, there were some limitations that could not be addressed by the research design. First, the sociodemographic characteristics of participants recruited by the discussed methods may be different for topics other than active mobility and the corresponding health aspects. The study population in our sample was highly educated and younger than the general population. This may be because our recruitment partly addressed cyclists, and the subject may hold particular interest to those with higher education. Second, participants needed to have access to the internet to participate in the Web-based survey, which could also explain the high proportion of young participants in this study. Finally, the sample was limited to the adult population. The strategies used may be effective in recruiting children or adolescents.

These limitations are offset by several strengths. First, we were able to recruit one of the largest longitudinal samples in different European cities with a common recruitment strategy. Second, we were able to shed new light on the effectiveness and time efficiency of different recruitment approaches. We now have a large and very detailed database on response behavior as per the recruitment method for seven different cities in Europe. We observed that offering a mixed recruitment approach was very effective in reaching a high participation rate. The resulting database can answer research questions and analyze the effects of active mobility on people’s health, crash risks, and exposure to traffic-related air pollution (eg, for cyclists) because of its size and composition. Thus, overall, the use of a mixed-methods approach has been successful.