Allowing Physicians to Choose the Value of Compensation for Participation in a Web-Based Survey: Randomized Controlled Trial

Introduction

Understanding the opinions and practices of health care providers is essential for clinical research [1]. However, surveys of health care providers are plagued by declining response rates [2-4], and techniques for increasing response rates in the general public [5] frequently fail to motivate physicians [6]. Although low response rates do not necessarily bias results [7,8], they do increase the potential for nonresponse bias, and hamper publication [2,9-11].

While physician response rates are declining, inexpensive tools for conducting sophisticated Web-based electronic surveys are flourishing. Although physician response rates to electronic surveys have generally been lower than to postal surveys, many trials comparing postal versus electronic surveys were conducted a decade ago and targeted community-based physicians in regions where high-speed Internet access was unreliable or nonexistent [12-15]. As high-speed Internet access becomes ubiquitous and health care providers become more comfortable with Web-based technologies, electronic surveys have the potential to provide researchers with unique tools, including instant compensation for participation and data about how physicians interact with surveys.

A major challenge when designing a survey is determining the appropriate level of financial compensation required to incentivize participation [16-18]. Allowing respondents to choose how much they wish to be compensated provides insight into participant motivation and may maximize the cost-effectiveness of incentives by not spending funds on participants who do not require a large incentive to respond. An additional challenge posed by electronic surveys is the inability to provide prepayment. Prepayment in postal surveys, traditionally achieved by including cash in the survey envelope, is associated with significantly greater response rates among surveys of physicians versus providing payment contingent on survey completion [16,19,20].

We combined three techniques to address these challenges. First, we invited physicians to select their preferred level of instant compensation, up to US $50, for completing a short, electronic survey. Second, we attempted to engender altruism by reminding physicians that the study was funded by a limited student budget. Finally, compensation was only promised to the first 100 respondents, making it a scarce, time-limited incentive. To assess whether these three combined techniques affected response rate, time to response, survey completion rate, and time spent completing the electronic survey, we designed a randomized controlled trial of respondent-selected compensation.

Methods

Study Design

A previously described database of academic intensivists was used to recruit faculty from US hospitals with training programs accredited by the Accreditation Council for Graduate Medical Education in Internal Medicine-Critical Care Medicine, Anesthesiology-Critical Care Medicine, and Surgical Critical Care [21]. The database was updated in 2012 to include demographic and electronic contact information for 2482 physicians. Physicians were excluded from randomization if they (1) lacked electronic contact information (285/2482, 11.48%), (2) had been invited to participate in a pilot study (268/2482, 10.80%), (3) had made a previous request not to be contacted (76/2482, 3.06%), or (4) contributed to study design or survey development (3/2482, 0.12%). The remaining 1850 intensivists were potentially eligible to participate in a randomized trial of an intervention to increase communication about life support with families of critically ill patients [22], administered using the Qualtrics Web-based survey platform [23]. The Institutional Review Board (IRB) of Johns Hopkins School of Medicine approved the study. Intensivists were notified that survey completion served as consent to participate in the trial.

Randomization was blocked on intensivist sex, specialty—medicine, anesthesiology, or surgery—years since completing residency, and geographic region of residency [24,25]. Within each block, 45% of eligible intensivists were randomly assigned to the group with the ability to select their preferred level of compensation as an incentive to participate.

On November 20, 2012, each randomized intensivist was sent an invitation by email containing a unique link to the survey. All invitations included the survey topic, number of questions, expected time required to complete the survey (5 minutes), IRB approval, study confidentiality, number of follow-up/reminder emails for nonresponders, planned date for study closing (December 20, 2012), and names and affiliations of study investigators. Invitations for intensivists randomized to receive an incentive to participate also included the following text:

In appreciation for your participation, the first 100 respondents to complete the survey will be offered an Amazon.com gift code at the end of the survey. The code can be redeemed immediately for any amount up to $50. In selecting the compensation amount, please consider that this is a PhD thesis project being funded by a limited student budget.

Reminder emails were sent to all intensivists who had not completed the survey on days 13, 22, and 28 following the initial invitation. In each of the reminder emails, intensivists randomized to the incentive group were informed that funds for incentives were still available. Because not all respondents chose to take the full amount available, there were sufficient funds to offer the incentive to more than 100 respondents. To establish survey eligibility, participants were first asked if they had treated patients in the intensive care unit (ICU) setting during the previous 2 years. Those who had were then asked one question about practice history followed by 10 screens, each containing a brief clinical scenario for review.

Participants randomized to the incentive intervention who completed the survey had the option of entering the amount they wished to spend at Amazon.com [26] up to US $50 using the Amazon Gift Codes On Demand service, which allows study participants to claim incentives instantly. Among those participants who elected to receive one, incentives were claimed on the Amazon.com website. The value of created incentives was instantly deducted from a study fund containing US $5000. Study investigators could not access information on goods purchased by participants or the timing of purchases made using gift codes.

Statistical Analysis

The primary outcome measure was the difference in survey response rate between trial arms. Response rates were calculated in accordance with response rate three (RR3), defined by the American Association for Public Opinion Research guidelines for Internet surveys of specifically named persons [27] as follows:

RR3 = I/(I + P + eC[UH + UO] + eI[UH + UO])

The RR3 is equal to the number of eligible participants who responded to all survey questions (I) divided by the sum of eligible participants who responded to all survey questions (I), eligible participants who answered the eligibility screening question but did not answer all survey questions (P), and the estimated proportion in the control arm (eC) and intervention arm (eI) of nonresponders (UH) and responders with unknown eligibility (UO) who were eligible. The proportions of eligible nonresponders and responders with unknown eligibility were estimated based on the proportion of responders in each arm of the trial who answered the screening question and were known to be eligible.

Secondary outcome measures were defined as follows: survey completion rate (ie, number of intensivists known to be eligible who answered all survey questions divided by the number of eligible intensivists who clicked the link to the Web-based survey), time to response (ie, time of survey completion minus the time the initial email was sent among intensivists who completed the survey), and time spent completing survey (ie, time of survey completion minus time that the link in the invitation email was clicked by eligible intensivists).

Analyses were performed using the R programming language version 3.0.1 (Vienna, Austria) [28] using two-sided significance tests, with P<.05 indicating statistical significance, and data were analyzed on an intention-to-treat basis. Hypothesis tests for differences in proportion were performed using Pearson’s chi-square test. Fisher’s exact test was used when a cell within a contingency table contained fewer than 10 observations. Confidence intervals for differences in proportions were calculated using the Wald interval. Differences in the distribution of continuous variables were assessed using the Wilcoxon-Mann-Whitney test [29].

Results

Overview

The overall response rate was 35.90% (630/1755), with 92.0% (630/685) of eligible respondents answering all questions in the survey (see Figure 1 and Table 1). A total of 13 out of 991 (1.3%) respondents in the control arm and 22 out of 859 (2.6%) respondents in the incentive arm indicated that they had not treated patients in the ICU setting in the last 2 years and were deemed ineligible. Among the 55 known eligible respondents who did not complete the survey, 32 (58%) answered the screening question and the question about practice history, but did not answer any of the questions related to the brief clinical scenarios. The remaining 23 (42%) participants responded to a median of 4 scenarios (interquartile range [IQR] 3-6). The median time to response was 3.4 days (IQR 0.3-22.0) and the median amount of time eligible intensivists spent completing the survey was 3.9 minutes (IQR 2.5-5.5). Among eligible intensivists invited to participate, 80.49% (1489/1850) were male, 63.30% (1171/1850) specialized in internal medicine, and the median number of years since completing initial residency was 20 (IQR 13-28). The characteristics of intensivists randomized to the control versus incentive groups were similar (see Table 2).

Based on the RR3 equation, the overall response rate was equal to 35.90%:

(316+314)/([316+314]+[35+20]+ .96 [624+3]+.93[500+3])= 630/1755 = 35.90%

The control arm response rate was equal to 33.2%:

316/(316+35+ .96 [624+3])= 316/953=33.2%

The incentive arm response rate was equal to 39.2%:

314/(314+20+ .93[500+3])= 314/802=39.2%

‎View this figure

Primary and Secondary Outcomes

The overall response rate among eligible intensivists offered an incentive was 39.2% (314/802) versus 33.2% (316/953) in the control group (P=.01) (see Figure 1). The proportion of eligible respondents answering all survey questions was modestly greater in the incentive group (807/859, 94.0%) versus the control group (892/991, 90.0%) (P=.06). Among the known eligible participants who did not complete the survey, 65% (13/20) in the incentive group and 54% (19/35) in the control group did not answer questions related to the brief clinical scenarios (P=.44). In contrast to these relatively small effects on response, the incentive was associated with a large reduction in median time to response among responders (0.8 days for incentive group and 6.6 days for control group, P=.001; see Table 3). The median time required to complete the survey was 3.9 minutes in each group (P=.56).

Incentive Amount

Out of 314 intensivists who answered all survey questions and were offered an incentive, 111 (35.4%) chose to create one (see Table 4). All who chose to accept an incentive were able to do so, as US $95 remained at study end from the US $5000 originally budgeted for incentives. Overall, 80.2% (89/111) of accepted incentives were for the maximum value of US $50. Among the 22 intensivists out of 111 (19.8%) who chose incentives worth less than US $50, the median value was US $20 (IQR $11-$25). Intensivists randomized to the incentive group who accepted, versus did not accept, incentives completed their first residency more recently (median years since residency 15 vs 19, respectively; P=.004). A greater proportion of male versus female intensivists chose incentives (99/259, 38.2% versus 12/55, 22%, respectively; P=.03).

Discussion

Principal Findings

In a national randomized trial of 1850 academic intensivists, permitting these physicians to choose their preferred level of financial compensation for participating in a short Web-based survey resulted in a 6.0% (95% CI 1.5-10.5, P=.01) absolute increase (15.3% relative increase) in response rate, a 3.9% absolute increase in survey completeness, and a faster response time (0.8 vs 6.6 days), with no impact on the time spent completing the survey. Although 66.7% of intensivists offered compensation did not take it, those who did accept it generally took the maximum US $50 amount that was available to them.

Among intensivists offered an incentive, the only respondent characteristics associated with taking it was time since completing residency and gender. More recent graduates of medical training are likely to have lower salaries, higher educational debt levels, and greater electronic expertise, making US $50 more valuable and accessible. Although previous studies have found male health care workers to be less likely to respond to surveys than women [2,10], a sex-based difference in response to compensation has not been commonly reported in prior literature and merits greater investigation. The observed association between instant compensation and time to response is likely to have been influenced by the perceived scarcity of the compensation (ie, the invitation email said the incentive would be offered to the first 100 respondents) and the proximity of the study timing to annual holiday spending in December. This enhanced desirability of scarce resources is a well-known psychological effect [30], and an example of the larger phenomenon of loss aversion [31].

Shortening the time to survey response decreases the number of reminder or follow-up contacts required. Sending fewer reminders saves time and money when surveys are administered by post. Additionally, previous work suggests that late responders often differ from early responders both demographically and in their survey responses [32-35]. Techniques that recruit physicians who intend to respond, but are prone to delaying participation, help ensure their unique perspectives are represented in the study sample.

The fact that relatively few intensivists (33.3%) took the incentive may have meant that most respondents were sufficiently interested in the survey topic not to require any further motivation for participation, or that the survey was short enough—median completion time was 3.8 minutes—that most respondents did not require reimbursement. Participants who took the incentive may have been less interested in the survey topic, but this difference in interest did not lead these participants to spend less time considering or answering survey questions. Efficiently incentivizing participants to thoughtfully answer all questions may be more important for lengthier research questionnaires, although recent trials have reported no association between survey length and physician response for either postal [36-38] or Web-based surveys [39]. Decisions about whether to take the incentive also may have been influenced by altruistic or sympathetic sentiments created by disclosing the limited funding available for this student thesis project.

As physician response rates decline, leveraging available funds to incentivize survey participation becomes increasingly important. Allowing physicians in the incentive group to decide whether they wished to be compensated and explicitly mentioning that only the first 100 respondents could take the incentive allowed us to offer the full US $50 to the 859 targeted intensivists who were potentially motivated by financial gain, despite a budget of only US $5000. There are two alternative incentive strategies to consider. The first is providing US $50 to all 314 eligible respondents who completed the survey. This would have cost US $15,700. Compared to this incentive strategy, our approach produced an absolute savings of US $10,845 and a relative savings of 69.08%. The second incentive strategy is to provide a flat incentive of US $50 to the 111 eligible respondents who requested compensation which would have cost US $5500. By permitting these 111 respondents to determine their preferred compensation level up to a maximum of US $50, the total value of compensation requested was US $4855. Compared to the second strategy, providing the option to choose the value of compensation to those who requested it resulted in an absolute savings of US $645 and a relative savings of 11.73%.

Determining the appropriate amount of compensation to offer for survey completion remains challenging. Given that the vast majority of respondents who elected compensation took the maximum amount suggests that US $50 may not have been viewed as sufficient by the majority of intensivists requiring a financial incentive to participate in this very short survey. Future studies with the ability to offer greater incentives and, thus, subject to less of a ceiling effect could provide insight into the distribution of preferred compensation for survey participation among physicians.

It is important to consider the ethical and practical ramifications of perceived scarcity and providing different levels of financial compensation to members of the same study cohort. If provided as remuneration for a participant’s time, failing to provide sufficient compensation may be ethically untenable or impact data quality. In such cases, a preferable strategy would be to offer an ethically acceptable level of remuneration for all participants completing the survey and to permit the subset of participants who respond most quickly to choose a preferred level of compensation beyond the minimum remuneration level. In such cases, study investigators would need sufficient funds to cover the maximum possible cost of the total compensation and would effectively be incentivizing prompt responses.

Limitations and Strengths

A potential limitation of our study is the generalizability of our results to other groups of physicians and other health care providers. Additionally, as a bundled intervention comprised of three techniques to optimize survey response, we cannot isolate the impact of any one technique or detect any potential synergistic effects. Study strengths include the use of a national database of academic intensivists containing demographic information on physicians who are almost certain to have regular Internet access, and an electronic survey platform that provided important details regarding survey completion. A study of health care providers offered a gift card to a retail store chain in a postal survey found that provider response decreased in proportion to distance from the nearest store [40]. By offering an incentive that can be used to make Web-based purchases, it is unlikely that the decision to create an incentive was influenced by concerns about proximity to a physical location.

Conclusions

In conclusion, in this randomized controlled trial of 1850 US academic intensivists, giving physicians a time-limited opportunity to choose how much they wished to be compensated for participation in a brief, Web-based survey was associated with a small increase in response rate and a substantial decrease in time to response, without any decrease in how long physicians spent in completing the survey.