Response Rates for Patient-Reported Outcomes Using Web-Based Versus Paper Questionnaires: Comparison of Two Invitational Methods in Older Colorectal Cancer Patients

Introduction

The first Web-based questionnaires were posted in the mid-1990s, but they were only available for a select few with access to a computer and to the Internet [1]. Today, the Internet is accessible for more and more households. In the Netherlands, access to the Internet is high with 97% of the households having an Internet connection in 2013 [2]. To optimally utilize this high level of access, the population-based Patient-Reported Outcomes Following Initial treatment and Long-Term Evaluation of Survivorship (PROFILES) registry was developed in 2010. Its goal is to collect, preferably, online data on patient-reported outcomes (PRO) of cancer patients at least once a year [3]. However, paper questionnaires can be provided if preferred by the respondent. Offering different modes of administration is a way to improve response as is offering incentives and sending reminders [4].

Although it is widely accepted that Web-based questionnaires offer advantages, these advantages are not all scientifically proven. Advantages of Web-based questionnaires compared to paper questionnaires that are supported by literature are more complete data [5], less data entry errors [6], and questionnaires returned more quickly [7]. Several studies also show that reliability of Web-based questionnaires and paper questionnaires is comparable [8-11]. An often-described disadvantage of Web-based questionnaires is sample bias [1] because not all population groups have access to or are proficient in using the Internet. Additionally, Web-based questionnaires often have lower response rates than paper questionnaires [12].

Recent literature shows that the online respondent is most likely to be young and highly educated [10,13,14]. On the other hand, cancer patients tend to be older. There are a few studies that report on Internet use of older patients, but these patients are described as one group (eg, 50 years and older or 65 years and older) with response percentages varying from 58% to 63% [15-20]. Studies on computer and Internet use that are stratified by age and include a group older than 60 years are sparse. We found 6 studies that stratified older age into groups for computer or Internet use [21-26]. Percentages varied from 10% for those aged ≥85 years for Internet use in 2007 [22] to 35% in the 60-69 years age range for computer use for email in 2011 [21]. A previous study from our group, performed in 2007, observed Internet access for 3 age groups (<50 years, 50-59 years, and 60-69 years) of 81%, 65%, and 47%, respectively [24]. According to Statistics Netherlands, 54% of Dutch individuals between ages 65 and 75 years had access to the Internet in 2007 [27]. Since 2007, access to the Internet in the Netherlands has increased rapidly. In the older population aged 65-75 years, 80% currently have access to the Internet [28]. Therefore, we expect that more older respondents will be able to fill out Web-based questionnaires.

This paper describes a study in which our primary aim was to investigate whether including a paper questionnaire in the initial invitation would lead to a higher response rate. Furthermore, we wanted to compare patient characteristics and response rates between different modes of administration (Web-based vs paper). Our second aim was to evaluate the preference of administration mode for the older patient in particular. We hypothesized that including a paper questionnaire would increase the overall response rate because it may still be the preferred mode of administration for many older adults.

Methods

Setting

We used data from a large population-based survey conducted in 2010 among colorectal cancer (CRC) patients. Data were collected within the PROFILES registry [3]. The PROFILES registry collects data for the study of the physical and psychosocial impact of cancer and its treatment from a population-based cohort of short- and long-term cancer survivors. PROFILES contains a large Web-based component. However, because a large percentage of cancer patients are older, PROFILES also collects PRO data using traditional paper questionnaires. Collected PRO data are directly linked to clinical data from the Eindhoven Cancer Registry (ECR).

The questionnaire consisted of questions on work and lifestyle, health care use, comorbidity (the Self-Reported Comorbidity Questionnaire), diabetes (Problem Areas in Diabetes Questionnaire), quality of life (EORTC QLQ-C30), disease-specific symptoms (EORTC QLQ-CR38), health status (SF-12), personality (DS14), illness perception (Brief Illness Perception Questionnaire), fatigue (Fatigue Assessment Scale), and anxiety and depression (Hospital Anxiety and Depression Scale). A total of 182 items were to be answered. Respondents were informed that filling out the questionnaire could take up to 45 minutes. Online respondents could see the progress on a progress indicator and were able to log in again to continue completing the questionnaire from where they last left off. Online respondents were required to answer all questions, but could choose the option “I don’t want to say” when sensitive information was asked (eg, sexuality).

The ECR, which is part of the Comprehensive Cancer Center the Netherlands, compiles data of all newly diagnosed cancer patients in the southern part of the Netherlands, covering an area with 10 hospitals serving 2.3 million inhabitants [29]. All individuals aged between 18 and 85 years, diagnosed with CRC between January 2000 and June 2009, and registered in the ECR were eligible for participation in our study. Those with cognitive impairment, with stage 0/carcinoma in situ, and who died prior to start of the study were excluded, resulting in 3585 eligible patients. The CRC survey was approved by the Central Committee on Research Involving Human Subjects (approval number NL23463.015.08) and the Medical Ethics Committee of Máxima Medical Centre (approval number 0822). All patients signed informed consent. Further details on the method of data collection are published elsewhere [30].

Description of Study Groups

Patients were divided into 2 groups for this study (Figure 1). Patients in the paper-optional group (n=3406) were invited via a letter from their former attending specialist. The letter contained a website address and log-in instructions to fill out the Web-based questionnaire. It also contained a reply card with a return envelope (postage included) with which participants could request a paper version of the questionnaire. Patients in the paper-included group (n=179) received the same letter, but with a paper questionnaire and a return envelope (postage included) also included. Nonrespondents from both groups were sent a reminder letter together with a paper questionnaire and return envelope (postage included). A reminder was sent after 3 months, on average. The number of patients needed for the paper-included group was calculated in advance. We sampled the number of patients to be able to test a statistically significant higher (1-sided test based on our hypothesis) response rate of 10% between both groups, assuming a response rate of 75% in the paper-optional group (power 80%, alpha 10%).

‎View this figure

Demographics and Clinical Characteristics

Patients’ demographic and clinical information, including cancer stage, time since cancer diagnosis in years, and primary treatment were available from the ECR. The questionnaire contained questions about marital status and educational level. Information about response status, time of completion (either before or after the reminder), and the chosen mode of administration (paper or Web-based) was gathered from the PROFILES data manager application.

Statistical Analyses

Differences in characteristics of respondents and nonrespondents and between groups were analyzed using independent t tests and chi-square tests where appropriate. Further analyses within the paper-optional group were conducted to assess differences in clinical and demographic characteristics between online and paper respondents. All differences with a P value <.05 were considered statistically significant. To assess the difference in online response between the 2 groups, logistic regression models were constructed. An unadjusted and a logistic model adjusted for age, sex, educational level, and having a partner or not were used to assess differences in Web-based response. Odds ratios (ORs) and 95% confidence intervals (95% CI) were reported. All statistical analyses were performed using SAS v9.2 for Windows (SAS institute Inc, Cary, NC, USA).

Results

Of the 3406 invited CRC patients in the paper-optional group, 2491 (73.14%) responded. In the paper-included group, a similar response rate was found with 134 (74.9%) respondents of the 179 invited CRC patients.

Statistically significant differences in characteristics between respondents and nonrespondents were seen for gender (male: 55.16%, 1448/2625 vs 48.0%, 297/619, P<.001), age (mean 69.4, SD 9.53 vs mean 72.4, SD 9.9, P<.001), and cancer type (colon: 61.18%, 1606/2625 vs 66.9%, 414/619, P=.03) (Table 1). The age difference between the 2 groups was more pronounced after age was stratified into categories. The biggest response difference was found in the age category 60-70 years (32.27%, 847/2625 vs 23.4%, 145/619, P<.001) and ≥80 years (13.26%, 348/2625 vs 25.8%, 160/619, P<.001).

Differences in Response Rates Between Groups

No differences in overall response rate were found between the paper-optional and the paper-included groups, with 73.14% (2491/3406) and 74.9% (134/179, P=.57), respectively (Figure 2). For respondents aged 70 years and older, no difference in response rate was found, with a 68.84% (1290/1847) response rate in the paper-optional group and 75.7% (78/103) in the paper-included group (P=.38).

Characteristics of the respondents in the paper-optional group were comparable with those of the paper-included group, except for age which was slightly older in the paper-included group (≥70 years: 51.79%, 1290/2491 vs 58.2%, 78/134, P=.04) (Table 2). The unadjusted logistic regression model showed patients in the paper-optional group were 4.82 times (95% CI 2.88-8.07, P<.001) more likely to fill out the Web-based questionnaire compared to patients in the paper-included group; this effect remained after adjustments for age, sex, educational level, and having a partner or not (OR 5.81, 95% CI 3.37-10.01, P<.001). In the paper-optional group, online response was significantly higher compared to the paper-included group (41.23%, 1027/2491 vs 12.7%, 17/134, P<.001).

Sending a reminder increased the response by 30% in both arms. Due to local logistical issues, the time of sending a reminder varied between 2 and 5 months. However, this variation did not influence overall response rates (data not shown) or mean time until response (Table 2).

‎View this figure

Comparison of Characteristics of Online and Paper Respondents in the Paper-Optional Group With a Focus on Older Patients

In the paper-optional group, men were more likely to complete the Web-based questionnaire than to return the paper questionnaire (61.73%, 634/1027 vs 50.14%, 734/1464, P<.001) (Table 3). Compared to paper respondents, online respondents were younger (mean 65.72, SD 9.28 vs mean 71.85, SD 8.89, P<.001), more often highly educated (30.34%, 311/1027 vs 11.88%, 171/1464, P<.001), more often married (83.93%, 862/1027 vs 70.69%, 1020/1464, P<.001), more often recently diagnosed (time since diagnosis: mean 4.94, SD 2.74 vs mean 5.31, SD 2.83, P<.001), and more often had a rectal cancer diagnosis compared to paper respondents (41.38%, 425/1027 vs 36.75%, 538/1464, P=.02). The majority of the online respondents responded after the first invitation (95.42%, 979/1027), which was significantly higher than the paper respondents (52.36%, 764/1464, P<.001).

After age was stratified, Web-based versus paper response differed per age group (P<.001, Figure 3). We saw that the turning point of filling out a Web-based questionnaire was approximately age 70 years: the majority of respondents younger than 70 years filled out a Web-based questionnaire and the majority older than 70 years chose a paper questionnaire. Among those aged ≥80 years, 18.2% (61/336) preferred a Web-based questionnaire.

‎View this figure

Discussion

Principal Findings

Our study showed that including a paper questionnaire with the first invitation did not increase overall response rates. In contrast, it negatively influenced the online response. Sending a reminder improves response rates by 30%. Compared to patients responding on paper, online respondents were more often male, younger, married, and highly educated. The majority of respondents in both arms chose to fill out a paper questionnaire. The turning point of preference for a Web-based questionnaire was approximately age 70 years. The majority of respondents who were younger than 70 years preferred to fill out the Web-based questionnaire. The majority of respondents older than 70 years chose the paper questionnaire. We did not find evidence that including a paper questionnaire led to a higher response among older patients.

We expected the overall response rate in the paper-included group to be higher than in the paper-optional group because respondents in the paper-included group received the paper questionnaire immediately at invitation. However, we observed similar overall response rates. A previous review of the literature showed that when respondents can choose between paper and Web-based questionnaires, paper response is higher than online response in most studies [31]. A recent literature review confirmed this, although they expect the difference to diminish in the near future [6]. The absence of this expected difference in response rates in our study could not be explained by differences in patient characteristics between the 2 groups. A possible explanation for the comparable response rates could be the willingness of respondents to participate because the subject of the questionnaire (ie, cancer and health-related quality of life) felt relevant to them. Furthermore, the respondents received the invitation directly from their medical specialist, so they might have felt a moral obligation to participate. The lack of difference in response rates implies that researchers can leave out a questionnaire at first invitation without lowering response. It is preferable not to include a paper questionnaire because more respondents will fill out the Web-based version of a questionnaire, which will enable researchers to access data more quickly and to have a more complete dataset.

Several studies have compared response rates between patients invited via paper only and Web only, or mixed-mode and Web only, or paper only and mixed-mode [5,7,8,10,11,13,14,32-36]. However, few studies are available that address the influence of including a paper questionnaire on response rate in the invitation for a mixed-mode survey. We found an American study that compared the response rates of 3 modes of administration, namely paper only, paper with an Internet option, or Internet with a paper option [37]. The response for the Internet with a paper option and for the paper with an Internet option was 37% and 42%, respectively. These are the same manners of invitation we used in our study. The difference with our study is that instead of sending 1 reminder, the other study sent 4 reminders. Only the last reminder for the Internet with a paper option contained a paper questionnaire. A second difference is that this study was done in 2000 in the United States, so the use of Internet was lower than in 2010 in the Netherlands, when our study was done. Internet use in the United States in 2000 was 51% compared to 90% in the Netherlands in 2010 [38,39]. This might explain the lower response rates for both groups and the bigger difference in response rates between the groups in that study. A Dutch study among 277 long-term childhood cancer survivors in 2010 used a comparable invitation approach and mode of administration [40]. The study used a mixed invitation group (paper with the option of Internet) and a Web-only invitation group (Internet with the option of paper) leading to a response of 83% and 89%, respectively. A different approach with regard to reminders was chosen in that study compared to ours; after sending 1 reminder letter, nonrespondents were contacted by telephone in their study. Another difference is that only young women were included in that study. Both studies [37,40] did not address the (preference of) the older patient.

When studying different age groups, we found that almost 20% of the respondents aged ≥80 years filled out the questionnaire online. We expected a lower percentage because of the so-called “grey digital divide” referring to the low use of computers and the Internet in the older population [41]. This grey digital divide is also confirmed by a British study that found that only 10% of respondents aged ≥85 years have used the Internet at any point in their lives [22]. To fill out a Web-based questionnaire, a respondent must not only be able to use a computer, but also be skilled on the Internet. The high number of older respondents who used the Internet in our study might imply that the grey digital divide is closing in the Netherlands and more older people are becoming familiar with the Internet. Numbers from Statistics Netherlands (CBS) confirm this, showing that there is an increase in Internet use among individuals aged between 65 and 75 years in recent years [28]. Daily use of the Internet among these individuals increased from 15% in 2005 to 55% in 2013. Eurostat Statistics also show these numbers: a rise in frequent use of the Internet among people aged 65 to 74 years from 41% in 2008 to 73% in 2013 in the Netherlands [42]. Unfortunately, a group of users is left out in these statistics, namely the people older than 75 years. The statistics do, however, indicate a trend of older people being more online. With this in mind, researchers could more easily consider using the Internet as a primary mode for data collection without the inclusion of a paper questionnaire with the first invitation.

Strengths of this study are that it is population-based, including (very) older people, has a high overall response rate, and the cooperation of medical specialists. Furthermore, the influence of sending a paper questionnaire in 2 mixed-mode groups has rarely been studied. Thirdly, our results are more recent than other studies that compare paper versus Web-based questionnaires, which is important because of the rapid changes in Internet access. Lastly, we have looked at many patient characteristics to assess the differences in patient characteristics of online and paper respondents.

A limitation of this study is that the time a reminder was sent varied per hospital due to local logistical issues. However, analyses showed that the difference in reminder time did not have any effect on outcomes. A second limitation is that the comparison between the paper-optional group and the paper-included group shows a slight discrepancy in the age categories, although mean age did not differ. There is a slightly higher percentage of respondents older than 70 years in the paper-included group. Although an age difference existed before data collection in the initial random selection of this group, it was not significant (results not shown). Thus, the significant discrepancy found in our results after data collection is a consequence of (un)willingness to cooperate. In the future, further evaluation of nonrespondents may clarify this difference. It is not unlikely that the results found in our study are applicable to other populations, for example, patients with a different type of cancer, a different disease (eg, diabetes), or a normative population. However, further research is needed to confirm this.

Conclusion

Although this study was on a CRC survivor population, we are of the opinion that the significant lack of difference in response rates between invitation modes implies that researchers may leave out a paper questionnaire at invitation without lowering the response rate. It may even be more preferable not to include a paper questionnaire because more respondents then will fill out a questionnaire online, which will lead to faster available data. However, due to respondent preference, it is not likely that paper questionnaires can be left out completely in the near future.