The Current Knowledge, Attitudes, and Practices of the Neglected Methodology of Web-Based Questionnaires Among Chinese Health Workers: Web-Based Questionnaire Study

Introduction

Questionnaire surveys are an important method in medical epidemiological research and have been applied to scientific research for approximately 100 years [1]. In recent decades, web-based questionnaire (WBQ) survey methods (using computerized self-administered questionnaires stored on a server and accessed via a browser [2]) have gradually become popular worldwide [3,4]. In 2019, the World Health Organization (WHO) published the first digital health guideline, recommending “digital tracking of patients’/clients’ health status and services via mobile devices” [5], which required reliable data collection through digital methods. Thus, with the advent of the digital health era, WBQs are expected to become one of the most acceptable data collection methods.

Currently, WeChat is becoming the most promising scientific social media platform worldwide, especially in China [6]. It has been reported that the number of internet citizens in China has reached 989 million, with an internet penetration rate of 70.4% [7], and the number of WeChat active users worldwide has reached 1.26 billion, most of whom are in China [8]. More importantly, given the popularity of the internet and WeChat and the compatibility between WeChat and WBQ software, the potential difference in population coverage between paper-based questionnaires and WBQs is gradually narrowing [9], which effectively reduces the risk of selection bias of WBQ survey research. Therefore, the platform and audience of WBQ surveys in China have reached an unprecedentedly high level, implying the emergence of a large number of WBQ survey research in the future.

The WBQ has many advantages that traditional questionnaires do not have, but its disadvantages are also obvious (such as selection bias, information bias, problems of data reliability, and repeatability) [2,3,10,11]. Therefore, acceptable WBQ survey research must have a standardized research design, implementation, and reporting to ensure the reliability of the results [12]. Unfortunately, only a few of the existing books and guidelines on questionnaire survey methodology involve a WBQ [2], which may not be compatible with the experience of traditional questionnaire surveys. Bennet et al [13] and Turk et al [14] showed disappointing results in the quality of reporting the studies (most of the studies were underreported, with some items even reaching 98%) after evaluating published WBQ survey research in 2010 and 2018, respectively, indicating that the current reporting quality is unsatisfactory. However, little is known about the reasons that exactly lead to the underreported situation in WBQ survey research.

It seems that in the digital health era with the wide use of WBQs, researchers are more likely to pay attention to the advantages rather than the disadvantages of WBQs, and few researchers recognize the differences between WBQs and traditional questionnaires, suggesting that the WBQ methodology may have long been neglected. Therefore, it is urgent to investigate the current knowledge, attitudes, and practices (KAP) of WBQ methodology among health workers to further guide and improve this situation. For this reason, we developed a WBQ according to the recommendations and current knowledge of WBQ methodology [3,10,11,15] and conducted a cross-sectional survey on the KAP of WBQ methodology among Chinese health workers for the first time. The novelty and importance of our study is that we tried to clarify what makes the current quality of reporting WBQ survey research uneven and unsatisfactory from the root causes (researchers’ KAP in WBQ methodology) to provide a basis for improvement in the future.

Methods

Study Design and Participants

From July 8 to July 10, 2022, we held the 2022 Chongqing’s Academic Annual Conference of Child Health Care (web-based) and delivered a lecture on WBQ methodology at the conference on July 8 (Multimedia Appendix 1) [16]. The participants of the conference were mainly health workers of child health care and pediatrics in institutions at all levels. Notably, in China, child health care workers mainly conduct epidemiological research and are one of the most experienced groups in the practice of traditional questionnaires and WBQ surveys [17]. Thus, we chose these participants as the target population to understand the KAP of WBQ among Chinese health workers.

This study was set as the login part of the conference and investigated the participants through a self-developed WBQ from June 7 to July 6, which was before the conference. All participants were recruited without selection and were allowed to decide whether to complete the questionnaire or skip and jump to the conference page directly (Figure 1). This study was an exploratory study, and the main outcome was the KAP of WBQ among Chinese health workers, while the characteristics of the participants who had difficulties in conducting WBQ survey research were the secondary outcomes.

We referred to the ethical issues and suggestions of using social media for recruitment proposed by Gelinas et al [18] and followed the Checklist for Reporting Results of Internet E-Surveys (CHERRIES; Multimedia Appendix 2) and Strengthening the Reporting of Observational Studies in Epidemiology (STROBE; Multimedia Appendix 3) checklists to report the results [15,19].

‎View this figure

Ethics Approval

Our study was reviewed and approved by the Ethics Committee of the Children's Hospital of Chongqing Medical University (No. 2022-216).

Development of the Questionnaire

We developed a structured WBQ through “Lediaocha.com” (an open web-based questionnaire platform developed by Shanghai Feiguan Information Technology Co, Ltd) and displayed the description of the survey on the cover page (Multimedia Appendix 4). The questionnaire consisted of 48 required questions on 4 pages (14 questions related to general characteristics, 8 questions related to attitude, 11 questions related to knowledge, and 15 questions related to practices). The questions were based on the CHERRIES checklist [15], the recommendations (from Minto et al [3], Ball et al [10], and Regmi et al [11]), and our current knowledge.

Logical display rules were set between relevant questions, which were displayed only after specific conditions were met to avoid false positive answers and save time. We added spaces next to the core vocabulary of each topic to shorten the reading time according to psychological theory (appropriate word spacing is beneficial to Chinese reading) [20], and the whole questionnaire took approximately 5 minutes to complete.

In addition, we set 3 logical progressive questions (“Have you heard of the WBQ?” “Have you conducted the WBQ survey?” “Have you participated in training on WBQs?”), 1 repeated question about age at the beginning and end, and 1 self-evaluation question about the response quality (“The answer is completed and cannot be modified after submission; please help us evaluate the quality of your answer”) to evaluate the reliability of the responses [21].

Settings of the Questionnaire

The questionnaire was completed anonymously and voluntarily and was accessed and answered through the browser provided with WeChat on a mobile phone or computer. The participants would automatically jump to the conference page after submitting the questionnaire and were allowed to choose to skip the questionnaire directly on the cover page (Figure 1A). The number of answers per account was not limited, but the WeChat nickname and device ID were recorded on the first visit for a duplicate response check. The questionnaire was open access without passwords, which allowed WeChat users to share and promote it. However, the questionnaire could not be retrieved directly by the search engine and was not provided to the public template library.

The serial numbers of questions and completion indicators were displayed while answering, submitted responses could not be reviewed or changed, and unsubmitted responses were not automatically saved. The data stored in the cloud server of “Lediaocha.com” were downloaded and saved by the researchers and then deleted after the survey to avoid potential data security issues. According to the responses, we gave a lecture on WBQ methodology at the conference after the survey, which was used as a response inducement measure.

Pretest and Promotion

A pretest was conducted before the formal survey. A total of 10 health workers came from the research team, and other institutes were recruited to test the expression of the questions and the process of the survey. We promoted the questionnaire for 30 days before the opening of the conference. Because the hot topics of online public opinion usually have the characteristics of rapid growth and rapid decline [22], we conducted 4 centralized promotions (including WeChat Groups, Moments, and official accounts). The interval between each promotion was no more than 1 week.

Sample Size Estimation

Currently, there is no clear sample size calculation method for KAP studies. We referred to the sample size estimation method for the quality-of-life study (the sample size is recommended to be 5-10 times the number of items concerned) [23], as the KAP study is a multi–end point questionnaire study and is similar to the quality-of-life study. Thus, the minimum number of valid questionnaires was 170 (5 times of the 34 questions about KAP) to analyze the KAP. In addition, since the number of health workers in China and the proportion of their use of WBQ are unknown, we assumed that the proportion was 50% (which provided the largest calculated sample size compared to other proportions) [24] with an allowable error of 0.05 and a confidence level of 95%. The calculation identified 402 as the minimum sample size through PASS 11.0, and 201 health workers who had used WBQs were expected to be recruited.

Data Processing

The collected data were directly exported as an Excel (Microsoft Inc) file and checked for inclusion criteria, including health workers, nonduplicate responses, and reliable responses. Duplicate responses were screened by WeChat nickname and device ID and then verified by gender and age. Furthermore, we analyzed the 22 duplicate responses to decide which one should be excluded and found that only 4 duplicate responses were not easy to judge (Multimedia Appendix 5). Thus, we excluded the 4 later duplicate responses according to psychological theory, which indicates that the first impression is more accurate [25], as the small proportion (4/458, 0.9%) was unlikely to affect the results. A data reliability check was conducted through the 3 groups of preset questions, and the response was determined to be unreliable and excluded when any of the preset questions did not meet the requirements. Finally, the responses of nonhealth workers were excluded.

Statistical Analysis

Data analysis was performed with GraphPad Prism 9. Qualitative data were described as frequencies (percentages), and quantitative data were described as medians (interquartile range, IQR) after normality testing. In this study, 50% and 75% were defined as “qualified” and “satisfactory,” respectively, for the proportion of knowledge and practice questions. The chi-square test and Mann-Whitney U test were used to analyze the general characteristics (including sex, age, occupation, specialized subject, positional title, and hospital level) between the users and nonusers of WBQ. In addition, multivariate logistic regression was used to analyze the general characteristics of the users who had difficulties in WBQ survey research. Further subgroup analysis was not conducted, as the sample size would be insufficient. P<.05 was considered statistically significant.

Results

General Characteristics of the Participants

Figure 2 shows the visits and responses to the questionnaire. The number of recruits increased and then decreased rapidly in a pulse manner after each promotion. The questionnaire was visited 2041 times by 1055 participants (the duplicate visit rate was 48.3%) and was responded to 480 times by 458 participants (the duplicate response rate was 4.6% and the unique response rate was 43.4%). A total of 381 valid questionnaires were finally included in the analysis (the effective rate was 83.2%), and the response time was 3.7 (IQR 2.6-5.1) minutes (Figure 1B).

Table 1 shows the general characteristics of the participants. Women accounted for 90.6% (345/381), most respondents were 37 (IQR 11) years old, and doctors accounted for 70.1% (267/381). A total of 91.6% (349/381) of the participants had heard of WBQs, 88.3% (308/349) believed that WBQs were more suitable for investigation, and 51.9% (181/349) believed that they were easy to use. A total of 56.4% (215/381) of the participants had used WBQs for investigation (users). Among them, 11.6% (25/215) had used a WBQ once, 35.3% (76/215) had used it 2-3 times, and 53.0% (114/215) had used it more than 4 times. However, only 44.2% (95/215) had ever received training. Further analyses were based on the 215 users of WBQ, and the general characteristics (including sex, age, occupation, specialized subject, positional title, and hospital level) between the users and nonusers were not significantly different (P>.05).

‎View this figure

Knowledge and Practices of WBQ Methodology

Figure 3 shows the knowledge and practices of the methodology among the users (N=215). In the knowledge topics, “detection of duplicate responses” (86, 40.0%) and “guidelines for results reporting” (105, 48.8%) did not reach a qualified level, and no topics reached a satisfactory level. In the practice topics, “ethics committee review” (71, 33.0%), “conduct pretests” (91, 42.3%), “response inducement measures” (68, 31.6%), and “proper disposal of cloud data” (66, 30.7%) did not reach a qualified level. Only “set anonymous” (176, 81.9%) and “explain incomprehensible topics” (180, 83.7%) reached a satisfactory level.

‎View this figure

Attitude on WBQs

Table 2 shows the attitude toward conducting web-based surveys among the users (N=215). Among them, 72.6% (156) believed that a WBQ was easy to accept, and 62.3% (134) believed that a WBQ was practical, but 58.6% (126) had doubts about the reliability of the results. In addition, 88.8% (191) of the users believed that training was necessary. Regarding the difficulties of WBQ surveys, “quality control” (136, 63.3%), “study and questionnaire design” (134, 62.3%), “data processing and statistical analysis” (128, 59.5%), and “response rate of participants” (126, 58.6%) were the most difficult aspects.

The multivariate analyses (Table 3) showed that users who were male (odds ratio [OR] 0.35, 95% CI 0.14-0.87, P=.03) and younger (OR 0.95, 95% CI 0.90-0.99, P=.03) were less likely to report difficulties in response issues. Doctors were more likely to report difficulties in quality issues than nurses (OR 0.34, 95% CI 0.15-0.76, P=.009) and other health workers (OR 0.32, 95% CI 0.11-0.97, P=.04). Pediatric workers (OR 2.54, 95% CI 1.15-5.61, P=.02) were more likely to report difficulties in design issues than child health care workers. Users with senior positional titles were less likely to report difficulties in promotion issues (OR 0.26, 95% CI 0.08-0.87, P=.03) but were more likely to report difficulties in response issues (OR 3.26, 95% CI 1.01-10.60, P=.049) than users with junior/none titles.

Discussion

Principal Findings

In 2010, van Gelder et al [26] proposed that WBQs could be expected to become the future of epidemiology, but the design of the questionnaire would have a significant impact on the results. Since then, Bennet et al [13] and Turk et al [14] have reviewed the published WBQ survey research and found that the quality of reporting the research was not optimal. To clarify the possible reasons for the unsatisfactory reporting quality of WBQ survey research, we described the KAP of the WBQ methodology among Chinese health workers for the first time. Our study found that the participants who had heard of WBQs generally believed it was suitable for investigation (308/349, 88.3%), and more than half (181/349, 51.9%) believed it was easy to use. However, further analysis of the users showed that most of the knowledge and practice topics did not reach a satisfactory level, and some even did not reach a qualified level. Therefore, although WBQs are regarded as the most promising epidemiological research method, the methodology is seriously underestimated and neglected, which may explain why the research is underreported.

The Differences Between WBQs and Traditional Questionnaires

It is known that the WBQ methodology directly affects the reliability of the results [10]. Our study showed that 62.3% (134/215) of the users believed that it was very practical, but more than half of the users (126/215, 58.6%) also had doubts about the reliability of the results. The possible reason is that WBQ users do not fully understand the influences that WBQs bring to the results, causing a lack of consideration and confidence in the reliability. More importantly, our study showed that more than half of the users (110/215, 51.2%) did not even know the guidelines for reporting WBQ survey research, indicating that they do not treat WBQ as a survey method different from traditional questionnaires. However, the current research shows that there are many differences between WBQ and traditional questionnaires. Thus, researchers need to know the potential influences when applying WBQ to research.

The most concerning topic is that which is more accurate in the results of WBQs and traditional questionnaires. Although some studies have shown that WBQs and traditional questionnaires might obtain different results when dealing with subjective or sensitive topics [21,27,28], many studies have shown that WBQs have the same internal consistency as paper questionnaires and have even higher reliability when dealing with private topics [29-33]. In fact, the reasons why different survey methods produce different results are partly related to the complexity and variability of human psychology. For some topics related to privacy or social stigmatization, face-to-face surveys may be more vulnerable to social expectation bias than WBQs [34-36], but some studies have shown that WBQs have no advantage in this regard [37]. In addition, for some topics that are difficult to understand, WBQ may be more prone to information bias than face-to-face surveys. A small number of WBQ participants (5/515, 1%) reported comprehension problems even after careful cultural adjustment and pretesting, as previously described [21]. Our study showed that “set anonymous” (176/215, 81.9%) and “explain difficult topics” (180/215, 83.7%) reached a satisfactory level, indicating that the researchers had fully recognized these two aspects. However, anonymity also brings its problems, that is, it is difficult to verify when the data are difficult to explain.

The other topic of concern could be that different survey methods could cover different populations, that is, researchers should consider the sample size and representativeness of the sample. In this regard, our study showed that “estimate sample size” (150/215, 69.8%) and “consider sample representativeness” (146/215, 67.9%) had reached a qualified level. In fact, studies have shown that some groups of people are not able to easily access WBQs (such as older adult people) [38-40]. However, in recent years, an increasing number of older adult people have begun to use WeChat and carry out communications about health issues [41]. At the same time, the coverage of WBQs has been further expanded, so the low response rate does not necessarily increase selection bias [9]. However, it is not enough to rely only on previous research results to determine the selection bias of WBQ. Researchers should consider creating advertisements targeting underrepresented populations [40] or adjusting publicity strategies [42].

Neglect of the WBQ Methodology

To standardize the research using WBQs, Eysenbach et al [15] formulated a checklist (CHERRIES) of the results reported in 2004, which provided several recommendations to improve the quality of WBQ survey research and are still highly applicable. In the past decade, Mario et al [2], Minto et al [3], Ball et al [10], and Regmi et al [11] all proposed recommendations on the design and implementation of WBQ surveys. However, given the unsatisfactory reporting quality of WBQ survey research according to the literature reviews [13,14], these recommendations seemed to have little effect on improving the reporting quality. Our study further showed that the current KAP of the WBQ methodology among Chinese health workers was not ideal, as most of the items were far from satisfactory. Although our study focused on Chinese health workers, it provided evidence on the lack of researchers’ awareness of the WBQ methodology, which may explain the insufficient effects of current recommendations.

It is worth mentioning that with the development of internet companies, an increasing amount of research has been conducted through open web-based survey platforms, which may bring additional problems for the WBQ methodology. The most prominent problems involve privacy protection and data security and have gradually become the focus of ethics committees [1]. Our study showed that only 30.7% (66/215) of the WBQ users would dispose of cloud data after the survey, and 33.0% (71/215) would submit it to the ethics committee for review, which could increase the potential risk of data disclosure. Although the related items in the CHERRIES checklist are currently insufficient [15], we recommend that researchers properly deal with cloud data after the survey (such as saving it by a specially assigned researcher after downloading).

Another problem could be the strategy of recruiting through social media, which is related to the response rate that was reported with difficulties by 58.6% (126/215) of the users. In 2005, Dannetun et al [43] showed that most of the responses (70%) to WBQs were received within 7 days after promotion. However, our study showed that when the questionnaire was promoted through social media, each promotion was only valid for 1-2 days, and the recruitment potential was basically lost after the valid period. This may be related to the fact that hot topics of web-based public opinion have their own life cycle [22], the topics of our survey were not attractive enough, and the response inducement measures were not immediate. However, it may also be related to the current popular fast-food culture. Therefore, although WBQs are easy to promote, researchers may not obtain the expected sample size if they do not know these characteristics when formulating promotion strategies, which may lead to excessive promotions. However, it remains unclear whether excessive promotion will have a negative impact on the sample.

Requirement for Methodological Training of WBQs

The current KAP of the WBQ methodology is not ideal, suggesting that education or training is obviously insufficient. The surveys of Edirippulige et al [44] and Machleid et al [45] showed that medical students had a great demand for digital health education and training and were eager to improve their knowledge of digital health. Our study also showed that less than half (95/215, 44.2%) of the users had received training before conducting WBQ surveys, while most of them (191/215, 88.8%) believed that training was necessary. Considering the frequency with which the users used WBQs in our study (190/215, 88.4% had used them more than 2 times and were considered well experienced), their significant demand for methodological training is credible.

For the education or training of WBQ methodology, our study provided content (such as the unqualified topics) that should be strengthened. In addition, “quality control” (136/215, 63.3%), “study and questionnaire design” (134/215, 62.3%), “data processing and statistical analysis” (128/215, 59.5%), and “response rate of participants” (126/215, 58.6%) were the most difficult aspects according to the users’ self-reports. Different users had different difficulties in WBQ surveys. For example, participants with senior positional titles reported fewer difficulties in promotion issues but more difficulties in response issues, which might be related to the resources, experiences, and skills that users had. Therefore, future medical educators should also consider targeted content while increasing the number of methodological trainings on WBQs.

Strengths and Limitations

The strengths and limitations of this study require careful consideration. First, this study focused on the situation of unsatisfactory reporting quality of WBQ survey research, which is an international problem and provided unique insights on improving the quality of WBQ survey research from the perspective of researchers. Second, this study was designed and conducted by a well-experienced WBQ survey team, which guaranteed the reliability of the results. Third, this study discussed the current WBQ survey methods, which are meaningful not only for medical educators but also for WBQ users.

Despite its strengths, this study has some limitations. First, for selection bias, this study is focused on Chinese health workers, so that the results could not be simply extended to other populations but still provided clues. This study took an academic conference as the context, which may make the participants more experienced in WBQs than the average level and cause overestimation of the real KAP of Chinese health workers. In addition, this study did not analyze the potential differences between participants and nonparticipants due to lack of data on the nonresponders. However, since our outcomes were based on WBQ users, we analyzed the differences between users and nonusers and found no significant difference. Second, for response rate, although the response rate of our study (458/1055, 43.4%) was lower than the mean level (51.3%) of public health research using WBQ from 2010 to 2015 [46], it could be underestimated because many nonparticipants in social media could also access the questionnaire. Third, for network technology, this study was based on an open WBQ platform, which improved the convenience but also increased the lack of specific data (such as login files) [47], leading to the unknown characteristics of nonresponders.

Conclusions

In summary, this study found that Chinese health workers seriously underestimated and neglected the importance of the WBQ methodology, which may be an important reason for the reduced reporting quality of WBQ survey research. The accessibility and convenience of WBQs should not be the reason researchers neglect their methodology. Medical educators need to strengthen methodological training on WBQs, which may help to improve the reliability of a large number of WBQ survey studies in the digital health era.