Using the Ensuring Quality Information for Patients Tool to Assess Patient Information on Appendicitis Websites: Systematic Search and Evaluation

Introduction

In the modern era, the increasing accessibility and availability of information has promoted the internet as the primary source for patient information. The access to countless sources of information can cater to every need by providing jargon-free material for the wider public while making details available for those who seek in-depth knowledge. Thus, many patients search online for medical information on their symptoms prior to consulting medical professionals, many of whom subsequently self-diagnose based on these online sources [1,2]. Consequently, access to online medical information is critical to their decision-making process. However, with many sites containing potentially irrelevant or incorrect information, their credibility and reliability may present a barrier against seeking early medical help [3]. Furthermore, the use of unreliable websites may undermine patient relationships with health care professionals; at best, trust in healthcare may be affected, and at worst, presentations delayed by misinformed self-diagnoses may lead to poorer outcomes [4].

With a lifetime incidence of 8%, appendicitis is an affliction that is common enough to be familiar to the general public [5]. Hence, appendicitis, a leading cause of acute abdominal pain, and appendicectomy, its treatment, are likely to be terms that are searched by patients seeking further information. It has been shown that through providing education via high quality information on appendicitis and its risks, awareness and outcomes may be improved [6,7]. In order to make informed and competent health decisions, safe, reliable, and easily accessible information is essential; thus, the variable quality of internet patient resources warrant an evaluation of the quality and readability of this medical information.

The original Ensuring Quality Information for Patients (EQIP) tool is a checklist of 20 items used to assess written health care information [8]. Various aspects are considered, such as clarity of information, quality of written work, and website design. The EQIP tool has been used to evaluate information sources related to gallstone disease, transplant surgeries, eczema, liposuction, and, more recently, COVID-19 [9-13], demonstrating its applications across various disciplines and information types. We assessed the top-indexed websites related to appendicitis and appendicectomy using the modified EQIP tool for evaluation. The objective of our study was to evaluate the quality of information found on the top-searched websites that aim to provide patients with information on appendicitis.

Methods

Eligibility Criteria, Information Sources, and Data Selection

The most popular search engine, Google [14-16], was used to obtain a database of websites. Other search engines were not used in this study, as this would only lead to duplicate results. The searched terms were “appendix,” “appendicitis,” “appendicectomy”, and “appendectomy”. These were obtained using the Google AdWords Keyword Planner [17]. Formation of the database, analysis, and eligibility assessment of the websites were performed between September 2019 and February 2020. Previous work has suggested that patients limit their searches to well within the first 100 hits; therefore, the hits on each page were obtained until this target was reached [10]. The inclusion criterion was any website with information intended for patients. Websites were excluded if the literature was intended for scholars in scientific journals or if they were in a language other than English. Furthermore, links which directed individuals to purely video content or which were used for marketing purposes were also excluded. Upon exclusion, 119 websites were identified as eligible for analysis.

Website Scraping

To obtain the URL database from the top 100 hits for each search engine, a website scraping tool was developed. This reduced the amount of time required for cutting and pasting links to the database. The custom Hypertext Preprocessor (The PHP Group) tool was designed to make HTTP requests to the search engines, mimicking the requests web browsers make when using a search engine. This tool made repeated requests, logging all of the hits per page with a target of 100 unique URLs. During this process, any duplicates were automatically removed within the individual search. If there were more hits on a page after the target of 100 websites was reached, these websites were also collected. The tool was run using a server based in Texas in the United States, although no preferences were chosen to limit searches to certain geographical areas.

Data Entry

Each website was assessed independently by 4 assessors, SAG, KSF, KHF, and LL, all of whom are fluent in English. To assess each website, a Google Form containing the 36 EQIP items was used to evaluate criteria through yes, no, or “N/A” responses. Assessors also recorded the country of origin and the following source types: academic center, encyclopedia, health department, hospital, industry, news service, patient group, practitioner, professional society, or other. After the initial round of data entry, the websites were reassessed by another assessor, and any contradictory results were resolved by consensus.

EQIP Tool

The original EQIP tool has been expanded to 36 criteria to provide a more robust and effective analysis of patient information. The modified EQIP tool sets out to satisfy the patient information collaboration guidelines of both the British Medical Association (BMA) [18] and the International Patient Decision Aids Standards (IPDAS) [19]. The modified EQIP tool consists of 36 items split into 3 domains: content (items 1-18), identification (19-24), and structure (items 25-36). Similar to previous studies, only yes or no options were provided for each item to avoid assessor subjectivity in partial answers. The option for “N/A” was also included if items were not relevant for the type of source. Websites which scored above the 75th percentile were deemed high-scoring websites.

Additional Items Describing Mortality and Complication Rates of Surgery and Emergency Information

Questions were added to the Google Form to assess the variation in reported complication and mortality rates published by differing sources. The additional questions identified those websites that included rates for mortality and complications, and recorded the values given. Furthermore, a question was added to identify websites that included advice in the case of an emergency.

Statistical Analysis

Continuous variables are reported as median and IQR and categorical variables as numbers and proportions in percentages. Continuous variables were compared with the Mann-Whitney and Kruskal-Wallis tests where appropriate. Proportions were compared with the Fisher exact test or chi-square test where appropriate. All P values were 2-sided and considered statistically significant when P<.05. A decision was made to dichotomize the EQIP score by using the 75th percentile as a cutoff point for discriminating high-scoring from low-scoring websites, as previously described and defined [9]. Statistical analysis was performed using R version 3.3.2 (The R Project for Statistical Computing, GNU General Public License version 2) and R Studio version 1.0.44 (RStudio) with the graphical user interface, rBiostatistics.com alpha version [20].

Results

Gathering of Websites With Information on Appendicitis and Its Management

To obtain a database of websites for analysis, the unique hits from each page using the search terms (“appendix,” “appendicitis,” “appendicectomy,” and “appendectomy”) were gathered. The workflow of this is shown in Figure 1. Although a target of 100 websites per term was used, additional hits on the last page of each search were gathered if they were unique, resulting in a total of 435 websites. Duplicate results obtained between search terms and websites failing to meet inclusion criteria were removed, resulting in 119 websites for analysis.

‎View this figure

Overall Quality of the Websites According to the Modified EQIP Tool

The distribution in EQIP score between all websites meeting inclusion criteria is shown in Figure 2. The country of origin and source of information for the database are shown in Table 1. Hospitals were the most common source of information, accounting for 45.4% (54/119) of the database, and 52% (28/54) of websites by hospitals originated from the United States. The country with the most websites was the United States, representing 54.6% (65/119) of the total, of which 26% (17/65) were high scoring. This represented 61% (16/26) of the total high-scoring websites. The distribution of EQIP scores by country of origin with more than one website is shown in Figure 3. Websites from the United Kingdom demonstrated the greatest variance in scores ranging from 9-28 with a median of 18 (Figure 3). Overall EQIP scores from countries with only 1 website were 24 for New Zealand, 19 for South Africa, 16 for Singapore, and 15 for India.

‎View this figure

‎View this figure

Complication rates were included in 12.6% (15/119) of websites and varied between 0.2%-26%. Mortality rates were included in 3.3% (4/119) of websites and ranged between 0.001%-1.8%. Emergency advice was provided in 44.5% (53/119) of websites.

EQIP Content Data

The items for the content domain of the EQIP tool are shown in Table 2. The median score achieved was 9 (50%), and the maximum score obtained was 16 (89%) of a possible 18 (Table 3). All of the high-scoring websites provided a description of the medical problem (item 3), definition of the purpose of intervention (item 4), and description of the qualitative risks and complications of appendicectomy (item 6; Table 2). High-scoring websites were found to describe how complications are handled (item 12) and provided details of other sources of reliable information (item 17) in 77% (20/26) and 73% (19/26) of cases, respectively; this was significantly less in the low-scoring websites (P<.001; Table 2). IQRs for the content domain are included in Table 3.

Among all websites (including low scoring), 95.7% (114/119) failed to address the costs and insurance issues related to appendicectomy (item 15). Furthermore, 97.4% (116/119) and 86.5% (103/119) of websites failed to describe the quantitative benefits (item 8) and risks (item 10) of appendicectomy, respectively (Table 2). Of note, 23 of the 54 hospital-published sources (43%) did not include any description of the qualitative risks and complications of surgery (item 9).

EQIP Identification

In the identification domain, the median score obtained was 3 (50%), and the maximum score obtained was 6 (100%; Table 3). Two websites, both of which were high scoring (overall score >75th percentile) obtained maximum points for this section. High-scoring websites were significantly better (58%, 15/26) than low-scoring websites (20%, 19/93) in providing a short bibliography of the evidence base for the information (P<.001; item 23; Table 4). IQRs for the identification domain are included in Table 3.

Furthermore, 97.4% (116/119) of all websites failed to include a statement about how patients were involved or consulted in the document’s production (item 24), and 68.0% (81/119) did not explicitly provide names of the persons or entities that financed the document (item 22; Table 4).

EQIP Structure

The median score obtained for the structure domain was 8 (66%), and the maximum score obtained was 10 (83%) of a possible 12 (Table 3). All high-scoring websites used a respectful tone (item 29), presented clear information (item 30), and delivered information in a logical order (item 32). High-scoring websites were found to include balanced information on risks and benefits (item 31) more frequently (24/26, 92%) than low-scoring websites (40/93, 43%; Table 4). In addition, 89.9% (107/119) of all websites failed to include a named space for readers’ questions, and 71.4% (85/119) failed to include clear and relevant figures or graphs (item 34; Table 4). IQRs for the structure domain are included in Table 3.

Top 3 Websites According to the EQIP Tool

The websites scoring above the 99th percentile (EQIP score of 28) are shown in Table 5. The top-ranked resource, which scored 29 out of 36, was produced by the American College of Surgeons, displaying a comprehensive guide for patients to understand appendicitis and its management. Bupa health insurance and Medical News Today, each with a with an EQIP score of 28, were tied for the second ranking. All 3 sources provided information on symptoms, details of the procedure, preoperative and postoperative instructions, and guidance on complications.

Discussion

Principal Findings

Based on our analysis, the quality of patient information regarding appendicectomy was shown to be of a moderate level, as reflected by the median overall score of 20 (IQR 18-22). Generally, websites tended to score well in the structure domain, reflected by the median score of 8 (IQR 7-9). This domain focuses on the ability of websites to display their information in a clear and logical manner. As technology is improving, it is becoming easier to produce websites and leaflets to a higher visual standard with minimal computer literacy. This presents a new challenge to patients with limited clinical understanding, as websites of poorer quality may appear similar to high quality sources of information. The increase in total EQIP score due to higher marks being obtained from the structure domain from the improvement of website quality is a phenomenon which has also been seen in regards to COVID-19 [13].

In the identification domain, high-scoring websites were better than low-scoring websites at providing a bibliography of the evidence base, potentially suggesting that these were written by individuals with experience in academia. It is therefore unsurprising that high-scoring websites more often included the names of the persons that produced the document. Contrastingly, the majority of websites (68.0%, 81/119) both high- and low-scoring, failed to include the names of the persons or entity that financed the document.

As appendicitis commonly affects younger individuals, it is possible that the decision to go to hospital is being made by a parent or guardian of the patient. This decision may be the result of a combination of factors, such as the severity of symptoms experienced by the patient, the health beliefs of the parent or guardian, and the quality of any information they may seek. Delays in presentation to hospital may increase the risk of complications, such as rupture; therefore, it is important that information regarding complications and warning signs is clearly described. Despite this clear need for quality, the scores achieved for the content domain were generally low. Only 67.2% (80/119) of websites mentioned warning signs to detect appendicitis, although 44.5% (53/119) of the websites did provide some form of advice in case of emergencies.

Hospitals contributed 45.4% (54/119) to the total database of websites, but 43% (23/54) of these failed to the specify qualitative risks of complications after surgery. It is also concerning that only 30.2% (36/119) of all websites provided a description of how complications may be handled. Over half of the websites (52%,28/54) from hospitals originated from the United States, possibly reflecting a degree of hesitancy by private hospitals to include information that may adversely affect customer decisions. This is further supported by the low number of websites reporting the incidence rate of complications. Although mortality rates are low for appendicitis, it is disappointing that this information was only provided by 4 websites, as this should be standard practice with the aim to fully inform patients.

To the best of our knowledge this is the only study to date to evaluate patient information regarding appendix surgery; therefore, it is not possible to evaluate how the EQIP tool compares to other scoring systems used for this condition. However, the EQIP tool has been described previously when evaluating the quality of information for gallstone disease (median EQIP score 15, IQR 13-18) [9], for clefts of the lip and palate (median EQIP score 19, IQR 16-22) [24], for bariatric surgery (median EQIP score 17, IQR 15-19) [25], for phalloplasty (median EQIP score 17.5, IQR 13-21) [26], for Dupuytren disease (median EQIP score 16, IQR 13-19) [27], for breast augmentation (median EQIP score 15, IQR 13-17) [28], donor information for living liver transplantation (median EQIP score 16, IQR 13-20) [10], and for COVID-19 (median EQIP score 18, IQR 15-20) [13]. The median score and IQR of this study are slightly higher than those in these studies. This may suggest that the general quality of information for appendicitis on the internet is higher than that of the previously studied diseases; however, this relatively high quality is likely due to the points gained in the structure domain in comparison to previous studies. The maximum score obtained in this study was 29, which was only achieved by 1 website and was still considerably lower than the maximum possible score of 36.

The first and second highest EQIP scores were achieved by websites from the United States and the United Kingdom, respectively. The United Kingdom also had the lowest-scoring website and lowest median among countries with more than one website. This indicates that the high-scoring websites in the United Kingdom are diluted by a majority of poor quality websites. The IQR range of scores for websites originating from Canada and Australia were much smaller, suggesting that, although there are fewer websites, they are generally of a higher quality.

Limitations

There were a number of limitations for this study. Identification of search terms with Google AdWords Keyword Planner only provides commonly used search phrases by the wider public and may not be able to truly predict the search patterns of individuals seeking health information. Another limitation is that only websites in English language were evaluated; thus, the conclusions drawn might not be representative of patient websites in other languages. Furthermore, we have described the use of the EQIP in relation to appendicitis although the tool was not originally created for this specific purpose, and therefore this may be considered as a limitation. However, as the EQIP tool has previously been shown to be robust and effective for a number of surgical conditions [9,10,26,28], it is reasonable to expect this to also hold true for appendicitis. One may argue that there is no clear reason for why tools such as EQIP exist to evaluate patient information. The purpose of this tool is to enable a method of categorizing information so that we may learn what areas can be improved upon to produce patient information websites of higher quality. Our next step is to use the information gathered from this paper to design a website that will include information which has been found to be regularly omitted from previous work. Using solely the Google search engine introduced another limitation, as it is possible that the search engine had listed results not simply by popularity, but also by the geographical location of the requesting computer. Therefore, although geolocation features were disabled, the websites extracted could have still been centered around a particular location or continent, preventing a truly representative analysis of the top websites used globally. Finally, it is important to note that the findings of this study act as a snapshot of a particular point in time when the search was used; however, while search engine results do change over time, we consider the findings of this study to be representative of the information available to patients.

Conclusions

In conclusion, the internet has become an essential source of information for our society. Our study showed that despite a growing body of web-based resources on appendicitis and appendicectomy, the currently available websites are generally of poor quality and inform patients inadequately. Although the clinicians responsible for each patient can provide patients with important, accurate, and relevant clinical information, the ability to direct patients to trustworthy internet resources may lead to better patient education and long-term outcomes. Online information is playing an increasingly significant role in patients’ attitude and can potentially affect willingness to accept and comply with medical advice [29]. With emergencies such as appendicitis, it is paramount to make emergency guidance widely available, especially when websites may be a patient’s first point of contact for information. Health care professionals should strive to educate patients on how to navigate and appraise internet-based resources in order to access the highest quality of information. Many studies have also identified similar problems and have not acted upon them, thus highlighting the urgent need to establish high quality websites and specific information evaluation tools to ensure the optimal patient education for a disease as common as appendicitis.