Efficacy of Quality Criteria to Identify Potentially Harmful Information: A Cross-sectional Survey of Complementary and Alternative Medicine Web Sites

Introduction

Online health information can harm as well as heal. Many quality criteria have been suggested to help consumers identify misleading, inaccurate, or harmful information. Objective quality criteria that offer a limited number of options are particularly promising since they are easier to assess. For example, it is easier to assess whether an author is identified than to determine whether the author is qualified. However, even seemingly objective quality criteria have proven unreliable without specific operational definitions [1]. Further, there is little evidence that these criteria, known as "technical criteria," actually filter out undesirable health information. The few studies that have attempted to evaluate technical criteria reported conflicting results [2-4]. If harmful information can be effectively identified, this should be publicized. If, on the other hand, currently available quality criteria cannot identify potentially harmful information, then we should caution consumers and work on finding other ways of identifying problematic information online.

In this study, we analyze Web sites that display information about complementary and alternative medicine (CAM). CAM includes "diverse medical and healthcare systems, practices and products that are not presently considered to be a part of conventional medicine," such as dietary supplements, aromatherapy, chiropractic, and homeopathy [5]. Assessing accuracy and quality of CAM Web sites poses unique challenges as there is less documented research on the efficacy of CAM products, yet use is common and the potential for harm remains. There is also no gatekeeper to control and monitor access to CAM. Consumers can choose the product and dosage without having to encounter a healthcare professional. In fact, patients often fail to report CAM use to their physicians [6]. On the other hand, consumers frequently turn to the Internet to answer questions about CAM, and trust and act upon what they see online [7]. However, CAM information online has been found to be commercially driven [8], to be poorly referenced [8], and to contain illegal claims [9], and it may therefore be dangerous to consumers [10]. The combination of accessible, unproven CAM therapies and poor quality online CAM information is dangerous.

"Accuracy is a function of whether a site reflects the use of … agreed-upon benchmark[s] such as clinical practice guidelines." [11] The accuracy of CAM information, which is often not evidence-based and lacks support from the peer-reviewed biomedical literature, is not testable. However, we can assess the potential harm of displayed information, even if we cannot verify its accuracy. Further, if information regarding the safety and efficacy of a product is available, it should be displayed.

Our previous work provides preliminary evidence that breast cancer Web sites that meet more technical quality criteria are less likely to contain false statements [12]. Motivated by a desire to help consumers, we sought to determine whether current technical quality criteria can identify potentially harmful CAM information.

Materials and Methods

Selection of Web Sites

Consumers use general-purpose search engines rather than medical sites or portals to find information, and most do not go beyond the first page of search results [13]. Therefore, we chose the ten most popular search engines (Table 1) to select Web sites that consumers are likely to encounter [14]. The three most popular herbs in the United States (in terms of dollars spent) [15], ginseng, ginkgo, and St. Johns wort, and their most common uses formed the search query. The following three queries were executed in each search engine on July 15, 2003: "ginseng and cancer," "ginkgo and memory loss," and "St. John's wort and depression." All Web sites listed on the first results page, including sponsored or paid links, were analyzed.

A Web site was included if it contained at least one sentence or phrase of health information on the search topic. Health information was defined as "information intended to be used to maintain or improve health, including to understand disease processes, health care issues, etc… to prevent, diagnose, or treat health problems, to be rehabilitated from the effect of diseases, or treatments, and to seek and select health care plans, providers, and other resources." [16] Duplicate URLs were removed. HTTrack [17], a Web site copier was used to permanently capture each Web site and every directly linked page.

Assessing Technical Quality Criteria

In prior work, we assessed inter-rater agreement for popular technical quality criteria [1]. We assessed the degree to which two raters agreed upon the presence or absence of 22 quality criteria selected from Eysenbach's systematic review [17] of a sample of 21 CAM Web sites. Our preliminary analysis showed poor inter-rater agreement on 10 of the 22 criteria. Therefore, we created operational definitions for each of the criteria, decreased the allowed choices, and defined a location to look for the information. As a result, 15 out of the 22 quality criteria had acceptable inter-rater agreement (kappa > 0.6).

For this study, one evaluator (MW) analyzed all Web sites for compliance with 15 technical quality criteria (Table 2) that we previously determined to be reliably assessable. Therefore, in this study we did not re-calculate inter-observer reliability for these technical criteria.

Assessing Potential Harm

First, a set of critical facts for each of the three herbs was determined by consensus of two clinically trained reviewers (SS, DS); please see appendices 1-3. The sets of critical facts were extracted from two independent sources of CAM information: the Physician Desk Reference (PDR) for Herbal Medicines [19] and the Sloan Kettering database of herbs [20]. After the sets of critical facts were determined, the CAM content displayed on each Web site was independently evaluated by both reviewers. Cases where reviewers disagreed were resolved by consensus. In order to minimize bias, materials identifying each Web site's origin, such as organization name, logo, footers, URLs and hyperlinks were removed. However, no changes were made to the design or layout.

In order to verify the concordance between reviewers, two additional clinically trained evaluators (validation reviewers), who were not aware of the study hypothesis or quality criteria tested, were given 30 randomly selected sites from the same sample looked at by primary reviewers (SS, DS). Inter-rater agreement between the validation reviewers was calculated. The validation reviewers were given the same critical facts documents as the primary reviewers and each validation reviewer assessed every site independently. After each reviewer independently evaluated the Web sites, inter-rater agreement was calculated between the two validation reviewers. Subsequently, cases of disagreement were resolved by consensus. A second inter-observer agreement measure was calculated between the pairs of reviewers (primary reviewers vs. validation reviewers) based on the consensus data.

Content on each page was scrutinized for the presence of misleading statements likely to cause physical harm (acts of commission) and for vital information that was missing (acts of omission). Commission may be thought of as a surrogate for accuracy, while omission has been referred to as completeness, coverage, or comprehensiveness [21]. We based our evaluation on the following framework, adapted from Markman [22]:

1. 1. Direct toxicity
   2. Interaction with conventional medical therapy
   3. Delay in diagnosis or conventional treatment
   4. Avoidance of conventional treatment
2. 1. Warnings
   2. Drug interactions
   3. Contraindications
   4. Side effects

Statements that suggest use of higher doses of herbs than recommended in the critical facts documents (appendices 1-3) were categorized as causing "direct toxicity." Statements suggesting that the herb protects against disease and encouraging patients to self-medicate instead of seeing a physician were placed in the "delay in diagnosis or conventional treatment" category. Statements that project herbs as an "alternative to conventional treatment" (for example, "the herb is the first choice of treatment for the disease") were categorized as potentially causing "avoidance of conventional treatment." Statements that suggested using herbs with medications known to have drug interactions (for example, using St. John's wort with monoamine oxidase inhibitors) were classified as causing potential harm due to "interaction with conventional therapy." However, while evaluating potential physical harm due to omission of information about interactions, we did not expect Web sites to list all the drug interactions listed in the critical facts documents. Web sites that noted at least one drug interaction were considered not to omit drug interaction information. Web sites with vague statements such as "there are many interactions," were categorized as having "omitted drug interactions." Potential physical harm was present if any error of commission or omission was found.

We recognize that in addition to physical harm due to either commission or omission, CAM information on the Internet may cause other types of harm, such as emotional and financial. Emotional harm may occur because of inaccurate perception of disease or conventional therapy such as exaggeration of side effects of conventional treatment and presentation of alternative treatment as a "natural cure." Financial harm may be caused by the purchase of ineffective or harmful yet expensive CAM products. However, we did not evaluate emotional and financial harm in this study because of the inherent subjectivity involved, and difficulty in quantifying and assessing such measures.

Statistical Analyses

The dichotomous (yes/no) dependent variables were: 1) physical harm from commission and 2) physical harm from omission. The independent variables were also dichotomous and consisted of the 15 technical quality criteria listed in Table 2. In addition, these 15 criteria were grouped into 5 categories [23]: authority, transparency and honesty, updating of information, editorial policy, and other. Web sites were classified into two groups based on whether they complied with the median number of quality criteria. The first group complied with six or fewer technical quality criteria, the second group complied with more than six technical quality criteria.

Inter-observer agreement measures were calculated to assess a) the degree to which validation reviewers agreed among themselves in their assessments of these dichotomous dependent variables (Table 3) and, b) the degree to which the validation reviewers agreed with the primary reviewers (Table 4). Cohen's kappa (K) is a commonly used measure of inter-observer agreement between two observers for dichotomous data. However, because K is affected in complex ways by the presence of bias between observers and by the distributions of data across the categories [24], we computed the prevalence-adjusted bias-adjusted kappa (PABAK), the bias index (BI) and the prevalence index (PI), as well as K, as recommended by Byrt et al [24].

The bias index (BI) is defined as the difference between the proportions of "Yes" for the two raters. The prevalence index (PI) is defined as the difference between the probability of "Yes" and the probability of "No." A BI close to 0 indicates less bias, while values closer to 1 (absolute value) indicate greater bias. Similarly, a PI close to 1 (absolute value) indicates high prevalence, while a PI closer to 0 indicates lower prevalence. The BI then measures the degree to which one reviewer tends to identify more or fewer occurrences than the other, while the PI measures the degree to which "Yes" agreements or "No" agreements predominate. The PABAK index of agreement between two observers is a measure that adjusts for both bias and prevalence. Although the derivation of the PABAK index is somewhat more complex, in practice it can be calculated as 2P₀ - 1, where P₀ is the proportion of observed agreement. Consequently, PABAK ranges from -1 to +1 and like K, a value of 0 represents no better than chance agreement, while magnitudes approaching 1 indicate maximal agreement.

Chi-square was calculated for each pairing of an independent variable with a dependent variable. Given the large number of statistical tests performed, significance was set at α<0.01. All analyses were performed using SPSS 11.0 statistical software.

Results

A total of 546 Web sites were retrieved. After removing duplicates and checking for eligibility, 150 Web sites remained: 54 for the query "ginseng and cancer," 46 for "ginkgo and memory loss," and 50 for "St. John's wort and depression."

Technical Quality Criteria

Most Web sites did not comply with technical quality criteria. On average, a Web site complied with 6.3 (SD±2.6) of 15 criteria. One site failed to comply with any criteria, while three sites complied with 13 criteria. Only 27% of sites disclosed authorship, 36% provided references and 6% mentioned an editorial review process. Table 2 shows the number of Web sites that complied with each of the 15 quality criteria.

Assessing Potential Harm: Agreement among Reviewers

As shown in Table 3, agreement between the two evaluation reviewers was high (all PABAK > 0.67). Although there was little bias, there was a strong prevalence effect. Therefore, the two validation reviewers had a high degree of agreement for all measures of harm from commission and omission. Similarly, as shown in Table 4, consensus agreement between the primary and validation reviewers was also high (all PABAK > 0.73).

Potential Harm

Potential physical harm from omission was more prevalent than from commission (97% vs. 25%, Table 5). However, a substantial number of Web sites (25%) displayed statements that could lead to physical harm. Statements that may cause toxicity if acted upon (direct toxicity) were present in 13% of CAM Web sites, while 7% of Web sites included statements encouraging the avoidance of conventional therapies. Eight percent of sites included information that may lead to harm from interactions if the advice were followed. Most CAM Web sites (97%) omitted vital information such as contraindications (89%) and drug interactions (83%).

Technical Quality Criteria

We found that individual technical quality criteria did not identify sites with the potential to cause physical harm from commission or omission (Table 6). Similarly, when technical criteria were grouped into categories (such as authority, transparency and honesty, etc.), no significant association was found with potential physical harm (Table 7). Even when Web sites were classified into two groups, those complying with more criteria (≥ 6) versus fewer criteria (<6), there was no significant relationship. Overall, 44 hypotheses were tested but none were significant at the α<0.01 level, despite our study having 0.80 power to detect significance. Surprisingly, the presence of two quality criteria where a significant association was found at α<0.05 ("sources clear" and "editorial review process") indicated a greater chance of potential harm; the reverse of their original intent. However, it is possible that these two significant results may be due to chance since we conducted numerous statistical analyses.

Top Level Domain

We also explored the relationship between top level domain and potential harm. Seventy-seven percent of the 150 Web sites analyzed were commercial (.com), 10% organizational (.org), 7% network (.net), 3% educational (.edu), 2% governmental (.gov) and 1% unknown (numerical IP address only). Fisher's exact test statistic was calculated as expected values in some cases were <5, and significance was set at α = 0.05 level. Only the network top level domain had a significant relationship with physical harm from omission (Table 8). Of the 10 Web sites with the network top level domain, 20% did not contain harm from omission. In contrast, only 2% of sites that had a top level domain other than network did not have harm from omission (p<0.04). However, there was no statistically significant relationship between network and non-network sites with respect to physical harm from commission. Although there were few educational and government sites in our study, it is notable that there were no identified cases of potential harm by commission in these sites. As most Web sites were commercial, it is difficult to draw meaningful conclusions from this analysis.

Intent to Sell Products

In order to explore the relationships between Web sites that sold products and those that did not, two evaluators independently revisited each Web site and identified Web sites that allowed the ordering of products. Agreement between reviewers was high (K=0.95). Fifty-three percent of Web sites (n=79) sold products. There was no significant relationship between selling products and potential harm due to omission (P=0.56) or commission (P=0.02). Although not statistically significant at the α = 0.01 level, selling products was actually related to less harm from commission, the reverse of what we would expect. In fact 63% (n=24) of the harmful Web sites from commission were found on sites that did not sell products, while 37% (n=14) were found on Web sites that sold products. Therefore, in our sample there does not appear to be more harmful information on sites that sell products.

Discussion

We found that most CAM Web sites were potentially harmful either by displaying statements which could cause harm, or by omitting vital information. However, our data suggest that available technical quality criteria fail to identify potentially harmful information online.

We found that one quarter of CAM Web sites present information that may cause physical harm if acted upon. These sites encouraged consumers to avoid conventional therapy, presented information on products that may be directly toxic, or presented information on products that may cause interactions with conventional medications. This is potentially dangerous because consumers have easy access to CAM products online and act upon what they see on the Internet [7], often do so without the knowledge or advice of clinicians [25].

Almost all (97%) CAM Web sites omitted vital warnings, drug interactions, contraindications, or adverse reactions. This is concerning because many consumers perceive "natural" products as safe. Further, many herbs that may be safe when used alone interact with conventional medications.

Previous studies have found scientific references [4], absence of financial interest [4], display of copyright [2], and display of editorial policy [3] to correlate with information accuracy. Technical quality criteria evaluated in this study may be unsuitable for CAM information as they seek to identify accuracy, which is difficult to determine for CAM. Surprisingly, even generally accepted measures of content quality such as disclosure of authorship and updating of information had no relationship to potential harm. Other researchers have also encountered difficulty in developing guidelines to evaluate CAM information [26].

Our previous study of breast cancer information online found that sites which complied with 3 JAMA benchmarks [27] (authorship, references, currency, and disclosure) were more accurate than lower quality sites (3 JAMA benchmarks) [12]. However, in this sample of CAM Web sites we found no such relationship for potential harm resulting from commission (p=0.31) or omission (p=0.21). We are forced to question the assumption, at least for CAM information, that consumers can be taught to discern good content from bad by looking at domain-independent quality criteria. Recommending such criteria may convey a false sense of security, inadvertently causing consumers to trust harmful CAM websites. Although the technical criteria we assessed had no relationship to potential harm, other criteria or tools not tested may have some value.

Five Web sites contained no harmful information from either commission or omission at the time of our study (Table 9). Four of the five best performing Web sites were retrieved from a search for St. John's wort, and one from a search on ginseng. One of these Web sites was from the American Cancer Society. However, the remaining four Web sites were from commercial or for-profit entities, two of which sold products. We note that Web site content changes frequently. Therefore, it is difficult to endorse any list of Web sites.

The major limitation of our study is the inherently subjective domain. Whether or not information has the potential to harm a consumer is a subjective clinical judgment which defies strict definition. However, relatively high inter-observer agreement among clinically trained reviewers suggests that our definitions were consistent.

Our study was also limited by our sample, which was restricted to Web sites displaying information about three popular herbs. Searches on other herbs or different alternative therapies may have different results. Also, we did not evaluate all possible technical quality criteria. Instead, we evaluated only criteria that were used in three or more studies as reviewed by Eysenbach et al [18] and were found to be reliably assessable using pre-determined operational definitions [1]. It is possible that other quality criteria will be more effective.

Since the primary reviewers (SS, DS) were aware of the study hypotheses, they may have been biased by this knowledge. However, inter-observer agreement between the primary and validation reviewers (who were unaware of the hypotheses) was high. Therefore this potential bias appears to have minimal effect on the results.

As we search for quality measures, we must keep in mind that some potentially useful criteria are easily manipulated. For example, one study found sites that claimed copyright were more accurate [2]. Such very specific and objective criteria are appealing since they may be automatically assessed using software, and evaluated by consumers by simply searching for the word "copyright" or © symbol. However, it is easy for site builders to claim copyright without changing the health information displayed on their site.

Although we restricted our analysis to individual sites, consumers may not make health-care decisions on the advice of one site, but rather on the collective information learned, confirmed or refuted from a multitude of online sources. Future work can assess the degree to which confirmatory evidence present on a range of sites can screen out undesirable information. In addition, it would also be important to understand why consumers search for CAM information. After all, some may turn to CAM only after conventional treatment fails, whereas others may reject traditional therapies.

The Internet provides a constantly changing, endless variety of information from innumerable sources. Ideally, we would like to empower consumers to evaluate health information for themselves. Currently available technical quality criteria, however, are not adequate to evaluate CAM information. For the time being, it may be prudent to recommend that consumers looking for CAM information online rely on known, authoritative providers of information. With this in mind, we must continue to search for ways of alerting consumers to potentially harmful information without restricting them to known sources.