Original Paper
Abstract
Background: Concerns over online health information–seeking behavior point to the potential harm incorrect, incomplete, or biased information may cause. However, systematic reviews of health information have found few examples of documented harm that can be directly attributed to poor quality information found online.
Objective: The aim of this study was to improve our understanding of the quality and quality characteristics of information found in online discussion forum websites so that their likely value as a peer-to-peer health information–sharing platform could be assessed.
Methods: A total of 25 health discussion threads were selected across 3 websites (Reddit, Mumsnet, and Patient) covering 3 health conditions (human immunodeficiency virus [HIV], diabetes, and chickenpox). Assessors were asked to rate information found in the discussion threads according to 5 criteria: accuracy, completeness, how sensible the replies were, how they thought the questioner would act, and how useful they thought the questioner would find the replies.
Results: In all, 78 fully completed assessments were returned by 17 individuals (8 were qualified medical doctors, 9 were not). When the ratings awarded in the assessments were analyzed, 25 of the assessments placed the discussion threads in the highest possible score band rating them between 5 and 10 overall, 38 rated them between 11 and 15, 12 rated them between 16 and 20, and 3 placed the discussion thread they assessed in the lowest rating band (21-25). This suggests that health threads on Internet discussion forum websites are more likely than not (by a factor of 4:1) to contain information of high or reasonably high quality. Extremely poor information is rare; the lowest available assessment rating was awarded only 11 times out of a possible 353, whereas the highest was awarded 54 times. Only 3 of 78 fully completed assessments rated a discussion thread in the lowest possible overall band of 21 to 25, whereas 25 of 78 rated it in the highest of 5 to 10. Quality assessments differed depending on the health condition (chickenpox appeared 17 times in the 20 lowest-rated threads, HIV twice, and diabetes once). Although assessors tended to agree on which discussion threads contained good quality information, what constituted poor quality information appeared to be more subjective.
Conclusions: Most of the information assessed in this study was considered by qualified medical doctors and nonmedically qualified respondents to be of reasonably good quality. Although a small amount of information was assessed as poor, not all respondents agreed that the original questioner would have been led to act inappropriately based on the information presented. This suggests that discussion forum websites may be a useful platform through which people can ask health-related questions and receive answers of acceptable quality.
doi:10.2196/jmir.5051
Keywords
Introduction
Background
Over the past 2 decades in England and Wales, consultation rates within general practitioners’ (GP) surgeries have increased from approximately 220 million in 1995 to 300 million in 2008 and are estimated currently at approximately 340 million [
]. Over the last decade, the number of attendances at accident and emergency (A&E) units in the National Health Service has increased more than 30%, from 14 million per year prior to 2003/2004 to 21.7 million in 2013/2014, and numbers are continuing to grow [ ]. Pressure on GP surgeries may be one of the reasons for the increasing pressure on hospital A&E departments: 22% of patients report that it is not easy to get through to their GP’s surgery on the telephone and 9.8% of people who are unable to get a convenient GP appointment go to an A&E walk-in center instead [ ].Since 2008, online health-seeking information in the United Kingdom has increased dramatically, from 18% of UK adults saying they looked for health information online in 2008 to 43% in 2013, with an increase of 59% among the 25 to 29 years age group [
]. Health information seeking represented one of the fastest growing areas of Internet use measured by the UK government during the period from 2008 to 2013 and in 2014; 8% of people aged 16 to 35 years and 15% of those aged 55 to 64 years made a GP appointment using the Internet. The United Kingdom is the second highest country globally for Internet health searches; in a recent survey, “Google my symptoms” was a more common first action than “book a doctor’s appointment” or “visit a pharmacy for advice” [ ]. In 2014, the number of health searches carried out in the United Kingdom increased by 19% [ ].Quality Considerations for Online Health-Seeking Behavior
Supporting individuals to shift at least some of their health-seeking behavior from a face-to-face consultation with a medically qualified practitioner to seeking information online, both before and following diagnosis, provides opportunities to relieve the pressure on GP surgeries and A&E departments. However, it is also dependent on the information found online being of sufficiently high quality that following it does not pose a health risk. Prior studies of health information online have shown that it is of variable quality [
- ]. Although much concern has been expressed over this [ - ], few examples of actual rather than potential harm have been documented [ , ].Internet users often seek disease-specific information [
, ], including information that will enable them to diagnose a particular health problem [ ]. Because trusted brands play an important role in health-seeking behavior [ - ], one way to make health-seeking behavior more comfortable for the Internet user may be to encourage them to turn to known and trusted websites when seeking health information, leveraging trusted brands to help them feel confident about the information they find there. If the brand is not health-specific, but is a source of information on a range of topics that the patient already trusts, they may be more likely to turn to it for information when they engage in online health-seeking behavior for the first time. Respondents to a short study on health information-seeking behavior during the 2014-2015 Ebola crisis in West Africa largely did not begin to use new modes of communication to seek out health information. Instead, they searched for health information through platforms and media they were already familiar with, turning first to trusted health and information brands, such as the World Health Organization, the BBC, and government ministries of health in addition to knowledgeable friends [ ].Characteristics to Support Health-Seeking Behavior
Online discussion forums have a number of characteristics that could benefit online health information seekers. Discussion is known to enable better learning and absorption of knowledge [
, ] and this has been identified as a benefit of discussion forums in general [ ] and of online discussion forums specifically [ ]. The emergence of Web 2.0 has provided new opportunities to gain and share knowledge about health issues [ - ] and discussion forums display positive attributes relating to all 4 website characteristics (source, medium, message, and receiver) that have been identified as important to engendering trust during online health-seeking behavior [ ]. In particular, discussion forums can act as both the medium for and source of health information. Because both doctors and friends can be accessed through online discussion forums, the Internet should not be seen as a competing category to face-to-face interaction with such sources, but rather an enabler of it.A weakness of the existing literature is the tendency to approach the Internet as if it is a homogenous environment where every website can be trusted or mistrusted equally until trust is added on by accreditation seals or source authority. This does not consider whether some characteristics, such as voted discussion forums that offer the ability to counter a previous post with more accurate information or to fill-in the information missing from a previous incomplete answer, make discussion forums inherently more conducive to the transfer of good quality information than other types of websites. All spaces exert influences on the choices that people make in those spaces. The more designers, owners, operators, and users of online discussion forums are aware of what these influences are likely to be, the more able they will be to consider how they can influence users’ choices [
, ].The aim of this study is to provide an assessment of the quality and quality characteristics of information found in online discussion forums so that doctors, patients, and health care policymakers can better understand the online discussion forum environment and the information found there.
Methods
Selection Criteria
Our study involved UK-qualified medical doctors and UK (London)-based nonmedically qualified individuals assessing the information found in 3 online discussion forums (Reddit [
], Mumsnet [ ], and Patient [ ]) relating to 3 health conditions. We selected 3 health conditions that affect a high number of individuals in the United Kingdom: diabetes, chickenpox, and human immunodeficiency virus (HIV). According to the most recent figures from Public Health England, an estimated 107,800 individuals in the United Kingdom were living with HIV in 2013 [ ]. An estimated 3.2 million (7%) of the UK population is living with diabetes [ ], of whom 10% have type 1 diabetes and the remaining 90% have type 2. An estimated 90% of all Britons will have had chickenpox by the age of 15 [ ], although no exact figures on infection exist for the United Kingdom because not all cases receive clinical attention. It is important to note that although US health policy positions chickenpox as a dangerous disease for which childhood vaccination is recommended [ ], this is not the case in the United Kingdom, where it is positioned as a mild childhood disease for which vaccination is only necessary for high-risk groups. Professional medical consultation is considered necessary only in cases of complications listed on the website of the UK National Health Service [ ].There is evidence that a high volume of health-seeking information occurs in relation to all 3 conditions. Diabetes and HIV both feature in the top 10 most searched for diseases on Google (diabetes at number 2 with more than 9 million monthly global searches in 2013, HIV at number 4 with more than 6 million, and acquired immune deficiency syndrome [AIDS] at number 6 with 5 million) [
]. Although chickenpox appears lower on the list (at 43 with more than half a million global monthly searches), it is one of only a handful of communicable diseases found there and is the most significant childhood disease in the United Kingdom. Vaccination is uncommon (ie, the majority of the population are likely to catch it), but many parents do seek professional advice when their children develop symptoms. Therefore, it was likely that a considerable volume of health information would exist online for these 3 conditions and that forums where health is discussed were likely to have discussion threads related to them.Selection of Discussion Forum Websites and Discussion Forum Threads
Three online discussion forum websites were selected based on their popularity and common usage by the UK population (rather than among specialist interest groups or social media superusers). These included 2 general discussion websites (Reddit and Mumsnet) and a health-specific site (Patient). We investigated each of the 3 websites to see if their message forums had existing discussion threads related to these conditions and found that all 3 health conditions were discussed on all 3 forums.
We selected specific discussion threads for the survey subjectively by undertaking a basic search inside each selected website on the chosen health conditions between February 15 and 17, 2015, and reading through the returned results to find questions for which we felt the original poster could (and probably should) have sought advice from a qualified medical practitioner. Discussion threads were rejected if the question did not require a medical or scientific reply (eg, a diabetic asking others whether they thought disclosing his diabetes on job applications would be disadvantageous) and if the question had received less than 2 replies. We selected 25 suitable questions (Reddit: n=9; Mumsnet: n=8; Patient: n=8; diabetes: n=8; HIV: n=9; chickenpox: n=8) according to the order they appeared in the search results, favoring discussions in which the question had been posted within the previous 12 months. If no suitable questions matching these criteria appeared within the first 50 search results, questions posted earlier or from beyond the first 50 search results were selected. Eleven of 25 questions selected appeared to be prediagnosis and were asking if symptoms they or a friend/family member were exhibiting might be indicative of a certain condition (eg, diabetes), 9 of 25 appeared to be postdiagnosis asking for advice on how to act in light of a condition (eg, whether certain exercise routines were suitable for diabetics), and 4 were asking for general advice on topics such as vaccination. Each question and the discussion thread that followed it was then assessed by more than one assessor. In total, 79 assessments were returned (mean 3.2, range 2-7 for the 25 questions).
Selection of Study Participants
We aimed to have the information in the forums assessed for quality by UK-qualified medical doctors (1 GP, 2 hospital infection specialists, 1 hospital-based diabetes consultant, 4 who did not give exact details) and also by London-based individuals who were not medically qualified, but who had experience with the health issue being discussed as a patient or as a carer of a patient.
The majority of the doctors were recruited through Ashford and St Peter’s Hospital, which has links with the Health, Human Body and Behaviour (H2B2) program at Royal Holloway, University of London. Two other medical doctors, known personally by the authors, were also invited to participate.
The nonmedically qualified participants for diabetes were recruited by contacting the chairs of 2 (offline) support groups for diabetics. Contact addresses for support groups are given on the website of the diabetes support charity Diabetes UK, which enabled group coordinators to be contacted personally and asked to take part. For chickenpox, parents of children in the common age group for contracting chickenpox (age 2-10 years) were recruited through the Parents and Friends Associations of 2 local West London schools (Lovelace Primary School in Chessington and Putney Girls High School). The Terrence Higgins Trust, a charity that supports people living with AIDS and HIV, was also approached and asked to contact people living with HIV who would be willing to take part, but they did not reply. As such, no HIV-positive patients participated and the questions relating to HIV were answered by doctors only. Participants were self-selecting and, therefore, may be subject to selection bias. Demographic data collected on the participants was minimal: it recorded whether or not they were medically qualified and confirmed they were adults older than age 18 years before taking part, but no other particulars were recorded because these were not deemed necessary for this part of the study. Participants were given the option of taking part anonymously; 12 chose to disclose no other information than their level of qualification. Five of the doctors, but only one of the nonmedically qualified participants, provided a contact email address.
Participants were sent, by email, a list of paired URL links for each discussion thread they were asked to assess. One linked to the actual online forum discussion thread, which they saw in situ with no modification made to it for the sake of the study, and the other to an online assessment form. Each discussion thread was assessed against the same criteria. The discussion threads were assessed according to 5 criteria and the participant responded by rating the information from highest quality to lowest quality (range 1-5) on:
- The medical/scientific accuracy of the information found there [ , , ];
- The medical/scientific completeness of the information [ , , ];
- How sensible they considered the answers provided to be;
- Whether they thought someone reading the website would act appropriately based on the information provided; and
- How useful they felt the answers given would be to the original poster.
An additional question was asked to check that the respondents found the discussions easy to follow; only 4 assessments recorded any level of difficulty in following the discussions.
The responses assessed perceived factual quality of the answer (accuracy and completeness), gave a subjective assessment on that information (how sensible was it?), and subjective assessments of how the reader might respond (would they act appropriately and would they find the information useful?). We included this differentiation in the questions because, although many previous studies have criticized online health information for being of poor or variable quality [
, , ], far less have found actual evidence of poor information leading to inappropriate or dangerous health decisions being made [ , , , ]. Because even fewer studies focused on how likely it is that Internet discussion forum readers will take action based on the information they found there, exploring perceptions around this is of particular interest.In each assessment, the discussion threads could be assigned 1 of 5 rating values, for which the highest (1) related to the best quality information and the lowest (5) to information that was considered to be inaccurate or ill advised. The criteria for marking were consistent across each health topic and website, and provided a potential overall score of between 5 (5*1, top rating for each criteria) and 25 (5*5, lowest rating for each criteria) to each discussion forum thread.
Participants were invited to participate between May 12 and June 4, 2015, and were given 2 to 3 weeks to reply. The final survey assessments were accepted on June 13, 2015. Participants were sent a mean 8 (range 7 to 25) discussion threads to assess (each of which required assessments of the 5 separate criteria) based on their particular area of experience or medical expertise, with only one participant—a recently retired GP—offered all surveys to complete. The assessments were completed, and results collected, using the free online survey software SmartSurvey. A generic version of the assessment questionnaire is available in
.Results
Survey Data Returned
A total of 79 assessments of discussion threads were returned at least partially completed (as of June 13, 2015). For 78 assessments, all 5 criteria were assessed and rated, but on 1 assessment, 2 of the criteria were skipped. Seventeen separate individuals took part, 8 of whom identified themselves as medically qualified. The qualified medical doctors completed 58 of 79 (73%) returned surveys and 21 of 79 (27%) surveys were completed by the nonmedically qualified respondents.
- show the data from all survey responses by health condition and website. includes a visual representation of the quality scores and a reference to the actual question as it appeared on the Internet discussion forum website (as the assessors saw it when they made their assessment). shows a visual comparison of ratings in each category across all websites and health conditions.
Website, question, and respondent | Quality score (1=high, 5=low) | Total | ||||||
Accurate | Complete | Sensible | Appropriate | Useful | ||||
Q1. First party since being diagnosed, need advice? | ||||||||
Doctor 1 | 2 | 1 | 1 | 1 | 1 | 6 | ||
Doctor 2 | 2 | 2 | 1 | 1 | 1 | 7 | ||
Public 9 | 2 | 2 | 1 | 2 | 2 | 9 | ||
Q2. Advice for exercise and midnight lows? | ||||||||
Doctor 1 | 2 | 3 | 3 | 2 | 2 | 12 | ||
Doctor 2 | 2 | 2 | 1 | 1 | 1 | 7 | ||
Mumsnet | ||||||||
Q3. Are anger outbursts normal with diabetes? | ||||||||
Doctor 1 | 2 | 3 | 2 | 2 | 2 | 11 | ||
Doctor 2 | 3 | 3 | 2 | 2 | 2 | 12 | ||
Public 1 | 2 | 2 | 2 | 2 | 2 | 10 | ||
Public 9 | 1 | 4 | 3 | 3 | 2 | 13 | ||
Q4. Signs of diabetes or paranoid Mummy? | ||||||||
Doctor 1 | 2 | 2 | 2 | 3 | 3 | 12 | ||
Doctor 2 | 2 | 2 | 1 | 1 | 1 | 7 | ||
Public 1 | 2 | 3 | 2 | 2 | 2 | 11 | ||
Patient | ||||||||
Q5. Longer to get over a cold with diabetes? | ||||||||
Doctor 1 | 5 | 4 | 4 | 3 | 3 | 19 | ||
Doctor 3 | 3 | 3 | 2 | 2 | 2 | 12 | ||
Public 2 | 3 | 1 | 3 | 3 | 4 | 14 | ||
Public 3 | 4 | 3 | 3 | 2 | 2 | 14 | ||
Q6. Can this be diabetes? | ||||||||
Doctor 1 | 2 | 3 | 3 | 3 | 3 | 14 | ||
Public 2 | 3 | 1 | 3 | 2 | 4 | 13 | ||
Q7. Do I have Type 1 diabetes? | ||||||||
Doctor 1 | 2 | 2 | 1 | 1 | 1 | 7 | ||
Doctor 2 | 2 | 2 | 1 | 1 | 1 | 7 | ||
Public 1 | 2 | 2 | 2 | 2 | 2 | 10 | ||
Q8. Diabetes: advice please? | ||||||||
Doctor 1 | 2 | 3 | 2 | 3 | 2 | 12 | ||
Public 2 | 3 | 1 | —a | —a | 4 | — |
a Scores are missing because respondent did not answer for these criteria.
Website, question, and respondent | Quality score (1=high, 5=low) | Total | ||||||
Accurate | Complete | Sensible | Appropriate | Useful | ||||
Q9. I found out I had HIV; not clear about the stage | ||||||||
Doctor 1 | 2 | 3 | 2 | 2 | 2 | 11 | ||
Doctor 4 | 2 | 2 | 2 | 2 | 2 | 10 | ||
Q10. HIV and depression | ||||||||
Doctor 1 | 3 | 5 | 4 | 4 | 5 | 21 | ||
Doctor 4 | 3 | 4 | 3 | 3 | 2 | 15 | ||
Q11. Question about HIV and personal fitness | ||||||||
Doctor 1 | 2 | 2 | 2 | 1 | 1 | 8 | ||
Doctor 4 | 2 | 1 | 1 | 1 | 1 | 6 | ||
Q12. FAQ: Worried? Risk, testing, and anxiety | ||||||||
Doctor 1 | 1 | 2 | 1 | 1 | 2 | 7 | ||
Doctor 4 | 1 | 2 | 1 | 2 | 1 | 7 | ||
Mumsnet | ||||||||
Q13. Babysitter has just announced he’s HIV positive | ||||||||
Doctor 1 | 3 | 2 | 3 | 3 | 4 | 15 | ||
Doctor 4 | 3 | 3 | 2 | 2 | 2 | 12 | ||
Q14. Question about HIV (and partner) | ||||||||
Doctor 1 | 3 | 2 | 3 | 3 | 3 | 14 | ||
Doctor 4 | 2 | 3 | 2 | 2 | 2 | 11 | ||
Q15. Children and HIV | ||||||||
Doctor 1 | 3 | 2 | 3 | 2 | 3 | 13 | ||
Doctor 4 | 2 | 4 | 2 | 3 | 3 | 14 | ||
Patient | ||||||||
Q16. HIV question | ||||||||
Doctor 1 | 2 | 2 | 2 | 2 | 2 | 10 | ||
Doctor 4 | 2 | 3 | 1 | 2 | 2 | 10 | ||
Q17. HIV infection: intestinal yeast after 4 months? | ||||||||
Doctor 1 | 3 | 3 | 3 | 3 | 4 | 16 | ||
Doctor 4 | 4 | 3 | 3 | 2 | 2 | 14 |
Website, question, and respondent | Quality score (1=high, 5=low) | Total | ||||||
Accurate | Complete | Sensible | Appropriate | Useful | ||||
Q18. Is this chickenpox? Help!! | ||||||||
Doctor 1 | 5 | 4 | 4 | 5 | 5 | 23 | ||
Doctor 4 | 3 | 3 | 3 | 2 | 2 | 13 | ||
Doctor 5 | 4 | 4 | 3 | 3 | 4 | 18 | ||
Doctor 6 | 5 | 4 | 4 | 2 | 3 | 18 | ||
Public 4 | 3 | 4 | 3 | 2 | 3 | 15 | ||
Public 5 | 4 | 2 | 2 | 2 | 2 | 12 | ||
Public 6 | 3 | 2 | 2 | 1 | 2 | 10 | ||
Q19. Did you give your child the chickenpox vaccine? | ||||||||
Doctor 1 | 2 | 2 | 2 | 1 | 2 | 9 | ||
Doctor 4 | 2 | 2 | 1 | 2 | 1 | 8 | ||
Doctor 5 | 2 | 2 | 1 | 1 | 1 | 7 | ||
Doctor 6 | 3 | 3 | 4 | 2 | 3 | 15 | ||
Public 4 | 3 | 3 | 3 | 3 | 3 | 15 | ||
Public 5 | 2 | 2 | 1 | 1 | 1 | 7 | ||
Public 6 | 3 | 4 | 3 | 3 | 3 | 16 | ||
Q20. Chickenpox: why more dangerous to adults? | ||||||||
Doctor 1 | 3 | 3 | 3 | 3 | 4 | 16 | ||
Doctor 5 | 3 | 4 | 3 | 3 | 4 | 17 | ||
Doctor 7 | 3 | 3 | 3 | 3 | 3 | 15 | ||
Q21. Chickenpox: is 5 months too young to expose? | ||||||||
Doctor 1 | 3 | 3 | 3 | 3 | 3 | 15 | ||
Doctor 4 | 2 | 2 | 2 | 2 | 2 | 10 | ||
Doctor 5 | 5 | 4 | 4 | 5 | 5 | 23 | ||
Public 4 | 3 | 3 | 3 | 3 | 3 | 15 | ||
Public 7 | 3 | 3 | 2 | 2 | 2 | 12 | ||
Public 8 | 4 | 4 | 4 | 3 | 3 | 18 | ||
Q22. Has your child had the chickenpox vaccine? | ||||||||
Doctor 1 | 3 | 4 | 3 | 3 | 3 | 16 | ||
Doctor 4 | 2 | 2 | 2 | 1 | 2 | 9 | ||
Doctor 5 | 3 | 3 | 3 | 3 | 3 | 15 | ||
Public 6 | 3 | 4 | 3 | 3 | 3 | 16 | ||
Public 7 | 3 | 2 | 4 | 3 | 3 | 15 | ||
Q23. Is it normal to be so very ill with chickenpox? | ||||||||
Doctor 1 | 4 | 4 | 3 | 2 | 3 | 16 | ||
Doctor 4 | 2 | 2 | 3 | 1 | 2 | 10 | ||
Doctor 8 | 2 | 2 | 1 | 1 | 2 | 8 | ||
Public 4 | 3 | 3 | 1 | 3 | 3 | 13 | ||
Q24. Should toddler get the chickenpox vaccine? | ||||||||
Doctor 1 | 2 | 2 | 2 | 3 | 3 | 12 | ||
Doctor 4 | 2 | 3 | 2 | 2 | 2 | 11 | ||
Doctor 5 | 1 | 2 | 1 | 3 | 4 | 11 | ||
Q25. Strange symptom with chickenpox | ||||||||
Doctor 1 | 4 | 4 | 4 | 2 | 4 | 18 | ||
Doctor 4 | 3 | 3 | 3 | 2 | 2 | 13 | ||
Doctor 5 | 3 | 4 | 2 | 3 | 2 | 14 |
On average (excluding the retired GP), medically qualified respondents completed 5 surveys each and nonmedically qualified respondents completed 2 to 3 assessments each. When asked, the reason why respondents did not complete all assessments they were offered was lack of time. In total, this provided 393 criteria ratings across all 79 assessments (see
).Condition | Discussion forum, n | Total, n | ||
Mumsnet | Patient | |||
Diabetes | 25 | 35 | 53a | 113 |
HIV | 40 | 30 | 20 | 90 |
Chickenpox | 85 | 75 | 30 | 190 |
Total | 150 | 140 | 103 | 393 |
a Two questions on a survey were skipped.
Of the 353 assessments made overall, assessors rated the majority as a score of 2 (n=149) or 3 (n=137) on the scale of 1 to 5. Some information was rated the highest score of 1 (n=54) and a smaller proportion was rated the lowest scores (n=42 for a score of 4; n=11 for a score of 5). The lowest possible rating was given only 11 times out of 393 (2.8%) across the entire survey in comparison to 54 instances (13.5%) in which the highest possible rating was given. No discussion thread was given the lowest possible rating across all 5 categories (which would have given it an overall score of 25 suggesting that the information was inaccurate, incomplete, and likely to lead to harm) or the highest possible rating (which would have given it an overall score of 5 suggesting that the information was entirely accurate, complete, and would lead to appropriate action being taken).
When ratings were considered across all 5 criteria on which discussion forums were assessed and grouped into 4 value groups (5-10: threads were predominantly given 1 of 2 highest ratings for all criteria; 11-15: threads were largely given the high-middle ratings; 16-20: threads were often rated in the lower categories; 20-25: threads were assessed poorly on all 4 criteria) across the 79 surveys completed, 25 scored between 5 and 10, 38 scored between 11 and 15, 12 scored between 16 and 20, and 3 scored between 21 and 25. The one assessment that was only partially completed and could not be awarded an overall score was excluded from this stage.
High ratings were awarded more often than low ratings by a factor of 4:1. Ratings of 1 or 2 were awarded 203 of 393 times (51.7%), whereas ratings of 4 or 5 were awarded 53 of 393 times (13.5%). Overall ratings of 5 to 10 or 11 to 15 were awarded 63 of 78 times (81%), whereas ratings of 16 to 20 or 21 to 25 were awarded 15 of 78 times (19%).
How Is “Bad” Information “Bad”?
Previous studies tended to assess information against a single criterion for quality (ie, completeness of information or accuracy of information) across all websites available regardless of their characteristics. This study enabled a comparison across a number of criteria and considered how website characteristics specific to discussion forums may influence this.
Although 18 of 79 surveys (23%) were considered to be incomplete (receiving scores of 4 or 5), covering “very little” or “none” of the information the assessor would expect to see, a smaller number than this (11/79, 15% ) considered the information given to be “somewhat” or “very” medically inaccurate (receiving scores of 4 or 5 for those criteria). The findings are consistent with the existing literature in that information marked on quality is generally more related to the incompleteness of the information rather than actual inaccuracy [
, , ]. An even smaller number (3/79, 4%) thought that poor information may lead to someone acting in a way that may put their health at risk. The results suggest that even if information is considered to be inaccurate or incomplete, this may not necessarily result in poor advice being given.Because all message threads were rated by at least 2 assessors, and some were rated by 5 to 7 different assessors, this offered an opportunity to note whether information assessed as poor was marked consistently (or inconsistently) by all assessors, which would suggest an element of subjectivity in the assessment. In general, assessors scored consistently. For example, Q1 was awarded overall scores of 6, 7, and 9 by its 3 assessors; Q3 scored 10, 11, 12, and 13; and Q20 scored 15, 16, and 17. Some threads scored consistently high by all assessors (eg, Q1, Q7, Q12) and some scored consistently average (Q3, Q20, Q25). No discussion thread was consistently awarded the lowest band range by all its assessors.
Assessors who marked more than 3 discussion threads awarded marks reasonably evenly across the possible rating bands: Doctor 1, the only respondent who assessed all 25 threads, gave ratings ranging from 6 to 24; Doctor 4 gave a range from 6 to 15; and Doctor 6 gave a range from 7 to 23. The nonmedically qualified respondents tended to be more cautious awarding ratings marks; they awarded across a range of 7 to 18. Only 1 of the 10 lowest-rating assessments was returned by a nonmedically qualified respondent (Q21/Public 8). This could be seen as nonmedically qualified respondents being less able to recognize poor quality information or less confident about highlighting it as poor quality. However, the nonmedically qualified respondents tended to agree with the medically qualified respondents when assessing information as being inaccurate and/or incomplete, but they gave different responses on how likely someone would be to act inappropriately on the information provided (eg, see Q21/Doctor 5 and Public 8).
In the 11 cases in which the lowest possible rating was returned for a discussion thread, other assessors (including doctors) of the same thread rated it more favorably. If a thread was rated in the highest score group (5-10) by one respondent, the lowest rating it received from any of the other assessors was 16, with the largest range 7 to 16 (Q19), whereas the 3 threads that rated in the lowest band (21-25) received a much broader range of scores (Q18: 10-24; Q21: 10-23). When information was considered to be of middling quality overall, it was more likely that some assessors would consider it to be very poor. However, in most cases, low ratings were outliers in a broad range; for instance, the low rating of 23 awarded to Q21 by assessor Doctor 5 and the rating of 24 awarded to Q18 by assessor Doctor 1 were outliers to a range of 10 to 18 in both cases from the same thread’s other assessors.
Assessment of Quality by Message Forum
There was some variation in quality between the message forums, with Reddit containing the highest quality information (rating of 1: n=33) more often than either Mumsnet (n=9) or Patient (n=12), but also being more likely to contain the lowest quality information (rating of 5: Reddit=7; Mumsnet=3; Patient=1) than the other 2 websites.
Assessment of Quality by Health Condition
Message threads related to chickenpox were less likely to be awarded high ratings (score of 1 or 2) than discussion threads related to either HIV or diabetes. The middle rating (score of 3) was most often awarded to discussions on chickenpox, whereas both HIV and diabetes were mostly likely to be rated 2. Eight of the total 11 lowest ratings were awarded against chickenpox discussion threads and 8 of the 10 lowest-rated threads overall were chickenpox threads.
displays all scores of the individual health topics to show the variation in results across the different health conditions.
Discussion
Principal Results
The results of this survey suggest that, in general, the health information found in discussion forums is of reasonably good quality and only rarely does it contain information that is very inaccurate (4/79) and which some reviewers (3/79) feel may lead someone to act in a way that may put their health at risk.
These results are broadly consistent with those found elsewhere in the existing literature on the completeness and accuracy of health information found online. Previous studies on a diverse range of health conditions, including cancer, managing fever in children, and childbirth, have consistently suggested that approximately 60% to 70% of information is generally of good quality [
, , ] with only approximately 5% to 7% considered genuinely inaccurate [ , , ]. The results also suggest that people may be more able to make sensible decisions when faced with poor quality information than doctors give them credit for. This warrants further study.Rating the discussion threads on different criteria enables us to look more closely than previous studies at how, where, and why poor ratings are awarded. It is interesting to note, for example, that the controversial discussions around vaccination and herbal/natural remedies in the chickenpox discussions led to 36 separate low ratings of 4 or 5 being awarded. These were more often awarded against the inaccuracy of the information (n=11) or incompleteness (n=18) than the information being likely to lead the poster to make a somewhat inappropriate or very ill-advised decision (n=3).
It was not within the scope of the study to compare the questions that appeared to be asked prediagnosis with those that appeared to have been asked postdiagnosis. It was also not within the scope to compare those questions that were asked about the more serious conditions (ie, HIV and diabetes) with those that were asked about the milder condition (ie, chickenpox). We appreciate that these may be important factors in influencing the replies given and they warrant further research.
Q18. Reddit/Chickenpox
The discussion thread that was rated most poorly was Q18 (“Is this Chickenpox? Help!!”) for chickenpox on Reddit, on which a parent had posted a photo of spots their child had developed and asked, “Is this chickenpox?” Two respondents considered the information given to be very medically/scientifically inaccurate, one of whom also considered the information to be very ill advised, likely to lead the questioner to make a very ill-advised decision, and to act in a way that may put their health in danger.
In total, 8 respondents completed this questionnaire. Although more than half (4/7) considered the information given to be “somewhat” or “very” scientifically inaccurate and to cover “very little” of the medical information they would expect to see, 6 of 7 respondents did not think this would actually lead to harmful behavior. It is also worth noting that some posters did encourage the original questioner to go to the doctor, who later posted an update to say that they had taken this course of action. This is particularly interesting because it provides proof that although the information was assessed by some experts to be poor, it did not lead to dangerous behavior and the original poster was capable of sorting the sensible advice from the mix of replies given.
Q21. Mumsnet/Chickenpox
On Mumsnet, Q21 (“Chicken pox–is 5 months too young to expose?”) for chickenpox returned 3 lowest possible ratings against the inaccuracy of the information and in the 2 categories relating to how the poster might act. The survey was completed by 6 respondents in total and the low scores were awarded by only 1 of the 6; the other 5 rated the information more favorably. The discussion related to a parent’s question about the safety of exposing their 5-month-old child to someone who was infected with chickenpox in the hope of getting the disease “out of the way.” The discussion contained a range of views, from some parents who thought there would be little harm in it (largely due to experience of their own children having had the disease at a similar age with no problems arising) to those who considered it dangerous. Several replies actively discouraged the parent from exposing an infant so young. None of the discussions displayed antivaccination viewpoints and none actively encouraged the mother to go ahead. At the end of the discussions, the original poster summarized her understanding of the discussions and stated that, after reading the advice, she thought that the ideal age to catch chickenpox “is 2 to 6 years,” suggesting that she had been convinced that deliberately exposing a 5-month-old child would not be a good idea. Therefore, it is difficult to understand why one assessor felt that the poster would have made an ill-advised decision that would have put [her child’s] health at risk, rather than taking the view of another assessor who made the qualitative comment: “I think she came to the right conclusion based on the information given.”
Q10. Reddit/HIV
Two lowest ratings were given in response to Q10 (“I am somewhat prone to depression, but even more so now that I am HIV+. How do you guys deal with it?”) for HIV on Reddit, which asked about links between HIV diagnosis and depression. One respondent ranked this discussion in the lowest categories for completeness of information and how the poster might act based on information provided. A qualitative response given in the comments box explained that the low ratings had been given because none of the replies encouraged the poster to seek professional help, which the respondent (a GP) believed they needed. Therefore, it was not so much that poor information was given, but that the appropriate good information was not. Another respondent (a hospital consultant) marked the discussion more favorably.
Q5. Patient/Diabetes
The final low rating was recorded against a discussion on whether diabetes affects a person’s ability to recover from a cold (“Does it take longer to get over a cold if you have type 2 diabetes?”). One of 4 assessors felt that the information given was “very medically/scientifically inaccurate,” but in this case, they did not feel that the information would lead the poster to make an ill-advised decision or to act in a way that may put their health at risk.
Because 4 of 79 surveys were responsible for all 11 instances of low ratings and just one of those (Q18. Reddit/chickenpox) was responsible for 5 of 11 low ratings (see
), this warrants further research.Header | Inaccurate | Incomplete | Ill advised | Make bad decision | Dangerous to health |
Q5. Patient/diabetes | 1 | ||||
Q10. Reddit/HIV | 1 | 1 | |||
Q18. Reddit/chickenpox | 2 | 1 | 1 | 1 | |
Q21. Mumsnet/chickenpox | 1 | 1 | 1 |
There was no discussion thread that was consistently rated in the lowest (or even lowest plus second lowest) categories by all its respondents, suggesting that what constitutes poor information is as much a subjective judgment on the part of the reviewer as an absolute. Respondents disagreed more on how people who read the information may act based on it than on the accuracy or completeness of the information. Previous studies have suggested that there is an element of subjectivity attached to assessments of quality [
] and although the results of this study uphold this, further exploration is warranted of how health information in online discussion forums is received and acted on.Limitations of the Study
A number of limitations have to be taken into account when considering the results presented here. Firstly, the sample size was very small, consisting of only 17 individuals from a limited demographic (UK adults in West London). This cannot be considered to be representative; a much larger sample would need to be surveyed to ensure results could be applied more generally.
Secondly, recruiting respondents to the study was difficult, especially recruiting nonmedically qualified respondents. The diabetes support groups contacted were nervous about involving their members in a study that may direct them to incorrect and potentially harmful information. The HIV support charities contacted did not respond. Although it was easier to recruit parents to assess the chickenpox discussion, the numbers recruited were still fewer than hoped for. A larger future study would need to consider more efficient ways of recruiting higher numbers of participants.
Comparison With Prior Work
These results support other studies of online health information that found although online health information is of variable quality, the majority of it is of reasonably high quality with only a very small proportion considered to be factually incorrect (4/79) or potentially harmful to health (3/79 assessors thought the poor information given may lead someone to “act in a way that may put their health at risk”). Only 3 assessors awarded a discussion thread the lowest rating band overall, whereas 25 assessments rated a message thread in the highest band. This is broadly consistent with previous assessments about the quality of online health information in general [
, , , , ].Conclusions
The results of this study suggest that discussion forums are capable of producing health information of reasonably high quality. Of the 79 threads, 68 were assessed to contain at least some medically/scientifically accurate information and 61 of 79 were considered to contain at least some of the medical information that would be expected.
On only 3 occasions did an assessor think someone might make a “somewhat” or “very” ill-advised decision based on the information provided and there were only 3 occasions in which assessors felt the questioner may be led to act in a way that could put their health at risk. In each case, only one of the assessors felt this way when others did not; in the case of 2 of the 3 lowest ratings, comments made in the discussion forum by the original poster could be interpreted as meaning that they were not going to take a potentially harmful course of action. This challenges the assumption that the presence of poor information is automatically harmful.
The forums that contained the most inaccurate or controversial information also contained counterbalancing comments that appear able to dilute the potentially harmful consequences of the poor quality information. Comments made by the original poster and the majority of the respondents suggest that the better quality information was the more influential. This, in particular, warrants further study.
Online discussion forums do seem to be able to provide an opportunity for online health information seekers to access health information of acceptable quality. The findings suggest that there is merit in further exploring the possibilities of online discussion forums for providing peer-to-peer health information. In particular, there is a need to develop a better understanding of whether, and how, the small amount of incorrect or ill-advised information provided in a minority of answers is likely to result in adverse health outcomes or whether the discussion forum characteristics enable such messages to be counteracted and diluted. Most previous studies have tended to hone in on the small minority of poor quality examples and overemphasize the potentially detrimental impact they may have. This is despite the small number of studies that have found evidence of actual harm caused by poor health information found on the Internet. Thus, further analysis of the relationship between poor information and patient interpretation/action is crucial.
Acknowledgments
These results have been gathered through a pilot study designed to test the methodology for the main study undertaken for a PhD in Computer Science at Royal Holloway, University of London by the corresponding author, Jennifer Cole, which is supervised by the two co-authors, Dr. Chris Watkins and Dr. Dorothea Kleine. The PhD is funded through an internal college Reid Scholarship in Health, the Human Body and Behaviour.
Conflicts of Interest
None declared.
Multimedia Appendix 1
Supplementary File 1: Example Assessment Questionnaire.
PDF File (Adobe PDF File), 40KB
Multimedia Appendix 2
Data from all survey results including links to the actual question as it appeared on the discussion forum website (as the assessors saw it when they made their assessment).
PDF File (Adobe PDF File), 101KBReferences
- NHS England Analytical Service. Improving General Practice–A Call to Action: Evidence Pack. 2013 Aug. URL: http://www.england.nhs.uk/wp-content/uploads/2013/09/igp-cta-evid.pdf [accessed 2015-08-18] [WebCite Cache]
- The King's Fund. 2015 Oct 08. What's going on in A&E? The key questions answered URL: http://www.kingsfund.org.uk/projects/urgent-emergency-care/urgent-and-emergency-care-mythbusters [accessed 2015-08-18] [WebCite Cache]
- Office for National Statistics. 2013 Aug 08. Internet access-households and individuals, 2013 URL: http://www.ons.gov.uk/ons/rel/rdit2/internet-access---households-and-individuals/2013/stb-ia-2013.html [accessed 2015-08-18] [WebCite Cache]
- Last Night The Internet Saved My Life: Digital Health Report 2015. Stourbridge, UK: Pushdoctor; 2015 Aug. URL: https://www.pushdoctor.co.uk/Resources/PushDoctor-Health-report.pdf [accessed 2015-08-18] [WebCite Cache]
- Impicciatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the public on the World Wide Web: systematic survey of advice on managing fever in children at home. BMJ 1997 Jun 28;314(7098):1875-1879 [FREE Full text] [Medline]
- Suarez-Almazor ME, Kendall CJ, Dorgan M. Surfing the Net--information on the World Wide Web for persons with arthritis: patient empowerment or patient deceit? J Rheumatol 2001 Jan;28(1):185-191. [Medline]
- Air M, Roman SA, Yeo H, Maser C, Trapasso T, Kinder B, et al. Outdated and incomplete: a review of thyroid cancer on the World Wide Web. Thyroid 2007 Mar;17(3):259-265. [CrossRef] [Medline]
- Lupiáñez-Villanueva F. Health and the internet: beyond the quality of information. Rev Esp Cardiol 2011 Oct;64(10):849-850 [FREE Full text] [CrossRef] [Medline]
- Lawrentschuk N, Sasges D, Tasevski R, Abouassaly R, Scott AM, Davis ID. Oncology health information quality on the Internet: a multilingual evaluation. Ann Surg Oncol 2012 Mar;19(3):706-713. [CrossRef] [Medline]
- Whitelaw N, Bhattacharya S, McLernon D, Black M. Internet information on birth options after caesarean compared to the RCOG patient information leaflet; a web survey. BMC Pregnancy Childbirth 2014;14:361 [FREE Full text] [CrossRef] [Medline]
- Ahmad F, Hudak P, Bercovitz K, Hollenberg E, Levinson W. Are physicians ready for patients with Internet-based health information? J Med Internet Res 2006;8(3):e22 [FREE Full text] [CrossRef] [Medline]
- Ziebland S, Wyke S. Health and illness in a connected world: how might sharing experiences on the internet affect people's health? Milbank Q 2012 Jun;90(2):219-249 [FREE Full text] [CrossRef] [Medline]
- Sillence E, Hardy C, Briggs P. Why don't we trust health websites that help us help each other?: An analysis of online peer-to-peer healthcare. In: Proceedings of the 5th Annual ACM Web Science Conference. 2013 Presented at: 5th Annual ACM Web Science Conference; May 2-4, 2013; Paris p. 396-404 URL: http://dl.acm.org/citation.cfm?id=2464488 [CrossRef]
- Grant L, Hausman BL, Cashion M, Lucchesi N, Patel K, Roberts J. Vaccination persuasion online: a qualitative study of two provaccine and two vaccine-skeptical websites. J Med Internet Res 2015;17(5):e133 [FREE Full text] [CrossRef] [Medline]
- Biermann JS, Golladay GJ, Greenfield ML, Baker LH. Evaluation of cancer information on the Internet. Cancer 1999 Aug 1;86(3):381-390. [Medline]
- Nölke L, Mensing M, Krämer A, Hornberg C. Sociodemographic and health-(care-)related characteristics of online health information seekers: a cross-sectional German study. BMC Public Health 2015;15:31 [FREE Full text] [CrossRef] [Medline]
- Crocco AG, Villasis-Keever M, Jadad AR. Analysis of cases of harm associated with use of health information on the internet. JAMA 2002 Jun 5;287(21):2869-2871. [Medline]
- Dickerson S, Reinhart AM, Feeley TH, Bidani R, Rich E, Garg VK, et al. Patient Internet use for health information at three urban primary care clinics. J Am Med Inform Assoc 2004;11(6):499-504 [FREE Full text] [CrossRef] [Medline]
- Schwartz KL, Roe T, Northrup J, Meza J, Seifeldin R, Neale AV. Family medicine patients' use of the Internet for health information: a MetroNet study. J Am Board Fam Med 2006;19(1):39-45 [FREE Full text] [Medline]
- Ybarra ML, Suman M. Help seeking behavior and the Internet: a national survey. Int J Med Inform 2006 Jan;75(1):29-41. [CrossRef] [Medline]
- Luo WM, Najdawi M. Trust-building measures. Commun ACM 2004 Jan 01;47(1):108-113. [CrossRef]
- Ye Y. Correlates of consumer trust in online health information: findings from the health information national trends survey. J Health Commun 2011 Jan;16(1):34-49. [CrossRef] [Medline]
- Kim Y. Trust in health information websites: a systematic literature review on the antecedents of trust. Health Informatics J 2014 Dec 16:e1. [CrossRef] [Medline]
- Cole J, Watkins C. International employees' concerns during serious disease outbreaks and the potential impact on business continuity: Lessons identified from the 2014-15 West African Ebola outbreak. J Bus Contin Emer Plan 2015;9(2):149-162. [Medline]
- Laurillard D. Rethinking University Teaching: A Framework for the Effective Use of Educational Technology. London: Routledge/Falmer; 2002.
- Laurillard D. A conversational framework for individual learning applied to the 'learning organisation' and the 'learning society'. Syst Res Behav Sci 1999;16(2):113-122 [FREE Full text] [CrossRef]
- Rowntree D. Teaching and learning online: a correspondence education for the 21st century? Brit J Educ Technol 1995;26(3):205-215. [CrossRef]
- Thomas MJ. Learning within incoherent structures: the space of online discussion forums. J Comput Assist Lear 2002;18:351-366. [CrossRef]
- Sundar S, Rice R, Kim H, Sciamanna C. Online health information: conceptual challenges and theoretical opportunities. In: Thomas T, Parrott R, Nusbaum J, editors. Handbook of Health Communication, 2nd edition. London: Routledge; 2011:108-112.
- Hajli M, Bugshan H, Lin X, Featherman M. From e-learning to social learning–a health care study. Euro J Train Dev 2013 Nov 04;37(9):851-863. [CrossRef]
- Groselj D. A webometric analysis of online health information: sponsorship, platform type and link structures. Online Inform Rev 2014 Feb 25;38(2):209-231. [CrossRef]
- Wathen CN, Burkell J. Believe it or not: factors influencing credibility on the Web. J Am Soc Inf Sci 2002;53(2):134-144. [CrossRef]
- Sunstein TA, Thaler RH, Sunstein CR. Nudge: Improving Decisions About Health, Wealth, and Happiness. New York: Penguin; 2009.
- Munson SA, Cavusoglu H, Frisch L, Fels S. Sociotechnical challenges and progress in using social media for health. J Med Internet Res 2013;15(10):e226 [FREE Full text] [CrossRef] [Medline]
- Reddit. 2015. URL: https://www.reddit.com/ [accessed 2015-12-18] [WebCite Cache]
- Mumsnet. URL: http://www.mumsnet.com/ [accessed 2015-12-28] [WebCite Cache]
- Patient. URL: http://patient.info/ [accessed 2015-12-28] [WebCite Cache]
- HIV in the United Kingdom: 2014 Report. London: Public Health England; 2014 Nov. URL: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/401662/2014_PHE_HIV_annual_report_draft_Final_07-01-2015.pdf [accessed 2015-08-18] [WebCite Cache]
- State of the Nation: Challenges for 2015 and Beyond. London: Diabetes UK; 2015. URL: https://www.diabetes.org.uk/Documents/About%20Us/What%20we%20say/State%20of%20the%20nation%202014.pdf [accessed 2015-08-18] [WebCite Cache]
- Public Health Wales. 2015 Jan 10. Chickenpox URL: http://www.wales.nhs.uk/sitesplus/888/page/43921 [accessed 2015-08-18] [WebCite Cache]
- Centers for Disease Control and Prevention. 2011 Nov 16. Chickenpox (varicella) URL: http://www.cdc.gov/chickenpox/about/overview.html [accessed 2015-10-15] [WebCite Cache]
- NHS Choices. 2014 Jul 28. Chickenpox URL: http://www.nhs.uk/conditions/chickenpox/Pages/Introduction.aspx [accessed 2015-10-15] [WebCite Cache]
- PharmaForward LLC. Most Searched for Diseases on Google. 2013 May. URL: http://www.pharmaforward.com/wp-content/uploads/2013/05/top50conditions_may2013.pdf [accessed 2015-08-18] [WebCite Cache]
- Eysenbach G, Powell J, Kuss O, Sa E. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 2002;287(20):2691-2700. [Medline]
- Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002 Mar 9;324(7337):573-577 [FREE Full text] [Medline]
- Bansil P, Keenan NL, Zlot AI, Gilliland JC. Health-related information on the Web: results from the HealthStyles Survey, 2002-2003. Prev Chronic Dis 2006 Apr;3(2):A36 [FREE Full text] [Medline]
Abbreviations
A&E: accident and emergency |
AIDS: acquired immune deficiency syndrome |
GP: general practitioner |
HIV: human immunodeficiency virus |
Edited by G Eysenbach; submitted 27.08.15; peer-reviewed by J Batra, J Huh; comments to author 09.10.15; revised version received 20.10.15; accepted 09.11.15; published 06.01.16
Copyright©Jennifer Cole, Chris Watkins, Dorothea Kleine. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 06.01.2016.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.