Published on in Vol 13, No 1 (2011): Jan-Mar

Presenting Evidence to Patients Online:  What Do Web Users Think of Consumer Summaries of Cochrane Musculoskeletal Reviews?

Presenting Evidence to Patients Online: What Do Web Users Think of Consumer Summaries of Cochrane Musculoskeletal Reviews?

Presenting Evidence to Patients Online: What Do Web Users Think of Consumer Summaries of Cochrane Musculoskeletal Reviews?

Original Paper

1Clinical Epidemiology Program, Ottawa Hospital Research Institute, The Ottawa Hospital, Ottawa, ON, Canada

2Epidemiology and Community Medicine, University of Ottawa, Ottawa, ON, Canada

3Clinical Epidemiology Program, McMaster University, Hamilton, ON, Canada

4School of Nursing, University of Ottawa, Ottawa, ON, Canada

5Knowledge Synthesis and Exchange Branch, Canadian Institutes of Health Research, Ottawa, ON, Canada

6Human Oriented Technology Laboratory, Department of Psychology, Carleton University, Ottawa, ON, Canada

7Knowledge Translation Portfolio, Canadian Institutes of Health Research, Ottawa, ON, Canada

8Institute of Population Health, University of Ottawa, Ottawa, ON, Canada

Corresponding Author:

Jamie C Brehaut, PhD

Clinical Epidemiology Program

Ottawa Hospital Research Institute

The Ottawa Hospital

ASB 2-004

Box 693-1053 Carling Avenue

Ottawa, ON, K1Y 4E9


Phone: 1 613 798 5555 ext 13432

Fax:1 613 761 5402


Background: The Internet has the potential to be an effective medium for delivering health care knowledge to consumers. While computer usability research makes recommendations about how to present Web-based information generally, there remains no clear guidance on how to present specific forms of health care research evidence online in a way that facilitates understanding and good health care decision making.

Objective: The two goals of this study were to describe the Cochrane Musculoskeletal Group’s (CMSG’s) process for developing online patient-focused summaries of systematic reviews and to evaluate the impressions of these summaries formed by users.

Methods: A process for summarizing the results of systematic reviews via consumer summaries has evolved over 15 years. An evaluation of this approach took the form of Internet surveys on the Arthritis Society of Canada website and surveys of members of the Canadian Arthritis Patient Alliance (CAPA). Respondents provided information on background, relationship to the decision, their satisfaction with and preparation for decision making, and suggestions for improvements to the summaries. Survey data were collected between August 1, 2005, and February 28, 2006.

Results: A total of 261 respondents completed the survey. The majority (226/261 or 87%) of respondents reported having an arthritis-related condition. The consumer summary approach was generally reviewed favorably by respondents, with most agreeing that the summary provided appropriate information (177/261 or 68%), would be useful to others (160/261 or 61%), was well laid out (159/261 or 61%), was easy to learn from (157/261 or 60%), and was useful to the reader (153/261 or 59%). Areas of potential improvement were indicated by relatively fewer respondents agreeing that they could easily find all the information they wanted (118/261 or 45%), by a substantial proportion being unable to judge whether the providers of the information are reliable (80/261 or 31%), and by a similar proportion being unable to determine whether the information presented was the best available (68/261 or 26%).

Conclusions: The CMSG has developed an approach to summarizing the results of often-technical systematic reviews into public-friendly consumer summaries. Our online survey showed that this approach was generally well liked but identified specific areas for improvement. Feedback from this survey will help to reshape and improve the current template for consumer summaries used by the CMSG.

J Med Internet Res 2011;13(1):e5




A key aspect of the rapidly changing face of health care is the explosion of knowledge targeted at health care consumers. In part, because of advancing information technology allowing access to them, knowledge producers are increasingly seeing consumers as an important target group. In a recent review of 56 Canadian organizations producing practice guidelines between 2000 and 2005, it was found that 630 unique guidelines had been developed; of these, 42.7% included consumer versions or were intended for consumers [1]. Furthermore, many major governmental and nonprofit organizations such as the UK National Institute for Health and Clinical Excellence (NICE) [2], the US Agency for Healthcare Research and Quality (AHRQ) [3], the Journal of the American Medical Association [4], the Cochrane Collaboration [5], along with many patient condition-specific organizations (eg, Arthritis Society of Canada [6]) now disseminate research evidence directly to health care consumers, primarily via the Internet. Indeed, targeting patients can be an effective strategy to reduce the gap between research knowledge and clinical practice [7,8].

Development of consumer-targeted health knowledge is not only on the rise because of “push” from knowledge producers but also because of “pull” by consumers [9]. Consumers are often highly motivated to maximize the quality of their own care and are demanding greater involvement in decisions surrounding their own health care [10]. This is manifested as a demand for health care knowledge, with the Internet serving as an important mode of delivery. For example, about 80% of Canadians over the age of 16 now report using the Internet, with health information being the most commonly reported search topic [11,12].

The Internet has many advantages as a medium for delivering health care knowledge to consumers. For knowledge producers, the Web can provide wide distribution at relatively little cost. Its potential for interactivity can allow both an improved learning environment and data gathering alongside dissemination activities. Furthermore, knowledge can be updated with relative ease and low cost when compared to print or other media. For consumers, the Web is accessible, free, convenient, and allows for learning at a pace the individual finds most comfortable.

Despite these advantages, effective dissemination of online health care knowledge can be limited by variable quality and indeterminate reliability [13]. Relevant literatures exist, but have not been exploited. Computer usability research makes general recommendations about presenting Web-based information [14,15], but these lessons have often not been followed in presenting health information online. Similarly, considerable research from the patient decision-support and human decision-making literatures [16,17] has not been translated into specific recommendations for facilitating good decision making based on online health information. We propose that specific approaches to presenting health care research knowledge online to consumers need to be developed and evaluated.

The Cochrane Musculoskeletal Group (CMSG) has produced consumer summaries of over 100 of its systematic reviews of health care interventions for arthritis and other related conditions. These consumer summaries are targeted at patients but have only recently become widely available on the Internet and are now accessible by over half of the world’s population through country-level subscriptions to the Cochrane Library and consumer-targeted websites (eg, Arthritis Society of Canada [6] and Arthritis Victoria in Australia [18]).

While these summaries were developed with extensive consumer input, they were primarily developed in a paper format, which may not have translated well into effective online information tools. Guidance for producing effective online tools comes from at least two sources. First, the human-computer interaction literature identifies components of effective online information, as well as means to measure them [15]. For example, the extent to which a computer website is favorably rated by users has been categorized into 5 domains of satisfaction including aesthetics, likeability, usability, emotion, and expectation [14,19]. Second, a substantial literature has grown up around the Ottawa Decision Support Framework [16,20,21] focused on measuring constructs related to good-quality decision making. Based on this literature, we sought to examine the extent to which the CMSG summaries were seen to be relevant to such constructs.


This paper had two primary objectives: (1) to provide a narrative describing the work of the CMSG in creating online consumer summaries of the evidence from systematic reviews and (2) to evaluate the impressions of these summaries formed by users of the Arthritis Society of Canada website via an Internet survey. This knowledge will contribute toward establishing guidelines about how to summarize and present research evidence to consumers on the Internet.

The Cochrane Musculoskeletal Group Consumer Summaries

The CMSG consumer summaries have evolved over the last 15 years. Initially developed on an ad hoc basis, they have been revised and standardized based on the recommendations of a variety of guidelines for creating patient information [22-24], evidence from research [25], and user feedback. Summaries are regularly distributed and feedback sought from consumer members of the CMSG at annual CMSG meetings and from consumers and research professionals at workshops at Cochrane Colloquium meetings [26,27].

What has resulted is the standard 1-page summary now used by the CMSG (see example, Figure 1). Each summary usually consists of no more than 400 words and typically takes consumers approximately 5 minutes to read. Each summary is divided into short sections with illustrative, standardized questions as headers. An introductory section (section 1) provides background on what and who was studied and mentions the Cochrane Collaboration as the source of the information. Section 2 answers questions about the intervention and the condition, for example, “What are osteoarthritis and glucosamine?” Section 3 answers questions about the effectiveness of the intervention, for example, “How well does glucosamine work?” Section 4 answers questions about safety, explicitly addressing both benefits and harms, for example, “How safe is glucosamine?” The final section provides a single-statement summary of the overall meaning of the results and provides a Web link to the description of the level of evidence underlying these statements (ie, platinum, gold, silver, and bronze levels of evidence). This method for grading scientific evidence was derived by Tugwell et al and incorporates the types of studies and quality of evidence into the ranking [28]. This section answers the question, “What is the bottom line?”

Figure 1. Example of a consumer summary
View this figure

Sections are kept short, and information is provided in a logical flow to facilitate interpretation by a wide range of audiences. While formal readability evaluations have been carried out at various points during the development process of these summaries, questions about the validity of formal readability evaluations have been noted [29]. We found that for these summaries the technical terminology describing the diseases and treatments in CMSG reviews often inflated readability scores, while eliminating this terminology to reduce readability scores often caused more problems than it solved for users. Rather than relying on formal readability scores, we opted to ensure that all technical terms were clearly defined, and that all language was clear and readable. We saw readability analysis, therefore, as a means to end (a readable document dealing with technical issues) rather than an end itself (a document with a particular readability score).

We also strove to maintain the active voice throughout, based on the recommendations of a number of guidelines [30-32]. When available, outcome probabilities are presented in natural frequencies with consistent denominators (eg, “72 out of 100”), explicitly describe the time frame to which the results apply (eg, “after 6 months of treatment, 10 patients out of 100 will improve will improve”), and present the numbers in multiple ways (eg, “this means that 12 more people will improve”) to facilitate understanding [16]. Table 1 describes what we see as key components of successful summaries as informed by the experience of the CMSG development process. While these individual recommendations have not been empirically tested in the current context, their combination gave rise to the summaries evaluated here.

Table 1. Preliminary recommendations for presenting consumer summaries online
1Consider existing standards for clear presentation of risk information (eg, [16])
2Pilot test individual summaries on the target audience
3Provide links to definitions for technical terms
4Maintain consistent formatting between summaries
5Consider providing information-rich displays (eg, charts and graphs) in addition to clear language
6Consider the readability, or in cases where technical terms are unavoidable, the lexical density of the language
7Indicate level of evidence supporting risk estimates (eg, gold and silver)
8Provide links to more detailed information
9Prominently display information on the provider and timeliness of the information
10Keep summaries short (at approximately 400 words) and provide short bottom-line statements of key messages

In total, over 100 consumer summaries of systematic reviews for treatments of arthritis-related conditions have been produced and made available by the CMSG between 1993 and 2005. All are available online through the Arthritis Society of Canada website [6] and were the focus of our online survey. This site contains a wide variety of information about the many conditions related to arthritis, tips for living well, drug information, discussion forums, self-management programs, and research information. The consumer summaries are located in the latter section of the website and can be reached in 2 clicks from the home page. The summaries are organized by type of arthritis, and links are provided to related information on the website. To evaluate and improve the summaries, we conducted a Web-based survey of a subset of visitors to the Arthritis Society Website who read 1 or more CMSG summaries.

Survey: Overall Approach

Over the years, CMSG summaries have been subjected to many rounds of focus groups, interviews, and other forms of qualitative testing and evaluation. We chose to conduct an evaluation using a Web-based survey for 2 main reasons. First, the majority of this work was carried out on paper-based summaries, and we wanted to know how well the summaries translated to an online environment. Second, most of this qualitative testing was carried out on a select sample of people who were closely aligned with the CMSG, and we wanted to elicit the impressions of a wider range of types of respondents. In addition, while our survey should not be considered theory-derived, the choice of constructs was primarily informed by principles of computer usability [15,19] and the Ottawa Decision Support Framework [20,21], around which constructs relevant to good quality decision making have been developed.

Survey Development

This study was approved by the Ottawa Hospital Research Ethics Board. We designed an exploratory Web-based survey to evaluate user impressions of the CMSG consumer summaries. After the author group identified all the key constructs to be included in the survey, reviews of the literature were carried out for validated measures of the relevant constructs, and, where appropriate, such measures were included in the original survey or versions modified for Web-based administration. An iterative process of design, evaluation by the author team, and redesign resulted in a draft version of the survey that was programmed for the Web. The survey was then pilot tested at the Carleton University Human Oriented Technology Laboratory by 5 senior students trained in issues of computer usability. These students completed the survey and provided feedback on how to improve the aesthetic qualities, layout, content, and navigational ease of the survey.

The final version of the survey consisted of a cover letter and 5 sections and included both open-ended questions and closed-ended questions with pull-down menu or check box-type response options.

The cover letter of the survey included the names of principal investigators on the project, the purpose of the study, and the length of time the survey was expected to take. It also stated that completion and submission of the survey would serve as tacit consent that the subject’s responses could be used in the study. Details of duration and location of data storage were not provided unless specifically requested by the participant.

The survey consisted of 5 sections. The first addressed summary-specific issues such as how long participants had spent reading the summary. Subsequent sections included items on user experience, satisfaction with the way the information prepared them for decision making, experience with computers, background/demographic information, and suggestions for improvement. The complete survey consisted of 1 screen displaying a total of 53 items.

Survey Questions

Summary-specific items measured the time spent reading the summary as an indirect measure of the care with which reviewers read the items (categorized in 5-minute increments). Also measured was confidence in participants’ understanding of the key points of the summary (on a 5-point scale from “not at all” to “very” confident) as a key prerequisite of good decision making as described by the Ottawa Decision Support Framework [20,21]. Finally, items targeting the extent to which the respondent was closely related to the issue included how participants had discovered the treatment summaries (ie, by exploring, through a search engine, a recommendation of a physician, friends/family, or a support group, or other); participants’ main reasons for reading the summary (ie, for personal relevance, general interest, or other), and interest in arthritis (ie, “I have arthritis,” “I know someone with arthritis,” or other). The summary the respondent had read was logged automatically.

Computer user experience was measured using 14 items (Table 4) related to 5 domains of user satisfaction [19] including (1) aesthetics, (2) likeability, (3) usability, (4) emotion, and (5) expectation. Each item was rated on a 6-point scale from “strongly disagree” to “strongly agree” with an option of “no opinion/don’t know.” Also, 4 items (the summary is boring, the summary is frustrating, learning from the summary seems hard, I had to read too much) were phrased in the opposite direction from the rest of the items, and reported item means were reversed for ease of comparison.

Satisfaction in how the information materials prepared them for decision making was measured using 11 items adapted from Graham and O’Connor [33]. Items elicited respondents’ feelings of support and preparation to make decisions (Table 5). Items were rated on a 5-point scale from “not at all” to “a great deal.” Finally, an open-ended question asked for advice on how the consumer summaries could be improved. Note that items from both the user experience and satisfaction with preparation for decision-making instruments were analyzed separately, as neither scale had been validated in the form administered in this survey. Our goal for these items was to describe people’s impressions of the consumer summaries rather than target potential underlying constructs. Analysis and validation of underlying constructs for these items will be the subject of separate investigation.

We included 6 computer expertise items selected from a scale reported by Liaw et al [34], which asked respondents to self-report on their experience with (1) computers in general, (2) the Internet/World Wide Web, (3) Internet search engines, (4) word processing software, (5) database software, and (6) computer programming languages. Participants were asked to rate each of these 6 items on a 5-point scale ranging from 0, “none at all” to 4, “a great deal.” The 6 items were summed to produce a scale score ranging from 0 to 24. We then categorized this scale into thirds to indicate respondents with low, middle, and high levels of self-reported computer experience.

Background questions included demographic questions about sex, age, education, location of residence, and employment status.

Sampling and Recruitment

Sample size for this descriptive survey was based on estimates of the amount of traffic to the website and the response rate to the online questionnaire. We chose to aim for 300 respondents to yield a wide range of opinions on each of the 10 summaries attached to the survey. In prior years, approximately 17,000 users visited the relevant section of the Arthritis Society of Canada website. From this pool, previous (1-question) surveys administered on this website had typically yielded 200 to 500 responses per month. Based on an estimated completion rate of 30% of that number, we planned to collect 60 to 150 responses per month and take between 2 and 5 months to collect the data.

Recruitment to the survey proved a bigger challenge than expected. As a result, we engaged in 3 relatively separate recruitment strategies. First, we used a convenience sampling strategy in which the administrator of the Arthritis Society of Canada website posted a link on the home page of the site encouraging any visitors to the site to consider completing a voluntary, Web-based, open version of the survey. Respondents were assured of the confidentiality of their data, but this version did not specifically state that ethics board approval had been granted. Second, the society distributed a letter on behalf of the study authors to the provincial Arthritis Society educational team leaders asking them to encourage their local members and contacts to complete the Web-based survey. Finally, an electronic invitation to participate in a closed version of the survey was sent to all 463 members of the Canadian Arthritis Patient Alliance (CAPA), an advocacy group closely related to the Arthritis Society of Canada. Members of CAPA typically have arthritis or have a strong interest in arthritis advocacy.


The Web survey was linked to 10 of the most popular CMSG summaries on the Arthritis Society of Canada website. All 10 summaries were standardized with regard to format and content as of July 2005 (see Figure 1).

Respondents chose the summary that was most relevant to them either by clicking on it during their Internet session or by being directed by an email recruitment letter to a list of the 10 target summaries. A link to the survey itself was appended to the bottom of each selected summary, and instructions on the survey indicated that participants should complete the survey about the summary they had just read. Summary topics addressed issues around various treatments for osteoarthritis, rheumatoid arthritis, and shoulder and elbow pain. Each summary generally described the effects of a single intervention, such as drug interventions (eg, methotrexate or glucosamine) or nonpharmacologic treatments (eg, exercise or ultrasound).

To the extent possible, administration of the closed version of the survey to CAPA members was governed by Dillman’s Tailored Design Method [35]. A prenotification to complete the survey, an invitation to visit the summaries and complete the survey, and 2 reminder emails were sent via email. This version of the survey consisted of 1 screen displaying a total of 71 items including most questions from the earlier version (with the exception of 2 website feedback items) plus 1 additional scale, the Medical Data Interpretation Test [36], which was excluded from analysis due to technical problems. An appendix with the CAPA prenotification, CAPA cover letter, CAPA survey, Web link on the Arthritis Society of Canada website, Web survey, and the 10 consumer summaries can be obtained from the authors. Correspondence assured respondents of the confidentiality of their data and stated that the survey had been approved by the Ottawa Hospital Research Ethics Board.

All survey items were provided in a consistent order for all participants; no randomization of questions was carried out. All participants saw the same questions; no adaptive questioning or question branching was required. Completion of items was not enforced and the options “not applicable,” “no opinion,” and “rather not say” were included for all items comprising preexisting scales. Respondents were not asked to review responses before submitting the survey. No incentives were provided for respondents to complete the survey.

Data Collection and Statistical Analysis

The online survey compiled respondent data automatically into a Microsoft Excel file maintained by the Arthritis Society Web master and provided to the study researchers. Data collected were anonymous, organized by identification numbers created by the Arthritis Society. Access to data was limited to the principal investigators and one research assistant. No technical methods were used to prevent multiple entries from the same individual. The reviewing agency flagged the use of “cookies” (ie, small pieces of software code placed on the user’s machine from the survey server intended to track usage) as a potential ethical concern. The authors determined that the inclusion of cookies as a method of determining who had previously completed the survey adds little information over and above the use of the other acceptable methods (Web traffic logs or a specific question on the survey) and the use of cookies was not implemented for this study. As a result, the CAPA email survey provided a website link for respondents who previously completed the survey elsewhere, which advised against duplicate entries. Incomplete surveys were assessed manually by viewing Excel data files to determine whether any respondents had stopped short of completing the survey and simply submitted what had been completed. No statistical correction for nonrepresentative sampling was computed. Surveys were presented as a single webpage requiring 1 submission of data upon completion; therefore, it was not possible to determine rates of how many people had agreed to submit but had not submitted their final data. Nonresponder information was not available, making it impossible to compute view rates or participation rates for the Web administered survey. Time to complete the survey was not computed, as a time stamp was given at time of submission of the survey only.

Survey data were collected between August 1, 2005, and February 28, 2006. Closed-ended items were analyzed using frequencies and descriptive statistics using SPSS, version 16 (SPSS Inc, Chicago, IL, USA) and SAS, version 9.1 (SAS Institute Inc, Cary, NC, USA). Missing data were assessed on a question-by-question basis. Open-ended comments about how to improve the summaries were assembled into Microsoft Excel and examined for themes. In the next step, 3 coders (authors JB, AL, NS) reviewed all comments provided by respondents and coded them into 1 or more of 12 themes determined by an initial scan of the responses. Disagreements over how comments should be categorized into themes were resolved by consensus. We employed the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) reporting guideline to inform our report of this study [37].

Over the 7-month period from August 2005 through February 2006 that the survey was available on the Arthritis Society website, 162 site visitors responded to the survey; the number of hits to the survey-relevant summaries during this time period was not collected. Of the invitations we sent to the CAPA members, we obtained 99 responses out of 395 successfully delivered emails (25%). After comparisons of demographic characteristics showed no important differences between the website and CAPA respondents, responses from the 2 groups were combined for a total of 261 responses.

Table 2 describes the background and summary-specific characteristics of the respondents. The majority (176/261 or 67%) of respondents were female, and 56% (147/261) were between the ages of 45 and 64 years of age. Most were well educated, with 77% (202/261) of respondents reporting at least some postsecondary education, and 35% (92/261) were employed full-time. Respondents were distributed across western (British Columbia, Alberta, Saskatchewan, Manitoba; 49/261 or 19%), central (Ontario, Quebec; 61/261 or 23%), and eastern Canada (New Brunswick, Nova Scotia, Prince Edward Island, Newfoundland; 115/261 or 44%) and included respondents from all 10 provinces, but none from the 3 northern territories. Approximately 7% (17/261) of respondents were not from Canada.

Respondents showed variable computer experience, but 48% (125/261) reported a moderate level of experience. A substantial majority (226/261 or 87%) reported that they themselves had some form of arthritis. Of the 10 different target summaries, 48% (124/261) of respondents chose a summary focused on rheumatoid arthritis, while another 41% (107/261) chose osteoarthritis. Nearly half (111/261 or 48%) reported spending less than 5 minutes reading the summaries. The most common ways in which the summaries were discovered were by simple exploration of the Arthritis Society of Canada website (60/261 or 23%), upon physician recommendation (55/261 or 21%), or through support groups like CAPA (44/261 or 17%), while the most common reasons for reading the summaries were for personal relevance (169/261 or 65%) or general interest (51/261 or 20%).

Table 2. Characteristics of respondents
Characteristic% of Respondents
n = 261
Age (years)
< 4521.5
45 to 5426.4
55 to 6429.9
65 and over9.2
High school or less9.6
Some postsecondary36.0
Bachelor’s degree or higher41.4
Disability leave14.2
Geographic location in Canada
Outside Canada6.5
General computer experience
Arthritis experienced by
Information sought for
Rheumatoid arthritis47.5
Time spent reading the summary (minutes)
< 542.5
5 to 1034.1
> 1011.9
Summary discovered by
Exploring website23.0
Physician recommendation21.1
Support group recommendation16.9
Search engine5.4
Friend/family recommendation3.8
Reason for reading the summary
Personal relevance64.8
General interest19.5

Table 3 describes the number and percentage of the 261 respondents reviewing each consumer summary and self-reported time spent reviewing each summary. Of the 10 consumer summaries, 3 accounted for 54% (141/261) of respondents. A significant proportion of respondents spent less than 5 minutes reviewing each summary (summary-specific range of the percent of respondents who spent less than 5 minutes reviewing the summary, 25% to 70%).

Table 3. Number and percentage of respondents reviewing each consumer summary and self-reported time spent reviewing that summary page
Summary TitleReviewed the
n (% of 261)
Reported Spending
≤ 5 Minutes
Reviewing the
n (% of 261)
Reported Spending
> 5 Minutes
Reviewing the
n (% of 261)
Did Not Report
Time Viewing the
n (% of 261)
Does exercise help osteoarthritis of the hip or knee?52
Does glucosamine work for treating osteoarthritis?45
Does methotrexate work to treat rheumatoid arthritis?44
Does etanercept work to treat rheumatoid arthritis?30
Does physical therapy work to treat ankylosing spondylitis?20
Does folic acid decrease side effects in patients taking methotrexate for rheumatoid arthritis?19
Does occupational therapy help people with rheumatoid arthritis?17
Do antimalarials work to treat rheumatoid arthritis?14
Do steroid injections work to treat tennis elbow?10
Does ultrasound therapy work to treat osteoarthritis of the knee?10
Total number (%) of respondents261

Table 4 describes responses to the 14 user experience items. A number of items indicated favorable impressions by respondents, with a majority either strongly or moderately agreeing with statements that the summary used language that was appropriate (196/261 or 75%), provided expected information (178/261 or 68%), (didn’t) involve too much reading (172/261 or 66%), would be of use to many others (160/261 or 61%), was well laid out (159/261 or 61%), (wasn’t) hard to learn from (157/261 or 60%), (wasn’t) frustrating (154/261 or 59%), was useful to me (153/261 or 59%) and (wasn’t) boring (149/261 or 57%). Somewhat fewer respondents agreed that they could find all the information they wanted (118/261 or 45% strongly or moderately agreed). Finally, several items suggested that many respondents felt unable to judge the credibility or reliability of the summaries. Relatively few agreed that the summary clearly provided information on the providers of the information (112/261 or 43%). Furthermore, large proportions of respondents selected “no opinion/don’t know” to whether the information is up-to-date (54/261 or 21%), the providers of the information are reliable (80/261 or 31%), and the information presented was the best available (68/261 or 26%).

Table 4. Percentage of respondents for items of user satisfaction (higher percentages indicate greater satisfaction)
Item of User Satisfaction“Strongly Agree” to
“Moderately Agree”
% of 261
Less Than
“Moderately Agree”
% of 261
“No Opinion/ Don’t
% of 261
No Response
% of 261
The summary uses appropriate language.
The summary provided the info I expected.67.818.80.812.6
I had to read too much.a65.920.30.813.0
This summary would be useful to many others.61.324.11.513.0
The summary is well laid out.60.923.81.913.4
Learning from this summary is hard.a60.
This summary is frustrating. a59.026.11.913.0
The information was useful to me.58.628.40.412.6
This summary is boring. a57.128.70.813.4
The information is up-to-date.51.714.920.712.6
I can easily find all the information I want.
The summary clearly presents who provides the information.42.935.38.413.4
The providers of the information are reliable.42.914.230.712.3
The information presented was the best available.31.829.926.112.3

a These items are reversed, that is, to the respondent, the questions appeared as written, but the scores reported in this table are reversed to ensure agreement percentages reflect a positive opinion of the summary for all items.

Table 5 describes responses to the items related to preparation for decision making. In general, the responses indicated that the summary would improve preparation for decision making, particularly to help to identify questions to ask the physician (157/261 or 60%). Relatively few agreed with the statement that the summary helps you know that your values affect your decisions (103/261 or 40%).

To probe in more detail issues around people’s impressions of the summaries, we asked respondents to provide us with feedback about how we might improve the summaries. Of the 261 respondents, 131 provided 1 or more comments. Table 6 presents the themes identified, the relative frequencies with which those themes were mentioned, and shows examples of comments relevant to each theme. The most commonly cited themes were to provide more detail about the treatment and options, risks and the research; to provide additional interactivity or functionality to the summaries; to make the language clearer or simpler; and to increase the use of pictures, graphs or colors. Several people specifically requested the summaries use point form and provide more bottom lines about the treatments, and several suggested that more detailed information about the credibility of the information should be provided.

Table 5. Percentage of respondents by satisfaction with preparation for decision making
The Summary Would:“Quite a Bit” or “A Great Deal”
% of 261
Less Than “Quite a Bit”
% of 261
No Response
% of 261
Help identify questions you want to ask60.226.813.0
Help you think about how involved you want to be55.232.212.6
Prepare you to communicate your opinion54.432.613.0
Prepare you to make a better decision51.336.012.6
Help you prepare for a follow-up visit50.636.013.4
Help recognize that a decision needs to be made47.140.212.6
Help make a follow-up visit run more smoothly46.041.412.6
Help you think about pros and cons of the decision45.239.914.9
Help you organize your own thoughts44.842.512.6
Help you think about what is most important43.344.112.6
Help you know that your values affect decisions39.548.312.3
Table 6. Frequency of themes and examples of theme-relevant quotes of respondents’ impressions of the summaries (n = 131)
ThemeFrequencyExamples of Quotes
Need more information about treatment40length of time before effects are felt, how the meds can be taken...dosage of the meds (how often)... [Respondent 175]
Need more information about risks, side effects, or contraindications15The risk of treatment to patients should always be included. When I make a decision, I always want to know the risks involved. [Respondent 140]
Need more information about research details15Refer to other studies done or ones planned. More details on number of cases studied... [Respondent 198]
Need more information about other treatment options13It would be interesting to know how well treatments other than [X] perform for comparison. [Respondent 85]
Need more information about the disease8Try to explore the psychological issue with patients suffering [ankylosing spondylitis], such as depression, suicide, etc. [Respondent 251]
Need additional interactivity, website functionality20Maybe you could have a basic summary and have sections that expand if more detail is needed... [Respondent 212]
Use pictures, graphs, or colors12... though the statistics are interesting and indeed useful, perhaps presenting the information in a more visual manner... [Respondent 182]
Make language clearer or simpler12In an effort to use simple language, this summary was often vague and imprecise. [Respondent 203]
Improve the format of the material10A solid recommendation to do or not to do would help take away the uncertainty of decision making... [Respondent 257]
Need more information about credibility9Wasn’t real clear on the source of the information. [Respondent 241]
Compliments8I found [it] to be straightforward, easy to comprehend. [Respondent 223]

The 15-year evolution of the CMSG consumer summaries has resulted in a successful standardized presentation format that enables brief but clear presentation of research evidence for a wide range of treatments and interventions for musculoskeletal disorders. Because these consumer summaries were carefully tested as paper-based tools but hadn’t been tested as Web-based tools, we decided to evaluate how they performed on the Web and what specific areas needed work in order to improve them as Web-based tools. Our survey showed that the tools were generally rated favorably and identified specific areas for improvement, which we discuss below.

Amount of Detail

One key finding was that many respondents reported wanting additional information to be available from the summaries. Less than half of respondents (118/261 or 45%) agreed that they could easily find all the information they wanted. Open-ended comments also revealed that many respondents wanted more details about specific risks, about the types of studies comprising the research, and about the type of participants in the studies (presumably so that they could compare themselves to the study participants). Many respondents also wanted more information about the condition and its various treatment options.

We note that our sample of respondents was likely quite sophisticated in terms of its existing knowledge on arthritis-related issues. Over 77% (202/261) of respondents had some postsecondary education, and many came through recruitment from the Canadian Arthritis Patient Alliance group, members of which are likely to be actively engaged in issues of musculoskeletal disorders. This sample may, therefore, have been quite well informed already about issues around their illness (226/261 or 87% had personally experienced some arthritis condition) and, therefore, preferred relatively detailed information.

A key challenge when presenting any health information online is dealing with the variability in user needs. Nearly half (111/261 or 48%) spent less than 5 minutes reading the consumer summaries, and the clear majority (200/261 or 77%) reported spending 10 minutes or less reading them. The standardized format used here was designed primarily to yield clear, concise summaries of systematic reviews. Yet many users will wish to use these summaries as springboards for more detailed information searches. This need for more flexible, interactive information presentation was evident in our survey findings: 2 commonly cited themes were to increase use of interactivity and to include more in the way of graphs, charts, and other information-rich display formats. To facilitate such uses, it seems likely that rather than trying to create a one-size-fits-all solution that would be too detailed for some and not detailed enough for others, the challenge for developers is to provide a flexible, interactive approach that can allow users to tailor for themselves the amount and type of information they review [38]. This approach is becoming increasingly feasible on the Web, and suggests a clear avenue for future research.

We have begun this work in at least 3 ways. First, we are exploring the utility of providing links within these summaries to other, more detailed descriptions of the systematic reviews. Second, the consumer summaries are housed within the larger Arthritis Society website that includes a wealth of information on all aspects of musculoskeletal disorders; efforts to link from the summaries to this additional information may well improve areas identified in this survey. Third, we are currently conducting research on how to adapt more detailed decision support tools such as patient decision aids for use online, discussed in more detail below.

Layout and Language

While the majority of respondents felt our consumer summaries incorporated clear language and were well laid out, some respondents did feel that the language or formatting could be improved. Within the CMSG, we have avoided evaluating the summaries using readability algorithms due to the number of technical, often complex terms at the heart of the reviews. Instead, our goal has been to use relatively few content words per sentence (ie, use lower lexical density) and a clear, logical progression from background information to the effects of treatment [29]. While this approach appears to be a qualified success, there may be opportunity for linking terms; such an approach can allow optional, more detailed information and definitions to be provided without adding length or clutter. We are currently exploring the use of such techniques in another study.

Credibility of the Source

Many people were unable to judge the credibility or reliability of the summaries or did not know who the providers of the information were. This is a concern since assessment of the credibility of online information is a key component of evaluating health information [24]. Our summaries included a statement that the research is based on a Cochrane review, a reference to the review at the bottom of the summary, and a link to a website, About Cochrane, that describes the processes involved in writing a review. Clearly some respondents did not see or make use of this information; more investigation of how to make this information more salient to users will be important.

Preparation for Decision Making

While user experience was generally positive, fewer than half of the respondents felt that the summary helped them recognize that a decision needs to be made, think about pros and cons, know how their values affect their decision, organize their own thoughts, or prepare to make a better decision. It is unsurprising that these summaries should not have all information necessary to prepare people for a decision since they are limited chiefly to providing information about the treatment options and the pros and cons of the treatments [39]. We are currently assessing whether patient decision aids, that is, decision support tools designed to help people make specific and deliberative choices among options, may be useful when presented online. While their effectiveness has been demonstrated in a variety of other presentation formats [20], it remains an open question how such tools can most effectively be employed via the medium of the Web and for what situations such tools may be most useful. These tools not only present information on the options and outcomes relevant to the person’s health status but often also include exercises to help patients explicate factors such as how they value the different options, preference for role in decision making, or choice predisposition. Patient decision aids can be much more detailed and, therefore, longer than consumer summaries. Use of these more detailed tools may be warranted in situations where a decision cannot be made on the basis of a consumer summary.


The response rate and makeup of our sample of respondents is one clear limitation of the present study. The Web subsample was collected over a period of 7 months from summaries that see hundreds of visitors per month. While we have no information on nonresponders, we have to assume that our relatively slow accrual rate suggests that we were only obtaining data from a small, select group of visitors to the site. The CAPA survey subgroup likely exhibits similar biases. Despite our best efforts, logistical limitations prevented us from obtaining a high response rate from the CAPA survey sample. The response rate (99/395 or 25%) suggests that we may have a very select sample of the CAPA group, which itself is likely quite different from the target population of all arthritis patients using online information. In total, we must assume that our sample is biased with respect to our overall target population. We have chosen to interpret our results not as a sample representative of all arthritis sufferers, but as one of a relatively sophisticated sample of well-educated patients. Future work will be needed to assess whether these findings generalize to the wider population of arthritis patients who use online information.

A second clear limitation of the current work is the lack of a control group against which to compare the survey findings. We chose the single-group design in order to evaluate the CMSG model for presenting consumer information online as it is currently being implemented on the Canadian Arthritis Society website, and we chose to use the information derived from it to inform future controlled studies. Our use of previously validated measures (eg, computer experience and satisfaction with preparation for decision making) gives us confidence in the constructs we have measured, but ongoing work using controlled designs will assess the extent to which the levels of these constructs can be improved upon using other approaches.


The relationship between the Cochrane Musculoskeletal Group and the Arthritis Society has created an excellent opportunity for research producers to target those people who would benefit most from this research information. The current work focuses specifically on engaging in this knowledge translation process in an online environment and makes clear that while we are on the right track, there is more work to do in order to understand how best to communicate systematic review information online. We have begun this work and provided some initial recommendations about how consumer summaries should look. Feedback from this survey will help to reshape and improve on the current presentation format for consumer summaries used by the CMSG. Our results should also provide initial guidelines to other developers of patient information who wish to reach consumers via the Internet.


We would like to thank the Arthritis Society of Canada and the Canadian Arthritis Patient Alliance for their support of this survey and, in particular, Rob Watts, manager of Internet resources, Arthritis Society of Canada; Anne Dooley, cochair of research, Canadian Arthritis Patient Alliance; and Colleen Maloney, vice president, Canadian Arthritis Patient Alliance. Also thanks to Anton Saarimaki who programmed the survey for the Web and to Dr Ann Cranney for clinical input. This work was funded by an operating grant from the Canadian Institutes of Health Research (CIHR; MOP-68973). Dr Brehaut is a CIHR New Investigator in Knowledge Translation.

Conflicts of Interest

None declared

  1. Kryworuchko J, Stacey D, Bai N, Graham ID. Twelve years of clinical practice guideline development, dissemination and evaluation in Canada (1994 to 2005). Implement Sci 2009;4:49 [FREE Full text] [CrossRef] [Medline]
  2. National Institute for Health and Clinical Excellence (NICE).   URL: [accessed 2010-02-02] [WebCite Cache]
  3. AHRQ Agency for Healthcare Research and Quality.   URL: [accessed 2010-02-02] [WebCite Cache]
  4. JAMA. JAMA Patient Page   URL: [accessed 2010-02-02] [WebCite Cache]
  5. The Cochrane Collaboration.   URL: [accessed 2010-02-02] [WebCite Cache]
  6. The Arthritis Society.   URL: [accessed 2011-01-17] [WebCite Cache]
  7. Schuster MA, McGlynn EA, Brook RH. How good is the quality of health care in the United States? Milbank Q 1998;76(4):517-63, 509. [Medline]
  8. Grol R. Successes and failures in the implementation of evidence-based guidelines for clinical practice. Med Care 2001 Aug;39(8 Suppl 2):II46-II54. [Medline]
  9. Lavis JN, Lomas J, Hamid M, Sewankambo NK. Assessing country-level efforts to link research to action. Bull World Health Organ 2006 Aug;84(8):620-628 [FREE Full text] [Medline]
  10. Kassirer JP. Incorporating patients' preferences into medical decisions. N Engl J Med 1994 Jun 30;330(26):1895-1896. [CrossRef] [Medline]
  11. Statistics Canada. 2010 May 10. The Daily Canadian Internet Use Survey   URL: [accessed 2010-05-05] [WebCite Cache]
  12. Statistics Canada. Internet users at home   URL: [accessed 2010-05-05] [WebCite Cache]
  13. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 2002;287(20):2691-2700 [FREE Full text] [Medline]
  14. Hammerich I, Harrison C. Developing online content: the principles of writing and editing for the Web. New York: John Wiley & Sons; 2002.
  15. Lindgaard G. Usability testing and system evaluation: a guide for designing useful computer systems. London: Chapman & Hall; 1994.
  16. Elwyn G, O'Connor AM, Bennett C, Newcombe RG, Politi M, Durand MA, et al. Assessing the quality of decision support technologies using the International Patient Decision Aid Standards instrument (IPDASi). PLoS One 2009;4(3):e4705 [FREE Full text] [CrossRef] [Medline]
  17. Hibbard JH, Peters E. Supporting informed consumer health care decisions: data presentation approaches that facilitate the use of information in choice. Annu Rev Public Health 2003;24:413-433. [CrossRef] [Medline]
  18. Arthritis Victoria.   URL: [accessed 2010-02-02] [WebCite Cache]
  19. Lindgaard G, Dudek C. What is this evasive beast we call user satisfaction? Interacting with Computers 2003;15:429-454. [CrossRef]
  20. O'Connor AM, Bennett CL, Stacey D, Barry M, Col NF, Eden KB, et al. Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev 2009(3):CD001431. [CrossRef] [Medline]
  21. O'Connor AM, Tugwell P, Wells GA, Elmslie T, Jolly E, Hollingworth G, et al. A decision aid for women considering hormone therapy after menopause: decision support framework and evaluation. Patient Educ Couns 1998 Mar;33(3):267-279. [Medline]
  22. Duman M. In: Duman M, editor. Producing Patient Information: How to Research, Develop and Produce Effective Information Resources. London: King's Fund; 2003.
  23. Australian Government National Health and Medical Research Council. 2000. How to present the evidence for consumers: preparation of consumer publications   URL: [accessed 2010-10-10] [WebCite Cache]
  24. DISCERN Online: Quality criteria for consumer health information.   URL: [accessed 2010-02-02] [WebCite Cache]
  25. Moynihan R, Bero L, Ross-Degnan D, Henry D, Lee K, Watkins J, et al. Coverage by the news media of the benefits and risks of medications. N Engl J Med 2000 Jun 1;342(22):1645-1650 [FREE Full text] [CrossRef] [Medline]
  26. Santesso N, Maxwell L, Tugwell PS, Wells GA, O'connor AM, Judd M, et al. Knowledge transfer to clinicians and consumers by the Cochrane Musculoskeletal Group. J Rheumatol 2006 Nov;33(11):2312-2318. [Medline]
  27. Tugwell PS, Santesso NA, O'Connor AM, Wilson AJ, Effective Consumer Investigative Group. Knowledge translation for effective consumers. Phys Ther 2007 Dec;87(12):1728-1738 [FREE Full text] [CrossRef] [Medline]
  28. Tugwell P, Shea B, Boers M, Brooks P, Simon L, Strand V, et al. In: Wells G, editor. Evidence-based rheumatology. London: BMJ; 2004.
  29. Clerehan R, Buchbinder R, Moodie J. A linguistic framework for assessing the quality of written patient information: its use in assessing methotrexate information for rheumatoid arthritis. Health Educ Res 2005 Jun;20(3):334-344 [FREE Full text] [CrossRef] [Medline]
  30. Fisk D, Rogers WA, Charness N, Czaja SJ, Sharit J. In: Sharit J, editor. Designing for older adults: principles and creative human factors approaches. Boca Raton, FL: CRC Press; 2004.
  31. Nielsen J. In: Nielsen J, editor. Designing Web Usability: The Practice of Simplicity. Indianapolis: New Riders Publishing; 2000.
  32. World Wide Web Consortium. Web Accessibility Initiative (WAI)   URL: [accessed 2010-02-02] [WebCite Cache]
  33. Graham ID, O'Connor AM. User Manual - Preparation for Decision Making   URL: [accessed 2010-02-10] [WebCite Cache]
  34. Liaw SS. An Internet survey for perceptions of computers and the World Wide Web: relationship, prediction, and difference. Computers in Human Behavior 2002;18:17-35. [CrossRef]
  35. Dillman D. In: Dillman D, editor. Mail and Internet surveys: the tailored design method. New York: John Wiley & Sons, Inc; 2000.
  36. Schwartz LM, Woloshin S, Welch HG. Can patients interpret health information? An assessment of the medical data interpretation test. Med Decis Making 2005;25(3):290-300. [CrossRef] [Medline]
  37. Eysenbach G. Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res 2004 Sep 29;6(3):e34 [FREE Full text] [CrossRef] [Medline]
  38. Hibbard JH, Greene J, Tusler M. Improving the outcomes of disease management by tailoring care to the patient's level of activation. Am J Manag Care 2009 Jun;15(6):353-360 [FREE Full text] [Medline]
  39. O'Connor AM, Llewellyn-Thomas HA, Flood AB. Modifying unwarranted variations in health care: shared decision making using patient decision aids. Health Aff (Millwood) 2004;Suppl Web Exclusives:VAR63-VAR72. [CrossRef] [Medline]

AHRQ: (US) Agency for Healthcare Research and Quality
CAPA: Canadian Arthritis Patient Alliance
CHERRIES: Checklist for Reporting Results of Internet E-Surveys
CSMG: Cochrane Musculoskeletal Group
NICE: (UK) National Institute for Health and Clinical Excellence

Edited by G Eysenbach; submitted 17.03.10; peer-reviewed by A Gagliardi, M Sampson, R Bannuru; comments to author 21.04.10; revised version received 14.06.10; accepted 10.07.10; published 18.01.11


©Jamie C Brehaut, Nancy Santesso, Annette M O'Connor, Alison Lott, Gitte Lindgaard, Ania Syrowatka, Ian D Graham, Peter S Tugwell. Originally published in the Journal of Medical Internet Research (, 18.01.2011.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.