How Breast Cancer Patients Want to Search for and Retrieve Information From Stories of Other Patients on the Internet: an Online Randomized Controlled Experiment

Introduction

Patients value having access to stories of other patients as it provides them with emotional support, information, reassurance, and practical advice [1]. The Internet is a valuable resource for accessing stories because of its privacy and 24-hour availability without the need to leave one’s home [2]. Two well-known examples of web-based applications that include personal stories of patients are The Comprehensive Health Enhancement Support System (CHESS) [3] and The Database of Individual Patients’ Experiences of Illness (DIPEx) [4].

Studies of online patient stories have focused on several factors. These include why patients publish their stories online and what this means in a broader sociological context [5-7]. Wise et al [8] found that accessing personal stories in a computer-based patient support system had a positive effect on patients’ healthcare participation, which entailed participation preferences, confidence, and communication with their doctor. Little is known, however, about how patients search online for stories of other patients and whether they can find relevant ones.

Some qualitative studies have found that patients appreciate the ability to select stories of other patients of a particular age or who have opted for similar treatment [1,9]. In addition, searching by topics seems also to be of interest [10]. Some websites with patient stories provide a search facility to search for personal characteristics of the story writers and/or for topics written about in the stories [11-13]. However, to our knowledge, patients’ satisfaction and search success with these search facilities have not yet been studied.

In the present study, we examined which search facilities for patient stories resulted in satisfaction with the search process and the stories retrieved. We also studied the impact of the stories retrieved on coping with illness. Our expectation was that having a search facility would be an improvement compared with not having a search facility. Moreover, we expected that a combination of search facilities would result in higher satisfaction than a single search facility because a combination of search facilities may result in more opportunities to find a relevant story.

Methods

Design and Procedure

Study Design

We focused our study on patients with breast cancer. We contacted the board of The Amazones Foundation, which was founded by a group of young women with breast cancer to provide their peers with information and support. The Amazones Foundation developed a website for young women with breast cancer [14] that provides information and advice, a calendar of activities, an online support group, and links to other sites. The website also has a section with personal stories. Women can anonymously submit their own story to the site. The stories are presented alphabetically by writers’ nicknames. If a writer passes away, her story remains on the site accompanied by an obituary written by the website moderators.

We were granted permission by the board of The Amazones Foundation to conduct our study. In January 2007 we downloaded all 170 stories available at that time on their website for use in our study. We facilitated access to the stories in three ways: (1) with a search facility for story topics, (2) with a search facility for writer profiles, and (3) with a combination of these two search facilities. In addition, a control group could access the stories by means of the original alphabetical listing by story writer. We implemented these four ways of facilitating access to the stories on a separate study website. This resulted in four groups based on two independent factors: (1) no versus yes search by story topics, and (2) no versus yes search by writer profile (Table 1). In each of the four groups the same set of 170 stories could be searched. Figures 1 to 4 show screenshots of the search pages available to the four groups. We requested that the board of the Amazones Foundation not to participate in the study.

The present study is reported in accordance with the CHERRIES checklist, which is a checklist for reporting results of Internet e-surveys [15]. It was not registered as a clinical trial on ClinicalTrials.gov, a registry of clinical trials conducted around the world, because our study does not correspond to the definition of a clinical trial as provided in their glossary.

Recruitment Process

Recruitment announcements were disseminated online using banners on the websites of several Dutch patient and health organizations and offline using posters and flyers in waiting rooms of several hospitals. Dutch-speaking women with breast cancer were invited to participate irrespective of other personal characteristics. The offline recruitment announcements gave the URL of the study website, and the online announcements contained a hyperlink to the study website. The study was accessible to each visitor of the site, but only visitors who met the inclusion criteria were further directed to the informed consent page. After finishing the final questionnaire, participants could send a ready-made email message with the URL of the study website to other women who might be interested in participation. The study website was accessible in the period June through November 2007. During this time period, women could choose for themselves on what day and time they wanted to participate. No incentives were offered for participation.

Study Website

The first page of the study website provided the following information about the study: study objective, information about the researchers, study inclusion criteria, details about participation, expected time within which to complete the study, and contact details. When a website visitor chose to participate by clicking the button “I would like to participate”, two questions were presented to check whether the visitor met the inclusion criteria (ie, being female and having been diagnosed with breast cancer). If this was the case, the visitor was asked to read the informed consent statement. By agreeing with the informed consent statement, the visitor declared that her participation was voluntary, that she understood what participating entailed, and that she was aware of what data would be recorded. To agree with the informed consent statement, the visitor had to check the box “I agree” and then click on the button marked “Next”. After this the visitor was asked whether she was certain that she agreed with the informed consent statement. In this way, we assured that women did not automatically agree to participate. Before the participant was randomly assigned to one of the four groups, she was asked to provide a short description of the information she wanted to search for in the stories. Random assignment of each participant to a study group was carried out by an algorithm that was part of the study website. The chance of being assigned to a group was equal for all four groups, that is, 1 in 4 (except when more than one session was conducted from the same IP address; see the section below labeled “Technical Aspects”). Once assigned to a group, a participant could search for and read the stories as long as she liked. When finished with searching and reading she was asked to complete a final questionnaire posted on the study website about satisfaction with the search process and the stories retrieved and the stories’ impact on coping with breast cancer.

‎View this figure

‎View this figure

‎View this figure

‎View this figure

Ethical Aspects

Participants remained anonymous since no log-in, name, or address were required. In order to minimize traces on each client’s computer, no cookies were used. Recording of log data did not start until participants had agreed to the informed consent statement. Questionnaire responses were not saved until participants confirmed at the end of the final questionnaire that they agreed to submit their responses. Data were saved in a password protected SQL database only accessible to the researchers. Participants could stop at any time without receiving pop-ups or text when leaving the study website.

Before the start of the study, our research proposal (see Multimedia Appendix 1) was presented to the Ethical Committee of the Leiden University Medical Centre (archive number 06/43). The Committee concluded that our study involved no medical intervention and that we could proceed. Our intervention consisted of providing access to the stories already available on the website of the Amazones Foundation in several new ways.

Development of Intervention Groups

Search Facilities

To develop the search facilities, all 170 stories were coded according to a coding scheme for story topics and a coding scheme for writer profile (Table 2). The topics and personal characteristics in the coding schemes were chosen because they had been used on other websites that contained breast cancer stories [11-13] or by other authors of studies in this field [1,9,10]. For the characteristic “phase in the course of disease,” the category “passed away” was assigned to stories that contained an obituary.

Participants could search for age using the categories: 20-30 years, 30-40 years, 40-50 years, and over 50 years. Participants could search for time since diagnosis using the categories: less than half a year ago, ½ - 1 year ago, 1-3 years ago, 3-5 years ago, and more than 5 years ago. To ensure that participants were aware of all search facilities, the search button was placed at the bottom of the page. In the groups with access to a single search facility, it was possible to search for more than one topic or more than one writer characteristic. In the combination group, participants could choose whether they wanted to search for story topics only, for writer characteristics only, or for both. Searching for more items simultaneously was based on the OR Boolean operator.

Weight Assignment in Story Retrieval

For every search performed by the participants, a weight between 0 and 1 was assigned to each of the 170 stories in the database. If a story matched exactly with the search objectives, it received a weight of 1. Story weights were calculated with every new search. Therefore, the weight assigned to a story could change with every search.

In the story topics group, weights were calculated by dividing the number of topics found in a story by the number of topics that were searched for. For example, when a participant searched for four topics, all stories containing one of these four topics received a weight of ¼ (0.25).

In the writer profile group, a weight was assigned to each of the personal characteristics that a participant searched for. These weights were then multiplied with each other to calculate the weight of a story as a whole.

If the age of a writer fell in the age category that the participant was searching for, then “age” received a weight equal to 1. The more the age of the writer deviated from the age category that the participant was searching for, the lower the weight that “age” received. In a similar way, weights for “time since diagnosis” were assigned.

If the partner status of a writer exactly matched the partner status that the participant was searching for, then “partner” received a weight equal to 1. If the partner status of a writer is unknown, then this characteristic received a weight of 0.5 irrespective of the partner status that the participant was searching for. If the partner status of a writer was the opposite of the partner status that a participant was searching for, then this characteristic received a weight of 0.2. We did not assign a weight of 0 for the latter case because then the weight for the whole story would be 0. In similar ways, weights for the other categorical variables were assigned.

In the combination group, weight assignment was similar to that of the previous two groups or a multiplication of these two, depending on whether a participant searched for story topics only, for writer profile only, or for both.

Number of Stories Retrieved

It was decided to present participants with at least ten stories after each search. The total number of stories presented after a search depended on the distribution of the weights assigned to the stories. All stories with the same weight as the tenth story were presented because we saw no valid reason for presenting only a portion of the stories with that weight. For example, when five stories matched exactly with the search objectives, that is, weight equal to 1, 20 stories received a weight of 0.80, 45 stories received a weight of 0.60, and 100 stories received a weight of 0.40, then 25 stories (5 + 20) would be presented. Accordingly, if no stories exactly matched the search objectives, still at least ten stories were presented. The list of retrieved stories showed the extent to which the stories matched the search objectives. In this way we tried to present participants with neither too few nor too many stories.When participants did not fill in the search page, no stories were presented to them because we wanted to ensure that participants were aware of the search facility.

Story Display and Sequence

The retrieved stories were displayed as a list giving for each story the writer’s nickname and the story’s weight. Weights were represented as a number of pink ribbons. In addition, the search criteria that were fulfilled were given in each group, that is, the topics found, the writer’s characteristics, or both (see Multimedia Appendix 2). The list of stories was sorted by weight with the story with the heighest weight at the top. Stories with the same weight in the story topics group were displayed as follows. For each story, the percentage of text of the story relating to the topics the participant searched for was calculated. Stories with the highest percentage were ranked first. In the writer profile group, stories with the same weight were sorted by the age of the writers, and if age of writers was equal by time since diagnosis. Clicking on a story title from the list displayed the complete story.

Final Questionnaire

Demographic and Disease Characteristics

Participants were asked to provide demographic information such as age, marital status, children, religion, education, and employment status. They were also asked to report characteristics of their cancer, such as time since diagnosis, type of diagnosis, metastases in axillary lymph nodes or other parts of the body, treatment undergone, and prognosis.

Use of the Internet and the Amazones Website (Before Study Participation)

The participants were asked to indicate their frequency of Internet use, the type of activities in which they engage on the Internet, and whether they had read stories of other patients on the Internet before. Moreover, they were asked to report how often they had visited the Amazones website before, how familiar they were with this website, and how many of the stories on this website they read before.

Satisfaction With the Search Process, the Stories Retrieved, and the Stories’ Impact on Coping With Breast Cancer

The constructs listed below were used to measure the three outcomes. Cronbach alphas were calculated using SPSS version 16.0 (SPSS Inc, Chicago, IL, USA) by conducting reliability analyses. Reverse phrased items were recoded. We found the internal consistency for each construct to be good or satisfactory (Cronbach alpha = 0.71 to 0.88). The items “overall satisfaction with the search facility” and “overall satisfaction with the stories retrieved” were answered using 10-point Likert scales; all other items were answered using 5-point Likert scales. For an overview of the items belonging to all the below mentioned constructs, see the Multimedia Appendix 3.

To measure satisfaction with the search process, 13 items were formulated (partially based on [16,17]). “Opinion about the search facility” was measured using 5 items (Cronbach alpha = .88). To measure “the extent to which the search options enabled finding information one was looking for,” 4 items were formulated (Cronbach alpha = .75). “Recommendation to others and future own use” was measured with 2 items (Cronbach alpha = .82). “Opinion about the number of search options” and “overall satisfaction with the search facility” were measured with 1 item each.

To measure satisfaction with the stories retrieved, 18 items were formulated (partially based on [16,17]). “Opinion about the stories retrieved” was measured with 6 items (Cronbach alpha = .71). To measure “opinion about the list of stories displayed after a search” 4 items were formulated (Cronbach alpha = .76). “The extent to which the stories retrieved covered one’s information need” was measured with 4 items (Cronbach alpha = .82) and “recommendation to others and future own reading” with 2 items (Cronbach alpha = .77). “Opinion about the number of stories retrieved” and “overall satisfaction with the stories retrieved” were measured with 1 item each.

“The stories’ impact on coping with breast cancer” was measured with 6 items (Cronbach alpha = .85) which were based on an extensive literature on coping [18-21]. Two of the items were formulated to measure problem-focused coping (“By reading the stories I have learnt things” and “By reading the stories I know what to do”), another two items were formulated to measure emotion-focused coping (“By reading the stories I am more able to understand my feelings” and “By reading the stories I can see that certain emotions are part of learning to live with breast cancer”), one item was formulated to measure reappraisal (“By reading the stories I view things in a different way”) and one item was formulated to measure social comparison (“By reading the stories I see that others have experienced the same things”). The existing validated coping scales, such as the Ways of Coping checklist [18,19] and the COPE inventory [21], could not be used because they were too general for our research question.

Technical Aspects

We tested the usability and technical functionality of the study website, including the final questionnaire, multiple times, and we solved all appearing errors. During participants’ search processes, log data recorded how long participants surfed on the study website, how many searches they performed, how many stories they accessed, and how long the text of the stories was displayed on the screen. In the control group, clicking on a folder (A to Z) was regarded as performing a search, and subsequently clicking on a name was seen as accessing a story. Also, the time participants needed to fill in the final questionnaire was recorded.

Participants who were searching for or reading the stories were reminded to fill in the final questionnaire by a yellow figure on the left side of the screen with the text “Do not forget to complete the questionnaire,” which was highlighted every five minutes. Adaptive questioning was used to reduce the number and complexity of the questions. Questions were not randomized or alternated. The final questionnaire was distributed over five pages in the following sequence:(1) the search process, (2) the stories retrieved, including the stories’ impact on coping with breast cancer, (3) use of the Internet and the Amazones website, (4) disease characteristics, and (5) demographic characteristics. When participants clicked on the “Next” button at the end of a page, JavaScript was used to check for completeness. Unanswered questions were highlighted, and participants were asked to answer these. Yet, answering was not enforced, since by clicking on the “Next” button again, the next page was reached. Participants were not able to review and change their answers in previous parts to prevent a possible influence of questions asked later in the questionnaire.

Log data and questionnaire responses were saved automatically in an SQL database. In preparation for data analysis, sessions from the same IP address with a time interval of less than 20 minutes were merged, and those with a time interval of greater than 20 minutes were kept as two separate sessions. We assumed that in the former case the sessions were from the same participant and, in the latter, from different participants. Applying the first rule resulted in 23 merged sessions; the latter rule was applied to 6 pairs of sessions. Merging was possible because in all cases the questionnaire was filled out only once. Participants were only distinguished by IP address. A particular IP address was always assigned to the same intervention group. This was done to prevent women from participating multiple times when trying to get in another study group.

Data Analysis

The data were imported into SPSS version 16.0. Differences in the log data between questionnaire completers and noncompleters were assessed using Mann-Whitney tests. The noncompleters were excluded from further analyses, since no questionnaire responses for this group were available. For the completers, there was no time frame for filling in the questionnaire. Differences between the four groups in baseline characteristics were assessed using Chi square tests, 1-way ANOVA, or Kruskal-Wallis tests (depending on variable type and skewness).

Kruskal-Wallis tests were performed to assess differences between the four groups in search behaviour (ie, the log data). Significant differences were examined further by performing post hoc tests. We chose to use Mann-Whitney tests with a Bonferroni correction, and as as the critical level of significance we used .05/6 = .008 because with four groups six comparisons were performed.

For each construct of the three outcome measures (satisfaction with the search process, the stories retrieved, and the stories’ impact on coping with breast cancer) a mean total score was calculated. A higher mean indicated a higher satisfaction or impact respectively. The effects of the search facilities on the constructs of the three outcome measures were examined using ANOVA with two independent factors (search facility for story topics yes/no; search facility for writer profiles yes/no) to assess possible main and interaction effects. This analytical approach was chosen in order to examine the effects of the two search facilities both independently and in combination. P values above .05 were considered not significant.

Results

Participant Statistics

Informed consent was given by 353 people, of whom 182 (51.6%) completed the final questionnaire (Figure 5). No significant difference was found between the four groups in percentage questionnaire completers (χ²₃= 3.7, P = .30). The mean time that participants needed to fill in the final questionnaire was 15.3 minutes (SD = 12.7; min = 5.0, max = 138.4). In comparison with questionnaire noncompleters, questionnaire completers spent less time visiting the study website (mean = 809.1 seconds vs 928.0 seconds, P < .001), but completers performed more searches (mean = 2.1 vs 1.7, P < .001), accessed more stories (mean = 6.6 vs 3.4, P < .001), and their mean reading time per story was longer (mean = 92.8 seconds vs 63.5 seconds, P < .001).

Table 3 shows the baseline characteristics of the questionnaire completers. No significant differences between the four groups were found for demographic and disease characteristics and use of the Amazones websites. With respect to use of the Internet, the writer profile group was less familiar with accessing fellow patients’ stories on the Internet.

‎View this figure

Search Behaviour

Table 4 shows that there were no differences between the four groups in time spent on the study website or in the number of searches performed. However, we found differences between the four groups in the number of stories accessed and in the mean reading time per participant per story. Post hoc tests (with a critical level of significance of P = .008 due to Bonferroni correction) showed that compared with the control group, fewer stories tended to be accessed in both the writer profile group (P = .01) and the combination group (P = .02). In addition, in the control group, the mean reading time per participant per story was shorter than in the writer profile group (P = .007) and tended to be shorter compared with the story topics group (P = .02) and the combination group (P = .009).

Satisfaction With the Search Process

Table 5 shows that having access to the story topics search facility resulted in a more positive opinion about the search facility (1a), in a more positive opinion about the number of search options (1b), in being better enabled to find the information one was looking for (1c), in being more inclined to recommend it to others or to use it more often themselves in future (1d), and in a higher overall satisfaction with the search facility (1e), compared with not having access to this search facility (all comparisons were significant at P < .05). Having access to the writer profile search facility compared with not having access to this search facility resulted in a significantly more positive opinion about the search facility (1a), and in a significantly higher overall satisfaction score (1e).

An interaction effect was found for the overall satisfaction score (1e). When participants could search using the story topics, they were satisfied with this search facility regardless of whether (mean = 7.3, SD = 1.5) or not (mean = 7.2, SD = 1.4) they could also search with the writer profile. The effect of having access to the writer profile search facility when also having access to the story topics search facility was not significant (P = .90). However, when participants could not use the story topics to search the stories, they were more satisfied with having access to the writer profile as a search facility (mean = 6.8, SD = 1.6) compared with not having access to any search facility (mean = 5.7, SD = 2.3). The effect on satisfaction of having access to the writer profile search facility when not having access to the story topics search facility was significant (P = .009).

Satisfaction With the Stories Retrieved

Having access to the story topics search facility resulted in a more positive opinion about the list of stories displayed after a search (2c), a greater extent to which the stories retrieved covered one’s information need (2d), and a higher overall satisfaction score with the stories retrieved (2f) compared with not having access to this search facility (Table 5). Having access to the writer profile search facility compared with not having access to this search facility resulted in a more positive opinion about the list of stories displayed after a search (2c).

There were no interaction effects observed in satisfaction with the stories retrieved.

The Stories’ Impact on Coping With Breast Cancer

Table 5 shows that the stories retrieved using the story topics search facility had a greater impact on coping with breast cancer (3a). When analysing each of the six coping items individually, we observed that having access to the story topics search facility resulted in a significantly higher score for having learned things (3a.1).

Discussion

Principal Findings

To our knowledge, this study is the first randomized controlled experiment with a 2x2 factorial design that examined search facilities for accessing online patient stories. We observed that the story topics search factor had a strong impact on patient satisfaction and search success: participants were the most satisfied with this search facility and the stories retrieved. Also, the stories retrieved had a greater impact on coping with breast cancer. The effect of the writer profile search factor was limited. This search facility resulted only in a few effects, predominantly on satisfaction with the search process. The two search factors combined generally had no amplified effect on patient satisfaction or search success as we only had one significant interaction.

These findings are contrary to our expectation, which was that the combined search facilities (the interaction) would outperform a single search facility because this combination is more complete and differentiated resulting in greater opportunities to find a relevant story. Apparently, this quantity argument seems to be less important than the type of the search facility (quality). In line with our expectation was that a single search facility was an improvement compared with the alphabetically listed stories in the control group.

Participants in the three search facility groups accessed fewer stories and read longer per accessed story compared with the control group. An explanation for this might be that the stories retrieved in the search facility groups were more relevant to the participants. A search facility probably increases the proportion of the documents retrieved relevant to the user's information need [22].

The story topics search facility resulted not only in participants being more satisfied with the search process, but also in participants retrieving stories that better covered their information needs and retrieving stories from which they learned more. Patients might use online stories predominantly for information, and, therefore, the topics described in the stories might be more important for them than the writer’s profile. Patients’ profiles might be more important when seeking face-to-face contact. This difference between seeking information and seeking contact has also been noted by Bennenbroek et al [23] in their research on social comparison.

Our observation that the writer profile search facility compared with not having this search facility resulted in a more positive opinion about the search facility and in a higher overall satisfaction with the search facility is in line with the results of Rozmovits and Ziebland [1]. They found in interviews that patients positively evaluated the ability to select other patients of a particular age, stage of illness, or patients who were long-term survivors or who had opted for similar treatment. However, although our study also showed that participants were more satisfied with this search facility, they were not more satisfied with the stories retrieved using this facility.

Limitations of the Present Study

A considerable number of participants performed searches but did not complete the questionnaire. Compared with completers, noncompleters spent more time on the study website while they performed fewer searches, accessed fewer stories, and spent less time reading per story. Noncompleters might not have been sure about how to use the search facilities, or they might not have been as interested. Yet, we could not empirically evaluate these hypotheses nor perform any statistical analyses, since we had no further information about noncompleters. The number of completers and noncompleters was evenly distributed over the four study groups. Therefore, we believe that potential bias equally affected all four groups. In addition, the direction of the bias is probably twofold: dissatisfied participants might have stopped or they might have completed the questionnaire to express their annoyance.

More than half of the participants who completed the questionnaire (63.7%) had previously visited the Amazones website. This could introduce bias because participants familiar with the original disclosure of stories might be especially satisfied with the new search facilities. However, frequency of visiting the Amazones site, knowing the site “rather well” or “well,” and the number of Amazones stories read before, were all evenly distributed over the four study groups. Therefore, we do not think this previous experience with the Amazones website affected the results.

Since the experiment was conducted completely online [24], we cannot verify that all participants indeed had (or had had) breast cancer. However, we targeted this group for recruitment and asked relevant questions before randomization. We assume that all participants were sincere, because overall they spent 15 minutes filling in the final questionnaire. This suggests that participants were interested in the subject matter.

A limitation of the design was a possible confounding between type of search facility (story topics, writer profile) and number of search options (17 topics, 6 personal characteristics). The story topics search might have been more appreciated because it was more extensive than the writer profile search. However, an argument against this reasoning is that the most extensive search facility (ie, story topics in combination with writer profile) was not the most favourite.

In addition, one could question the content of the search facilities. Were the most appropriate topics and personal characteristics included in the facilities? Yet, the topics and characteristics we used were chosen based on other websites containing breast cancer stories [11-13] and other studies in this field [1,9,10].

A final limitation is that participants may have been annoyed when stories were presented that did not exactly match their search objectives. However, a search resulting in no stories could also be a cause of annoyance. This is why we chose to present at least ten stories after each search. In order to ease interpretation of the resulting list of stories, weights (as pictured in the form of pink ribbons) were used to indicate to what extent a story matched with the search objective.

Conclusions and Practical Implications

Earlier studies have shown that patients can benefit from stories of other patients, and that the Internet is an important source of these stories. Our current study suggests that a story topics search facility would be most helpful to patients. With a story topics search facility, participants were better enabled to find the information they were looking for. Also, they retrieved stories that more closely covered their information needs and they learned more from the stories retrieved.

Thus, patient organisations or website developers that offer patient stories on their websites can best provide access to them using a story topics search facility. However, constructing such a search facility is very time consuming and labour intensive since stories have to be coded for content. An efficient method might be to use a system analogous to social bookmarking/tagging [25] in which story readers assign keywords or tags to the stories, and the keywords or tags that are most often assigned are seen as most important in describing the content. Another possibility is to construct a list of items from which writers can compose descriptions of their stories. Finally, stories could also be classified by automatic full text indexing or clustering. This will be the subject of our next study.