Clinician Search Behaviors May Be Influenced by Search Engine Design

Introduction

Searching for information on the Web to support decision making is now an important part of clinician practice [1]. While much research focuses on the design of retrieval algorithms to identify potentially relevant documents, there has been little examination of the way that different search engine capabilities influence search behavior. Yet, to develop information retrieval systems that actively support decision making, it is necessary to understand the complex process of how people search for and review information when making decisions [2] and to design search user interfaces appropriate for these needs.

Recent studies of clinical search strategies have concentrated on methods of optimizing queries sent to information retrieval systems that enhance the performance of the retrieval. Hoogendam and colleagues conducted a prospective observational study of how physicians at a hospital used PubMed to search for information during their daily clinical activities [3]. They found that the likelihood of physicians viewing article abstracts returned from PubMed increased as the number of terms contained in a search query increased. Haase and colleagues investigated the optimal performance for different search engines in retrieving clinical practice guidelines by combining different search query terms [4]. Our own prior analysis of information searching by clinicians used a Bayesian belief revision framework to retrospectively model how documents might influence decisions during and after a search session [5]; the analysis demonstrated that clinicians can experience cognitive biases while searching for online information to answer clinical questions [6].

Few studies have looked at how clinicians reformulate queries and select sources to retrieve information during a search session to answer clinical questions. In previous studies, we have shown that a task-based search engine design allows for faster clinical decision making (ie, “decision velocity”) compared with purely resource-based engines at no cost in correctness of answers [7]. Similar results with respect to search times have been noted by others for the use of topic-specific “infobuttons” [8]. In the current study, we sought to understand the basis for these performance variations, by testing whether differences in search engine interface design are associated with any differences in user search behaviors.

Methods

Participants and Study Design

In all, 75 clinicians (44 doctors and 31 clinical nurse consultants) practicing in the state of New South Wales, Australia, were recruited to use an online information retrieval system to answer questions on 8 clinical scenarios within 80 minutes in a controlled setting in a university computer laboratory (Table 1) [9]. Participants had an average of 17 years of clinical experience, with the majority having rated their computer skills as good to excellent and having reported use of an online information retrieval system once per week or more.

Participants were randomly allocated to use either a resource-based or a task-based version of an online information retrieval system to answer the 8 questions. All participants were given a brief written orientation tutorial regarding their allocated system. Questions were presented in random order. Each participant was asked to use the allocated system to locate documentary evidence to help answer each question. Participants were asked to work through the questions as they would in a real clinical setting and not spend more than 10 minutes on any one question.

Resource-based System Versus Task-based System

The search systems used by participants were essentially identical in that both systems allowed users to first select a profile (ie, search filter) to delimit their search and then to enter keywords to specify the focus of their search. The resource-based system first required clinicians to select a profile by specifying one of six online resources. These included PubMed, MIMS (a pharmaceutical database), Therapeutic Guidelines (an Australian synthesized evidence source focusing on guidelines for therapy), the Merck Manual, Harrison’s Principles of Internal Medicine, and HealthInsite (a government-funded consumer-oriented health database). Of the six resources, five presented evidence in a predigested, summarized form with references available for follow-up.

The task-based system first required the clinicians to select a profile by selecting one of six clinical tasks: diagnosis, drug information, etiology, patient education, treatment, and other (Figure 1). Four keyword categories were available for both systems: disease, drug, symptom, and other. Clinicians could enter keywords under one or more of these categories. Quick Clinical (University of New South Wales, Sydney, Australia), the task-based information retrieval system, utilized meta-search filters to simultaneously search across a set of disparate information sources [10]. This task-based system has been demonstrated to be effective and efficient in searching and delivering information in various technical, laboratory, and longitudinal evaluation studies [9-14].

‎View this figure

Coding of Search Actions

System logs unobtrusively capturing participants’ interactions with the systems were coded and analyzed for their search actions and query reformulation strategies. For each clinical scenario question, participants were able to reformulate queries and conduct a sequence of searches as they explored information to assist in answering the question. We first coded these query reformulations by the change in profile selection (task or resource) between consecutive searches in a session as “new profile,” “same profile,” or “previously used profile.” We next coded the keyword changes, as indicating a syntactic and/or a semantic reformulation [14]. Examples of syntactic reformulations include changing the following: the use of capitalization, the order of words, the conjunctions used between words, word spacing, or the typographic of the words (ie, variants of the base form of the word) used in the query (Table 2). Semantic reformulations include adding, removing, or replacing keywords.

Quantitative Analyses

Chi-square analyses and the test for difference between proportions were conducted to detect statistically significant differences in profile and query search actions between clinicians using the resource-based and task-based systems.

Results

Of 75 clinicians, 39 were randomly allocated to use of the resource-based system and 36 to use of the task-based system. Two resource-based scenarios were not completed, giving a total of 310 (ie, 39×8−2) search sessions, 1708 searches, and 1455 document accesses using the resource-based system. The task-based system generated 288 (ie, 36×8) search sessions, 873 searches, and 1136 document accesses.

Next Action in a Search Sequence

Chi-square analyses conducted of data presented in Table 3, Table 4, and Table 5 showed statistically significant differences in the next action in a search sequence between the resource-based and task-based systems. These significant differences included (1) selecting the next profile in a search sequence (χ²₂ = 103.45, P < .001) (Table 3), (2) reformulating keywords (χ²₃ = 59.37, P < .001) (Table 4), and (3) both selecting the next profile and reformulating keywords (χ²₁₁ = 165.33, P < .001) (Table 5).

The test for difference between proportions revealed that clinicians using the resource-based system were 19.5% more likely to select a new profile and apply no changes to keywords (Z =11.43, P < .001), and 5.9% more likely to select a profile that was previously visited and apply no changes to keywords (Z = 5.80, P < .001) (Table 5). Also, clinicians using the task-based system were 7.8% more likely to keep the same profile in a sequence of search actions (Z = –5.28, P < .001), 7.5% to keep the same profile and apply both syntactic and semantic changes to the query (Z = –4.69, P < .001), and 6.5% to keep the same profile and apply semantic changes to the query (Z = –3.37, P < .001) (Table 5). Further, task-based clinicians seldom accessed a profile that had been previously visited (9.4%, 95% CI 7.29-12.04) (Table 3).

Search Actions During a Session

We examined search behaviors at the beginning, middle, and end of a search sequence. At the beginning of a search sequence, query reformulation was the most frequent choice for both systems (Table 6). In the middle of a session, clinicians using the resource-based system were 26.6% more likely to change profile only (Z = 10.21, P < .001) (Table 6), and clinicians using the task-based system were 20.7% more likely to reformulate query only (Z = –6.06, P < .001) (Table 6). At the end of a sequence, clinicians using the resource-based system were 26.7% more likely to change profile only (Z = 6.50, P < .001) (Table 6), and clinicians using the task-based system were 14.9% more likely to reformulate query only (Z = –2.75, P = .006) (Table 6).

Consecutive Search Actions

Table 7 displays comparisons of the frequencies of use of consecutive pairs of actions anywhere within a sequence between the two systems. For clinicians using the resource-based system, the pair “change profile only” followed by “change profile only” was 18.6% more likely (Z = 13.88, P < .001) (Table 7). Among clinicians using the task-system, the pair “change query only” followed by “change query only” was used 17.8% more frequently compared with clinicians using the resource-based system (Z = –6.95, P < .001) (Table 7).

Discussion

Clinicians using the resource-based system appeared to favor a “breadth-first” search strategy, exploring different resources with the same keywords in the query before searching in a specific resource with query reformulations. Clinicians using the task-based system were provided with results from multiple resources in each search and so appeared to favor a “depth-first” search strategy, searching in the same task profile exhaustively with different keyword reformulations in the query before moving to other profiles.

We have previously shown that changes in search engine design and interface were associated with changes in clinical decision velocity, number of search actions undertaken, and ultimate decision outcome [7]. To understand the basis for such differences, we have now looked at the type of actions undertaken by users of two different systems and the sequences of these actions. While it was the intention of the experiment to detect changes in search behavior, our present analysis extends the analytic framework of the original experiments and may thus suffer from being a post hoc explanation of the observed differences. This limitation may readily be addressed by further experiments specifically designed to test for changes in search strategy.

Further study is needed to understand how clinicians assess the results of a search and formulate the next step in their strategy. We have discussed elsewhere that the process of searching can be thought of as a conversation [15] where individuals ask questions of knowledgeable agents (eg, information retrieval systems or people) to help find answers to their questions. Thinking of the interaction with a search engine as a conversation between a human with a question and a search engine with capabilities to help find an answer may help us understand the human behaviors observed in this study.

According to Grice’s conversational maxims [15], (which were originally created to describe the “rules” for effective human conversations), an answer to a question may be inappropriate for a number of reasons. The respondent may be poorly qualified to answer the question (eg, the respondent may be an inappropriate, out of date, or otherwise misleading information source); may misunderstand the question (eg, the query may not be well expressed in terms understandable by the resource); or may reply with unhelpful or irrelevant information (eg, because of poor relevance metrics of the search algorithm). We can speculate that the search actions taken by clinicians are in response to judgments they make about the progress of their “conversation” with the information retrieval system.

One can hypothesize, when clinicians are faced with a choice of several resources with no clear indication of which is the best, they scan multiple resources to gauge the "competence" of each before committing to a detailed conversation with the resource they feel best qualified to help. In contrast, clinicians with a task-based system are simultaneously receiving answers from multiple resources and so should be able to quickly form a view of the overall capabilities of the group of resources being simultaneously searched. Not faced with concerns about the competence of the system they are interacting with, clinicians focus on improving the dialogue with the system. This is done by finding different ways to ask the same question or by changing the question focus if there has been a “misunderstanding.” As a result, this could explain why users of task-based systems conduct fewer searches and consult fewer documents [7], that is, these users may not need to credential the resources they are interacting with in the same way that users of resource-based systems appear to do.

Overall, given the clear differences in the styles of user-system dialogue demonstrated in this study, and the impact of such behavior on the clinical utility of information retrieval systems, discovering ways of optimizing the dialogue between knowledge sources and users seems a productive line of further enquiry.