This paper is in the following e-collection/theme issue:

Information Retrieval (284) Information Seeking, Information Needs (461) mHealth in a Clinical Setting (609) Technology in Physiotherapy (133)

Published on in Vol 22, No 12 (2020): December

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/19747, first published .
Physiotherapists’ Use of Web-Based Information Resources to Fulfill Their Information Needs During a Theoretical Examination: Randomized Crossover Trial

Physiotherapists’ Use of Web-Based Information Resources to Fulfill Their Information Needs During a Theoretical Examination: Randomized Crossover Trial

Physiotherapists’ Use of Web-Based Information Resources to Fulfill Their Information Needs During a Theoretical Examination: Randomized Crossover Trial

Authors of this article:

Cailbhe Doherty1, 2 Author Orcid Image ;   Arash Joorabchi2, 3 Author Orcid Image ;   Peter Megyesi2 Author Orcid Image ;   Aileen Flynn4 Author Orcid Image ;   Brian Caulfield2 Author Orcid Image
Article Authors Cited by (3) Tweetations (10) Metrics

    Original Paper

    1School of Public Health, Physiotherapy and Sports Science, University College Dublin, Dublin, Ireland

    2Insight Centre for Data Analytics, O'Brien Centre for Science, University College Dublin, Dublin, Ireland

    3Department of Electronic & Computer Engineering, University of Limerick, Limerick, Ireland

    4Beacon Hospital, Dublin, Ireland

    Corresponding Author:

    Cailbhe Doherty, BSc, PhD

    School of Public Health, Physiotherapy and Sports Science

    University College Dublin

    A308, Health Science Building

    Belfield

    Dublin, D4

    Ireland

    Phone: 353 17166511

    Email: cailbhe.doherty@ucd.ie


    Background: The widespread availability of internet-connected smart devices in the health care setting has the potential to improve the delivery of research evidence to the care pathway and fulfill health care professionals’ information needs.

    Objective: This study aims to evaluate the frequency with which physiotherapists experience information needs, the capacity of digital information resources to fulfill these needs, and the specific types of resources they use to do so.

    Methods: A total of 38 participants (all practicing physiotherapists; 19 females, 19 males) were randomly assigned to complete three 20-question multiple-choice questionnaire (MCQ) examinations under 3 conditions in a randomized crossover study design: assisted by a web browser, assisted by a federated search portal system, and unassisted. MCQ scores, times, and frequencies of information needs were recorded for overall examination-level and individual question-level analyses. Generalized estimating equations were used to assess differences between conditions for the primary outcomes. A log file analysis was conducted to evaluate participants’ web search and retrieval behaviors.

    Results: Participants experienced an information need in 55.59% (845/1520) MCQs (assisted conditions only) and exhibited a mean improvement of 10% and 16% in overall examination scores for the federated search and web browser conditions, respectively, compared with the unassisted condition (P<.001). In the web browser condition, Google was the most popular resource and the only search engine used, accounting for 1273 (64%) of hits, followed by PubMed (195 hits; 10% of total). In the federated search condition, Wikipedia and PubMed were the most popular resources with 1518 (46% of total) and 1273 (39% of total) hits, respectively.

    Conclusions: In agreement with the findings of previous research studies among medical physicians, the results of this study demonstrate that physiotherapists frequently experience information needs. This study provides new insights into the preferred digital information resources used by physiotherapists to fulfill these needs. Future research should clarify the implications of physiotherapists’ apparent high reliance on Google, whether these results reflect the authentic clinical environment, and whether fulfilling clinical information needs alters practice behaviors or improves patient outcomes.

    J Med Internet Res 2020;22(12):e19747

    doi:10.2196/19747

    Keywords

    evidence-based medicine; knowledge discovery; information seeking behavior; information dissemination; information literacy; online systems; point-of-care systems; mobile phone 


    Health Care Professionals’ Information Seeking Behaviors: A Brief History

    The ubiquity of internet-connected smart devices in the clinical setting [1] and the availability of preappraised point-of-care evidence summaries [2,3] have changed how health care professionals seek, consume, and implement information in the care pathway [4]. For instance, a generation now separates the sample of health care professionals who participated in the seminal work of Covell et al [5] in 1985 and their digitally native descendants. That sample (47 medical physicians) reported having 2 clinical questions for every 3 patient visits and preferred textbooks, journals, and drug information indexes to fulfill their information needs [5]. Importantly, these findings are now irreconcilable with modern clinical archetypes as “computered sources were reported to be used least often” [5].

    Since then, the delivery of clinically relevant information to the care pathway has progressed to smart devices capable of running stand-alone software apps. At present, an increasing proportion of health care professionals use these devices to inform their practice [1]. This change has been charted in thousands of studies [6] and syntheses of studies [7,8], which have demonstrated the use of web-connected smart devices for fulfilling clinical information needs [5,8-11]

    Empirical Research of Health Care Professionals’ Information Seeking Behaviors

    Unfortunately, the current understanding of information-seeking and utilization behaviors is shaped by suboptimal empirical models. Self-report questionnaires [5,9], interviews [12], and think-aloud [13-15] methodologies may offer a feasible way to estimate how health care professionals perceive their own information-seeking behaviors, their frequency, the topics to which they relate, the resources they use to fulfill these needs, and the strategies they employ to harvest information from these resources. However, these behaviors can now be more directly observed using web server log file analysis [16-18].

    To date, researchers have leveraged web logs to evaluate usage patterns for specific websites such as Wikipedia [19,20], PubMed [21,22], UpToDate [23], and web-based institutional health science libraries [16,17]. Typically reported usage metadata include the number of time- or user-dependent hits or sessions on the website [24,25], the most frequently searched topics [18,22], and the number of click-throughs from one section of the website to another [18].

    Importantly, these analyses tend to be constrained to individual websites [22] or institutional portals [23], which redirect to a limited group of external websites. These constraints undermine the validity of modern information-seeking paradigms and bely our understanding of the specific digital information resources that are used in an unconstrained web environment to fulfil information needs. Health care professionals in the real world have unbounded web access; this does not reflect the sandboxes of preselected information resources that are evaluated in the available body of research. Furthermore, little work in the web log analysis literature has focused on how these resources can be used to answer clinical questions.

    Aims and Objectives of the Study

    Therefore, the aim of this study is to evaluate the use of web-based resources for fulfilling clinical information needs. Specifically, our objectives were to conduct a randomized crossover trial, whereby a group of physiotherapists were subjected to a multiple-choice questionnaire (MCQ) examination, which they completed under 3 conditions: (1) assisted via a web browser with unconstrained web access, (2) assisted via a federated search engine “portal” app, and (3) unassisted.

    By including both a federated search system and unconstrained web use in a single study design, we sought to evaluate the potentially mediating effects of the access tool on the kinds of resources being used and the time spent doing so. This study aims to address both hypothesis-confirming and hypothesis-generating research questions.

    Experimental Hypotheses

    The confirmatory hypotheses were as follows:

    1. Physiotherapists frequently encounter information needs when presented with simulated clinical questions [5,9-11].
    2. Digital information resources available on the web can be used to fulfill these needs, resulting in a higher MCQ examination score in the assisted conditions [26].

    The hypotheses that this research may generate relate to the following:

    1. The specific web-based resources used by physiotherapists to fulfill their information needs.
    2. The rate of answering correctly in the presence or absence of an information need and its relationship with the use or nonuse of a digital information resource.
    3. Whether constraining physiotherapists to a limited number of web-based digital information resources (as in the federated search condition) is associated with a difference in the rate with which questions are answered correctly or the time spent in searching for information.

    Ethics

    Ethical approval for this randomized crossover trial was granted by the affiliate review board of the institution at which the authors were based (reference: LS-18-25). Study design, conduct, analysis, and results are reported according to the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth) checklist.

    Recruitment

    Prospective participants included practicing physiotherapists who were recruited with the help of collaborating health care institutions. The study was advertised on the websites of these institutions, and a notice was sent throughout the recruitment period to prospective participants via the “staff bulletin” of the respective institutions. Physiotherapists were chosen for evaluation, as their information behavior has not been well addressed in the research literature to date, despite their prominent role in health care [27].

    All individuals met the following inclusion criteria: adults (>18 years of age), currently employed by collaborating health-related institutions and involved with the management and treatment of patients, provided informed consent (which was obtained digitally during authentication with the federated search app).

    Protocol

    Information needs were induced via the administration of an MCQ examination completed at 3 time points under 3 conditions in random order: (1) assisted by a standard web browser, (2) assisted by a federated search engine, and (3) unassisted.

    One investigator (CD) organized an appropriate time and location to conduct all test sessions; to maximize participant retention, testing was conducted at a time and place that suited participants. Short-term rescheduling (ie, within 1 week of the original designated test time) was facilitated in the event of unexpected delays or schedule conflicts.

    For the purpose of this study, we defined an information need in the assisted conditions as any instance in which a participant used an assistive technology to access a digital information resource to inform their choice of answer in the MCQ. In the “assisted by a web browser” condition, participants were provided with a web-connected laptop preinstalled with a web browser (Google Chrome). For the “assisted by a federated search engine” condition, participants were required to install and register an account with a free experimental federated search engine called SciScanner [28] on their smartphone.

    The Multiple-Choice Questionnaire

    Each theoretical examination comprised a 20-item MCQ with one best answer, derived from the Physiotherapy Competency Exam (PCE) [29] prepared by the Canadian Alliance of Physiotherapy Regulators and the United States National Physical Therapy Exam (NPTE) [30]. A bank of >1000 questions was compiled for each of the PCE and NPTE examinations [31,32]. During each examination, a different random subset was taken from this question bank for each participant for each of the 3 examinations they completed. The MCQ was administered on a tablet device using a commercially available testing software that administered random subsets of topic-tagged questions. Participants were assigned questions based on their chosen area of practice using these tags. The software also documented the question that was administered, its associated response, whether that response was correct or incorrect, and the time spent answering it [33]. Participants were informed that the examination was negatively marked to discourage guessing; however, they were not informed about the penalty for an incorrect answer. Participants were also informed that time was not to be considered a metric for performance, whereby they would not be penalized for finishing the examination in a longer period. In the “assisted” experimental conditions, participants were informed that they could opt to use their assigned system but did not have to if they did not deem it necessary or if they were sufficiently confident of their answer. In either case, participants were informed that they would not be penalized for using or not using assistance if it was available to them. The examination was proctored by the first author, who manually documented the frequency of information needs (defined by instances where participants chose to use either the standard web browser or the federated search engine for a given examination question).

    Web and Federated Search Logs

    Participants’ web logs and federated search logs were captured during the 2 assisted conditions.

    Federated Search Logs

    Federated searching describes a system that implements a single query concurrently across multiple disparate collections of information. The federated search engine used in this study is a free experimental platform called SciScanner that centralizes popular health-information resources [34]. At the time of the study, searches were implemented across a series of authoritative (PubMed and the Cochrane library), nonauthoritative (YouTube), and community‐built (Wikipedia) resources in tandem. These were chosen based on prior research identifying the most commonly used, free resources among health care professionals [35-39] and because of the ease with which a simple search string could be used and adapted for each data source while maximizing the consistency of these results [40]. For example, the “Clinical Queries” search strategies were implemented in the system to aid users in finding different types of content on PubMed (eg, related to diagnosis, clinical prediction, or treatment best practices) [41]. The results for a single search query could be accessed for each resource on the app home screen (Figure 1).

    ‎
    Figure 1. Search results for “Deep Vein Thrombosis” on the federated search platform. At the time of the study, the platform queried Wikipedia (top, left), PubMed (“diagnosis” category; top, right), the Cochrane library (bottom, left), and YouTube (bottom, right).
    View this figure

    Following recruitment and before completion of the protocol, participants were required to download and register an account with the SciScanner federated search platform. This authentication process facilitated the tracking of participants’ account-linked information search and retrieval behaviors. The timestamps of all search events, the accounts associated with these events, the search queries themselves, and the resources used for each search query were logged by the system and could be exported in a comma-separated values (CSV) file format for further statistical analysis.

    Web Logs

    Web logs were acquired using a web browser extension. This browser extension was installed on the computer provided to participants during the completion of their examination. When opened, the browser loaded a blank homepage; participants were allowed to select their preferred sites and search engines. The browser extension captured a similar collection of metrics as those described previously, which were used to quantify information search and retrieval behaviors: search queries, associated timestamps, and the URLs of the webpages that were accessed were logged by the extension. These logs were manually exported in a CSV file format following completion of each examination for further analysis.

    Data Preparation

    All web and federated search log files were accrued in a single database after participants had completed the test protocol for each “assisted” condition and were filtered to remove setup (eg, log-in events to institutional networks) and duplicate events. All log files were then grouped into sessions based on the timestamps of the first and last log, which were cross-referenced with the recorded date, start time, and duration of the examination. Each session was then manually segmented by the primary author into individual information-seeking events based on search queries and the timestamps of every event; consecutive search queries that were deemed to be thematically related (eg, “sensation to medial tibia,” “nerve supply to medial leg,” and “saphenous nerve”) and those that occurred within a similar timeframe were grouped as a single information-seeking event. Information-seeking events were then partitioned into 3 periods: primary, intermediate, and terminal. The primary period related to the first search that was logged by participants in an information-seeking session or the first search logged in a thematically similar series of consecutive search segments. Terminal periods were defined as the last search logged by participants in an information-seeking session or the last search logged in a thematically similar series of consecutive search segments. Intermediate periods included consecutive logs between the primary and terminal periods.

    For each period of an information-seeking event, the resources that were accessed were identified. This was automatically recorded in the federated search logs by design and was determined based on the website URL for the web logs. The number of hits and the accumulated time for each hit were then determined for each resource for the entire test cohort. Segmentation, partitioning, and preparation for data analysis are displayed in Figure 2.

    ‎
    Figure 2. An example collection of search sessions segmented into primary, intermediate, and terminal search periods. Strikethrough text depicts that events would have been removed during data preparation, as they coincided with log-in and authentication to an institutional website.
    View this figure

    Sample Size

    Previous research on knowledge acquisition using web-based resources among medical trainees identified a 15% difference between assisted and nonassisted assessment scores (Cohen d=0.86) [21,26]. On this basis, a sample of 18 pairs of physiotherapists was deemed sufficient to achieve experimental power. To account for a 5% (2 of 36) potential dropout, 38 participants (19 pairs) were recruited; however, there were no dropouts, so all participants were included in the analysis.

    Outcomes

    We adopted a 3-tiered analysis paradigm for our outcomes. The first tier was related to the MCQ results (examination-level analysis), the second to the individual questions of the MCQ (question-level analysis), and the third was related to the log files recorded in instances where there was an information need (log-level analysis; assisted conditions only).

    Examination-Level Analysis

    The salient outcomes for the examination-level analysis included participants’ overall MCQ scores and their examination times for each condition. In addition, the total number of information needs was a salient outcome for examinations completed in the “assisted” conditions. Total examination scores were computed without implementing any negative marking.

    Question-Level Analysis

    The salient outcomes for the question-level analysis included the presence or absence of an information need for each question, the time spent answering each question, and whether the question was answered correctly or incorrectly.

    Log-Level Analysis

    The salient outcomes for the log-level analysis included the number of hits, the time spent per hit, and the digital information resource that was used.

    Statistical Analysis

    The demographics of the participating sample were represented using descriptive statistics.

    Examination-Level Analysis

    Generalized estimating equations (GEEs) were used to assess differences in examination scores (assisted with a web browser, assisted with a federated search engine, and unassisted conditions) and information needs (assisted with a web browser and assisted with federated search engine conditions) with examination time included as a covariate using an exchangeable correlation structure. The model was corrected for dependent observations by including the participants’ identifying code as a subject effect. The a priori p value for this analysis was set at P<.05.

    Question-Level Analysis

    A separate model GEE was defined to evaluate the effect of different conditions on individual answers in the presence and absence of an information need with question time included as a covariate in the model. The model was corrected for dependent observations by including the participants’ identifying code as a subject effect. The a priori p value for this analysis was set at P<.05.

    Log-Level Analysis

    Log data for all web-based and federated search–based information resources were represented using means with SDs or medians with IQR, where appropriate, for each period of the information trail. GEEs were used to assess the differences in the time spent per “hit” in each of the 3 periods of the information trail (primary, intermediate, and tertiary) for each of the assisted conditions using an exchangeable correlation structure. This model was corrected for dependent observations by including the participants’ identifying code as a subject effect. The a priori p value for this analysis was set at P<.05.

    All data analyses were performed using Statistical Package for the Social Sciences version 26 (IBM Corp) and Microsoft Excel.


    Participant Characteristics

    A total of 38 physiotherapists fully participated in the study, completing an examination under each of the 3 experimental conditions in random order using Vickers’ block randomization [42]. The demographic characteristics of the included sample are presented in Table 1. All tests were conducted in the 6-month period from June 2019 to January 2020. All participants completed all 3 conditions within a 1-month (4 week) period of enrolling in the study. A CONSORT diagram is presented in Figure 3 [43].

    Table 1. Characteristics of participants.
    CharacteristicsValues
    Male, n19
    Female, n19
    Age (years), mean (95% CI)28.6 (27.5-29.7)
    Number of years practicing, mean (95% CI)5.4 (4.4-6.3)
    Number of individual patient encounters per week, mean (95% CI)26.1 (22.5-29.7)
    ‎
    Figure 3. CONSORT (Consolidated Standards of Reporting Trials) diagram of the study design. Note that information needs were considered in the assisted conditions only. MCQ: multiple-choice questionnaire.
    View this figure

    Examination-Level Analysis

    The examination-level GEE estimated a main effect for condition (P<.001). Examination scores differed significantly between the unassisted condition (mean examination score 47.11%, SD 12.7) and both assisted conditions (mean federated search condition 57.5%, SD 12.8, P=.001; web search condition 63.3%, SD 12%, P=.02). The mean examination score and examination duration for the 3 experimental conditions, and the mean number of information needs for the assisted conditions are presented in Table 2.

    Table 2. Results of the examination-level analysis for the 3 experimental conditions including the mean examination score, duration, and number of information needs for the assisted conditions (with 95% CI).
    Exam parameterNumber of information needs, mean (95% CI)Time (min), mean (95% CI)Score (%), mean (95% CI)
    Assisted

    		Federated search11 (10-12)34 (29-39)58 (53-62)

    		Web search11 (10-12)27 (23-31)63 (59-67)
    UnassistedN/Aa12 (11-14)47 (43-51)

    aN/A: not applicable.

    Question-Level Analysis

    Participants experienced 845 information needs out of a total of 1520 questions, with a rate of 55.6% (assisted conditions only). The question-level GEE was used to further investigate the relationship between individual information needs and answers (correct and incorrect). This GEE estimated a main effect for condition (P<.001) and the presence of an information need (P=.009). An analysis of the parameter estimates associated with these main effects revealed that participants were more likely to answer correctly in both the federated search condition (P=.003) and the web browser condition (P<.001) compared with the unassisted condition.

    Despite the observation of a main effect for the presence of an information need, there were no significant differences based on the parameter estimates at the level of P<.05 in the rate of answering individual questions correctly in the presence or absence of an information need (P=.06).

    The average rate of answering correctly for each condition in the absence or presence of an information need is presented in Table 3.

    Table 3. Mean rate of answering correctly (with 95% CI) for each condition in the absence or presence of an information need.
    ConditionRate of answering correctly (%), mean (95% CI)

    		Federated search conditionWeb search conditionUnassisted condition
    Yes62.1 (57.4-66.7)65.2 (60.6-69.7)N/Aa
    No72.5 (67.6-77.2)65.1 (60-70.2)43.7 (40.1-47.2)

    aN/A: not applicable.

    Log Analysis

    The 845 information needs corresponded to 1987 web logs and 3277 app logs. This data set of app and web logs comprised all participants’ information-seeking sessions, which themselves included multiple information-seeking events (or information trails) split into primary, intermediate, and terminal periods.

    Table 4 presents the most popular resources used during the primary, intermediate, and terminal periods in the web and app log conditions.

    In the web-assisted condition, participants spent a total of 00:45:25 min (383 hits) in the primary period, 04:14:55 hours in the intermediate period (1356 hits), and 06:26:43 hours in the terminal period (248 hits). This corresponded with a median time per resource of 1 second in the primary period, 4 seconds per resource in the intermediate period, and 86 seconds per resource in the terminal period.

    In the federated search condition, participants spent a total of 01:47:21 hours (486 hits) in the primary period, 08:42:24 hours in the intermediate period (2076 hits), and 04:24:06 hours in the terminal period (668 hits). This corresponded with a median time per resource of 3 seconds in the primary period, 4 seconds per resource in the intermediate period, and 5 seconds per resource in the terminal period.

    The log GEE estimated a main effect for condition and for the period of the information trail (P<.001). There was also a significant main effect for the interaction between condition and period (P<.001). An analysis of the parameter estimates associated with these main effects revealed that participants spent less time per hit in the primary and intermediate periods for the web search condition compared with the federated search condition, and more time in the terminal period in the web search condition than in the federated search condition. The median and mean values for each condition stratified by search period are presented in Table 5.

    Table 4. Most popular resources used during the primary, intermediate, and terminal periods for the web and federated search conditions.
    Period, search condition, and resourceHits, nCumulative (%)Time (h:min:s)Cumulative (%)
    Primary

    		Web browser


    		Google Search3599400:39:5588


    		PubMed-NCBIa99600:02:3594


    		Physio-pedia99800:01:3297


    		ResearchGate19900:00:5499


    		Google Scholar29900:00:1599


    		Ovid210000:00:09100

    		Federated search


    		Wikipedia3046200:53:3550


    		PubMed1318800:36:3284


    		YouTube369500:12:1395


    		Cochrane2310000:05:03100
    Intermediate

    		Web browser


    		Google Search8586301:33:5037


    		PubMed-NCBI1417400:49:2356


    		Wikipedia577800:19:3864


    		Physio-pedia488100:18:5771


    		Google Scholar478500:10:0775


    		British Journal of Sports Medicine228700:01:1376


    		ResearchGate148800:04:2577


    		Mayo Clinic98800:03:2279


    		Sci-hub88900:00:5679


    		Cochrane library88900:01:5480

    		Federated search


    		PubMed9374603:56:4846


    		Wikipedia8488703:20:3885


    		Cochrane1339400:25:3190


    		YouTube12910000:51:18100
    Terminal

    		Web browser


    		Google Search562301:33:4924


    		PubMed-NCBI454101:20:1245


    		Physio-pedia335400:53:5359


    		Wikipedia266500:35:3568


    		British Journal of Sports Medicine126900:18:0573


    		Mayo clinic57100:05:1774


    		ResearchGate57300:01:2775


    		Teachmeanatomy47500:06:1976


    		Google Scholar37600:03:0477


    		Mananatomy37700:05:1678

    		Federated search


    		Wikipedia3665501:47:4842


    		PubMed2058501:22:5475


    		YouTube579400:50:4994


    		Cochrane4210000:14:04100

    aNCBI: National Center for Biotechnology Information.

    Table 5. Median and mean values of time spent per resource for the web and federated condition stratified by search period.
    Search periodPrimary (second)Intermediate (second)Tertiary (second)
    Federated search

    		Median (IQR)3 (11)4 (12)5 (26)

    		Mean (95% CI)13.3 (10.6-15.9)15.1 (13.6-16.6)24 (20.3-27.2)
    Web

    		Median (IQR)1 (7)4 (14)86 (55)

    		Mean (95% CI)7.1 (5.3-9)11.3 (10.4-12.2)169.8 (40-299.5)

    Principal Findings

    Frequency of Information Needs

    In this study, we sought to confirm a series of hypotheses. The first concerned the frequency with which physiotherapists experience information needs. Information needs were specifically defined: they were induced during a theoretical examination and anchored to participants’ access or nonaccess of digital information resources in the 2 “assisted” experimental conditions. Under this definition, participants experienced an information need in 55.59% (845/1520) of theoretical examination questions. In clinical environments, previous research has shown that medical physicians report experiencing an information need with varying frequencies. In the study by Covell et al [5], medical physicians had an information need in 67% of their patient encounters, whereas Sackett and Straus [44] observed 98 questions during the care of 166 hospitalized patients in a 30-day period (a rate of 59%). Among family care physicians, Ely et al [45] observed information needs at a rate of 3.2 (or 32%) for every 10 patient visits, while, more recently, Izcovich et al [46] documented 1.2 questions per patient encounter. It is important to note that by anchoring the existence of a need to information-seeking behavior, we could not evaluate information needs in the unassisted condition.

    Regardless of how information needs are defined, timely translation of research evidence to the care pathway is a policy priority of many health research systems [47], yet many obstacles block the channels by which such translation occurs [48]. Ultimately, these obstacles impede effective evidence translation [49].

    The Use of Web-Based Resources to Fulfill Information Needs

    Barriers to obtaining information include time [40,50], accessibility [40,50,51], and limited personal skills [40], yet web-connected digital technologies could overcome these barriers and enable clinicians to fulfill their information needs at the point of care [1]. This contributed to our second confirmatory hypothesis that digital information resources accessed via the web could be used to fulfill information needs, thus improving the overall MCQ examination score. In agreement with this hypothesis, participants exhibited a mean improvement of 10% (for the federated search condition) and 16% (for the web condition) in the overall examination score compared with the unassisted condition. Our analysis at the resolution of individual questions further revealed that participants were more likely to answer correctly in either of the assisted conditions but that there were no differences in the rate of answering correctly in the presence or absence of an information need. Specifically, participants answered 63.7% (538/845) of questions correctly in the presence of an information need, compared with 68.7% (464/675) of questions in the absence of an information need. There was no statistically significant difference between these rates, suggesting that the availability of assistance improved participants’ likelihood of answering a question correctly at a similar rate to that when they were confident of knowing the answer (and therefore did not seek assistance, even if it was available) or to a level where they were sufficiently confident to guess despite the expectation that they would be marked negatively for doing so.

    Previous researchers have sought to affect knowledge acquisition as assessed via MCQ examinations among health care professionals with seminars [52], tutorials [53], and course modules [54]. Course materials are generally developed using the frameworks of evidence-based medicine [55,56]. However, this does not reflect real-world practice, as health care professionals [57,58] autonomously use search engines and nonauthoritative, community‐built content to fulfill their information needs, prioritizing efficiency, familiarity, accessibility, and organization with “just the right” volume, variety, and scope to fulfill their needs [40]. The apparent dissonance between the types of resources that academic staff and researchers encourage health care professionals to use [55,56], those they report using [59,60], and those they actually use [61] informed our hypothesis-generating research questions. Through the log file analysis, we evaluated the digital information resources used by our cohort of physiotherapists in an unconfined web environment and those accessed via a federated search engine that was developed for the purposes of this study.

    Reflecting the available body of observational research evaluating the browsing behaviors of the general population for health-related information [62,63], we observed a preference mainly for search engines and nonauthoritative, community‐built content at all stages of the information-seeking journey. Specifically, in the web-assisted condition, Google was the most popular resource used at the start of the information-seeking journey (359/383, 93.7% of total hits; 00:39:55 of 00:45:25, 88% of the total time). The variety of resources then increased in the “intermediate” period, yet Google was still the most popular (858/1351, 63.51% of total hits; 01:33:50 of 04:13:03, 37% of total time), followed by PubMed (141/1351, 10.44% of total hits; 00:49:23 of 04:13:03, 20% of total time) and Wikipedia (57/1351, 4.22% of total hits; 00:19:38 of 04:13:03, 8% of total time). The terminal period contained the last resources used by participants before their submission of an answer in the MCQ. Therefore, these resources were assumed to be principally responsible for fulfilling participants’ information needs, an assumption reflected in the relative amount of time spent in this period (median duration of 86 seconds), compared with the primary (median duration of 1 second) and intermediate (median duration of 4 seconds) periods. In the terminal period, Google searches again accounted for the largest number of hits with 22.5% (56/248) of the total, which corresponded to 01:33:49, or 25% of the total duration (06:18:13), followed by PubMed (45/248, 18.1% of total hits; 01:20:12 of 06:18:13, 21% of total duration), Physio-pedia (33/248, 13.3% of total hits; 00:53:53 of 06:18:13, 14% of total duration), and Wikipedia (26/248, 10.4% of total hits; 00:35:35 of 06:18:13, 9% of total duration). That Google, PubMed, and Wikipedia were the most popular resources among a cohort of physiotherapists is in agreement with a small body of prior research in other health care professional groups [61]. However, that Google in particular is heavily relied upon as both a directory to other resources and as a resource itself is potentially problematic, owing to its propensity to display results in a “filter bubble” [64]. Indeed, Google was the only search engine used by participants in this study, and its tendency to display previews of the information contained in individual websites potentially bypasses the need to visit these websites and to properly apprise the evidence [65]. The impact of this in the context of health care delivery has yet to be formally evaluated, but the available body of research suggests that the use of Google may not always be conducive to acquiring valid and reliable health-related information [66,67].

    The federated search engine in this study was designed as an alternative mode of assistance, bypassing the need for traditional web search filter bubbles and improving the efficiency of navigating specific academic databases. Unsurprisingly, due to the constraints on the resources that were included in the federated search, a less diverse array of information resources were observed in the federated search condition. Specifically, Wikipedia accounted for the greatest number of hits in the primary period (304/494, 61.5% of hits; 00:53:35 of 01:47:23, 50% of time), PubMed accounted for the greatest number of hits in the intermediate period (937 of 2047 hits and 03:56:48 of 08:34:14, or 45.77% of both total hits and time), and Wikipedia accounted for the greatest number of hits in the terminal period (366/670, 54.6% of total hits; 01:47:48 of 04:15:35, 42% time), suggesting that participants sought out these resources despite the differences in their mode of delivery. Although the difference was not statistically significant, participants spent longer doing the examination in the federated search condition (34 min vs 27 min in the web condition). However, participants did spend significantly less time in the terminal period (a median duration of 5 seconds, compared with 86 seconds in the web condition) and clicked on a greater number of resources (there were 1290, or 65% more hits in the federated search condition compared with the web condition). In summary, these findings may suggest that the federated search was less effective in finding relevant and useful information, required more effort to locate that information, or that participants used the primary, intermediate, and terminal resources together to fulfill their information needs, rather than reaching an information end-point through a terminal resource as in the web condition. In relation to our second hypothesis-generating research question, that there was no significant difference in the examination scores between the 2 assisted conditions, and no significant difference in the rate with which questions were answered correctly or incorrectly in the presence or absence of an information need, would suggest the latter conclusions to be more likely.

    Limitations

    Despite the methodological strengths of the crossover experimental design and the insights garnered via the use of web log analysis to determine what digital information resources were used to fulfill information needs and the novelty of evaluating these outcomes among physiotherapists, this study is not without limitations. First, an MCQ was used as a surrogate stimulus for information needs and may not be an accurate representation of the clinical setting. The validity of the MCQ questions as a measure of physiotherapy knowledge and by extension, health care delivery, was assumed on the basis that they were from the examinations of recognized accreditation bodies. It must be acknowledged that although these questions are used to assess physiotherapists’ capacity to practice, they may not actually represent the information needs of the authentic clinical encounter. Second, this study does not address an enduring question as to whether the fulfillment of clinical information needs alters treatment practices and patients’ outcomes; it has always been assumed, but never proven [46], that the translation of the latest high-quality research evidence to the care pathway optimizes clinical practice behaviors and improves patients’ outcomes. Finally, although the use of a federated search portal system provides a useful comparator for unconstrained web browsing in evidence search and retrieval patterns and their effect on MCQ examination scores, the results presented in this study for this portal system (ie, SciScanner) are unique to it and have limited external applicability for other portal systems.

    Conclusions

    On the basis of an MCQ examination protocol, we identified that physiotherapists experienced an information need in 55.59% (845/1520) of theoretical questions and, when they were provided with access to digital information resources accessed via the web or a federated search software app, the fulfillment of these needs was associated with improved rates of answering examination questions correctly. The physiotherapists in this study exhibited a high preference for Google as both a directory and a resource, with Wikipedia and PubMed being the next most popular resources. The implications of relying heavily on Google as a search and retrieval mechanism for health-related information warrants further investigation, whereas the emulation of these findings in an authentic clinical setting would be an important research pursuit in the future.

    Acknowledgments

    This research was cofunded by the European Regional Development Fund under Ireland’s European Structural and Investment Funds Programmes 2014-2020 under the SciScanner project title and the Health Research Board (ARPP-A-2018-002).

    Conflicts of Interest

    The authorship team (PM, AJ, and CD) was responsible for the development of the federated search portal app. The authors have no conflicts of interest to declare as the portal system accesses publicly available digital information resources unrelated to this group.

    1. Mobasheri MH, King D, Johnston M, Gautama S, Purkayastha S, Darzi A. The ownership and clinical use of smartphones by doctors and nurses in the UK: a multicentre survey study. BMJ Innov 2015 Oct 7;1(4):174-181. [CrossRef]
    2. Campbell R, Ash J. An evaluation of five bedside information products using a user-centered, task-oriented approach. J Med Libr Assoc 2006 Oct;94(4):435-41, e206 [FREE Full text] [Medline]
    3. Dicenso A, Bayley L, Haynes RB. Accessing pre-appraised evidence: fine-tuning the 5S model into a 6S model. Evid Based Nurs 2009 Oct;12(4):99-101. [CrossRef] [Medline]
    4. Pettigrew K, Fidel R, Bruce H. Conceptual frameworks in information behavior. Annual Review of information Science and Technology (ARIST) 2001;35:43-78 [FREE Full text]
    5. Covell DG, Uman GC, Manning PR. Information needs in office practice: are they being met? Ann Intern Med 1985 Oct;103(4):596-599. [CrossRef] [Medline]
    6. Case DO. Looking for information: a survey of research on information seeking, needs, and behavior. In: Library Management. Bingley, UK: Emerald Group Publishing; 2012.
    7. Davies K, Harrison J. The information-seeking behaviour of doctors: a review of the evidence. Health Info Libr J 2007 Jun;24(2):78-94 [FREE Full text] [CrossRef] [Medline]
    8. Daei A, Soleymani MR, Ashrafi-Rizi H, Zargham-Boroujeni A, Kelishadi R. Clinical information seeking behavior of physicians: a systematic review. Int J Med Inform 2020 Jul;139:104144. [CrossRef] [Medline]
    9. Osheroff JA, Forsythe DE, Buchanan BG, Bankowitz RA, Blumenfeld BH, Miller RA. Physicians' information needs: analysis of questions posed during clinical teaching. Ann Intern Med 1991 Apr 01;114(7):576-581. [CrossRef] [Medline]
    10. Segev E, Sharon AJ. Temporal patterns of scientific information-seeking. Public Underst Sci 2016 May 20;26(8):969-985. [CrossRef]
    11. Ahamad A, Wallner P, Salenius S, Ross R, Fernandez E. Information needs expressed during patient-oriented oncology consultations: quantity, variation, and barriers. J Cancer Educ 2019 Jun;34(3):488-497. [CrossRef] [Medline]
    12. Swennen M, van der Heijden GJ, Boeije HR, van Rheenen N, Verheul FJ, van der Graaf Y, et al. Doctors' perceptions and use of evidence-based medicine: a systematic review and thematic synthesis of qualitative studies. Acad Med 2013 Sep;88(9):1384-1396. [CrossRef] [Medline]
    13. Currie L, Graham M, Allen M, Bakken S, Patel V, Cimino JJ. Clinical information needs in context: an observational study of clinicians while using a clinical information system. AMIA Annu Symp Proc 2003:190-194 [FREE Full text] [Medline]
    14. McKibbon KA, Fridsma DB. Effectiveness of clinician-selected electronic information resources for answering primary care physicians' information needs. J Am Med Inform Assoc 2006;13(6):653-659 [FREE Full text] [CrossRef] [Medline]
    15. Borycki E, Lemieux-Charles L, Nagle L, Eysenbach G. Novice nurse information needs in paper and hybrid electronic-paper environments: a qualitative analysis. Stud Health Technol Inform 2009;150:913-917. [Medline]
    16. Rozic-Hristovsk A, Hristovski D, Todorovski L. Users' information-seeking behavior on a medical library website. J Med Libr Assoc 2002 Apr;90(2):210-217 [FREE Full text] [Medline]
    17. Bracke P. Web usage mining at an academic health sciences library: an exploratory study. J Med Libr Assoc 2004 Oct;92(4):421-428 [FREE Full text] [Medline]
    18. Meats E, Brassey J, Heneghan C, Glasziou P. Using the Turning Research Into Practice (TRIP) database: how do clinicians really search? J Med Libr Assoc 2007 Apr;95(2):156-163 [FREE Full text] [CrossRef] [Medline]
    19. Lamprecht D, Lerman K, Helic D, Strohmaier M. How the structure of Wikipedia articles influences user navigation. New Rev Hypermedia Multimed 2017 Jan 2;23(1):29-50 [FREE Full text] [CrossRef] [Medline]
    20. Lamprecht D, Strohmaier M, Helic D, Nyulas C, Tudorache T, Noy NF, et al. Using ontologies to model human navigation behavior in information networks: a study based on Wikipedia. SW 2015 Aug 7;6(4):403-422. [CrossRef]
    21. Hersh W, Crabtree MK, Hickam DH, Sacherek L, Friedman CP, Tidmarsh P, et al. Factors associated with success in searching MEDLINE and applying evidence to answer clinical questions. J Am Med Inform Assoc 2002;9(3):283-293 [FREE Full text] [CrossRef] [Medline]
    22. Yoo I, Mosa AS. Analysis of pubmed user sessions using a full-day pubmed query log: a comparison of experienced and nonexperienced pubmed users. JMIR Med Inform 2015 Jul 2;3(3):e25 [FREE Full text] [CrossRef] [Medline]
    23. Maggio LA, Steinberg RM, Moorhead L, O'Brien B, Willinsky J. Access of primary and secondary literature by health personnel in an academic health center: implications for open access. J Med Libr Assoc 2013 Jul;101(3):205-212 [FREE Full text] [CrossRef] [Medline]
    24. Herskovic JR, Tanaka LY, Hersh W, Bernstam EV. A day in the life of PubMed: analysis of a typical day's query log. J Am Med Inform Assoc 2007 Mar 1;14(2):212-220. [CrossRef]
    25. Islamaj Dogan R, Murray GC, Névéol A, Lu Z. Understanding PubMed user search behavior through log analysis. Database (Oxford) 2009;2009:bap018 [FREE Full text] [CrossRef] [Medline]
    26. Scaffidi MA, Khan R, Wang C, Keren D, Tsui C, Garg A, et al. Comparison of the impact of Wikipedia, UpToDate, and a digital textbook on short-term knowledge acquisition among medical students: randomized controlled trial of three web-based resources. JMIR Med Educ 2017 Oct 31;3(2):e20 [FREE Full text] [CrossRef] [Medline]
    27. Kloda LA, Bartlett JC. A characterization of clinical questions asked by rehabilitation therapists. J Med Libr Assoc 2014 Apr;102(2):69-77 [FREE Full text] [CrossRef] [Medline]
    28. Joorabchi A, Doherty C, Dawson J. 'WP2Cochrane', a tool linking Wikipedia to the Cochrane Library: results of a bibliometric analysis evaluating article quality and importance. Health Informatics J 2020 Sep;26(3):1881-1897 [FREE Full text] [CrossRef] [Medline]
    29. Physiotherapy Competency Exam. Canadian Alliance of Physiotherapy Regulators. 2019 Jan 01.   URL: https://www.alliancept.org [accessed 2018-12-01] [WebCite Cache]
    30. National Physical Therapy Exam. National Physical Therapy Association.   URL: https://www.apta.org/Default.aspx [accessed 2018-12-01] [WebCite Cache]
    31. Fontinberry B. Q&A Review for the Physical Therapist Assistant Board Examination. Missouri, USA: Saunders; 2010.
    32. Tyson A. NPTE Study Guide 2017: Practice Questions for the National Physical Therapy Examination. -: -; 2017.
    33. ProProfs QuizMaker. ProProfs. 2019.   URL: https://www.proprofs.com [accessed 2018-12-01]
    34. Doherty C, Joorabchi P, Megyesi P. SciScanner: Search Science. 2020.   URL: www.sciscanner.com [accessed 2019-12-01]
    35. Alkhateeb F, Clauson KA, Latif DA. Pharmacist use of social media. Int J Pharm Pract 2011 Apr;19(2):140-142. [CrossRef] [Medline]
    36. Egle JP, Smeenge DM, Kassem KM, Mittal VK. The internet school of medicine: use of electronic resources by medical trainees and the reliability of those resources. J Surg Educ 2015;72(2):316-320. [CrossRef] [Medline]
    37. Allahwala UK, Nadkarni A, Sebaratnam DF. Wikipedia use amongst medical students – new insights into the digital revolution. Medical Teacher 2012 Nov 8;35(4):337. [CrossRef]
    38. Heilman JM, Kemmann E, Bonert M, Chatterjee A, Ragar B, Beards GM, et al. Wikipedia: a key tool for global public health promotion. J Med Internet Res 2011 Jan 31;13(1):e14 [FREE Full text] [CrossRef] [Medline]
    39. Cochrane and Wikipedia: Working Together to Improve Access to Health Evidence. Cochrane. 2017.   URL: https://www.cochrane.org/news/cochrane-and-wikipedia-working-together-improve-access-health-evidence [accessed 2018-12-01]
    40. Aakre C, Maggio LA, Fiol GD, Cook DA. Barriers and facilitators to clinical information seeking: a systematic review. J Am Med Inform Assoc 2019 Oct 1;26(10):1129-1140. [CrossRef] [Medline]
    41. -. PubMed clinical queries. Pediatric Neurology 2005 May;32(5):340. [CrossRef]
    42. Vickers AJ. How to randomize. J Soc Integr Oncol 2006;4(4):194-198 [FREE Full text] [CrossRef] [Medline]
    43. Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res 2011 Dec 31;13(4):e126 [FREE Full text] [CrossRef] [Medline]
    44. Sackett DL, Straus SE. Finding and applying evidence during clinical rounds: the 'evidence cart'. J Am Med Assoc 1998 Oct 21;280(15):1336-1338. [CrossRef] [Medline]
    45. Ely JW, Osheroff JA, Ebell MH, Bergus GR, Levy BT, Chambliss ML, et al. Analysis of questions asked by family doctors regarding patient care. Br Med J 1999 Aug 7;319(7206):358-361 [FREE Full text] [CrossRef] [Medline]
    46. Izcovich A, Malla CG, Diaz MM, Manzotti M, Catalano HN. Impact of facilitating physician access to relevant medical literature on outcomes of hospitalised internal medicine patients: a randomised controlled trial. Evid Based Med 2011 Oct;16(5):131-135. [CrossRef] [Medline]
    47. Westfall JM, Mold J, Fagnan L. Practice-based research—'Blue Highways' on the NIH Roadmap. J Am Med Assoc 2007 Jan 24;297(4):403. [CrossRef]
    48. Sung NS, Crowley WF, Genel M, Salber P, Sandy L, Sherwood LM, et al. Central challenges facing the national clinical research enterprise. J Am Med Assoc 2003 Mar 12;289(10):1278-1287. [CrossRef] [Medline]
    49. O'Connor PJ, Amundson G, Christianson J. Performance failure of an evidence-based upper respiratory infection clinical guideline. J Fam Pract 1999 Sep;48(9):690-697. [Medline]
    50. Dysart AM, Tomlin GS. Factors related to evidence-based practice among US occupational therapy clinicians. Am J Occup Ther 2002;56(3):275-284. [CrossRef] [Medline]
    51. Hall EF. Physical therapists in private practice: information sources and information needs. Bull Med Libr Assoc 1995 Apr;83(2):196-201 [FREE Full text] [Medline]
    52. Amsallem E, Kasparian C, Cucherat M, Chabaud S, Haugh M, Boissel J, et al. Evaluation of two evidence-based knowledge transfer interventions for physicians. A cluster randomized controlled factorial design trial: the CardioDAS Study. Fundam Clin Pharmacol 2007 Dec;21(6):631-641. [CrossRef] [Medline]
    53. Considine J, Brennan D. Effect of an evidence-based paediatric fever education program on emergency nurses' knowledge. Accid Emerg Nurs 2007 Jan;15(1):10-19. [CrossRef] [Medline]
    54. Widyahening IS, Findyartini A, Ranakusuma RW, Dewiasty E, Harimurti K. Evaluation of the role of near-peer teaching in critical appraisal skills learning: a randomized crossover trial. Int J Med Educ 2019 Jan 25;10:9-15. [CrossRef] [Medline]
    55. Ardern CL, Dupont G, Impellizzeri FM, O'Driscoll G, Reurink G, Lewin C, et al. Unravelling confusion in sports medicine and sports science practice: a systematic approach to using the best of research and practice-based evidence to make a quality decision. Br J Sports Med 2019 Jan;53(1):50-56. [CrossRef] [Medline]
    56. Crowell MS, Tragord BS, Taylor AL, Deyle GD. Integration of critically appraised topics into evidence-based physical therapist practice. J Orthop Sports Phys Ther 2012 Oct;42(10):870-879. [CrossRef] [Medline]
    57. Mikalef P, Kourouthanassis PE, Pateli AG. Online information search behaviour of physicians. Health Info Libr J 2017 Mar;34(1):58-73. [CrossRef] [Medline]
    58. Batt-Rawden S, Flickinger T, Weiner J, Cheston C, Chisolm M. The role of social media in clinical excellence. Clin Teach 2014 Jun 10;11(4):264-269. [CrossRef]
    59. de Leo G, LeRouge C, Ceriani C, Niederman F. Websites most frequently used by physician for gathering medical information. AMIA Annu Symp Proc 2006:902 [FREE Full text] [Medline]
    60. Shariff SZ, Bejaimal SA, Sontrop JM, Iansavichus AV, Weir MA, Haynes RB, et al. Searching for medical information online: a survey of Canadian nephrologists. JN 2011 Feb 23;24(6):723-732. [CrossRef]
    61. Hughes B, Joshi I, Lemonde H, Wareham J. Junior physician's use of web 2.0 for information seeking and medical education: a qualitative study. Int J Med Inform 2009 Oct;78(10):645-655. [CrossRef] [Medline]
    62. Lee K, Hoti K, Hughes JD, Emmerton L. Dr google is here to stay but health care professionals are still valued: an analysis of health care consumers' internet navigation support preferences. J Med Internet Res 2017 Jun 14;19(6):e210. [CrossRef] [Medline]
    63. Top 500. Alexa Company. 2019.   URL: https://www.alexa.com [accessed 2019-02-27]
    64. Measuring the 'Filter Bubble': How Google is Influencing What You Click. DuckDuckGo. 2018.   URL: https://spreadprivacy.com/google-filter-bubble-study/ [accessed 2019-12-01]
    65. Search Console Help. Google.   URL: https://support.google.com/webmasters/answer/7358659?hl=en [accessed 2018-12-01]
    66. Kingsley K, Galbraith GM, Herring M, Stowers E, Stewart T, Kingsley KV. Why not just Google it? An assessment of information literacy skills in a biomedical science curriculum. BMC Med Educ 2011 Apr 25;11:17 [FREE Full text] [CrossRef] [Medline]
    67. Pusz MD, Brietzke SE. How good is Google? The quality of otolaryngology information on the internet. Otolaryngol Head Neck Surg 2012 Sep;147(3):462-465. [CrossRef] [Medline]

    ‎
    CONSORT-EHEALTH: Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth
    CSV: comma-separated values
    GEE: generalized estimating equation
    MCQ: multiple-choice questionnaire
    NPTE: National Physical Therapy Exam
    PCE: Physiotherapy Competency Exam

    Edited by G Eysenbach, R Kukafka; submitted 30.04.20; peer-reviewed by E Prakash, E Burner, T Gladman; comments to author 30.06.20; revised version received 06.07.20; accepted 11.11.20; published 17.12.20

    Copyright

    ©Cailbhe Doherty, Arash Joorabchi, Peter Megyesi, Aileen Flynn, Brian Caulfield. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 17.12.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.