This paper is in the following e-collection/theme issue:

Clinical Information and Decision Making (1407)

Published on in Vol 15, No 11 (2013): November

Smartphones as Multimodal Communication Devices to Facilitate Clinical Knowledge Processes: Randomized Controlled Trial

Smartphones as Multimodal Communication Devices to Facilitate Clinical Knowledge Processes: Randomized Controlled Trial

Smartphones as Multimodal Communication Devices to Facilitate Clinical Knowledge Processes: Randomized Controlled Trial

Authors of this article:

Christoph Pimmer1 ;   Magdalena Mateescu2 ;   Carmen Zahn2 ;   Urs Genewein3
Article Authors Cited by (16) Tweetations (16) Metrics

    Original Paper

    1Institute for Information Systems, School of Business, University of Applied Sciences and Arts Northwestern Switzerland FHNW, Basel, Switzerland

    2Institute for Research and Development of Collaborative Processes, School of Applied Psychology (APS), University of Applied Sciences and Arts Northwestern Switzerland, Olten, Switzerland

    3Hightech Research Center, Cranio-Maxillofacial Surgery, University Hospital Basel, Basel, Switzerland

    Corresponding Author:

    Christoph Pimmer, Mag

    Institute for Information Systems

    School of Business

    University of Applied Sciences and Arts Northwestern Switzerland FHNW

    Peter Merian-Strasse 86

    Basel, 4002

    Switzerland

    Phone: 41 61 279 18 49

    Fax:41 61 279 17 98

    Email: Christoph.Pimmer@fhnw.ch


    Background: Despite the widespread use and advancements of mobile technology that facilitate rich communication modes, there is little evidence demonstrating the value of smartphones for effective interclinician communication and knowledge processes.

    Objective: The objective of this study was to determine the effects of different synchronous smartphone-based modes of communication, such as (1) speech only, (2) speech and images, and (3) speech, images, and image annotation (guided noticing) on the recall and transfer of visually and verbally represented medical knowledge.

    Methods: The experiment was conducted from November 2011 to May 2012 at the University Hospital Basel (Switzerland) with 42 medical students in a master’s program. All participants analyzed a standardized case (a patient with a subcapital fracture of the fifth metacarpal bone) based on a radiological image, photographs of the hand, and textual descriptions, and were asked to consult a remote surgical specialist via a smartphone. Participants were randomly assigned to 3 experimental conditions/groups. In group 1, the specialist provided verbal explanations (speech only). In group 2, the specialist provided verbal explanations and displayed the radiological image and the photographs to the participants (speech and images). In group 3, the specialist provided verbal explanations, displayed the radiological image and the photographs, and annotated the radiological image by drawing structures/angle elements (speech, images, and image annotation). To assess knowledge recall, participants were asked to write brief summaries of the case (verbally represented knowledge) after the consultation and to re-analyze the diagnostic images (visually represented knowledge). To assess knowledge transfer, participants analyzed a similar case without specialist support.

    Results: Data analysis by ANOVA found that participants in groups 2 and 3 (images used) evaluated the support provided by the specialist as significantly more positive than group 1, the speech-only group (group 1: mean 4.08, SD 0.90; group 2: mean 4.73, SD 0.59; group 3: mean 4.93, SD 0.25; F2,39=6.76, P=.003; partial η2=0.26, 1–β=.90). However, significant positive effects on the recall and transfer of visually represented medical knowledge were only observed when the smartphone-based communication involved the combination of speech, images, and image annotation (group 3). There were no significant positive effects on the recall and transfer of visually represented knowledge between group 1 (speech only) and group 2 (speech and images). No significant differences were observed between the groups regarding verbally represented medical knowledge.

    Conclusions: The results show (1) the value of annotation functions for digital and mobile technology for interclinician communication and medical informatics, and (2) the use of guided noticing (the integration of speech, images, and image annotation) leads to significantly improved knowledge gains for visually represented knowledge. This is particularly valuable in situations involving complex visual subject matters, typical in clinical practice.

    J Med Internet Res 2013;15(11):e263

    doi:10.2196/jmir.2758

    Keywords

    mobile health; mobile phone; telemedicine; educational technology; learning; problem solving; multimedia; audiovisual aids 


    Interclinician Communication and Mobile Phones

    Interclinician communication is a key component of health care systems. The significance becomes clear in light of its impact on patient care: poor communication between clinicians results in enormous costs and, more importantly, a high number of adverse clinical outcomes and deaths [1-4]. Typical forms of communication between medical professionals are shaped by the particularities of clinical work and can be characterized as instant and synchronous [2,3,5], interdisciplinary and interprofessional (between actors holding different domains and levels of knowledge [6,7]), and mobile (ie, between physically/locally mobile hospital workers [3,8] who increasingly communicate using mobile technologies). In particular, cellphones and smartphones are becoming increasingly popular in clinical settings, with adoption rates of up to 98% [9-12]. According to recent reviews, in light of these characteristics and recent developments, there is little known about how mobile phone-based communication can contribute to effective interclinician communication [13-15]. In many of the existing studies that have explored mobile clinical communication technologies, the methodological design was reported to be of lower quality and based on users’ perceptions [13]. Additionally, only a few randomized controlled experiments were identified [13,16].

    Rich Communication Modes: Speech, Images, and Annotation

    In recent publications, mobile phones have been considered as potentially efficient tools that enable instant location-independent communication [10,14,17-20]. In addition to supporting immediate communication in the form of speech, a number of studies have demonstrated how mobile phones and smartphones allow the ability to capture, exchange, and interpret images [14,21]. Comparisons to standard methods have demonstrated the suitability of mobile phones for use in assessment and diagnosis, such as when using computed tomography (CT), computed tomography angiography (CTA), and noncontrast computed tomography (NCCT) soft tissue or ophthalmic images [22-28]. A number of clinical information systems allow for the annotation of clinical images [29-31]; annotation refers to marking, drawing, highlighting, labeling, or otherwise describing (and enriching) aspects of visual material that should be the focus of attention. The creation of annotations is an essential part of clinical communication environments [32], and is considered a “fundamental task for clinicians, medical educators, or basic scientists” [30].

    Mobile Communication and Knowledge Exchange

    The aim of synchronous interclinician communication involves the building of a shared understanding and a “just-in-time grounding” [2] between clinical actors with varying levels and domains of knowledge and expertise for the well-being of patients. During this process, knowledge is exchanged and new knowledge is created. Less knowledgeable actors learn from the communication and may transfer what they have learned to future patient cases. In other words, interclinician communication represents a valuable opportunity for medical actors to learn from one another for present and for future patient cases, similar to problem-based learning (eg, [33]). Regarding knowledge exchange and learning, some research emphasizes the value of using images in mobile phone-based communication. For example, it has been suggested that mobile image messaging can serve as an instructional tool that allows for instant feedback and further insights for less knowledgeable actors [34]. In a study involving doctor-to-doctor consultations that were based on images taken and sent by means of mobile phones (in the form of Multimedia Messaging Service, MMS), it was concluded that multimedia consultation had a positive effect on patient management and led to an improvement of the service. In addition, 86% of the residents reported “the multimedia information contributed to their ability to independently handle similar cases in [the] future” [35].

    We conclude, despite the widespread use and advancements of mobile technology that enables rich communication modes, that there is a surprising paucity of evidence that demonstrates the actual value of mobile technology for effective interclinician communication, knowledge exchange, and learning.

    Objectives and Hypothesis

    To address this research gap, we delineated 2 sets of hypotheses based on cognitive and sociocognitive science approaches. The first set of hypotheses relates to theories of dual coding and multimedia learning [36-39] and the second set relates to the notion of guided noticing in communication [40,41].

    Concerning cognitive science, it has been shown that images support human information processing and understanding. This is not only because images convey different information than words (concrete vs abstract), but because text and images are processed in 2 different channels or modes [36-39]. The dual channel processing assumption asserts that humans have separate channels for processing words and images and that they use them both to construct coherent mental knowledge representations [36,38,39]. Empirical evidence has demonstrated picture superiority effects (ie, that people can recall information better from both words and images than from words alone) [37,42-44]. Moreover, a vast body of research on learning with multimedia [37,43] suggests that when words (written words or speech) and images are presented together and in a well-designed and integrated manner (eg, redundant and spatially contingent), they support the construction of coherent mental representations and, thus, learning (eg, learning about the functioning of physical systems) [45]. This idea is especially true for less knowledgeable learners (eg, [46]) and in situations in which test items were presented in image-based form [47]. However, research on dual coding and multimedia learning was primarily conducted in experimental studies by using simple content. There is little evidence that demonstrates learning effects in applied and more complex situations, such as that of interclinician communication and knowledge exchange. According to multimedia theory, we formulated our first assumption for the experimental design of mobile phone-based medical communication.

    Mobile phone-based medical communication, in which speech and images are combined to enhance the understanding of complex subject matter, leads to a better understanding by a less knowledgeable actor when compared to the speech-only condition. Therefore, knowledge recall and transfer of verbally and visually represented knowledge should increase when compared to the speech-only condition. Therefore, in our specific case of mobile communication and less knowledgeable actors, we hypothesize that mobile communication involving integrated speech and images leads to better recall and better transfer of visually represented and verbally represented medical knowledge than the speech-only condition.

    For our second set of hypotheses, we derived our assumptions from the concept of guided noticing. Guided noticing describes how specific digital technology affordances can be used in distributed collaborative settings to direct joint attention to notice specific elements in visual media (eg, videos, images). Guided noticing can involve highlighting (eg, annotation), naming, and commenting; thus, it allows the assignment of typological representations (culturally meaningful categories) to topological representations (eg, images). This is especially important when experts interpret visualizations in detail and explain their interpretations to others. Here, guided noticing can be a way to establish common ground [48] in communication between professionals (eg, in educational settings), for example, by using annotations. The creation of annotations is considered as a grounding or communication act by users (rather than designers) who intend to close gaps in common ground required to complete a task [32]. Guided noticing can also serve to develop domain expertise and a professional vision [49] (eg, the training of diagnostic skills with x-rays [41]). Professional vision was empirically investigated and characterized by Goodwin [49] as “...socially organized ways of seeing and understanding events that are answerable to the distinctive interests of a particular social group.” Case studies of professional practices (eg, developing coding schemes or highlighting) with domain experts (eg, archeologists in field research, lawyers in courtroom) were conducted analyzing these practices in great detail. Yet, empirical research on guided noticing [41] is still sparse. In our own previous research on history learning with advanced video tools, we identified episodes of guided noticing and analyzed them in relation to the successful acquisition of visual knowledge [50]. Our present study extends the existing qualitative research by adding original quantitative data on the effects of highlighting (by annotation) for guided noticing in the specific area of medical knowledge.

    Accordingly, we formulated the assumption that communication that is based on guided noticing, in which speech (typological representations) is linked with images (topological representations) in the form of annotation, leads to better knowledge recall and transfer by the less knowledgeable actor compared with speech-only communication or integrated speech and images without annotation. For our specific research of mobile phone-based communication in the context of less knowledgeable actors, we propose a second set of hypotheses that mobile communication that is based on guided noticing leads to better recall and better transfer of visually represented and verbally represented medical knowledge than speech-only communication or that of integrated speech and images without annotation.


    Experimental Design

    To test our hypotheses, an intervention study was designed. The experiment was conducted from November 2011 to May 2012 at the University Hospital Basel (Switzerland) with 42 medical students who were obtaining their master’s degrees. All medical students with a master’s curriculum were eligible to participate.

    Before starting the experiment, informed written consent was obtained from the participants. In the first step, an experimenter asked the participants to assume the role of an emergency assistant. They briefly analyzed a patient case about a subcapital fracture of the fifth metacarpal bone [51,52] to initiate a consultation with a hand specialist. The patient case included a short text with the initial information about anamnesis and status, as well as a radiological image and photographs showing the limited functional capabilities (impaired fist closure and extension deficit). We chose this case because it is a frequently encountered, yet complex, case for novices. The associated clinical reasoning and treatment involves a great deal of medical and clinical knowledge, including the consideration of radiological, functional, and sociodemographic indicators.

    In the second step, the participants were put in communication with a hand surgeon by means of an iPhone 4. To initiate the phone consultation, they were required to briefly characterize the patient case. The specialist provided pertinent standardized advice according to the 3 following experimental conditions/groups: group 1 received verbal explanations (speech only) from the specialist, group 2 received verbal explanations and the radiological image and photographs from the specialist, and group 3 received verbal explanations, the radiological image and the photographs, and annotations (by drawing structures/angle elements) on the radiological image from the specialist (speech, images, and image annotation as guided noticing). Although visual information was varied as described in groups 2 and 3, the verbal information (speech) remained constant in all 3 groups. The case information was prepared by a hand surgeon (UG) and, thereafter, the specialist’s role was assumed by MM.

    The technical setting was based on 2 widely available tools: Skype for the communication and Google Drive for the sharing and annotation of the images. Google Drive allows real-time collaboration, inter alia, in the form of sharing and annotating images on desktop and mobile technologies. To avoid variances in the annotations throughout the experiments, the images were preannotated and then displayed in a predefined sequence by the specialist as integral part of the communication setting.

    After the conversation with the expert, the participants were asked to complete 2 tasks. Task 1 (knowledge recall) required the participants to write a brief description of the case in which they summarized and justified the relevant points of the conversation (including the diagnosis) in a given time period of 90 seconds (ie, recall of verbally represented knowledge). Second, they were required to draw relevant angles/structures in the radiological image and to estimate the angle size (ie, recall of visually represented knowledge). Task 2 was to measure knowledge transfer. The participants were then given a second case that they had to solve in a similar manner. They were asked to analyze the case information and to develop and justify a diagnosis without specialist support. They wrote a short description of the case including the diagnosis (to determine the transfer of verbally represented knowledge) and drew the relevant angles/structures, and estimated the angle size (to determine the transfer of visually represented knowledge).

    In the last step, the participants completed an additional questionnaire about their previous knowledge and experience with fractures. In addition, they answered questions related to the task and experimental conditions. These questions were used as control variables to ensure that there were no differences between the 3 groups.

    Outcome Measures

    After the intervention, the participants were requested to estimate the (self-perceived) usefulness of the support provided by the specialist. We also tested the actual recall and transfer of visually and verbally represented medical knowledge. Sample solutions were elaborated by the research team, including medical (hand surgery), and education and psychology experts. For categories and descriptions, see Table 1. The participants’ answers were jointly rated by 2 raters. A negotiated coding approach was applied for the coding procedure because the raters had limited expertise in medical and health sciences and, although they were experienced coders, they were not familiar with applying medical coding schemes. Negotiated approaches have been proposed previously (eg, for transcript analyses when familiarity with a new coding scheme is low) [53]. Negotiated coding was enhanced by resolving disagreements upon discussion with a hand surgeon.

    To test the hypotheses with a 2-sided significance level of 5% and a power of 80%, a sample size of 45 participants was calculated, a priori, using G*Power-software [54]. The participants were randomly assigned to 1 of 3 groups using a 3-arm parallel design. A computer-generated list of random numbers was used to distribute the participants according to the principles of simple randomization with a 1:1 ratio.

    Statistical Methods

    The data were analyzed using a 3×2 repeated measures ANOVA with a between-subjects factor (group 1, group 2, and group 3) and a within-subjects factor (task 1 and task 2). The effect size (partial η2) and the observed power (1–β) for variables that addressed the recall and transfer of verbally and visually represented knowledge were reported. A 1-way ANOVA was calculated for the support offered by the specialist variable. Differences between conditions were assessed using a post hoc comparison. Statistical significance was determined by values P<.05 and all analyses were conducted using SPSS version 19 (IBM Corp, Armonk, NY, USA).

    Ethical Considerations

    Ethical approval was sought from the regional ethical review board. Because no patients were involved and the participants were considered health care professionals, the board ruled the experiment exempt from further ethical approval and trial registration requirements. Nevertheless, we consulted an expert outside the research team, a professor of ethics at a Swiss university who was part of a separate Swiss ethical board for ethical advice. The confidentiality of the participants was ensured and written informed consent was obtained from every participant before the experiment.

    Table 1. Measurements of verbally and visually represented knowledge.
    CategoryDescriptionExamples
    Recall and transfer of visually represented knowledge

    		Correctness/completeness of drawn angle elementsPositioning of the following elements: (1) point of intersection, (2) basis line, (3) fracture line, and (4) position of angleSee Figure 1

    		Correctness of drawn angle sizeMeasurement of the angle size drawn by the participanteg, 75 degrees

    		Correctness of estimated angle sizeAngle size estimated by the participanteg, 75 degrees
    Recall and transfer of verbally represented knowledge

    		Correctness/completeness of functional/radiological characteristicsMeasurement of written information, including identification of fracture, type of fracture and position, type of functional limitations (extension deficit and rotation error), and degree of functional restrictions and angulation of the fractureFracture of metacarpal V, extension deficit of 10 degrees, and rotation error of 10 degrees, representing 2 functional deficits

    		Correctness/completeness of individual sociodemographic patient characteristicsMeasurement of written information including age, profession, and dominant handYoung, employed patient, dominant hand

    		Correctness/completeness of overall written answersSummarizes both measurements
    ‎
    Figure 1. X-ray with fracture as used in the study.
    View this figure

    Participant Flow

    In all, 48 participants agreed to participate in the experiment. Pretests were conducted with 3 participants to validate the technical feasibility and time limits. The remaining 45 persons were randomized and assigned to the 3 groups. For a flow diagram of participants, see Figure 2.

    ‎
    Figure 2. Participant flow diagram.
    View this figure

    Recruitment

    Eligible participants were invited by means of email and direct invitation by the medical faculty of the University of Basel in November 2011, using a flyer with basic information. The participants did not receive monetary compensation, but a draw for an iPad was used as an incentive.

    Baseline Demographic Characteristics and Prior Knowledge for Each Group

    All 42 participants (23 male, 19 female) were between ages 19 and 38 years (mean 24.57, SD 2.95) and were medical students obtaining their master’s degrees (see Table 2). To determine the bias caused by potentially different levels of prior knowledge in the groups, we asked the participants about their interest in surgical topics (F2,39=0.10, P=.91), their knowledge on the evaluation of fractures (F2,39=2.17, P=.13), prior participation in the treatment of fractures of metacarpal bones (F2,39=0.96, P=.39), and prior experience with diagnosis and treatment of fractures of metacarpal bones (F2,39=0.58, P=.57). No significant differences between the groups were found (for average scores and standard deviation see Table 2).

    Outcomes

    Support Offered by the Specialist

    ANOVA revealed significant differences between group 1 and groups 2 and 3. No significant differences between groups 2 and 3 were observed; a test of between-subjects contrasts yielded the following: F2,39=6.76, P=.003; partial η2=0.26 and 1–β=.90. These results suggest that the students in groups 2 and 3 placed significantly more value on the support offered by the specialist. The students’ evaluations of the support offered by the specialist are displayed in Table 3.

    Table 2. Demographic characteristics and prior knowledge of participants (N=42).
    Demographic characteristicsGroup 1 (n=12)Group 2 (n=15)Group 3 (n=15)
    Age, mean (SD)24.17 (2.12)25.33 (4.39)24.13 (1.30)
    Sex, n (%)



    		Male7 (17)8 (19)8 (19)

    		Female5 (11)7 (17)7 (17)
    Years of study, mean (SD)5 (1)5 (1)4 (1)
    Prior knowledge, mean (SD)



    		Experience with touchscreena4.42 (1.16)3.53 (1.77)4.13 (1.50)

    		Interest in surgical topicsb3.17 (1.19)3.07 (1.39)3.27 (1.16)

    		Estimated knowledge on the evaluation of fractures (compared to peer students)b2.58 (0.67)2.20 (1.08)2.93 (1.03)

    		Prior participation in the treatment of fractures of metacarpal bonesb1.25 (0.62)1.13 (0.52)1.53 (1.13)

    		Prior experience in diagnosis and treatment of fractures of metacarpal bonesb1.25 (0.87)1.20 (0.77)1 (0.00)

    aScale: 1=no use; 5=daily use.

    bLikert scale from 1-5.

    Table 3. Measurement of specialist support.
    MeasureGroup, mean (SD)Mean difference (95% CI)

    		G1 (n=12)G2 (n=15)G3 (n=15)G1–G2G1–G3G2–G3
    Support offered by the specialist4.08 (0.90)4.73 (0.59)4.93 (0.26)–0.65a (–1.25, –0.05)–0.85a (–1.45, –0.25)–0.20 (–0.76, 0.36)

    aMean difference was determined as significant at the P<.05 level.

    Visually Represented Knowledge

    We used the following 3 measures to assess the recall and transfer of visually represented medical knowledge: correctness/completeness of drawn angle elements, correctness of drawn angle size, and correctness of estimated angle size (see Table 4). For all these measures, we performed a 2×3 ANOVA with a within-subjects factor (task 1 vs task 2) and a between-subjects factor (group 1 vs group 2 vs group 3). As predicted, for the correctness/completeness of drawn angle elements in both recall and transfer tasks, group 3 (supported with guided noticing) scored significantly higher than the other 2 groups as revealed by the test of between-subjects contrasts (F2,37=11.32, P<.001; partial η2=0.38, 1–β=.99). The means, standard deviations, and post hoc contrasts are shown in Table 4.

    To test the extent to which the experimental conditions affected the correctness of drawn angle size and the correctness of estimated angle size, we calculated z scores and then performed an ANOVA on the standardized data. As hypothesized, group 3 displayed better performance on both measurements: test of between-subjects contrasts relating to the correctness of drawn angle size (F2,37=8.81, P<.001; partial η2=0.32, 1–β=.96) and the test of between-subjects contrasts relating to the correctness of estimated angle size (F2,37=7.67, P<.001; partial η2=0.30, 1–β=.93). More precisely, for knowledge recall, post hoc tests showed that the ability to draw correct angle sizes and the ability to estimate the size of the drawn angle was significantly higher for group 3 when compared with the other 2 groups. However, a similar result was not supported in the transfer task, in which significant differences were observed only between groups 1 and 3.

    Table 4. Recall and transfer of visually represented medical knowledge.
    Submeasures of visually represented medical knowledgeGroup, mean (SD)Adjusted difference, mean (95% CI)

    		G1 (n=11)G2 (n=15)G3 (n=14)G1–G2G1–G3G2–G3
    Correctness/completeness of drawn angle elements

    		Task 1: Recall1.23 (1.54)1.83 (1.21)3.29 (1.14)–0.61 (–1.88, 0.67)–2.06a (–3.35, –0.76)–1.45a (–2.65, –0.26)

    		Task 2: Transfer1.82 (1.52)2.03 (1.32)3.57 (0.47)–0.22 (–1.38, 0.95)–1.75a (–2.93, –0.58)–1.53a(–2.62, –0.45)
    Correctness of drawn angle sizeb

    		Task 1: Recall0.54 (.91)0.36 (.95)–0.78 (0.63)0.18 (–0.66, 1.01)1.32a (0.47, 2.17)1.14a (0.36, 1.92)

    		Task 2: Transfer0.48 (0.98)0.15 (1.01)–0.56 (0.76)0.33 (–0.59, 1.25)1.05a (0.11, 1.98)0.71 (–0.15, 1.57)
    Correctness of estimated angle sizeb

    		Task 1: Recall0.51 (0.92)0.37 (0.96)–0.74 (0.69)0.14 (–0.73, 1.02)1.25a (0.38, 2.12)1.11a (0.29, 1.93)

    		Task 2: Transfer0.45 (0.98)0.15 (1.05)–0.55 (0.75)0.30 (–0.65, 1.25)1.00a (0.06, 1.95)0.70 (–0.18, 1.59)

    aThe mean difference was determined to be significant (P<.05).

    bSmall numbers indicate better performance.

    Verbally Represented Knowledge

    We used the following 2 submeasures to assess the recall and transfer of verbally represented medical knowledge: the correctness/completeness of functional/radiological characteristics and the correctness/completeness of individual sociodemographic patient characteristics (see Table 2). Statistical analysis was performed on the general measure of correctness and completeness of overall written answers (Table 5). Scores that assessed the recall and transfer of verbally represented knowledge were analyzed by means of a 2×3 repeated ANOVA with a within-subjects factor (task 1 vs task 2) and a between-subjects factor (group 1 vs group 2 vs group 3). The results showed no significant differences between the 3 groups, with respect to the correctness and completeness of verbally represented knowledge for both the recall and transfer tasks. The calculated tests of between-subjects effects (factor 1) did not reach statistical significance with a level of P<.05 (F2,39=0.58, P=.58; partial η2=0.03, 1–β=.13).

    Table 5. Correctness/completeness of overall written answers for recall and transfer.
    MeasureGroup, mean (SD)Mean difference (95% CI)

    		G1 (n=12)G2 (n=15)G3 (n=15)G1–G2G1–G3G2–G3
    Task 1: Recall5.92 (2.07)6.33 (2.02)6.67 (2.58)–0.42 (–2.60, 1.76)–0.75 (–2.93, 1.43)–0.33 (–2.39, 1.72)
    Task 2: Transfer3.00 (2.34)3.00 (2.54)3.67 (2.55)0.00 (–2.41, 2.41)–0.67(–3.08, 1.74)–0.67 (–2.94, 1.61)
    Performance Between Task 1 and 2

    With respect to verbally represented knowledge (correctness/completeness of overall written answers), we observed significant differences in participants’ performance between task 1 and 2 using a test of within-subjects effects (F1,39=43.72, P<.001; partial η2= 0.53, 1–β=1), whereby the performance decreased in task 2, independent of the experimental condition (task 1: mean 6.30, SD 0.39; task 2: mean 3.22, SD 0.38; adjusted difference: mean 3.08, 95% CI 2.14-4.03). We observed no significant differences with respect to visually represented knowledge relating to the correctness/completeness of drawn angle elements (F1,37=3.37, P=.06); the correctness of drawn angle (F2,37= 0.02, P=.89); and the correctness of estimated angle (F1,36=0.04, P=.84). We observed a tendency toward improved performance when assessing the variable correctness/completeness of drawn angle elements in task 2 (see Table 6).

    Table 6. Visually represented knowledge: performance between tasks 1 (knowledge recall) and 2 (knowledge transfer).
    MeasureTask, mean (SD)Mean difference (G1–G2) (95% CI)

    		Task 1: RecallTask 2: Transfer
    Correctness/completeness of drawn angle elements2.17 (1.51)2.51 (1.38)–0.36 (–0.73, 0.07)
    Correctness of drawn angle sizea0.01 (1.01)–0.01 (1.00)0.20 (–0.27, 0.31)
    Correctness of estimated angle sizea0.01 (1.01)–0.01 (1.00)0.30 (–0.26, 0.32)

    aSmall numbers indicate better performance.


    Principal Results

    In light of the rapid adoption of mobile technology, such as smartphones, for clinical communication, this study systematically compares different synchronous mobile phone-based communication modes, including (1) speech only, (2) speech and images, and (3) speech, images, and image annotation (guided noticing). Using an experimental approach, we investigated to what extent these modes affect knowledge processes in clinical communication as indicated by (1) recall and (2) transfer of visually and verbally represented knowledge by a less knowledgeable medical actor. Our variations and hypotheses were informed by psychological theories from related research in the cognitive and sociocognitive sciences.

    In the conditions where speech was integrated with images (groups 2 and 3), participants evaluated the support provided by the specialist as significantly more positive compared to the speech-only condition (group 1). However, in measuring actual knowledge gains, significant positive effects on the recall and transfer of visually represented medical knowledge were only measured in group 3, which integrated speech, images, and image annotation (guided noticing). With respect to verbally represented medical knowledge, no significant differences were measured. In other words, the presentation of visual information did not contribute to the recall and retention of verbally represented knowledge.

    These findings contribute original data to at least 2 broad discourses: (1) the practice of clinical communication and medical informatics, and (2) psychological, cognitive, and sociocognitive learning and communication theories.

    Implications for Medical/Clinical Communication and Medical Informatics

    Unlike most of the other studies that have investigated the contributions of mobile technologies to clinical communication and knowledge exchange, we used a randomized controlled experimental design. The findings suggest that, despite increased self-perceived effectiveness of the use of visual communication modes (compared to a speech-only mode), the understanding and knowledge gains of less knowledgeable medical actors are only enhanced if annotations are used to integrate speech with images. The finding that the actual effectiveness of mobile multimodal communication is different than the self-perceived effectiveness is critical and may negatively affect patient treatment, such as during a specialist consultation; if a requesting doctor were to deem phone-based advice from the specialist (without annotation) as sufficient, this could result in poor decision making with regard to the requesting doctor’s further treatment of the patient. In addition, the present results challenge studies in which positive outcomes of knowledge gains of rich smartphone-based consultations were determined using only self-perceived evaluation (eg, [35]). The present study also highlights the value of annotation for clinical communication and medical informatics. Hospital management may consider this mode of communication when developing or buying new communication solutions, including mobile solutions. Senior clinicians may learn from our findings that less knowledgeable actors, students, and younger colleagues are much more likely to benefit from explanations that integrate speech and visual information by means of labeling and annotating, which is a practice that should not be limited to mobile communication. Recently, it was shown how doctors who create rich multimodal representations by using gestures to connect speech with different computational and bodily representations can improve the understanding of less knowledgeable colleagues [55]. These aspects of communicating, which are highly relevant for clinical communication and learning, are widely neglected in the current clinical practice and should be more thoroughly considered in the future.

    Theoretical Implications and Discussion

    To our knowledge, our experimental study is the first to examine the dual channel and multimedia theories in applied, complex, and authentic situations, such as clinical communication. Our first set of hypotheses (that mobile communication involving integrated speech and images leads to better recall and better transfer of visually represented and verbally represented medical knowledge than the speech-only condition) was based on the dual channel and multimedia theories: we assumed that synchronous mobile communication which integrated images and speech (group 2; ie, contingent multimedia information) would enhance a less knowledgeable actor’s understanding of the subject matter compared with the speech-only condition (group 1). However, we found that using images in group 2 did not lead to significantly improved learning compared to the speech-only condition (group 1). This result challenges other studies from the related field of cognitive research (eg, [56]). A possible explanation lies in our specific experimental clinical setting. During the preparation phase, the images were shown to all participants (including those in the first condition) to initiate the specialist consultation. It is possible that the participants from the speech-only group could later recall these images during the communication phase, thus compensating to some extent for not having the real images available. In other words, the participants in the speech-only condition might have invested more mental effort by imagining the images.

    Our second set of hypotheses (that mobile communication based on guided noticing leads to better recall and better transfer of visually represented and verbally represented medical knowledge than speech-only communication or that of integrated speech and images without annotation) was derived from the concept of guided noticing [40]. We assumed that speech, images, and annotations (guided noticing) in group 3 would lead to knowledge gains with respect to the recall and transfer of both verbally and visually represented medical knowledge (compared with the other conditions). Our results confirm an increased ability to recall visually represented medical knowledge in the guided noticing condition for all 3 submeasures when compared with the other 2 conditions. With regard to the knowledge transfer, the correctness/completeness submeasure of drawn angle elements was significantly affected compared with the other conditions. However, the correctness of the drawn and estimated angle was significantly superior in group 3 (guided noticing) only compared to group 1 (the speech-only condition) and not when compared to group 2 (speech and images).

    Taken together, these results might be readily explained by the fact that the correctness/completeness of drawn angle elements submeasure (see Figure 1) requires a complex and comprehensive understanding of the image and knowledge that cannot be compensated for over time or by increased mental effort with imagination. With this difference in mind, the results demonstrate the importance of guided noticing as a concept and we recommend this mode of communication for the clinical practice.

    In summary, the simple integration of images and speech in mobile phone-based communication did not lead to significantly improved knowledge gains (compared with speech-only communication). Our results confirmed the value of guided noticing in communication. Moreover, they support the view that guided noticing is particularly important in situations involving complex visual subject matter, typical in clinical communication.

    Strengths, Limitations, Generalizability, and Future Research

    In addition to the strengths indicated (ie, using an experimental randomized design in authentic and complex settings), our study is based on firm theoretical underpinnings. Our design draws on and is discussed in terms of a broad theoretical spectrum, combining both cognitive (dual channel/multimedia theory) and sociocognitive (guided noticing) perspectives. This is an approach we deem suitable to address the complexities of clinical communication. However, the findings need to be interpreted and generalized with respect to several limitations. First, we recruited medical students as participants. This is a limitation because students represent a specific group of clinical actors. Although they have accumulated some clinical experiences in the course of their education, they represent a group with limited medical expertise and experience. Second, we tested the hypotheses exclusively against a specific clinical case, a specialist consultation. From a practical viewpoint, our study was conducted in a laboratory setting in a hospital (ie, in an environment that was relatively stable compared to the hectic, noisy, interrupted, and chaotic contexts of real clinical communication), where learning and teaching additionally depend on the nonlinear interplay of a variety of different factors [57,58]. In real clinical settings, it remains unclear, for example, to what extent clinicians are willing to use annotations and extended phone-based explanations. Similarly, to what extent an additional device (eg, a mobile phone) may contribute to cognitive overload in an environment where multiple sources already claim the full attention of a clinician needs to be explored [59]. A further limitation of our experimental design is that prior knowledge was not evaluated in the form of a test, but only as a self-reported measurement. It also needs to be acknowledged that the negotiated coding may represent a limitation because the widely recognized Guidelines for Reporting Reliability and Agreement Studies (GRRAS) recommend using the mean of 2 or more raters to increase reliability [60]. Although we fully agree with GRRAS, in our case we deem the negotiated coding approach a legitimate technique, as recently proposed in the research literature [53]. Moreover, negotiated coding was strengthened by resolving disagreement and uncertainty with a hand surgeon.

    In conclusion, the present study yields interesting insights into the knowledge-based effects of multimodal, phone-based communication, but it cannot provide definite accounts of all the observed phenomena. Future research addressing these questions may (1) use different user groups (eg, residents), (2) use different case representations (eg, MRI), (3) test the system not only in authentic but in real clinical settings (eg, by evaluating prior and post hoc knowledge), and (4) capture knowledge-based effects over a longer period of time.

    Acknowledgments

    The authors thank all research participants, the departments involved, and Andreas Brenner for their support of this study. The authors also thank the project sponsors and partners, the CTI - the Swiss Confederation’s Innovation Promotion Agency, AMTS, Agfa Healthcare, University Hospital Basel, Hightech Research Center of Cranio-Maxillofacial Surgery University of Basel, and the University of Applied Sciences Northwestern Switzerland for their support of this work.

    Conflicts of Interest

    None declared.

    ‎

    Multimedia Appendix 1

    CONSORT-EHEALTH checklist V1.6.2 [61].

    PDF File (Adobe PDF File), 1MB
    1. Parker J, Coiera E. Improving clinical communication: A view from psychology. J Am Med Inform Assoc 2000;7(5):453-461 [FREE Full text] [Medline]
    2. Coiera E. When conversation is better than computation. J Am Med Inform Assoc 2000;7(3):277-286 [FREE Full text] [Medline]
    3. Coiera E, Tombs V. Communication behaviours in a hospital setting: an observational study. BMJ 1998 Feb 28;316(7132):673-676 [FREE Full text] [Medline]
    4. Sutcliffe KM, Lewton E, Rosenthal MM. Communication failures: an insidious contributor to medical mishaps. Acad Med 2004 Feb;79(2):186-194. [Medline]
    5. Fitzpatrick LA, Melnikas AJ, Weathers M, Kachnowski SW. Understanding communication capacity. Communication patterns and ICT usage in clinical settings. J Healthc Inf Manag 2008;22(3):34-41. [Medline]
    6. Varpio L, Hall P, Lingard L, Schryer CF. Interprofessional communication and medical error: a reframing of research questions and approaches. Acad Med 2008 Oct;83(10 Suppl):S76-S81. [CrossRef] [Medline]
    7. Bezemer J, Cope A, Kress G, Kneebone R. “Do you have another Johan?” Negotiating meaning in the operating theatre. Applied Linguistics Review 2011;2:313-334. [CrossRef]
    8. Bardram JE, Bossen C. Mobility work: The spatial dimension of collaboration at a hospital. Comput Supported Coop Work 2005 Apr;14(2):131-160. [CrossRef]
    9. Norman N. Mobile Learning for the NHS: Research Report. Brighton, UK: Epic; 2011.   URL: http:/​/www.​workforce.southcentral.nhs.uk/​libraries__elearning/​elearning/​mlearning_research_report_2012.​aspx [accessed 2013-06-10] [WebCite Cache]
    10. Ramesh J, Carter AO, Campbell MH, Gibbons N, Powlett C, Moseley H, et al. Use of mobile phones by medical staff at Queen Elizabeth Hospital, Barbados: evidence for both benefit and harm. J Hosp Infect 2008 Oct;70(2):160-165. [CrossRef] [Medline]
    11. Baumgart DC. Smartphones in clinical practice, medical education, and research. Arch Intern Med 2011 Jul 25;171(14):1294-1296. [CrossRef] [Medline]
    12. Franko OI, Tirrell TF. Smartphone app use among medical providers in ACGME training programs. J Med Syst 2012 Oct;36(5):3135-3139. [CrossRef] [Medline]
    13. Wu RC, Tran K, Lo V, O'Leary KJ, Morra D, Quan SD, et al. Effects of clinical communication interventions in hospitals: a systematic review of information and communication technology adoptions for improved communication between clinicians. Int J Med Inform 2012 Nov;81(11):723-732. [CrossRef] [Medline]
    14. Prgomet M, Georgiou A, Westbrook JI. The impact of mobile handheld technology on hospital physicians' work practices and patient care: a systematic review. J Am Med Inform Assoc 2009;16(6):792-801 [FREE Full text] [CrossRef] [Medline]
    15. Ozdalga E, Ozdalga A, Ahuja N. The smartphone in medicine: a review of current and potential use among physicians and students. J Med Internet Res 2012;14(5):e128 [FREE Full text] [CrossRef] [Medline]
    16. Ortega GR, Taksali S, Smart R, Baumgaertner MR. Direct cellular vs. indirect pager communication during orthopaedic surgical procedures: a prospective study. Technol Health Care 2009;17(2):149-157. [CrossRef] [Medline]
    17. Spiegelman J, Detsky AS. Instant mobile communication, efficiency, and quality of life. JAMA 2008 Mar 12;299(10):1179-1181. [CrossRef] [Medline]
    18. Ammenwerth E, Buchauer A, Bludau B, Haux R. Mobile information and communication tools in the hospital. Int J Med Inform 2000 Jan;57(1):21-40. [Medline]
    19. Horwitz LI, Detsky AS. Physician communication in the 21st century: to talk or to text? JAMA 2011 Mar 16;305(11):1128-1129. [CrossRef] [Medline]
    20. Hanada E, Fujiki T, Nakakuni H, Sullivan CV. The effectiveness of the installation of a mobile voice communication system in a university hospital. J Med Syst 2006 Apr;30(2):101-106. [Medline]
    21. Knobloch K, Gohritz A, Vogt PM. Cell phone-based multimedia messaging service in reconstructive microsurgery: a novel telemedicine application. Plast Reconstr Surg 2009 Jun;123(6):220e-222e. [CrossRef] [Medline]
    22. Ng WH, Wang E, Ng I. Teleradiology multimedia messaging service in the provision of emergency neurosurgical service. In: Kumar S, Krupinski E, editors. Teleradiology. Heidelberg: Springer; 2008:77-87.
    23. Tsai HH, Pong YP, Liang CC, Lin PY, Hsieh CH. Teleconsultation by using the mobile camera phone for remote management of the extremity wound: a pilot study. Ann Plast Surg 2004 Dec;53(6):584-587. [Medline]
    24. Kumar S, Wang EH, Pokabla MJ, Noecker RJ. Teleophthalmology assessment of diabetic retinopathy fundus images: smartphone versus standard office computer workstation. Telemed J E Health 2012 Mar;18(2):158-162. [CrossRef] [Medline]
    25. Boyce Z, Gilmore S, Xu C, Soyer HP. The remote assessment of melanocytic skin lesions: a viable alternative to face-to-face consultation. Dermatology 2011;223(3):244-250. [CrossRef] [Medline]
    26. Filip M, Linzer P, Šámal F, Tesař J, Herzig R, Školoudík D. Medical consultations and the sharing of medical images involving spinal injury over mobile phone networks. Am J Emerg Med 2012 Jul;30(6):961-965. [CrossRef] [Medline]
    27. Mitchell JR, Sharma P, Modi J, Simpson M, Thomas M, Hill MD, et al. A smartphone client-server teleradiology system for primary diagnosis of acute stroke. J Med Internet Res 2011;13(2):e31 [FREE Full text] [CrossRef] [Medline]
    28. Ng WH, Wang E, Ng I. Multimedia Messaging Service teleradiology in the provision of emergency neurosurgery services. Surg Neurol 2007 Apr;67(4):338-341. [CrossRef] [Medline]
    29. Caruso RD, Postel GC, McDonald CS, Aronson BA, Christensen JD. Software-annotated, digitally photographed, and printed MR images: suitability for publication. Acad Radiol 2002 Mar;9(3):346-351. [Medline]
    30. Goede PA, Lauman JR, Cochella C, Katzman GL, Morton DA, Albertine KH. A methodology and implementation for annotating digital images for context-appropriate use in an academic health care environment. J Am Med Inform Assoc 2004;11(1):29-41 [FREE Full text] [CrossRef] [Medline]
    31. Jang SH, Kim WY. Defining a new annotation object for DICOM image: a practical approach. Comput Med Imaging Graph 2004 Oct;28(7):371-375. [CrossRef] [Medline]
    32. Coiera E. Communication spaces. J Am Med Inform Assoc 2013 Sep 4. [CrossRef] [Medline]
    33. Kilroy DA. Problem based learning. Emerg Med J 2004 Jul;21(4):411-413 [FREE Full text] [CrossRef] [Medline]
    34. Lam TK, Preketes A, Gates R. Mobile phone photo messaging assisted communication in the assessment of hand trauma. ANZ J Surg 2004 Jul;74(7):598-602. [CrossRef] [Medline]
    35. Farber N, Haik J, Liran A, Weissman O, Winkler E. Third generation cellular multimedia teleconsultations in plastic surgery. J Telemed Telecare 2011;17(4):199-202. [CrossRef] [Medline]
    36. Clark JM, Paivio A. Dual coding theory and education. Educ Psychol Rev 1991 Sep;3(3):149-210. [CrossRef]
    37. Mayer RE. Applying the science of learning to medical education. Med Educ 2010 Jun;44(6):543-549. [CrossRef] [Medline]
    38. Paivio A. Mental imagery in associative learning and memory. Psychological Review 1969;76(3):241-263. [CrossRef]
    39. Baddeley AD, Hitch GJ. Working memory. In: The Psychology of Learning and Motivation: Advances in Research and Theory. New York: Academic Press; 1974:8-89.
    40. Pea R, Mills M, Rosen J, Dauber K, Effelsberg W, Hoffert E. The diver project: interactive digital video repurposing. IEEE Multimedia 2004 Jan;11(1):54-61. [CrossRef]
    41. Pea RD. Video-as-data and digital video manipulation techniques for transforming learning sciences research, education, and other cultural practices. In: Weiss J, Nolan J, Hunsinger J, Trifonas P, editors. The International Handbook of Virtual Learning Environments. Dordrecht: Springer; 2006:1321-1393.
    42. Fletcher J, Tobias S. The multimedia principle. In: Mayer RE, editor. The Cambridge Handbook of Multimedia Learning. New York: Cambridge University Press; 2005:117-134.
    43. Mayer RE. Cognitive theory of multimedia learning. In: Mayer RE, editor. The Cambridge Handbook of Multimedia Learning. New York: Cambridge University Press; 2005:31-48.
    44. Baddeley A. Working memory: theories, models, and controversies. Annual Review of Psychology 2012;63(1):1-29. [CrossRef]
    45. Hegarty M. Multimedia learning about physical systems. In: Mayer RE, editor. The Cambridge Handbook of Multimedia Learning. New York: Cambridge University Press; 2005:447-465.
    46. Mayer RE, Gallini JK. When is an illustration worth ten thousand words? Journal of Educational Psychology 1990;82(4):715-726. [CrossRef]
    47. Brünken R, Steinbacher S, Schnotz W, Leutner D. Mentale Modelle und Effekte der Präsentations- und Abrufkodalität beim Lernen mit Multimedia. Zeitschrift für Pädagogische Psychologie 2001 Mar 1;15(1):16-27. [CrossRef]
    48. Clark HH. Using Language. Cambridge: Cambridge University Press; 1996.
    49. Goodwin C. Professional vision. American Anthropologist 1994 Sep;96(3):606-633. [CrossRef]
    50. Zahn C, Pea R, Hesse FW, Rosen J. Comparing simple and advanced video tools as supports for complex collaborative design processes. Journal of the Learning Sciences 2010 Jul 19;19(3):403-440. [CrossRef]
    51. Kuokkanen HO, Mulari-Keränen SK, Niskanen RO, Haapala JK, Korkala OL. Treatment of subcapital fractures of the fifth metacarpal bone: a prospective randomised comparison between functional treatment and reposition and splinting. Scand J Plast Reconstr Surg Hand Surg 1999 Sep;33(3):315-317. [Medline]
    52. Abdon P, Mühlow A, Stigsson L, Thorngren KG, Werner C, Westman L. Subcapital fractures of the fifth metacarpal bone. Arch Orthop Trauma Surg 1984;103(4):231-234. [Medline]
    53. Garrison DR, Cleveland-Innes M, Koole M, Kappelman J. Revisiting methodological issues in transcript analysis: Negotiated coding and reliability. The Internet and Higher Education 2006 Jan;9(1):1-8. [CrossRef]
    54. Baguley T. Understanding statistical power in the context of applied research. Appl Ergon 2004 Mar;35(2):73-80. [CrossRef] [Medline]
    55. Pimmer C, Pachler N, Genewein U. Reframing clinical workplace learning using the theory of distributed cognition. Acad Med 2013 Sep;88(9):1239-1245. [CrossRef] [Medline]
    56. Mayer RE, Anderson RB. Animations need narrations: An experimental test of a dual-coding hypothesis. Journal of Educational Psychology 1991;83(4):484-490. [CrossRef]
    57. Pimmer C, Pachler N, Genewein U. Contextual dynamics in clinical workplaces: learning from doctor-doctor consultations. Med Educ 2013 May;47(5):463-475. [CrossRef] [Medline]
    58. Dornan T, Boshuizen H, King N, Scherpbier A. Experience-based learning: a model linking the processes and outcomes of medical students' workplace learning. Med Educ 2007 Jan;41(1):84-91. [CrossRef] [Medline]
    59. Laxmisan A, Hakimzada F, Sayan OR, Green RA, Zhang J, Patel VL. The multitasking clinician: decision-making and cognitive demand during and after team handoffs in emergency care. Int J Med Inform 2007;76(11-12):801-811. [CrossRef] [Medline]
    60. Kottner J, Audige L, Brorson S, Donner A, Gajewski BJ, Hróbjartsson A, et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. Int J Nurs Stud 2011 Jun;48(6):661-671. [CrossRef] [Medline]
    61. Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res 2011;13(4):e126 [FREE Full text] [CrossRef] [Medline]

    ‎
    CT: computed tomography
    CTA: computed tomography angiography
    GRRAS: Guidelines for Reporting Reliability and Agreement Studies
    NCCT: noncontrast computed tomography

    Edited by G Eysenbach; submitted 17.06.13; peer-reviewed by I Mavridis, M Kalz; comments to author 12.07.13; revised version received 17.09.13; accepted 07.10.13; published 27.11.13

    Copyright

    ©Christoph Pimmer, Magdalena Mateescu, Carmen Zahn, Urs Genewein. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 28.11.2013.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.