This paper is in the following e-collection/theme issue:

Chatbots and Conversational Agents (463) Electronic/Mobile Data Capture, Internet-based Survey & Research Methodology (788) Artificial Intelligence (1198) Adoption and Change Management of eHealth Systems (674) E-Health / Health Services Research and New Models of Care (457)

Published on in Vol 24, No 11 (2022): November

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/40681, first published .
Investigating Patients' Continuance Intention Toward Conversational Agents in Outpatient Departments: Cross-sectional Field Survey

Investigating Patients' Continuance Intention Toward Conversational Agents in Outpatient Departments: Cross-sectional Field Survey

Investigating Patients' Continuance Intention Toward Conversational Agents in Outpatient Departments: Cross-sectional Field Survey

Authors of this article:

Xingyi Li1 Author Orcid Image ;   Shirong Xie2 Author Orcid Image ;   Zhengqiang Ye2 Author Orcid Image ;   Shishi Ma3 Author Orcid Image ;   Guangjun Yu4 Author Orcid Image
Article Authors Cited by (4) Tweetations (3) Metrics

    Original Paper

    1School of Public Health, Shanghai Jiao Tong University School of Medicine, Shanghai, China

    2Eye and Ear, Nose, and Throat Hospital, Fudan University, Shanghai, China

    3Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China

    4Shanghai Children’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China

    Corresponding Author:

    Guangjun Yu, PhD

    Shanghai Children’s Hospital

    Shanghai Jiao Tong University School of Medicine

    No 355 Luding Road

    Shanghai, 200062

    China

    Phone: 86 18917762998

    Email: gjyu@shchildren.com.cn


    Background: Conversational agents (CAs) have been developed in outpatient departments to improve physician-patient communication efficiency. As end users, patients’ continuance intention is essential for the sustainable development of CAs.

    Objective: The aim of this study was to facilitate the successful usage of CAs by identifying key factors influencing patients’ continuance intention and proposing corresponding managerial implications.

    Methods: This study proposed an extended expectation-confirmation model and empirically tested the model via a cross-sectional field survey. The questionnaire included demographic characteristics, multiple-item scales, and an optional open-ended question on patients’ specific expectations for CAs. Partial least squares structural equation modeling was applied to assess the model and hypotheses. The qualitative data were analyzed via thematic analysis.

    Results: A total of 172 completed questionaries were received, with a 100% (172/172) response rate. The proposed model explained 75.5% of the variance in continuance intention. Both satisfaction (β=.68; P<.001) and perceived usefulness (β=.221; P=.004) were significant predictors of continuance intention. Patients' extent of confirmation significantly and positively affected both perceived usefulness (β=.817; P<.001) and satisfaction (β=.61; P<.001). Contrary to expectations, perceived ease of use had no significant impact on perceived usefulness (β=.048; P=.37), satisfaction (β=−.004; P=.63), and continuance intention (β=.026; P=.91). The following three themes were extracted from the 74 answers to the open-ended question: personalized interaction, effective utilization, and clear illustrations.

    Conclusions: This study identified key factors influencing patients’ continuance intention toward CAs. Satisfaction and perceived usefulness were significant predictors of continuance intention (P<.001 and P<.004, respectively) and were significantly affected by patients’ extent of confirmation (P<.001 and P<.001, respectively). Developing a better understanding of patients’ continuance intention can help administrators figure out how to facilitate the effective implementation of CAs. Efforts should be made toward improving the aspects that patients reasonably expect CAs to have, which include personalized interactions, effective utilization, and clear illustrations.

    J Med Internet Res 2022;24(11):e40681

    doi:10.2196/40681

    Keywords

    conversational agent; continuance intention; expectation-confirmation model; partial least squares; structural equation modeling; chatbot; virtual assistant; cross-sectional; field study; optimization; outpatient; interview; qualitative; questionnaire; satisfaction; perceived usefulness; intention; adoption; attitude; perception 


    Background

    Tertiary hospitals in China are occupied with many outpatients every day, which results in long waiting times and limited physician-patient communication during consultations. This phenomenon is caused mainly by two aspects. First, the large population base has resulted in a growing demand for medical services. Second, physicians have to finish both consulting patients and filling out medical histories during a limited amount of time. A study found that in the consultation room, 66.5% of physicians’ time was spent on communication and the examination of patients, and 20.7% of their time was spent on writing medical records [1]. Long waiting times, together with limited consultation times, further result in insufficient physician-patient communication and an incomplete understanding of conditions and diagnoses (ie, knowing all of the facts) [2,3]. Besides, during the ongoing COVID-19 pandemic, long waiting times have also put patients at risk for cross-infection [4].

    Under national policies on the digital transformation of the health care industry, Shanghai, as a leading digital city, developed conversational agents (CAs) in outpatient departments, hoping to alleviate patient overload and improve communication efficiency. CAs are artificial intelligence programs that engage in dialogues with patients on mobile devices [5]. With these contextual question–answering agents, data on patients’ symptoms and medical histories can be captured and delivered to physicians’ workstations in structured forms before a consultation. During face-to-face consultations, physicians can rapidly gain an understanding of patients' general conditions and focus on other responsibilities [6], which has resulted in a man-machine integrated consultation model.

    Prior studies have indicated that CAs can save time by reducing the time required for history taking, improve consultation efficiency, and enhance the completeness and accuracy of medical histories [7-10]. However, their potential has not been fully exploited, as the usage of CAs is often limited; 6 months after tertiary hospitals in Shanghai established CAs, the usage rates fell short of expectations (26% and 20%, respectively, for the second- and fourth-ranked hospitals). As end users, patients’ continuance intention is essential for the sustainable development of CAs [11], yet limited studies are available.

    Based on the abovementioned research background and motivations, this study has 3 aims. First, it attempts to identify key factors influencing users’ continuance usage intention via a theoretical model. Second, it empirically examines the applicability of the model in the context of implementing CAs in outpatient departments. Third, it proposes corresponding managerial implications based on the results.

    Theoretical Background and Model

    The usage of information systems includes the following two stages: preacceptance (acceptance before a system’s initial use) and postacceptance (acceptance after a system’s initial use; ie, continuance).

    Even though initial use is an important first step toward realizing information systems’ success, it is mostly influenced by secondhand information from referent others or popular media rather than users’ actual interactions with the information system. In contrast, continuance after a system’s initial use is more realistic and unbiased, since it is grounded in users' firsthand experiences [12]. Therefore, the long-term viability and eventual success of information systems depend on users’ continued use.

    There has been a considerable body of theory-based research on information system use in recent years. Among the theoretical models, the Technology Acceptance Model (TAM) is commonly applied to understand the initial acceptance of information systems, including intelligent health service systems (eg, registration systems and patient portals) [13-16]. The TAM predicts users' initial use of information systems based on the following two constructs: perceived usefulness and perceived ease of use [17].

    To understand users’ continuance behavior in an information system context, an expectation-confirmation model (ECM) of information system continuance was proposed by Bhattacherjee [11]. This ECM has been empirically tested in a variety of contexts, including health services such as e-appointment systems and teleconsultations. The model predicts users' continuance intention via the following three antecedent constructs: perceived usefulness, confirmation, and satisfaction.

    The ECM only incorporated perceived usefulness from the TAM, as Bhattacherjee [11] considered it to be the more salient and consistent predictor of information system use intention. However, both perceived usefulness and perceived ease of use are the primary motivators of information system acceptance in the TAM [18,19]. The significant impact of perceived ease of use on both perceived usefulness and usage intention has been verified in previous research (eg, studies on electronic health record acceptance by physicians and self-management technology acceptance by patients [20,21]). Given the particularities of patients and medical professionalism, perceived ease of use might also have the potential to influence patients’ continuance intention. Therefore, this study extended existing ECM constructs by integrating perceived ease of use, hoping to provide a better understanding of patients' continuance intention in the context of CAs.

    Based on the abovementioned theoretical reasoning and the results of previous research, we propose the following theoretical model (Figure 1) and hypotheses: (1) perceived usefulness positively affects continuance intention (hypothesis 1), (2) perceived usefulness positively affects satisfaction (hypothesis 2), (3) perceived ease of use positively affects continuance intention (hypothesis 3), (4) perceived ease of use positively affects satisfaction (hypothesis 4), (5) perceived ease of use positively affects perceived usefulness (hypothesis 5), (6) the confirmation of initial expectations positively affects perceived usefulness (hypothesis 6), (7) the confirmation of initial expectations positively affects satisfaction (hypothesis 7), (8) the confirmation of initial expectations positively affects perceived ease of use (hypothesis 8), and (9) satisfaction positively affects continuance intention (hypothesis 9).

    ‎
    Figure 1. Research model. H: hypothesis; +: positive effect.
    View this figure

    Study Design and Setting

    Tertiary hospitals in Shanghai established CAs in early 2021 to improve medical service efficiency and alleviate the overload in outpatient departments. Before face-to-face consultations, patients can provide their symptoms and medical histories on mobile devices via a contextual question–answering agent. Afterward, physicians can rapidly gain an understanding of patients' general conditions with the structured information delivered by CAs.

    Shanghai Eye and ENT (ear, nose, and throat) Hospital has been a pioneer during the CA implementation process. Empirical data for this research were collected via a cross-sectional field survey that was conducted in the outpatient department of Shanghai Eye and ENT Hospital. The duration of this study was 3 months (November 2021 to January 2022). We invited patients and their companions who had used CAs. The survey was conducted near the pharmacy to make sure that patients finished their face-to-face consultations and minimize possible inconveniences.

    Sample Size and Sampling

    The minimum sample size for this research was 124, and according to Marcoulides and Saunders [22], the minimum sample size depends on the maximum number of arrows pointing at a latent variable. Hoyle [23] recommended a sample size of 100 to 200 when performing path modeling. The convenience sampling technique was used, and a total of 172 questionnaires were completed. This sample size met the requirements for obtaining sufficient statistical power.

    Measurement Tools

    The questionnaire included 3 parts—demographic characteristics, multiple-item scales, and the following optional open-ended question: “What are your expectations that CAs failed to meet?” The five constructs in the proposed model were measured by using multiple-item scales that were adapted from Davis [17] and Bhattacherjee [11], and the items were reworded to accommodate the context of CA use. Satisfaction items were scored on 5-point semantic differential scales. The remaining items were scored on 5-point Likert scales that ranged from 1 (strongly disagree) to 5 (strongly agree). Table 1 provides definitions and sources for the five constructs. The scale items were translated from English to Chinese because the survey was conducted in China. To avoid wording-related misapprehension, we used a back-translation process [24]. Multimedia Appendix 1 presents the items for each construct and their sources. A pretest was conducted among 25 patients to ensure the reliability and validity of the questionnaire.

    Table 1. Definitions of constructs.
    ConstructOperational definitionReference
    Continuance intentionPatients’ intention to continue using conversational agentsBhattacherjee [11]
    SatisfactionPatients’ affects (feelings) prior to using conversational agentsBhattacherjee [11]
    Perceived usefulnessPatients’ perceptions of the expected benefits of conversational agentsBhattacherjee [11]
    Perceived ease of useThe degree to which patients believe that using conversational agents would be free from effortDavis [17]
    ConfirmationPatients’ perceptions of the congruence between expectations for conversational agents and their actual performanceBhattacherjee [11]

    Data Collection

    To help patients better understand their choices and remain focused, two postgraduates from Shanghai Jiao Tong University School of Medicine conducted the in-person survey, using paper questionnaires. During this process, the investigators read all of the questions aloud to the patients and filled in the questionnaire with their answers, which saved patients the trouble of reading the items themselves. The questionnaires ended with an optional open-ended question (“What are your expectations that CAs failed to meet?”). The answers were collected through brief interviews and written down in the form of detailed summaries by the investigators. A total of 172 valid questionnaires were collected, with a 100% (172/172) response rate, and 74 participants answered the optional open-ended question.

    Data Analysis

    Descriptive statistics were performed by using SPSS 25.0 (IBM Corporation). A partial least squares structural equation model (PLS-SEM) analysis was performed in SmartPLS 3.3.3 (SmartPLS GmbH) to validate the research model and test the research hypotheses. A PLS-SEM was chosen because it is capable of producing robust results with restricted sample sizes and data lacking normality [25].

    The implementation of this method was performed in 2 steps [26]. The first step consisted of assessing the reliability and validity of the measurement model using the partial least squares algorithm, while the second step focused on assessing the fit of the structural model and the significance of the hypotheses by using bootstrapping (5000 bootstrap samples) [27].

    The qualitative data were analyzed via thematic analysis. Thematic analysis is a method for analyzing qualitative data that entails searching across a data set to identify, analyze, and report repeated patterns [28]. The initial codes were generated by deductively reading the manuscripts. This was done by a single coder, and the codes were reviewed by a second analyst [29]. After they reached consensus on the initial codes, the themes were extracted from and defined on the basis of the codes through group discussions.

    Ethics Approval

    This study was approved by Shanghai Children’s Hospital (approval number: 2022R092-E01). All respondents participated in this study voluntarily and anonymously on the basis of informed consent.


    Demographic Information

    A total of 172 questionnaires were complete and valid, with a 100% (172/172) response rate. The demographic information of CA users is listed in Table 2. Notably, 54.1% of the respondents were in the 25 to 35 years age group, and 63.4% (109/172) were women.

    Table 2. Demographic information.
    Participant characteristicsParticipants (N=172), n (%)
    Gender

    		Men63 (36.6)

    		Women109 (63.4)
    Age (years)

    		<2512 (7)

    		25-3593 (54.1)

    		36-4546 (26.7)

    		>4521 (12.2)
    Relationship with the patient

    		Patients themselves106 (61.6)

    		Patients’ children22 (12.8)

    		Patients’ parents44 (25.6)
    Visit type

    		First visit112 (65.1)

    		Return visit60 (34.9)
    Number of visits over the past half year

    		1101 (58.7)

    		2-347 (27.3)

    		>324 (14)
    Number of times that a participant used a conversational agent

    		1136 (79.1)

    		2-331 (18)

    		>35 (2.9)

    Measurement Model Assessment

    The measurement model was assessed in terms of construct reliability, convergent validity, and discriminative validity by performing a confirmatory composite analysis. The results are displayed in Table 3 and Table 4.

    Reliability can be evaluated with Cronbach α and composite reliability values [30]. Convergent validity can be accessed with factor loading and average variance extracted (AVE) values [27,31]. As shown in Table 2, all of the Cronbach α and composite reliability values were above 0.7, the AVE for each construct was above 0.5, and the factor loadings for each item were above 0.7, indicating good reliability and convergent validity [27].

    Discriminant validity reflects the extent to which constructs are significantly different from each other. To achieve discriminant validity, the square root of the AVE for a given construct must be higher than that construct’s correlation with other constructs, and this must hold true for all constructs [31]. As shown in Table 4, the results indicated that discriminate validity was achieved. Therefore, we concluded that the quality of the measurement model was sufficient for testing the hypotheses in the model.

    Table 3. Construct reliability and convergent validity.
    Constructs and itemsFactor loadingsCronbach αCRaAVEb
    CONFc.9380.9600.890

    		CONF10.947



    		CONF20.940



    		CONF30.943


    CId.9930.9960.993

    		CI10.996



    		CI20.996


    PEOUe.7920.8710.694

    		PEOU10.866



    		PEOU20.763



    		PEOU30.865


    PUf.7930.8790.710

    		PU10.825



    		PU20.761



    		PU30.933


    SATg.9590.9800.960

    		SAT10.980



    		SAT20.981


    aCR: composite reliability.

    bAVE: average variance extracted.

    cCONF: confirmation.

    dCI: continuance intention.

    ePEOU: perceived ease of use.

    fPU: perceived usefulness.

    gSAT: satisfaction.

    Table 4. Discriminant validity.

    		ConfirmationContinuance intentionPerceived ease of usePerceived usefulnessSatisfaction
    Confirmation0.943a—b———
    Continuance intention0.8530.996a———
    Perceived ease of use0.1770.1750.833a——
    Perceived usefulness0.8250.7590.1920.843a—
    Satisfaction0.8410.8570.1570.7830.980a

    aThe square root of the average variance extracted for each construct.

    bNot available.

    Structure Model Assessment

    The inner variance inflation factors were below 5, indicating that we were able to avoid construct collinearity in the model [27]. The research model was assessed by evaluating the path coefficients (β) and the coefficients of determination (R2). The path coefficients and their significance levels, as well as hypothesis outcomes and R2 values, are shown in Figure 2 and Table 5.

    β represents the direct effects of independent variables on dependent variables. The hypotheses that were based on the original ECM (hypotheses 1, 2, 6, 7, and 9) were all supported, while the hypotheses regarding the newly integrated construct—perceived ease of use (hypotheses 3, 4, and 5)—were rejected, except for hypothesis 8. R2 refers to the amount of explained variance for each endogenous latent variable. The entire model explained 75.5% of the variance in continuance intention and 73.2% of the variance in satisfaction, which was considered substantial.

    ‎
    Figure 2. Results of the structure model. *:P<.05; **:P<.01; *** P<.001.
    View this figure
    Table 5. Hypothesis test results.
    HypothesisPathCoefficient (β)P valueOutcome
    Hypothesis 1Perceived usefulness → continuance intention.221.004Supported
    Hypothesis 2Perceived usefulness → satisfaction.280<.001Supported
    Hypothesis 3 Perceived ease of use → continuance intention−.004.63Rejected
    Hypothesis 4Perceived ease of use → satisfaction.026.91Rejected
    Hypothesis 5Perceived ease of use → perceived usefulness.048.37Rejected
    Hypothesis 6Confirmation → perceived usefulness.817 <.001Supported
    Hypothesis 7Confirmation → satisfaction.610<.001Supported
    Hypothesis 8Confirmation → perceived ease of use.177.007Supported
    Hypothesis 9Satisfaction → continuance intention.680<.001Supported

    Qualitative Data on Patients’ Expectations

    A total of 3 themes were extracted from the 74 answers regarding patients’ specific expectations that CAs failed to meet. The first theme was personalized interaction (mentioned 50 times). Instead of the same interaction content and forms of interaction, patients expected to see more personalized conversations that were based on their previous medical histories and visit types. Older participants asked for a voice recognition function and larger font sizes. The second theme was effective utilization (mentioned 37 times). Patients expected CAs to have more useful functions, mainly focusing on self-assessments for referrals and self-management for follow-up treatments. The third theme was clear illustrations on the use and promises of CAs (mentioned 15 times). In some cases, CAs were easily mistaken as replacements for face-to-face consultations.


    Summary of Findings

    This study identified key factors influencing patients' continuance intention toward CAs through an extended ECM. Satisfaction (β=.68; P<.001) and perceived usefulness (β=.221; P=.004) were significant predictors of continuance intention, with satisfaction being the stronger predictor. Patients' extent of confirmation significantly affected both perceived usefulness (β=.817; P<.001) and satisfaction (β=.61; P<.001). These findings are consistent with the original ECM as well as the findings of previous research on information system usage (eg, telemedicine and health data reporting platform usage) among patients [32-35]. The confirmation of patients’ expectations has a positive effect on perceived usefulness and satisfaction, and the improvement of perceived usefulness and satisfaction can further enhance patients’ enthusiasm for continuing to use a system.

    Our qualitative data shed light on patients’ specific expectations that CAs failed to meet, including personalized interactions, effective utilization, and clear illustrations. Our findings can help administrators and researchers better understand low CA usage rates. After using CAs, if these expectations have not been positively confirmed, perceived usefulness and satisfaction among patients will drop accordingly and result in their unwillingness to continue using CAs.

    Although accumulated evidence has shown the significant impact of perceived ease of use on both perceived usefulness and usage intention [15-17], in this study, perceived ease of use turned out to be trivial in the context of CAs. Not coincidentally, some studies on information system usage in hospitals have also shown the insignificant relationship between perceived ease of use and usage intention [36-38]. This result indicates that once CAs prove to be useful and effective, patients will consider it worth their time and effort to learn how to use CAs. However, if CAs are easy to use but cannot collect useful medical histories from patients, patients’ continuance intention will not improve anyway [37].

    Managerial and Public Health Implications

    Our findings have important managerial implications. The proposed model provides a feedback channel that administrators can use to gain insight into patients’ actual experiences and expectations. To maximize patients’ satisfaction and continuance intention, efforts should be made toward improving the aspects that patients reasonably expect CAs to have. Offering personalized interactions based on patients’ histories and adding more functions can increase perceived usefulness among patients, while providing clear illustrations on the use and promises of CAs can result in patients having appropriate expectations, which allow for positive postuse confirmation.

    Contributions of This Study

    This study contributes to the body of knowledge about the determinants of CA continuance usage. Almost half of the existing literature on CA acceptance, adoption, and usage evaluates a specific CA artifact, while only 21% of studies put the user in the center of attention when investigating the determinants of their acceptance and usage of CAs [39]. Most of these user-focused empirical studies did not draw on specific concepts from theory for their evaluations [40-43], which makes the results hard to compare. The contribution of this paper is 2-fold. From a theoretical point of view, we identified key factors influencing users’ continuance usage intention through a theoretical model. The applicability and validity of the model was empirically tested via a cross-sectional field survey. From a practical point of view, corresponding managerial implications based on the results were proposed to facilitate the successful and continuous development of CAs.

    Study Limitations

    This study has several limitations that should be addressed. The digital transformation of CA systems started less than 1 year ago, and the progress of this transformation varies dramatically from hospital to hospital. Therefore, this study was conducted at a hospital with a relatively well-designed system and a larger user base. Further research is needed to confirm our findings in the context of different hospitals and different CAs. Additionally, the sample was not normally distributed in terms of gender and age. However, the data analysis was trustworthy, since a PLS-SEM is capable of producing robust results with restricted sample sizes and data lacking normality. Furthermore, a successful digital transformation in health care is a joint effort, and in terms of CAs, this effort depends not only on patients’ continuance but also on physicians’ utilization and administrators’ management of CAs. A multisource model is required to explore the relationships among the constructs.

    Conclusions

    This study intended to identify key factors influencing patients’ continuance intention toward CAs. Satisfaction and perceived usefulness were significant predictors of continuance intention (P<.001 and P<.004, respectively) and were significantly affected by patients’ extent of confirmation (P<.001 and P<.001, respectively). Developing a better understanding of patients’ continuance intention can help administrators figure out how to facilitate the effective implementation of CAs. Efforts should be made toward improving the aspects that patients reasonably expect CAs to have, which include personalized interactions, effective utilization, and clear illustrations.

    Acknowledgments

    This study was supported by the General Program of the National Natural Science Foundation (grants 71874110, 72074146) and the Major Research Plan Project of the National Natural Science Foundation (grant 91846302).

    Authors' Contributions

    We confirm that this paper has been read and approved by all named authors. XL conceived and designed this study. SX and ZY provided technical support. XL and SM acquired and analyzed the data. GY supervised this study. XL drafted this paper. All authors critically revised this paper.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Operationalization of the research variables.

    DOCX File , 16 KB
    1. Toscano F, O'Donnell E, Broderick JE, May M, Tucker P, Unruh MA, et al. How physicians spend their work time: an ecological momentary assessment. J Gen Intern Med 2020 Nov;35(11):3166-3172 [FREE Full text] [CrossRef] [Medline]
    2. Jabour AM. The impact of longer consultation time: A simulation-based approach. Appl Clin Inform 2020 Oct;11(5):857-864 [FREE Full text] [CrossRef] [Medline]
    3. Barber CEH, Barnabe C, Benseler S, Chin R, Johnson N, Luca N, et al. Patient factors associated with waiting time to pediatric rheumatologist consultation for patients with juvenile idiopathic arthritis. Pediatr Rheumatol Online J 2020 Mar 06;18(1):22 [FREE Full text] [CrossRef] [Medline]
    4. Stadler PC, Senner S, Frey S, Clanner-Engelshofen BM, Frommherz LH, French LE, et al. Teledermatology in times of COVID-19. J Dermatol 2021 May;48(5):620-624 [FREE Full text] [CrossRef] [Medline]
    5. McGreevey JD3, Hanson CW3, Koppel R. Clinical, legal, and ethical aspects of artificial intelligence-assisted conversational agents in health care. JAMA 2020 Aug 11;324(6):552-553. [CrossRef] [Medline]
    6. Qian H, Dong B, Yuan JJ, Yin F, Wang Z, Wang HN, et al. Pre-consultation system based on the artificial intelligence has a better diagnostic performance than the physicians in the outpatient department of pediatrics. Front Med (Lausanne) 2021 Nov 08;8:695185 [FREE Full text] [CrossRef] [Medline]
    7. de Cock C, Milne-Ives M, van Velthoven MH, Alturkistani A, Lam C, Meinert E. Effectiveness of conversational agents (virtual assistants) in health care: Protocol for a systematic review. JMIR Res Protoc 2020 Mar 09;9(3):e16934 [FREE Full text] [CrossRef] [Medline]
    8. Milne-Ives M, de Cock C, Lim E, Shehadeh MH, de Pennington N, Mole G, et al. The effectiveness of artificial intelligence conversational agents in health care: Systematic review. J Med Internet Res 2020 Oct 22;22(10):e20346 [FREE Full text] [CrossRef] [Medline]
    9. Zakim D, Brandberg H, El Amrani S, Hultgren A, Stathakarou N, Nifakos S, et al. Computerized history-taking improves data quality for clinical decision-making-Comparison of EHR and computer-acquired history data in patients with chest pain. PLoS One 2021 Sep 27;16(9):e0257677 [FREE Full text] [CrossRef] [Medline]
    10. Balaji D, He L, Giani S, Bosse T, Wiers R, de Bruijn GJ. Effectiveness and acceptability of conversational agents for sexual health promotion: a systematic review and meta-analysis. Sex Health 2022 Oct;19(5):391-405 [FREE Full text] [CrossRef] [Medline]
    11. Bhattacherjee A. Understanding information systems continuance: An expectation-confirmation model. MIS Q 2001 Sep;25(3):351-370. [CrossRef]
    12. Fazio RH, Zanna MP. Direct experience and attitude-behavior consistency. Adv Exp Soc Psychol 1981;14:161-202. [CrossRef]
    13. Mao CM, Hovick SR. Adding affordances and communication efficacy to the Technology Acceptance Model to study the messaging features of online patient portals among young adults. Health Commun 2022 Mar;37(3):307-315. [CrossRef] [Medline]
    14. McAlearney AS, Gaughan A, MacEwan SR, Fareed N, Huerta TR. Improving acceptance of inpatient portals: Patients' and care team members' perspectives. Telemed J E Health 2020 Mar;26(3):310-326 [FREE Full text] [CrossRef] [Medline]
    15. Tavares J, Oliveira T. New integrated model approach to understand the factors that drive electronic health record portal adoption: Cross-sectional national survey. J Med Internet Res 2018 Nov 19;20(11):e11032 [FREE Full text] [CrossRef] [Medline]
    16. Portz JD, Bayliss EA, Bull S, Boxer RS, Bekelman DB, Gleason K, et al. Using the Technology Acceptance Model to explore user experience, intent to use, and use behavior of a patient portal among older adults with multiple chronic conditions: Descriptive qualitative study. J Med Internet Res 2019 Apr 08;21(4):e11604 [FREE Full text] [CrossRef] [Medline]
    17. Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 1989 Sep;13(3):319-340. [CrossRef]
    18. Liu K, Or CK, So M, Cheung B, Chan B, Tiwari A, et al. A longitudinal examination of tablet self-management technology acceptance by patients with chronic diseases: Integrating perceived hand function, perceived visual function, and perceived home space adequacy with the TAM and TPB. Appl Ergon 2022 Apr;100:103667. [CrossRef] [Medline]
    19. Knox L, Gemine R, Rees S, Bowen S, Groom P, Taylor D, et al. Using the Technology Acceptance Model to conceptualise experiences of the usability and acceptability of a self-management app (COPD.Pal®) for chronic obstructive pulmonary disease. Health Technol (Berl) 2021;11(1):111-117 [FREE Full text] [CrossRef] [Medline]
    20. Beglaryan M, Petrosyan V, Bunker E. Development of a tripolar model of technology acceptance: Hospital-based physicians' perspective on EHR. Int J Med Inform 2017 Jun;102:50-61. [CrossRef] [Medline]
    21. Gagnon MP, Ghandour EK, Talla PK, Simonyan D, Godin G, Labrecque M, et al. Electronic health record acceptance by physicians: testing an integrated theoretical model. J Biomed Inform 2014 Apr;48:17-27 [FREE Full text] [CrossRef] [Medline]
    22. Marcoulides GA, Saunders C. PLS: A silver bullet? MIS Q 2006 Jun;30(2):iii-iix. [CrossRef]
    23. Hoyle RH. The structural equation modeling approach: Basic concepts and fundamental issues. In: Hoyle RH, editor. Structural Equation Modeling: Concepts, Issues, and Applications. Thousand Oaks, CA: Sage Publications Inc; 1995:1-15.
    24. Brislin RW. Back-translation for cross-cultural research. J Cross Cult Psychol 1970;1(3):185-216. [CrossRef]
    25. Chin WW, Newsted PR. Structural equation modeling analysis with small samples using partial least squares. In: Hoyle RH, editor. Statistical Strategies for Small Sample Research. Thousand Oaks, CA: Sage Publications Inc; 1999:307-337.
    26. Anderson JC, Gerbing DW. Structural equation modeling in practice: A review and recommended two-step approach. Psychol Bull 1988;103(3):411-423. [CrossRef]
    27. Hair JFJ, Hult GTM, Ringle CM, Sarstedt M. A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM), Second Edition. Thousand Oaks, CA: Sage Publications Inc; 2017.
    28. Clarke V, Braun V. Thematic analysis. J Posit Psychol 2016 Dec 09;12(3):297-298. [CrossRef]
    29. Kiger ME, Varpio L. Thematic analysis of qualitative data: AMEE Guide No. 131. Med Teach 2020 Aug;42(8):846-854. [CrossRef] [Medline]
    30. Nunnally JC, Bernstein IH. The assessment of reliability. In: Psychometric Theory, Third Edition. New York, NY: McGraw-Hill; 1994:248-292.
    31. Chin WW. How to write up and report PLS analyses. In: Wang H, Henseler J, Chin WW, Vinzi VE, editors. Handbook of Partial Least Squares: Concepts, Methods and Applications. Berlin, Germany: Springer; 2010:655-690.
    32. Reychav I, Arora A, Sabherwal R, Polyak K, Sun J, Azuri J. Reporting health data in waiting rooms with mobile technology: Patient expectation and confirmation. Int J Med Inform 2021 Apr;148:104376. [CrossRef] [Medline]
    33. Ouimet AG, Wagner G, Raymond L, Pare G. Investigating patients' intention to continue using teleconsultation to anticipate postcrisis momentum: Survey study. J Med Internet Res 2020 Nov 26;22(11):e22081 [FREE Full text] [CrossRef] [Medline]
    34. Liu J, Wang J. Users' intention to continue using online mental health communities: Empowerment theory perspective. Int J Environ Res Public Health 2021 Sep 07;18(18):9427 [FREE Full text] [CrossRef] [Medline]
    35. Almutairi ILFH, Alazemi BF, Almutairi FLFH. Kuwaiti hospital patients' continuance intention to use telemedical systems in the wake of the COVID19 pandemic. Healthc Technol Lett 2021 Dec 08;8(6):159-168 [FREE Full text] [CrossRef] [Medline]
    36. Kao HY, Yang YC, Hung YH, Wu YJ. When does Da Vanci robotic surgical systems come into play? Front Public Health 2022 Jan 31;10:828542 [FREE Full text] [CrossRef] [Medline]
    37. Sun SL, Hwang HG, Dutta B, Peng MH. Exploring critical factors influencing nurses' intention to use tablet PC in patients' care using an integrated theoretical model. Libyan J Med 2019 Dec;14(1):1648963 [FREE Full text] [CrossRef] [Medline]
    38. Alam MZ, Khanam L. Comparison of the young aged and elderly female users' adoption of mHealth services. Health Care Women Int. Epub ahead of print 2022 Feb 23:1-25. [CrossRef] [Medline]
    39. Riefle L, Benz C. User-specific determinants of conversational agent usage: A review and potential for future research. In: Innovation Through Information Systems. 2021 Presented at: WI 2021: International Conference on Wirtschaftsinformatik; March 9-11, 2021; Germany p. 115-129. [CrossRef]
    40. Gong E, Baptista S, Russell A, Scuffham P, Riddell M, Speight J, et al. My Diabetes Coach, a mobile app-based interactive conversational agent to support type 2 diabetes self-management: Randomized effectiveness-implementation trial. J Med Internet Res 2020 Nov 05;22(11):e20322 [FREE Full text] [CrossRef] [Medline]
    41. Ponathil A, Ozkan F, Bertrand J, Agnisarman S, Narasimha S, Welch B, et al. An empirical study investigating the user acceptance of a virtual conversational agent interface for family health history collection among the geriatric population. Health Informatics J 2020 Dec;26(4):2946-2966 [FREE Full text] [CrossRef] [Medline]
    42. Hurmuz MZM, Jansen-Kosterink SM, den Akker HO, Hermens HJ. User experience and potential health effects of a conversational agent-based electronic health intervention: Protocol for an observational cohort study. JMIR Res Protoc 2020 Apr 03;9(4):e16641 [FREE Full text] [CrossRef] [Medline]
    43. Spinazze P, Aardoom J, Chavannes N, Kasteleyn M. The computer will see you now: Overcoming barriers to adoption of computer-assisted history taking (CAHT) in primary care. J Med Internet Res 2021 Feb 24;23(2):e19306 [FREE Full text] [CrossRef] [Medline]

    ‎
    AVE: average variance extracted
    CA: conversational agent
    ECM: expectation-confirmation model
    ENT: ear, nose, and throat
    PLS-SEM: partial least squares structural equation model
    TAM: Technology Acceptance Model

    Edited by G Eysenbach; submitted 30.06.22; peer-reviewed by M Rampioni, M Ozan-Rafferty, M Mbwogge; comments to author 23.07.22; revised version received 31.08.22; accepted 20.10.22; published 07.11.22

    Copyright

    ©Xingyi Li, Shirong Xie, Zhengqiang Ye, Shishi Ma, Guangjun Yu. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 07.11.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.