This paper is in the following e-collection/theme issue:

Extended Reality, Virtual Reality and Virtual Worlds (560) Virtual Reality Interventions in Mental Health (122) Digital Pain Assessment and Management (164) Adoption and Change Management of eHealth Systems (712) Qualitative Methods (70) Focus Groups and Qualitative Research for Human Factors Research (851) Vulnerable Populations in Health Research (99) Clinical Integration of XR and Spatial Computing (32)

Published on in Vol 23, No 9 (2021): September

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/26623, first published .
Barriers and Facilitators to the Implementation of Virtual Reality as a Pain Management Modality in Academic, Community, and Safety-Net Settings: Qualitative Analysis

Barriers and Facilitators to the Implementation of Virtual Reality as a Pain Management Modality in Academic, Community, and Safety-Net Settings: Qualitative Analysis

Barriers and Facilitators to the Implementation of Virtual Reality as a Pain Management Modality in Academic, Community, and Safety-Net Settings: Qualitative Analysis

Authors of this article:

Urmimala Sarkar1, 2 Author Orcid Image ;   Jane E Lee1, 2 Author Orcid Image ;   Kim H Nguyen1, 2, 3 Author Orcid Image ;   Sarah Lisker1, 2 Author Orcid Image ;   Courtney R Lyles1, 2, 3 Author Orcid Image
Article Authors Cited by (14) Tweetations (15) Metrics

    Original Paper

    1Department of Medicine, University of California San Francisco, San Francisco, CA, United States

    2Center for Vulnerable Populations, University of California San Francisco, San Francisco, CA, United States

    3Department of Epidemiology and Biostatistics, University of California San Francisco, San Francisco, CA, United States

    Corresponding Author:

    Urmimala Sarkar, MD, MPH

    Department of Medicine

    University of California San Francisco

    1001 Potrero Avenue

    Box 1364

    San Francisco, CA, 94143

    United States

    Phone: 1 6282064273

    Email: urmimala.sarkar@ucsf.edu


    Background: Prior studies have shown that virtual reality (VR) is an efficacious treatment modality for opioid-sparing pain management. However, the majority of these studies were conducted among primarily White, relatively advantaged populations and in well-resourced settings.

    Objective: We conducted a qualitative, theory-informed implementation science study to assess the readiness for VR in safety-net settings.

    Methods: Using the theoretical lens of the Consolidated Framework for Implementation Research (CFIR) framework, we conducted semistructured interviews with current VR users and nonusers based in safety-net health systems (n=15). We investigated barriers and facilitators to a commercially available, previously validated VR technology platform AppliedVR (Los Angeles, CA, USA). We used deductive qualitative analysis using the overarching domains of the CFIR framework and performed open, inductive coding to identify specific themes within each domain.

    Results: Interviewees deemed the VR intervention to be useful, scalable, and an appealing alternative to existing pain management approaches. Both users and nonusers identified a lack of reimbursement for VR as a significant challenge for adoption. Current users cited positive patient feedback, but safety-net stakeholders voiced concern that existing VR content may not be relevant or appealing to diverse patients. All respondents acknowledged the challenge of integrating and maintaining VR in current pain management workflows across a range of clinical settings, and this adoption challenge was particularly acute, given resource and staffing constraints in safety-net settings.

    Conclusions: VR for pain management holds interest for frontline pain management clinicians and leadership in safety-net health settings but will require significant tailoring and adaption to address the needs of diverse populations. Integration into complex workflows for pain management is a significant barrier to adoption, and participants cited structural cost and reimbursement concerns as impediments to initial implementation and scaling of VR use.

    J Med Internet Res 2021;23(9):e26623

    doi:10.2196/26623

    Keywords

    virtual reality; medical informatics; information technology; implementation science; qualitative research 


    Immersive virtual reality (VR) has emerged as an efficacious treatment modality for a wide range of medical and neurocognitive conditions including pain. VR is delivered via a headset that displays computer-generated and oftentimes interactive audio and visual content that can be designed to reduce pain through techniques such as distraction, relaxation, and mindfulness. Randomized controlled trials and effectiveness studies demonstrate that VR improves pain scores among inpatients and during medical procedures [1-4]. Additionally, studies indicate that patients can use VR to help manage chronic pain [5-8]. In a prior study among predominantly non-Hispanic White patients, most patients found VR use for pain to be a positive experience [9]. VR is generally considered safe, with the most common adverse effects being dizziness and nausea. Motion sickness and seizure disorders are the most prevalent of the few medical contraindications to VR use.

    Clinicians have demonstrated interest in VR therapy as a safe and effective adjunctive or replacement to opioid agents to avoid adverse consequences of opioid use [10]. Opioid misuse represents a growing epidemic in the United States, affecting 10.3 million people, 48,000 of whom died in 2018. Even in acute-care settings, initiation of opioids can lead to long-term use, the consequences of which include dependence and hyperalgesia [11]. There is an urgent need for safer pain management for the millions of chronic pain sufferers worldwide. This represents a public health challenge as well, as racial and ethnic minorities, the elderly, and patients who speak English as a second language are more likely to experience suboptimal treatment for their chronic pain [12].

    Despite studies proving the efficacy of and great promise for VR therapy as a nonpharmacologic approach to pain management, VR has not become part of routine chronic pain management [2,13,14]. As with other complex interventions, there is little evidence to guide implementation [15]. Although we know that VR has been effective in clinical trial settings, understanding why and how it has been effective is required to translate this modality to diverse practice settings [16]. Most VR studies have also been conducted in settings that serve ethnically homogenous, relatively advantaged populations with high health literacy and educational attainment.

    There are specific considerations in adapting digital innovations to diverse populations and to safety-net health care settings that disproportionately care for them [17]. A systematic review of digital innovations in the safety net found that externally developed interventions face significant challenges, including acceptability to providers, staff, and patients; staffing needs; and implementation costs [18]. We sought to elucidate the implementation climate specifically for VR [19]. Therefore, we conducted a qualitative, theory-informed implementation science study to assess the readiness for VR in safety-net settings.


    This study aims to determine barriers and facilitators to the implementation of VR as a treatment for pain in safety-net settings. We obtained institutional review board approval (#19-29025) for this project from the University of California, San Francisco (UCSF) Human Research Protection Program.

    Conceptual Framework and Interview Guide

    We developed an interview guide based on the Consolidated Framework for Implementation Research (CFIR) [20,21]. We chose this framework because it accounts for a wide range of contextual factors and is widely used to characterize complex interventions in health care settings [22-26]. The CFIR framework includes 5 overarching domains: (1) the characteristics of the intervention itself; (2) the inner setting, that is, the specific environment in which the intervention will take place; (3) the outer setting, or larger environment, encompassing the broader landscape in which the inner setting sits; (4) the individuals involved in intervention delivery and the target population; and (5) the process by which the intervention is introduced.

    Data Collection

    We used the snowball method of sampling to conduct 15 semistructured interviews (Multimedia Appendix 1) of 5 health care providers, leaders, and administrators who currently provide VR to patients (users) and 10 who do not use VR (nonusers). Most but not all nonusers practiced in safety-net settings, defined as public hospitals or federally qualified health centers serving the majority of patients on Medicaid or those who are uninsured [27]. We intentionally sampled nonusers as health care providers or staff without experience in using VR for clinical pain treatment, ensuring that every participant had expertise working with diverse patient populations in their practice (i.e., safety net or academic medical center sites serving Medicaid or racially/ethnically or linguistically diverse patients). We oversampled safety-net sites in particular because they are generally the last to be considered for implementation of innovation programs, and we sought to seek out their perspectives early to improve eventual broadscale implementation of this technology for diverse patient populations [28]. We conducted interviews from October 2019 to September 2020. Two members of the study team conducted the interviews via videoconferencing with screen sharing because of either logistical barriers or public health restrictions due to the COVID-19 pandemic for meeting in person. The team recorded the interviews with participant consent and transcribed the interviews without any identifying information. The participants received an incentive for their participation, with e-gift cards sent via email following interview completion.

    Data Analysis

    We used the qualitative software Dedoose (SocioCultural Research Consultants, Hermosa Beach, CA, USA) for analysis. Two authors, KN and JL, read all the transcripts in detail. For deductive coding, they assigned excerpts to 1 of the 5 overarching CFIR domains (intervention characteristics, individual characteristics, implementation process, inner setting, and outer setting) and met to discuss codes and resolve inconsistencies (Figure 1). They then independently analyzed 3 interview transcripts inductively to identify themes that arose from the interviews. They subsequently met to reach an agreement on a comprehensive list of potential themes. One investigator (author JL) coded the remaining (12 total) transcripts under the supervision of KN for any relevant CFIR constructs and new inductive codes that emerged from participant responses. The entire study team met regularly to discuss discrepancies or resolve disputes regarding the coding. All authors iteratively revised and agreed upon the final list of themes and representative quotes. We reached thematic saturation after 11 interviews and elected to complete 4 additional interviews that had already been scheduled.

    ‎
    Figure 1. Overview of Consolidated Framework for Implementation Research (CFIR) domains and examples of barriers and facilitators.
    View this figure

    Data Availability

    The qualitative data generated and analyzed during the current study are not publicly available due to the risk of re-identification of participants, but codebooks are available upon reasonable request.


    We interviewed 15 participants from a variety of roles and organizations (Table 1).

    We defined “user” as a health care provider or staff member interviewee with previous research experience implementing the AppliedVR platform for pain treatment at their delivery system; we defined “nonuser” as a health care provider or staff member interviewee without experience with AppliedVR or other VR platforms for clinical pain treatment.

    Underlying these results was a strong sense of an unmet need for chronic pain management among users and nonusers. There was agreement among all interviewees that addressing pain management differently represents a high organizational/clinical priority. Building on that universally acknowledged need and prioritization, both users and nonusers of VR identified a range of factors related to the intervention itself and to the milieu in which it would be implemented that they expected would impact VR implementation. We described the major themes within each CFIR domain and provided exemplar quotes.

    Table 1. Participant roles, user status, and organization characteristics.
    Participant IDParticipant roleUser/nonuserOrganization characteristics
    1Chief Medical Information OfficerUserSafety-net health system
    2Research physicianUserAcademic medical center
    3Clinical research coordinatorUserAcademic medical center
    4aDirector for Innovation and Digital HealthaUserAcademic medical center
    4bProgram Manager for Innovation and Digital HealthaUserAcademic medical center
    5Director of surgical subspecialty serviceUserNonprofit regional tertiary medical center
    6Primary care and addiction services physicianNonuserSafety-net health system
    7Associate Director of Research and Health EquityNonuserAcademic medical center
    8Chief Medical OfficerNonuserSafety-net health system
    9Chief Medical Informatics OfficerNonuserSafety-net health system
    10Primary care physician, pain clinicNonuserSafety-net health system
    11Director of TelehealthNonuserSafety-net health system
    12Psychologist, internal medicineNonuserAcademic medical center
    13Registered nurse, palliative careNonuserSafety-net health system
    14Associate Chief Medical Informatics OfficerNonuserSafety-net health system
    15Psychologist, internal medicineNonuserSafety-net health system

    aThese participants were interviewed simultaneously.

    Characteristics of the Intervention

    The VR intervention itself has attributes that both facilitate and inhibit implementation (Multimedia Appendix 2, Table S1). A key factor for the intervention is its relative advantage over existing approaches for pain. VR users and nonusers both mentioned the favorable safety of using VR compared to pain medications. Additionally, VR users also mentioned the option for patients to use the technology on their own on an ongoing basis after initial training from staff, as opposed to ongoing, clinician-delivered pain interventions, such as cognitive behavioral therapy. Similarly, nonusers appreciated the potential for scaling pain management activities through VR-enabled self-management approaches. Nonusers of VR commented on the face validity of VR as a chronic pain treatment. In terms of the intervention content, both users and nonusers noted that the platform is available only in English and therefore not usable in limited English–proficiency populations. The difference between users and nonusers was in how they prioritized this limitation. Interviewees based in the safety net described non-English language as a prerequisite, whereas it was considered a limitation of varying importance among those who were already using VR. The participants discussed the ability to culturally tailor VR scenarios in opposing ways. VR nonusers expressed some skepticism about the content’s appropriateness for culturally diverse populations and for those with significant trauma histories. VR users noted that the currently available content does not always resonate across cultures, particularly for minoritized populations. However, they did consider the ability of VR to offer many culturally tailored content offerings to be a key attribute of the approach. Both users and nonusers acknowledged the specific contraindication of motion sickness and the overall low risk of the modality.

    Individual Characteristics

    For this study, we considered the CFIR domain of “individual characteristics” to refer to target patients for VR as a pain intervention (Multimedia Appendix 2, Table S2). A prevalent theme was comfort with technology, including challenges for older adults in navigating technology and digital literacy more broadly. Nonusers, in particular, raised concerns about patient mistrust of new or experimental treatments. Despite this, several interviewees emphasized that they would expect a significant proportion of low-income and diverse patients cared for in safety-net settings to willingly engage with VR. Participants who used VR expressed near-universal high levels of patient satisfaction with VR, and nonusers expected that patients with chronic pain would appreciate having additional treatment options. Nonusers did cite physical opioid dependence and a fear of opioid de-prescribing as expected barriers from patients, and users and nonusers alike referred to patients’ more pressing social and health needs as significant barriers to VR implementation for pain. Nonusers specifically mentioned trade-offs for health systems in deciding whether to invest in innovations such as VR versus approaches to address patient needs such as housing and food insecurity.

    Implementation Process

    Both VR users and nonusers considered VR implementation to be a complex process (Multimedia Appendix 2, Table S3). There was consensus that patients would require specific orientation from staff in order to initiate VR use for pain and that staff support would be required for coaching and troubleshooting on an ongoing basis. Both users and nonusers pointed to the requirement that frontline staff interact with VR and have a personal buy-in to facilitate successful implementation. Similarly, all participants highlighted the need for champions among clinicians who can share both evidence for VR as a pain treatment and successful treatment experiences. The availability of staff time was a dominant concern, with VR users emphasizing the need to plan for adequate staff effort to implement VR and nonusers expressing doubt about staff availability. Integration of VR into existing pain management workflows, a key aspect of the implementation process, highlighted barriers and facilitators. It emerged that implementation success would depend on the extent to which VR could be integrated into specific clinical settings. Some, but not all, nonusers focused on integration into the electronic health record. The participants cited some especially challenging settings, such as primary care, and some more feasible settings, such as the outpatient pain clinic. More specifically, the intervention requires a multistep process with each patient, which both users and nonusers believe complicates more widespread implementation.

    Inner Setting

    The inner setting for VR implementation is the specific health system and, within each health system, the specific clinical settings that currently treat pain (Multimedia Appendix 2, Table S4). One overarching inner setting barrier, seen across all participants, is that within health systems, pain management is a cross-cutting issue and therefore requires collaboration among departments and stakeholders, in contrast to other clinical conditions for which one clinical unit is entirely responsible. Both users and nonusers cited pressure to prescribe opioids and the providers’ lack of familiarity with VR as barriers to use. Several participants mentioned the default habit and culture of addressing pain with medication. Among the nonusers, participants noted that their peers hold skepticism about corporate approaches to treatment. Both users and nonusers emphasized the importance of leadership attitudes. They described interest in VR as part of a larger mindset around openness to innovation. All participants emphasized the need for dedicated staff time and training and space for VR treatment, which require a leadership commitment. Resource limitations in safety-net settings were seen to preclude these needed actions for implementation. Most participants suggested that an innovative culture is a prerequisite for VR implementation, but safety-net participants described this as pervasive: “We seem to be late adopters.” In contrast, VR users described few individuals as having an anti-innovation attitude. Some nonusers described local leaders as not being innovation minded but instead being focused on delivery of long-standing services. Similarly, nonusers described the wide array of stakeholders (eg, frontline staff, information technology [IT], and clinical leaders) within the local environment whose engagement was needed for success. In terms of concrete needs, both users and nonusers cited the need for adequate physical space for VR treatment and up-front investment in the VR hardware as inner-setting requirements. However, VR users experienced these needs as surmountable challenges, while VR nonusers highlighted space and initial costs as formidable obstacles.

    Outer Setting

    Challenges related to the outer setting of the intervention, such as external policies and incentives, arose as well (Multimedia Appendix 2, Table S5). In the broader health care landscape, use of VR is not a billable service. The participants viewed this lack of insurance reimbursement as a critical barrier to implementation at scale. Both VR users and nonusers mentioned philanthropic and private funding as a current or possible approach for the short term. There was consensus that expecting patients to bear the cost of pain treatment is a significant implementation barrier. Beyond cost, concerns around data privacy and security in digital health surfaced among nonusers. Nonusers also pinpointed the digital health industry’s focus on wellness over developing treatments for specific conditions as a barrier to widespread use of digital therapies more broadly.


    Principal Results

    Our results suggest that many of the factors affecting VR implementation, or, indeed, implementation of novel, technology-enabled therapeutic approaches, exist in both safety-net settings and more advantaged, resource-rich health care systems. We did not identify a completely different set of issues relating to VR in the safety net; rather, similar challenges seem to be magnified in safety-net settings. For example, among safety-net stakeholders who do not currently use VR, an English-only intervention would be impossible to implement for their linguistically diverse patient populations, whereas non-safety-net participants saw this limitation as less significant in impacting adoption. Similarly, the cultural relevance and acceptability of the content was a source of concern for safety-net interview participants but appeared to be surmountable to non-safety-net participants who were already testing VR at their sites. In other words, current VR users saw further cultural tailoring as an area for future improvement, whereas our safety-net participants saw it as a prerequisite for use because of the diverse populations served in safety-net settings. Both VR-use and VR-nonuse sites acknowledged that competing demands in patients’ lives, relating to housing status, income instability, and other social challenges, would interfere with using a novel therapeutic approach like VR. Again, these challenges seemed more prevalent in the safety net.

    Our findings in the CFIR domain of “inner setting” suggest that the process of implementing a complex intervention like VR requires a formal change management strategy encompassing leadership support, frontline champions, and a campaign for staff buy-in. Safety-net respondents reported a lack of capacity for implementing new approaches, particularly with regard to leadership commitment to innovation and resources to conduct change management activities. The outer setting or larger context for VR implementation has facilitators, such as the desire to appear innovative, while reducing the harm from opioid use. However, the lack of insurance reimbursement and available funding for VR interventions is a significant barrier. Competing priorities for leadership can also stymie innovations such as VR implementation, especially in safety-net health care settings where resource limitations and “keeping the doors open” are perennially in question.

    Our findings echo prior work about innovation in safety-net settings. Dating back to the early days of health IT, safety-net settings have lagged in implementation [29]. Prior studies suggest that frontline patient engagement and trust, appropriateness of innovative tools for diverse end users, and dedicated resources for innovation are particularly salient for the implementation of new health IT approaches in the safety net [18,28,30,31]. Implementation science studies in safety-net settings should specifically emphasize these concepts in applying, and, where necessary, adapting, conceptual frameworks such as the CFIR.

    Our study lends weight to a previously developed evaluation framework for VR [19]. The VR1 study design phase emphasizes the importance of working directly with end users, and we found that implementation in safety-net health settings will require significant tailoring and adaptation of the VR intervention to address the needs of diverse populations (eg, racial and ethnic minorities, language barriers, and social complexity). Several actions can address the current barriers to VR use for pain management in safety-net health care settings. For example, our results underscore the need for cultural tailoring and translation (eg, VR content that features voices and images of racially and ethnically diverse individuals and is provided in languages other than English). Furthermore, more specific adaptations or considerations addressing structural issues disproportionately facing lower-income and racial/ethnic minorities in the United States (eg, unjust incarceration practices and historical trauma) might require deeper qualitative exploration with patients about their preferences or comfort in using VR headsets. A prior study of predominantly non-Hispanic White hospitalized patients demonstrated that many refuse to try VR [9]; the acceptability of VR to patients should be tested in more diverse patient groups. Following this, usability testing with diverse populations could evaluate concerns about the acceptability and usability of technology, disinfection, and adverse effects such as motion sickness. Any deployment strategy for VR should take into account the resource and workforce constraints of the safety-net environment. Developers of VR for chronic pain management should address implementation considerations in parallel with design and evaluation in order to foster higher uptake of VR and digital tools overall.

    Strengths and Limitations

    Our study has strengths, such as inclusion of multiple health systems and geographic areas. In addition, we compared and contrasted the experiences of health care system participants who are currently overseeing the use of a single VR platform with those of safety-net and academic providers across multiple sites who are not currently using VR. Furthermore, we used a well-known conceptual framework and reached thematic saturation in content analysis. Despite these strengths, the study does have limitations. The sample size was circumscribed, and we do not have information about the extent of VR use among the participants who are currently using the tool in their health systems.

    Although our discussion focused on one technology approach, VR, for a single clinical problem (pain management), many of the themes that emerged are relevant across multiple innovative approaches to clinical care. The spread of innovations to safety-net health settings requires attention to patients’ language and culture. Efforts to address resource constraints, such as providing staff training and detailed protocols encompassing workflow integration, can lower barriers to innovation. Inclusion of diverse populations and the health care settings where they disproportionately receive care will address issues that are relevant across other health systems as well.

    Conclusion

    This study contributes to the literature by revealing barriers and facilitators to implementing VR for pain management in safety-net settings. We demonstrate that frontline pain management clinicians and leaderships are interested in VR, but it will require significant tailoring and adaptation to address the specific needs of the diverse populations they serve. The participants cited integration into complex workflows, structural costs, and reimbursement concerns as major barriers to implementing and scaling VR use. Future studies should augment this approach with direct observation, quantitative measures of VR use, and direct usability assessment of the AppliedVR platform with diverse patient populations.

    Acknowledgments

    We would like to thank all interviewees for participating.

    This publication was supported by the Food and Drug Administration (FDA) of the US Department of Health and Human Services (HHS) as part of a financial assistance award (Center of Excellence in Regulatory Science and Innovation Grant to University of California, San Francisco (UCSF) and Stanford University, U01FD005978) totaling $50,000, with 50% funded by the FDA/HHS, and $50,000, with 50% funded by a nongovernment source (AppliedVR). The contents are those of the author(s) and do not necessarily represent the official views of, nor an endorsement by, the FDA/HHS or the US government.

    Conflicts of Interest

    The study team received funding from AppliedVR, but AppliedVR was not involved in data collection, interpretation, or dissemination of findings.

    ‎
    Multimedia Appendix 1

    Consolidated Framework for Implementation Research (CFIR) structured interview guides for users and nonusers.

    DOCX File , 37 KB
    ‎
    Multimedia Appendix 2

    Participant quotes.

    DOCX File , 56 KB
    1. Carrougher GJ, Hoffman HG, Nakamura D, Lezotte D, Soltani M, Leahy L, et al. The effect of virtual reality on pain and range of motion in adults with burn injuries. J Burn Care Res 2009;30(5):785-791 [FREE Full text] [Medline]
    2. Dascal J, Reid M, IsHak WW, Spiegel B, Recacho J, Rosen B, et al. Virtual reality and medical inpatients: a systematic review of randomized, controlled trials. Innov Clin Neurosci 2017;14(1-2):14-21 [FREE Full text] [Medline]
    3. Spiegel B, Fuller G, Lopez M, Dupuy T, Noah B, Howard A, et al. Virtual reality for management of pain in hospitalized patients: a randomized comparative effectiveness trial. PLoS One 2019;14(8):e0219115 [FREE Full text] [CrossRef] [Medline]
    4. Wismeijer AAJ, Vingerhoets AJJM. The use of virtual reality and audiovisual eyeglass systems as adjunct analgesic techniques: a review of the literature. Ann Behav Med 2005 Dec;30(3):268-278. [CrossRef] [Medline]
    5. Bani Mohammad E, Ahmad M. Virtual reality as a distraction technique for pain and anxiety among patients with breast cancer: a randomized control trial. Palliat Support Care 2019 Feb;17(1):29-34. [CrossRef] [Medline]
    6. Matheve T, Bogaerts K, Timmermans A. Virtual reality distraction induces hypoalgesia in patients with chronic low back pain: a randomized controlled trial. J Neuroeng Rehabil 2020 Apr 22;17(1):55 [FREE Full text] [CrossRef] [Medline]
    7. Sarig BH, Takasaki H, Chen X, Bet-Or Y, Treleaven J. Cervical kinematic training with and without interactive VR training for chronic neck pain: a randomized clinical trial. Man Ther 2015 Feb;20(1):68-78. [CrossRef] [Medline]
    8. Sarig Bahat H, Croft K, Carter C, Hoddinott A, Sprecher E, Treleaven J. Remote kinematic training for patients with chronic neck pain: a randomised controlled trial. Eur Spine J 2018 Jun;27(6):1309-1323. [CrossRef] [Medline]
    9. Mosadeghi S, Reid MW, Martinez B, Rosen BT, Spiegel BMR. Feasibility of an Immersive Virtual Reality Intervention for Hospitalized Patients: An Observational Cohort Study. JMIR Ment Health 2016 Jun 27;3(2):e28 [FREE Full text] [CrossRef] [Medline]
    10. FDA identifies harm reported from sudden discontinuation of opioid pain medicines and requires label changes to guide prescribers on gradual, individualized tapering. U.S. Food and Drug Administration. 2019 Apr 09.   URL: https:/​/www.​fda.gov/​drugs/​drug-safety-and-availability/​fda-identifies-harm-reported-sudden-discontinuation-opioid-pain-medicines-and-requires-label-changes [accessed 2021-08-30]
    11. Larochelle MR, Liebschutz JM, Zhang F, Ross-Degnan D, Wharam JF. Opioid prescribing after nonfatal overdose and association with repeated overdose. Ann Intern Med 2016 Sep 06;165(5):376. [CrossRef] [Medline]
    12. Institute of Medicine (US) Committee on Advancing Pain Research, Care, and Education. Pain as a Public Health Challenge. Washington DC: National Academies Press; 2011.
    13. Morris LD, Louw QA, Grimmer-Somers K. The effectiveness of virtual reality on reducing pain and anxiety in burn injury patients. Clin J Pain 2009;25(9):815-826. [CrossRef] [Medline]
    14. Shahrbanian S, Ma X, Aghaei N, Korner-Bitensky N, Simmonds M. Use of virtual reality (immersive vs. non immersive) for pain management in children and adults: a systematic review of evidence from randomized controlled trials. Eur J Exp Biol 2012 Jan 10;2:1408-1422.
    15. Greenhalgh T, Robert G, MacFarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 2004 Dec;82(4):581-629. [CrossRef]
    16. Dixon-Woods M, Bosk CL, Aveling EL, Goeschel CA, Pronovost PJ. Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Q 2011 Jun;89(2):167-205. [CrossRef]
    17. Lyles CR, Handley MA, Ackerman SL, Schillinger D, Williams P, Westbrook M, et al. Innovative implementation studies conducted in US safety net health care settings: a systematic review. Am J Med Qual 2019 Jun;34(3):293-306 [FREE Full text] [CrossRef] [Medline]
    18. Nouri SS, Adler-Milstein J, Thao C, Acharya P, Barr-Walker J, Sarkar U, et al. Patient characteristics associated with objective measures of digital health tool use in the United States: a literature review. J Am Med Inform Assoc 2020 May 01;27(5):834-841 [FREE Full text] [CrossRef] [Medline]
    19. Birckhead B, Khalil C, Liu X, Conovitz S, Rizzo A, Danovitch I, et al. Recommendations for methodology of virtual reality clinical trials in health care by an international working group: iterative study. JMIR Ment Health 2019 Jan 31;6(1):e11973 [FREE Full text] [CrossRef] [Medline]
    20. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci [Internet] 2009;4:50 [FREE Full text] [CrossRef] [Medline]
    21. Damschroder LJ, Lowery JC. Evaluation of a large-scale weight management program using the consolidated framework for implementation research (CFIR). Implement Sci 2013 May 10;8:51 [FREE Full text] [CrossRef] [Medline]
    22. Cilenti D, Brownson RC, Umble K, Erwin PC, Summers R. Information-seeking behaviors and other factors contributing to successful implementation of evidence-based practices in local health departments. J Public Health Manag Pract 2012 Nov;18(6):571-576 [FREE Full text] [CrossRef] [Medline]
    23. Garg SK, Lyles CR, Ackerman S, Handley MA, Schillinger D, Gourley G, et al. Qualitative analysis of programmatic initiatives to text patients with mobile devices in resource-limited health systems. BMC Med Inform Decis Mak 2016 Feb;16:16 [FREE Full text] [CrossRef] [Medline]
    24. Khoong EC, Cherian R, Smith DE, Schillinger D, Wolf MS, Sarkar U. Implementation of patient-centered prescription labeling in a safety-net ambulatory care network. Am J Health Syst Pharm 2018 Aug 15;75(16):1227-1238. [CrossRef] [Medline]
    25. Kirk MA, Kelley C, Yankey N, Birken SA, Abadie B, Damschroder L. A systematic review of the use of the Consolidated Framework for Implementation Research. Implement Sci 2016 May 17;11:72 [FREE Full text] [CrossRef] [Medline]
    26. Zulman DM, Damschroder LJ, Smith RG, Resnick PJ, Sen A, Krupka EL, et al. Implementation and evaluation of an incentivized Internet-mediated walking program for obese adults. Transl Behav Med 2013 Dec;3(4):357-369. [CrossRef] [Medline]
    27. Lewin ME, Altman S, editors. America’s Health Care Safety Net: Intact but Endangered. Washington, D.C: National Academies Press; 2000.
    28. Lyles CR, Aulakh V, Jameson W, Schillinger D, Yee H, Sarkar U. Innovation and transformation in California's safety net health care settings: an inside perspective. Am J Med Qual 2014;29(6):538-545 [FREE Full text] [CrossRef] [Medline]
    29. Jha AK, DesRoches CM, Shields AE, Miralles PD, Zheng J, Rosenbaum S, et al. Evidence of an emerging digital divide among hospitals that care for the poor. Health Aff (Millwood) 2009 Nov;28(6):w1160-w1170. [CrossRef] [Medline]
    30. Moore SL, Fischer I, Havranek EP. Translating health services research into practice in the safety net. Health Serv Res 2016 Feb;51(1):16-31 [FREE Full text] [CrossRef] [Medline]
    31. Lyles CR, Nelson EC, Frampton S, Dykes PC, Cemballi AG, Sarkar U. Using electronic health record portals to improve patient engagement: research priorities and best practices. Ann Intern Med 2020 Jun 02;172(11 Suppl):S123-S129 [FREE Full text] [CrossRef] [Medline]

    ‎
    CFIR: Consolidated Framework for Implementation Research
    IT: information technology
    UCSF: University of California, San Francisco
    VR: virtual reality

    Edited by R Kukafka, G Eysenbach; submitted 18.12.20; peer-reviewed by B Spiegel, A Tapuria; comments to author 18.01.21; revised version received 26.01.21; accepted 25.05.21; published 22.09.21

    Copyright

    ©Urmimala Sarkar, Jane E Lee, Kim H Nguyen, Sarah Lisker, Courtney R Lyles. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 22.09.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.