Published on in Vol 25 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/45821, first published .
Medical Extended Reality Trials: Building Robust Comparators, Controls, and Sham

Medical Extended Reality Trials: Building Robust Comparators, Controls, and Sham

Medical Extended Reality Trials: Building Robust Comparators, Controls, and Sham

Authors of this article:

Susan Persky 1 Author Orcid Image ;   Luana Colloca 2 Author Orcid Image

Viewpoint

1Social and Behavioral Research Branch, National Human Genome Research Institute, Bethesda, MD, United States

2Department of Pain and Translational Symptom Science, School of Nursing, University of Maryland Baltimore, Baltimore, MD, United States

Corresponding Author:

Susan Persky, PhD

Social and Behavioral Research Branch

National Human Genome Research Institute

31 Center Drive

B1B36

Bethesda, MD, 20892

United States

Phone: 1 3014430098

Email: perskys@mail.nih.gov


The explosive pace of development and research in medical extended reality (MXR) is a testament to its promise for health care and medicine. In comparison with this growth, there is a relative sparsity of rigorous clinical trials that establish the efficacy and effectiveness of these interventions. Explicating mechanisms of action across clinical areas and MXR applications is another major area of need. A primary impediment to these goals is a lack of frameworks for trial design, more specifically, the selection of appropriate controls that effectively address unique elements of MXR. This paper delineates a framework for designing controls, sham conditions, and comparators, as well as proposed considerations for MXR trial designs. Special consideration is given to the design of sham conditions. Improved designs would enable more robust findings and the development of generalizable knowledge that could be adopted across MXR interventions.

J Med Internet Res 2023;25:e45821

doi:10.2196/45821

Keywords



Overview

Medical extended reality (MXR), the application of extended reality (XR) technology such as virtual reality (VR) and augmented reality to health care and medicine, spans a variety of domains, including patient treatment, procedure support, health education, and provider training. MXR is an area of long-anticipated promise that is finally beginning to materialize and make its way into clinical implementation [1-6]. Accordingly, the pace of development and research in MXR has accelerated dramatically in the past few years, resulting in a flood of new products and scientific papers. By comparison, there is yet a sparsity of rigorous clinical trials that establish the efficacy and effectiveness of these interventions. There is also a major need for research that establishes mechanisms of action across clinical areas and application types. This work will allow for better targeting and tailoring of future intervention designs [7,8]. A primary impediment to these goals is the lack of frameworks for trial designs and the selection of appropriate controls that effectively address the unique elements of MXR. This paper posits the challenges inherent in clinical trial designs for MXR and proposes approaches for choosing controls, sham conditions, and comparators that account for the technology’s complexity and multifaceted nature.

General Considerations

The clinical trial literature discusses comparators, an umbrella term describing another treatment, product, or experience to which the focal intervention is compared as a frame of reference to determine its efficacy, safety, or other benchmarks. Comparators as a class encompass the more specific concept of a control condition. A control condition provides a comparison that includes no active components of the intervention (ie, elements that underlie effectiveness) and, as such, allows researchers to determine that the treatment or intervention itself causally influenced patient outcomes. A control condition, therefore, includes elements present in the intervention that differ from the proposed primary active components (eg, intervention delivery vehicle and interactions with the study team), allowing researchers to rule out alternative explanations for patient outcomes. This may take the form of a sham (where study activities are performed without the performance of the active component activities) or a placebo (where an inactive component is substituted for an active one). In the context of MXR trials, there is little practical difference between sham and placebo control, and we use them interchangeably here.

The control condition essentially defines the claims that can be made about the efficacy or effectiveness of an MXR intervention in the case of a positive finding (ie, finding that the intervention is more effective than the control). As such, this is the primary consideration underlying control condition design or comparator choice. Due to variation in trial goals and variation in the nature of individual MXR interventions, there is no universal, optimal control for MXR interventions as a class. Control suitability depends on features of the intervention, the setting, and the target population and, ultimately, on potential sources of patient and health care provider expectancies (ie, beliefs about whether and how well a treatment will work).

There are many other factors that influence choice of comparator or control, and these should be considered in light of the trial’s primary objective. For example, different considerations are typically applied when designing control conditions for proof-of-concept, feasibility, and acceptability studies. There are also situations in which the profile of the intervention, the health condition under study, or the patient population dictates what is possible and ethically acceptable [9,10]. For example, it may be unethical to assign patients in ill-health to a no-treatment control when treatments are otherwise available. In addition to many potential benefits, there are several potential risks associated with MXR use that must be considered in the process of trial design. These include cybersickness (similar to motion sickness, stemming from XR use); strain or discomfort to the head, neck, or face; and distraction or disorientation [11]. Some approaches to MXR provision could also introduce risks to patient privacy if adequate data collection and protection measures are lacking or not possible in a given situation [12]. The risks present in the specific study and intervention context must be weighed against potential benefits for the ethical design of MXR trials.

Given competing demands, choosing the comparator or designing the control condition can be one of the most difficult, yet vitally important, parts of designing an MXR study, as it dictates what can be concluded from the study in terms of outcomes such as efficacy, effectiveness, acceptability, safety, and mechanisms of action.

Existing Guidance

Given the unique profile of MXR, the research community has recognized the need for specialized study design. Birckhead et al [13] made recommendations for clinical trial design, covering multiple design questions, including choice of control group. The authors discuss a range of forms that a control condition can take inside or outside of a headset, including both passive and interactive features. Ultimately, they recommend a study design that considers purported MXR mechanisms and targets of action. Beams et al [14] similarly address a variety of challenges associated with the evaluation of MXR applications from a regulatory perspective. Among other possibilities, these authors describe the promise of sham control interventions that involve the provision of an XR headset that administers 2D visualizations. Such sham groups (discussed in detail below) have become popular in VR-based clinical trials, including the high-profile trials associated with early US Food and Drug Administration–cleared XR applications [14]. More broadly, scientists and regulators have noted that the subject of clinical trial design, and control conditions for MXR trials, needs more exploration and explication in the scientific literature [6,13-15]. Such consideration would greatly strengthen the quality of trials evaluating MXR treatments and interventions. Herein we focus on frameworks and control conditions for MXR generally, although many details and examples are based on VR given its current dominance of the MXR literature.

Frameworks for MXR Trials

Although there are many variations in frameworks, the broad classes of randomized controlled trial comparators can be boiled down to those that are created and administered by the research team (eg, a control condition developed to parallel the intervention) and those that are not created by the research team (eg, treatment as usual, wherein participants are asked to continue their existing treatment approach). Gold et al [16] further break these areas down into several specific examples in the domain of traditional clinical trials. These include placebo control, specific factor component control, active comparator, treatment as usual, no treatment control, and waitlist control. We expand on these examples in Table 1 to consider how they can be applied to trials evaluating MXR interventions.

Table 1. Comparator group types, associated claims, and considerations.
Categories and definitionsMXRa exampleExample claim from a positive randomized controlled trialConsideration for use in MXR trials
Placebo control: nonspecific elements of the intervention, such as attention or delivery mechanism, without the proposed active elementsSham virtual realityThe MXR intervention is effective, and this is due to the purported active components
  • There is a clear set of proposed active components that can be separated from inactive elements
  • Placebo has been tested, and its effects are understood
  • Placebo can be made to appear credible to participants
Specific factors component control: reduced number of active intervention factors (ie, components) in addition to inactive factorsIntervention content delivered in 2D on a tabletActive components of the intervention benefit from XRb delivery
  • XR features are likely to boost or underlie effectiveness
  • Some active components (eg, patient education content) can be disentangled from XR delivery
Active comparator: a different, evidence-based treatmentIn-person physical therapy, provided by studyXR intervention works better than another specific available treatment
  • Quality alternatives are available
  • Looking for equivalence (XR is “as good as”)
  • Existing treatment is variable, or there are other benefits of standardizing the comparator
Treatment as usual: continued treatment as is typical for the health condition under studyInstructions to continue current therapy with the personal health care providerXR intervention works better than typical available treatment in general
  • Alternatives are available
  • Existing treatments are relatively standardized, or variance can be reduced or accounted for
No treatment control: no intervention elements providedProviding nothingXR intervention works better than doing nothing
  • There are no alternatives available
  • Low potential for placebo effects
Waitlist control: no intervention, but with expectation of future treatmentProvide nothing but promise XR intervention laterXR intervention works better than doing nothing
  • There are no alternatives available
  • Potential for nocebo (ie, negative expectations for outcomes) if not receiving treatment
No control: treatment group not compared with another groupA single group is assessed before and after use of XR interventionOutcomes reported to be different after using XR than they were before
  • Inability to run control group
  • Generally, this method is not acceptable for establishing causal relationships

aMXR: medical extended reality

bXR: extended reality

Special Considerations for MXR Trials

Because a central question in MXR trial designs often involves determining whether and to what extent the XR delivery mechanism is an active element of the intervention, it is crucial to examine what is unique about XR in its contribution to interventions and treatments. MXR technologies combine aspects of other intervention platforms that are not typically co-occurring (eg, experiencing digital content through behaviorally driven action) and present new elements, such as the potential to embody a user in a separate digital body. Several frameworks present unique features of XR for various use cases that may be instructive [3,17-20]. At their core, these unique features are the elements of MXR that could either elicit significant placebo effects (eg, novelty and modulated input from closed-loop systems) or be an active component through which applications in XR achieve effectiveness (eg, experienced presence in a simulated environment and behaviorally driven interactivity), as depicted in Table 2.

Table 2. Selected features of extended reality (XR) and their implications for evaluating interventions.
XR featuresDescriptionImplications for MXRa interventionsExample approaches for control conditions
Immersion
  • Use of XR equipment can, to various degrees, surround the user or limit access to the physical surroundings and allow sensory access only to digital content
  • Direct influence on distraction and attention due to augmented or reduced access to the physical environment
  • Blocking of other contextual stimuli orients attention and reduces distractions
  • Reducing outside sensory input such as using an XR headset for control stimuli
  • Providing control stimuli in a darkened or bare (physical) room
Presence
  • XR engenders a sense of “being” in the virtual digital environment such that users feel they are existing within the virtual world or along with its elements
  • Presence is considered the hallmark of virtual reality
  • Makes possible bodily illusions and experience of having personally engaged in a digital simulation, can underlie motivation and psychological influence, and can engender responses similar to real-world settings
  • Narrative approaches outside XR that cause presence and mental transportation
  • Use of nontherapeutic XR applications that are similarly presence-inducing
  • Use of the XR intervention environment with therapeutic elements removed or replaced
Embodiment
  • The ability for XR to make users feel as if they are existing within a digital body other than their own (eg, an avatar), enacting behaviors in the virtual environment
  • Ability to change perception and viewpoint and have experiences that are depersonalized or repersonalized in another body or entity
  • Central to therapy related to absent body parts (eg, phantom limb syndrome) and some empathy-inducing experiences
  • Body transfer to a digital body or entity without the therapeutic element present
  • Transfer to a digital body that engages in nontherapeutic activity
  • Directed body movements or scripted experiences outside of the XR environment
Multisensory stimulation
  • XR can engage multiple senses such as visual (3D and stereoscopic), aural, olfactory, haptic, as well as senses not typically engaged by media (eg, proprioception)
  • Ability to present media interventions through more sensory channels than typical and engage more brain areas
  • Engage key sensory channels in the physical environment (eg, providing ambient scent or sound)
  • Provide sensory content (eg, music) inside of XR that differs from active content but is similar on key factors (eg, salience and valence)
Behavior-driven
  • Real-time physical behaviors in XR environments have direct implications for experience and drive action
  • Allows multiple modes of interaction with whole-body input and facilitates an ability to engage in naturalistic behavioral activity
  • Allows for fine-grained, continuous behavioral measurement
  • Engage physical movement and behavior outside of the XR environment (eg, guided movements or screen-based demonstration)
  • Behavior in XR can drive nontherapeutic elements of experience (eg, moving hands to draw a picture rather than perform an exercise)
Interactivity
  • Response of XR environments to user input including physical movement and input from hand controllers
  • Integrates users directly into the action in a naturalistic way, increases engagement, and enables personalization
  • Interactive applications on non-XR platforms
  • Interactive elements can trigger nontherapeutic elements such as trivia questions rather than patient education
Closed-loop systems
  • Behavior, performance, physiology, or other metrics drive stimuli provided by XR intervention
  • Ability to adjust intervention to user behavior or performance in real time for adaptive learning or training and provision of biofeedback
  • Provide noncontingent or faux feedback to the participant on performance
  • Provide contingent feedback on a nontherapeutic factor (eg, based on EMGb when intervention uses EEGc)
Novelty
  • Consumer adoption is growing, and familiarity is variable
  • Expected to change over time with increased adoption
  • There is often an intangible “wow” effect, especially for newer users
  • Wide-ranging attitudes toward application of the technology from skepticism, given entertainment applications, to increased expectations for efficacy and excitement
  • Possibility for disconnect between user expectation and MXR intervention capabilities
  • Integrate aspects of the novel technology without specific features (eg, sham)
  • Use other novel, emerging digital technology approaches (eg, wearables and artificial intelligence–based elements)
Form factor
  • The primary form factor is head mounted, although not exclusive
  • Can cause discomfort to the head and neck, eye strain, and heat retention, especially with extended use
  • Integrate a head-worn element with or without XR capabilities

aMXR: medical extended reality.

bEMG: electromyography.

cEEG: electroencephalography.

The role of XR elements in patient outcomes is likely to vary substantially between interventions. The goal for trial design, then, is to include elements that are likely to be confounding or elicit placebo effects while avoiding those that underlie the efficacy of active components. This last piece is critical because the closer a control condition matches the active elements of an intervention, the more difficult it may be to find an effect [21]. Arriving at a control that will ensure validity and support claims about an MXR intervention while also avoiding the suspected active components of the trial can be difficult, and trade-offs may be necessary. For example, a trial assessing whether a new XR intervention for surgical education is better than the leading computer-based approach will typically include the standard version of the computer-based approach as an active comparator and thus will not control for behaviorally driven XR interaction even though it is a purported active element of the intervention.

It is also crucial to note that XR itself, in terms of the equipment and general software approaches, is never “the intervention.” Rather it is the delivery vehicle for the content comprising the intervention, and it may or may not enable unique intervention features. Content can range widely across MXR interventions, from relaxing 360° nature videos to interactive experiences with artificial intelligence–based virtual humans and therapeutic video games, each of which requires different considerations. While this is not yet discussed explicitly in clinical research, XR design models in education specify the need to consider the interaction between XR hardware approaches and specific media content to support and optimize outcomes [20]. The content of the intervention, the chosen control condition, and the alignment between the 2 must be central considerations underlying decision-making.

Although each individual XR-relevant factor can be addressed by feature choices in a control condition, the reality is that features typically co-occur within a given intervention and must be considered in conjunction with one another. Features become important for interventions depending on their content. For example, in the case of relaxation and distraction-based pain management, features such as presence and multimodal sensory elements may be paramount active components for the intervention’s effect, whereas in a physical, gamified rehabilitation application, the behaviorally driven nature of XR and the ability to create a closed-loop system may be central. The novelty and form factor associated with the equipment could be useful to control across both interventions. Certainly, there are also many elements that are not specific to MXR that should also be considered for inclusion in controls, such as staff attention and interaction, training content, and educational content. The standardization of general intervention features such as these should not be overlooked in MXR interventions.

Sham Control Design and Testing

While there are clearly a host of potential control condition designs, the sham control approach is worth deeper examination due to its popularity and promise. The term “sham” in the XR context has been used to describe a variety of approaches, including headsets with the power off, VR-based simulations of the physical research environment, and 2D content viewed outside of a headset [22-27]. Consensus in defining sham versus control in MXR is needed. Most often, a sham VR condition refers to watching a 2D video on a virtual screen in an XR headset, similar to watching a large screen television in the physical world. The headset provides a stereoscopic view and may track head rotation or position. This sham procedure is different from a control that involves watching a 2D video on a tablet or monitor outside of a headset. Using a true XR sham approach will allow a better understanding of both active and placebo components. It can be helpful to include an intervention arm in which no headset is worn, as this can function as a no-treatment control or a specific factor component control (Table 1). Such a control assesses the influence of XR itself on participant outcomes and can aid in assessing the functioning of the sham. Indeed, a poorly designed sham condition can create more problems than it solves and can fail to solve the problems it seeks to address. As newly developed sham and control conditions become evaluated, validated, and normed, it may become possible to apply them across trials that share features. Additionally, within a trial, it is helpful to monitor patient (and provider) outcome expectations and consider whether the novelty of XR tools is affecting those expectations [28]. Assessing expectations of benefit before starting the trial, perception of benefit during the trial, and requesting that participants guess their trial arm allocation at the end of the trial may conjointly help account for and increase the understanding of expectation effects [28].

As per the definition above, sham approaches involve the viewing of 2D content that contains features similar to MXR content in the active trial arm on a virtual screen within a VR-based room where user orientation is tracked [14,29-31]. At face value, this approach provides reasonable control over many factors involved in MXR delivery, including novelty and immersion, and can account for the look and feel of the content (eg, using nature videos in 2D in parallel to nature-based 360° videos and 3D sounds). As implemented, however, there are several places where sham conditions can fall short, which require further consideration. First, the appropriate sham experience for a given intervention is rarely ready-made. It can be tempting to cobble together an experience that checks boxes for appropriateness; however, we argue that this can backfire and reduce trial quality. In the sham, the 2D video content truly takes center stage and forms the bulk of the experience for the participant in the absence of interactivity and other features. To the extent that this content is uninteresting, poorly curated, or repetitive over time, it has the capacity to result in negative emotions, including frustration and boredom, which are far less likely to be present in active XR interventions [32]. The spare, passive nature of the sham experience is at odds with several tenants of XR design practice [33,34], and extra care may be needed to ensure a positive or neutral experience with the sham over time. Educational XR research has also highlighted the importance of meeting users’ expectations of XR to encourage engagement and effectiveness [35,36]. An XR experience that is disappointing may deflate expectations [33,37] rather than maintaining them like a placebo is meant to do. Therefore, far from serving as a neutral comparison, a sham that violates participants’ expectancies for XR technology capabilities or that makes participants question those capabilities could result in negative responses not typically seen in the active arm, which may be associated with disengagement or drop out [38-41].


While the question of appropriate controls has been extensively addressed in the clinical trial literature, the use of controls in MXR-focused trials requires careful consideration of unique and relevant XR aspects in combination with other study features. Through the processes of careful design, refining the sham and control conditions, and assessing expectations of improvement and allocation beliefs, we may build a set of evidence-based best practices for moving this field forward. Better controls will enable more robust trials and the development and distribution of more universal knowledge that can be adopted across MXR interventions and their evaluation process. Ultimately, improving our MXR methodology will foster interventions that mitigate suffering, improve clinical outcomes, and optimize treatment performance.

Acknowledgments

This work was supported in part by the Intramural Research Program of the National Human Genome Research Institute and the National Center for Complementary and Integrative Health (R01 AT011347-01A1, LC).

Conflicts of Interest

None declared.

  1. Laver KE, Lange B, George S, Deutsch JE, Saposnik G, Crotty M. Virtual reality for stroke rehabilitation. Cochrane Database Syst Rev. 2017;11(11):CD008349. [FREE Full text] [CrossRef] [Medline]
  2. Lungu AJ, Swinkels W, Claesen L, Tu P, Egger J, Chen X. A review on the applications of virtual reality, augmented reality and mixed reality in surgical simulation: an extension to different kinds of surgery. Expert Rev Med Devices. 2021;18(1):47-62. [CrossRef] [Medline]
  3. Martingano AJ, Persky S. Virtual reality expands the toolkit for conducting health psychology research. Soc Personal Psychol Compass. 2021;15(7):e12606. [FREE Full text] [CrossRef] [Medline]
  4. Fodor LA, Coteţ CD, Cuijpers P, Szamoskozi Ş, David D, Cristea IA. The effectiveness of virtual reality based interventions for symptoms of anxiety and depression: a meta-analysis. Sci Rep. 2018;8(1):10323. [FREE Full text] [CrossRef] [Medline]
  5. Rutkowski S, Kiper P, Cacciante L, Cieślik B, Mazurek J, Turolla A, et al. Use of virtual reality-based training in different fields of rehabilitation: a systematic review and meta-analysis. J Rehabil Med. 2020;52(11):jrm00121. [FREE Full text] [CrossRef] [Medline]
  6. Honzel E, Murthi S, Brawn-Cinani B, Colloca G, Kier C, Varshney A, et al. Virtual reality, music, and pain: developing the premise for an interdisciplinary approach to pain management. Pain. 2019;160(9):1909-1919. [FREE Full text] [CrossRef] [Medline]
  7. Campbell NC, Murray E, Darbyshire J, Emery J, Farmer A, Griffiths F, et al. Designing and evaluating complex interventions to improve health care. BMJ. 2007;334(7591):455-459. [FREE Full text] [CrossRef] [Medline]
  8. Staley K, Buckland SA, Hayes H, Tarpey M. 'The missing links': understanding how context and mechanism influence the impact of public involvement in research. Health Expect. 2014;17(6):755-764. [FREE Full text] [CrossRef] [Medline]
  9. Miller FG, Brody H. What makes placebo-controlled trials unethical? Am J Bioeth. 2002;2(2):3-9. [CrossRef] [Medline]
  10. Freedland KE, King AC, Ambrosius WT, Mayo-Wilson E, Mohr DC, Czajkowski SM, et al. The selection of comparators for randomized controlled trials of health-related behavioral interventions: recommendations of an NIH expert panel. J Clin Epidemiol. 2019;110:74-81. [FREE Full text] [CrossRef] [Medline]
  11. Souchet AD, Lourdeaux D, Pagani A, Rebenitsch L. A narrative review of immersive virtual reality’s ergonomics and risks at the workplace: cybersickness, visual fatigue, muscular fatigue, acute stress, and mental overload. Virtual Real. 2022;27(1):19-50. [FREE Full text] [CrossRef]
  12. Evans J. The IEEE global initiative on ethics of Extended Reality (XR) report--Extended Reality (XR) ethics in medicine. IEEE. 2022:1-31. [FREE Full text] [CrossRef]
  13. Birckhead B, Khalil C, Liu X, Conovitz S, Rizzo A, Danovitch I, et al. Recommendations for methodology of virtual reality clinical trials in health care by an international working group: iterative study. JMIR Ment Health. 2019;6(1):e11973. [FREE Full text] [CrossRef] [Medline]
  14. Beams R, Brown E, Cheng WC, Joyner JS, Kim AS, Kontson K, et al. Evaluation challenges for the application of extended reality devices in medicine. J Digit Imaging. 2022;35(5):1409-1418. [FREE Full text] [CrossRef] [Medline]
  15. Persky S. Validation and evaluation - Why and how. In: Greenleaf W, Roberts L, Fine R, editors. Applied Virtual Reality in Healthcare: Case Studies and Perspectives. United States. Cool Blue Media; 2021.
  16. Gold SM, Enck P, Hasselmann H, Friede T, Hegerl U, Mohr DC, et al. Control conditions for randomised trials of behavioural interventions in psychiatry: a decision framework. Lancet Psychiatry. 2017;4(9):725-732. [CrossRef] [Medline]
  17. Rizzo AS, Kim GJ. A SWOT analysis of the field of virtual reality rehabilitation and therapy. Presence. 2005;14(2):119-146. [FREE Full text]
  18. Riva G, Wiederhold BK, Mantovani F. Neuroscience of virtual reality: from virtual exposure to embodied medicine. Cyberpsychol Behav Soc Netw. 2019;22(1):82-96. [FREE Full text] [CrossRef] [Medline]
  19. Wienrich C, Döllinger N, Hein R. Behavioral Framework of Immersive Technologies (BehaveFIT): how and why virtual reality can support behavioral change processes. Front Virtual Real. 2021;2:627194. [FREE Full text] [CrossRef]
  20. Makransky G, Petersen GB. The Cognitive Affective Model of Immersive Learning (CAMIL): a theoretical research-based model of learning in immersive virtual reality. Educ Psychol Rev. 2021;33(3):937-958. [FREE Full text] [CrossRef]
  21. Cohen SP, Vase L, Hooten WM. Chronic pain: an update on burden, best practices, and new advances. Lancet. 2021;397(10289):2082-2097. [CrossRef] [Medline]
  22. Hughes SW, Zhao H, Auvinet EJ, Strutton PH. Attenuation of capsaicin-induced ongoing pain and secondary hyperalgesia during exposure to an immersive virtual reality environment. Pain Rep. 2019;4(6):e790. [FREE Full text] [CrossRef] [Medline]
  23. Mehesz E, Karoui H, Strutton PH, Hughes SW. Exposure to an immersive virtual reality environment can modulate perceptual correlates of endogenous analgesia and central sensitization in healthy volunteers. J Pain. 2021;22(6):707-714. [FREE Full text] [CrossRef] [Medline]
  24. Boonreunya N, Nopawong E, Yongsiriwit K, Chirapongsathorn S. Virtual reality distraction during upper gastrointestinal endoscopy: a randomized controlled trial. J Gastroenterol Hepatol. 2022;37(5):855-860. [CrossRef] [Medline]
  25. Zhu B, Zhou Y, Weng Q, Luo W, He X, Jin L, et al. Effects of biophilic virtual reality on cognitive function of patients undergoing laparoscopic surgery: study protocol for a sham randomised controlled trial. BMJ Open. 2022;12(7):e052769. [FREE Full text] [CrossRef] [Medline]
  26. Austin PD, Craig A, Middleton JW, Tran Y, Costa DSJ, Wrigley PJ, et al. The short-term effects of head-mounted virtual-reality on neuropathic pain intensity in people with spinal cord injury pain: a randomised cross-over pilot study. Spinal Cord. 2021;59(7):738-746. [FREE Full text] [CrossRef] [Medline]
  27. Morris NA, Wang Y, Felix RB, Rao A, Arnold S, Khalid M, et al. Adjunctive virtual reality pain relief after traumatic injury: a proof-of-concept within-person randomized trial. Pain. 2023;164(9):2122-2129. [FREE Full text] [CrossRef] [Medline]
  28. Colloca L, Barsky AJ. Placebo and Nocebo effects. N Engl J Med. 2020;382(6):554-561. [CrossRef] [Medline]
  29. Garcia LM, Darnall BD, Krishnamurthy P, Mackey IG, Sackman J, Louis RG, et al. Self-administered behavioral skills-based at-home virtual reality therapy for chronic low back pain: protocol for a randomized controlled trial. JMIR Res Protoc. 2021;10(1):e25291. [FREE Full text] [CrossRef] [Medline]
  30. Garcia LM, Birckhead BJ, Krishnamurthy P, Sackman J, Mackey IG, Louis RG, et al. An 8-week self-administered at-home behavioral skills-based virtual reality program for chronic low back pain: double-blind, randomized, placebo-controlled trial conducted during COVID-19. J Med Internet Res. 2021;23(2):e26292. [FREE Full text] [CrossRef] [Medline]
  31. Birckhead B, Eberlein S, Alvarez G, Gale R, Dupuy T, Makaroff K, et al. Home-based virtual reality for chronic pain: protocol for an NIH-supported randomised-controlled trial. BMJ Open. 2021;11(6):e050545. [FREE Full text] [CrossRef] [Medline]
  32. Sarker B. Decoding the user experience in mobile virtual reality narratives. Vancouver, Canada. Springer International Publishing; Presented at: Virtual, Augmented and Mixed Reality: 9th International Conference, VAMR 2017, Held as Part of HCI International 2017; July 9-14, 2017, 2017;437-452; Vancouver, BC. [CrossRef]
  33. Madary M, Metzinger TK. Real virtuality: a code of ethical conduct. Recommendations for good scientific practice and the consumers of VR-technology. Front Robot AI. 2016;3:3. [FREE Full text] [CrossRef]
  34. Salcedo D, Regan J, Aebersold M, Lee D, Darr A, Davis K, et al. Frequently used conceptual frameworks and design principles for extended reality in health professions education. Med Sci Educ. 2022;32(6):1587-1595. [CrossRef] [Medline]
  35. Rupp MA, Odette KL, Kozachuk J, Michaelis JR, Smither JA, McConnell DS. Investigating learning outcomes and subjective experiences in 360-degree videos. Comput Educ. 2019;128(5):256-268. [CrossRef]
  36. Parong J, Mayer RE. Cognitive and affective processes for learning science in immersive virtual reality. Comput Assist Learn. 2020;37(1):226-241. [CrossRef]
  37. Smits M, van Goor H, Kallewaard JW, Verbeek PP, Ludden GDS. Evaluating value mediation in patients with chronic low-back pain using virtual reality: contributions for empirical research in value sensitive design. Health Technol (Berl). 2022;12(4):765-778. [FREE Full text] [CrossRef] [Medline]
  38. Igarzábal FA, Hruby H, Witowska J, Khoshnoud S, Wittmann M. What happens while waiting in virtual reality? A comparison between a virtual and a real waiting situation concerning boredom, self-regulation, and the experience of time. Technol Mind Behav. 2021;2(2):1-11. [FREE Full text] [CrossRef]
  39. Sprenger DA, Schwaninger A. Technology acceptance of four digital learning technologies (classroom response system, classroom chat, e-lectures, and mobile virtual reality) after three months’ usage. Int J Educ Technol High Educ. 2021;18(1):8. [FREE Full text] [CrossRef]
  40. Elfeky A, Treweek S, Hannes K, Bruhn H, Fraser C, Gillies K. Using qualitative methods in pilot and feasibility trials to inform recruitment and retention processes in full-scale randomised trials: a qualitative evidence synthesis. BMJ Open. 2022;12(4):e055521. [FREE Full text] [CrossRef] [Medline]
  41. Skea ZC, Newlands R, Gillies K. Exploring non-retention in clinical trials: a meta-ethnographic synthesis of studies reporting participant reasons for drop out. BMJ Open. 2019;9(6):e021959. [FREE Full text] [CrossRef] [Medline]


MXR: medical extended reality
VR: virtual reality
XR: extended reality


Edited by T Leung, SR Ramos; submitted 18.01.23; peer-reviewed by A King, D Nguyen ; comments to author 15.08.23; revised version received 21.08.23; accepted 17.09.23; published 22.11.23.

Copyright

©Susan Persky, Luana Colloca. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 22.11.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.