Medical Extended Reality Trials: Building Robust Comparators, Controls, and Sham

Introduction

Overview

Medical extended reality (MXR), the application of extended reality (XR) technology such as virtual reality (VR) and augmented reality to health care and medicine, spans a variety of domains, including patient treatment, procedure support, health education, and provider training. MXR is an area of long-anticipated promise that is finally beginning to materialize and make its way into clinical implementation [1-6]. Accordingly, the pace of development and research in MXR has accelerated dramatically in the past few years, resulting in a flood of new products and scientific papers. By comparison, there is yet a sparsity of rigorous clinical trials that establish the efficacy and effectiveness of these interventions. There is also a major need for research that establishes mechanisms of action across clinical areas and application types. This work will allow for better targeting and tailoring of future intervention designs [7,8]. A primary impediment to these goals is the lack of frameworks for trial designs and the selection of appropriate controls that effectively address the unique elements of MXR. This paper posits the challenges inherent in clinical trial designs for MXR and proposes approaches for choosing controls, sham conditions, and comparators that account for the technology’s complexity and multifaceted nature.

General Considerations

The clinical trial literature discusses comparators, an umbrella term describing another treatment, product, or experience to which the focal intervention is compared as a frame of reference to determine its efficacy, safety, or other benchmarks. Comparators as a class encompass the more specific concept of a control condition. A control condition provides a comparison that includes no active components of the intervention (ie, elements that underlie effectiveness) and, as such, allows researchers to determine that the treatment or intervention itself causally influenced patient outcomes. A control condition, therefore, includes elements present in the intervention that differ from the proposed primary active components (eg, intervention delivery vehicle and interactions with the study team), allowing researchers to rule out alternative explanations for patient outcomes. This may take the form of a sham (where study activities are performed without the performance of the active component activities) or a placebo (where an inactive component is substituted for an active one). In the context of MXR trials, there is little practical difference between sham and placebo control, and we use them interchangeably here.

The control condition essentially defines the claims that can be made about the efficacy or effectiveness of an MXR intervention in the case of a positive finding (ie, finding that the intervention is more effective than the control). As such, this is the primary consideration underlying control condition design or comparator choice. Due to variation in trial goals and variation in the nature of individual MXR interventions, there is no universal, optimal control for MXR interventions as a class. Control suitability depends on features of the intervention, the setting, and the target population and, ultimately, on potential sources of patient and health care provider expectancies (ie, beliefs about whether and how well a treatment will work).

There are many other factors that influence choice of comparator or control, and these should be considered in light of the trial’s primary objective. For example, different considerations are typically applied when designing control conditions for proof-of-concept, feasibility, and acceptability studies. There are also situations in which the profile of the intervention, the health condition under study, or the patient population dictates what is possible and ethically acceptable [9,10]. For example, it may be unethical to assign patients in ill-health to a no-treatment control when treatments are otherwise available. In addition to many potential benefits, there are several potential risks associated with MXR use that must be considered in the process of trial design. These include cybersickness (similar to motion sickness, stemming from XR use); strain or discomfort to the head, neck, or face; and distraction or disorientation [11]. Some approaches to MXR provision could also introduce risks to patient privacy if adequate data collection and protection measures are lacking or not possible in a given situation [12]. The risks present in the specific study and intervention context must be weighed against potential benefits for the ethical design of MXR trials.

Given competing demands, choosing the comparator or designing the control condition can be one of the most difficult, yet vitally important, parts of designing an MXR study, as it dictates what can be concluded from the study in terms of outcomes such as efficacy, effectiveness, acceptability, safety, and mechanisms of action.

Existing Guidance

Given the unique profile of MXR, the research community has recognized the need for specialized study design. Birckhead et al [13] made recommendations for clinical trial design, covering multiple design questions, including choice of control group. The authors discuss a range of forms that a control condition can take inside or outside of a headset, including both passive and interactive features. Ultimately, they recommend a study design that considers purported MXR mechanisms and targets of action. Beams et al [14] similarly address a variety of challenges associated with the evaluation of MXR applications from a regulatory perspective. Among other possibilities, these authors describe the promise of sham control interventions that involve the provision of an XR headset that administers 2D visualizations. Such sham groups (discussed in detail below) have become popular in VR-based clinical trials, including the high-profile trials associated with early US Food and Drug Administration–cleared XR applications [14]. More broadly, scientists and regulators have noted that the subject of clinical trial design, and control conditions for MXR trials, needs more exploration and explication in the scientific literature [6,13-15]. Such consideration would greatly strengthen the quality of trials evaluating MXR treatments and interventions. Herein we focus on frameworks and control conditions for MXR generally, although many details and examples are based on VR given its current dominance of the MXR literature.

Frameworks for MXR Trials

Although there are many variations in frameworks, the broad classes of randomized controlled trial comparators can be boiled down to those that are created and administered by the research team (eg, a control condition developed to parallel the intervention) and those that are not created by the research team (eg, treatment as usual, wherein participants are asked to continue their existing treatment approach). Gold et al [16] further break these areas down into several specific examples in the domain of traditional clinical trials. These include placebo control, specific factor component control, active comparator, treatment as usual, no treatment control, and waitlist control. We expand on these examples in Table 1 to consider how they can be applied to trials evaluating MXR interventions.

Special Considerations for MXR Trials

Because a central question in MXR trial designs often involves determining whether and to what extent the XR delivery mechanism is an active element of the intervention, it is crucial to examine what is unique about XR in its contribution to interventions and treatments. MXR technologies combine aspects of other intervention platforms that are not typically co-occurring (eg, experiencing digital content through behaviorally driven action) and present new elements, such as the potential to embody a user in a separate digital body. Several frameworks present unique features of XR for various use cases that may be instructive [3,17-20]. At their core, these unique features are the elements of MXR that could either elicit significant placebo effects (eg, novelty and modulated input from closed-loop systems) or be an active component through which applications in XR achieve effectiveness (eg, experienced presence in a simulated environment and behaviorally driven interactivity), as depicted in Table 2.

The role of XR elements in patient outcomes is likely to vary substantially between interventions. The goal for trial design, then, is to include elements that are likely to be confounding or elicit placebo effects while avoiding those that underlie the efficacy of active components. This last piece is critical because the closer a control condition matches the active elements of an intervention, the more difficult it may be to find an effect [21]. Arriving at a control that will ensure validity and support claims about an MXR intervention while also avoiding the suspected active components of the trial can be difficult, and trade-offs may be necessary. For example, a trial assessing whether a new XR intervention for surgical education is better than the leading computer-based approach will typically include the standard version of the computer-based approach as an active comparator and thus will not control for behaviorally driven XR interaction even though it is a purported active element of the intervention.

It is also crucial to note that XR itself, in terms of the equipment and general software approaches, is never “the intervention.” Rather it is the delivery vehicle for the content comprising the intervention, and it may or may not enable unique intervention features. Content can range widely across MXR interventions, from relaxing 360° nature videos to interactive experiences with artificial intelligence–based virtual humans and therapeutic video games, each of which requires different considerations. While this is not yet discussed explicitly in clinical research, XR design models in education specify the need to consider the interaction between XR hardware approaches and specific media content to support and optimize outcomes [20]. The content of the intervention, the chosen control condition, and the alignment between the 2 must be central considerations underlying decision-making.

Although each individual XR-relevant factor can be addressed by feature choices in a control condition, the reality is that features typically co-occur within a given intervention and must be considered in conjunction with one another. Features become important for interventions depending on their content. For example, in the case of relaxation and distraction-based pain management, features such as presence and multimodal sensory elements may be paramount active components for the intervention’s effect, whereas in a physical, gamified rehabilitation application, the behaviorally driven nature of XR and the ability to create a closed-loop system may be central. The novelty and form factor associated with the equipment could be useful to control across both interventions. Certainly, there are also many elements that are not specific to MXR that should also be considered for inclusion in controls, such as staff attention and interaction, training content, and educational content. The standardization of general intervention features such as these should not be overlooked in MXR interventions.

Sham Control Design and Testing

While there are clearly a host of potential control condition designs, the sham control approach is worth deeper examination due to its popularity and promise. The term “sham” in the XR context has been used to describe a variety of approaches, including headsets with the power off, VR-based simulations of the physical research environment, and 2D content viewed outside of a headset [22-27]. Consensus in defining sham versus control in MXR is needed. Most often, a sham VR condition refers to watching a 2D video on a virtual screen in an XR headset, similar to watching a large screen television in the physical world. The headset provides a stereoscopic view and may track head rotation or position. This sham procedure is different from a control that involves watching a 2D video on a tablet or monitor outside of a headset. Using a true XR sham approach will allow a better understanding of both active and placebo components. It can be helpful to include an intervention arm in which no headset is worn, as this can function as a no-treatment control or a specific factor component control (Table 1). Such a control assesses the influence of XR itself on participant outcomes and can aid in assessing the functioning of the sham. Indeed, a poorly designed sham condition can create more problems than it solves and can fail to solve the problems it seeks to address. As newly developed sham and control conditions become evaluated, validated, and normed, it may become possible to apply them across trials that share features. Additionally, within a trial, it is helpful to monitor patient (and provider) outcome expectations and consider whether the novelty of XR tools is affecting those expectations [28]. Assessing expectations of benefit before starting the trial, perception of benefit during the trial, and requesting that participants guess their trial arm allocation at the end of the trial may conjointly help account for and increase the understanding of expectation effects [28].

As per the definition above, sham approaches involve the viewing of 2D content that contains features similar to MXR content in the active trial arm on a virtual screen within a VR-based room where user orientation is tracked [14,29-31]. At face value, this approach provides reasonable control over many factors involved in MXR delivery, including novelty and immersion, and can account for the look and feel of the content (eg, using nature videos in 2D in parallel to nature-based 360° videos and 3D sounds). As implemented, however, there are several places where sham conditions can fall short, which require further consideration. First, the appropriate sham experience for a given intervention is rarely ready-made. It can be tempting to cobble together an experience that checks boxes for appropriateness; however, we argue that this can backfire and reduce trial quality. In the sham, the 2D video content truly takes center stage and forms the bulk of the experience for the participant in the absence of interactivity and other features. To the extent that this content is uninteresting, poorly curated, or repetitive over time, it has the capacity to result in negative emotions, including frustration and boredom, which are far less likely to be present in active XR interventions [32]. The spare, passive nature of the sham experience is at odds with several tenants of XR design practice [33,34], and extra care may be needed to ensure a positive or neutral experience with the sham over time. Educational XR research has also highlighted the importance of meeting users’ expectations of XR to encourage engagement and effectiveness [35,36]. An XR experience that is disappointing may deflate expectations [33,37] rather than maintaining them like a placebo is meant to do. Therefore, far from serving as a neutral comparison, a sham that violates participants’ expectancies for XR technology capabilities or that makes participants question those capabilities could result in negative responses not typically seen in the active arm, which may be associated with disengagement or drop out [38-41].

Conclusion

While the question of appropriate controls has been extensively addressed in the clinical trial literature, the use of controls in MXR-focused trials requires careful consideration of unique and relevant XR aspects in combination with other study features. Through the processes of careful design, refining the sham and control conditions, and assessing expectations of improvement and allocation beliefs, we may build a set of evidence-based best practices for moving this field forward. Better controls will enable more robust trials and the development and distribution of more universal knowledge that can be adopted across MXR interventions and their evaluation process. Ultimately, improving our MXR methodology will foster interventions that mitigate suffering, improve clinical outcomes, and optimize treatment performance.