A Scalable Service to Improve Health Care Quality Through Precision Audit and Feedback: Proposal for a Randomized Controlled Trial

Background Health care delivery organizations lack evidence-based strategies for using quality measurement data to improve performance. Audit and feedback (A&F), the delivery of clinical performance summaries to providers, demonstrates the potential for large effects on clinical practice but is currently implemented as a blunt one size fits most intervention. Each provider in a care setting typically receives a performance summary of identical metrics in a common format despite the growing recognition that precisionizing interventions hold significant promise in improving their impact. A precision approach to A&F prioritizes the display of information in a single metric that, for each recipient, carries the highest value for performance improvement, such as when the metric’s level drops below a peer benchmark or minimum standard for the first time, thereby revealing an actionable performance gap. Furthermore, precision A&F uses an optimal message format (including framing and visual displays) based on what is known about the recipient and the intended gist meaning being communicated to improve message interpretation while reducing the cognitive processing burden. Well-established psychological principles, frameworks, and theories form a feedback intervention knowledge base to achieve precision A&F. From an informatics perspective, precision A&F requires a knowledge-based system that enables mass customization by representing knowledge configurable at the group and individual levels. Objective This study aims to implement and evaluate a demonstration system for precision A&F in anesthesia care and to assess the effect of precision feedback emails on care quality and outcomes in a national quality improvement consortium. Methods We propose to achieve our aims by conducting 3 studies: a requirements analysis and preferences elicitation study using human-centered design and conjoint analysis methods, a software service development and implementation study, and a cluster randomized controlled trial of a precision A&F service with a concurrent process evaluation. This study will be conducted with the Multicenter Perioperative Outcomes Group, a national anesthesia quality improvement consortium with >60 member hospitals in >20 US states. This study will extend the Multicenter Perioperative Outcomes Group quality improvement infrastructure by using existing data and performance measurement processes. Results The proposal was funded in September 2021 with a 4-year timeline. Data collection for Aim 1 began in March 2022. We plan for a 24-month trial timeline, with the intervention period of the trial beginning in March 2024. Conclusions The proposed aims will collectively demonstrate a precision feedback service developed using an open-source technical infrastructure for computable knowledge management. By implementing and evaluating a demonstration system for precision feedback, we create the potential to observe the conditions under which feedback interventions are effective. International Registered Report Identifier (IRRID) PRR1-10.2196/34990

1 R01 LM013894-01 3 BLR LANDIS-LEWIS, Z precision A&F in email messages in anesthesia care. Preliminary data show that provider preferences are not uniform, suggesting that a platform for computable knowledge is necessary to support scalable precision A&F. The Knowledge Grid platform, developed at the University of Michigan, has been shown to support "precisionizing" for clinical decision support systems34-36. Based on our prior work, the proposed project will advance the creation of more general services for precision A&F, applying the service in anesthesia care as a demonstration domain.

PUBLIC HEALTH RELEVANCE:
Healthcare professionals often receive inadequate feedback about the quality of care they provide. We propose to build a software service that customizes feedback for healthcare professionals based on their requirements and preferences, and to evaluate the effect of using the service on care quality.

CRITIQUES:
The criterion scores provided below are given by individual reviewers assigned to this application and the critiques from reviewers are presented "as is", without significant modification or editing by NLM staff. These individual critiques and criterion scores reflect the opinions of these assigned reviewers, which may or may not reflect the final Impact/Priority Score or final outcome/decision of the whole committee.

CRITIQUE 1
Significance: 3 Investigator(s): 1 Innovation: 4 Approach: 4 Environment: 1 Overall Impact: This R01 proposal from an ESI and NI from a K-award recipient seeks to develop a quality improvement system with "precision" audit & feedback, using anesthesia care as a demonstration domain. Aim 1 systematically captures user preferences and requirements. Aim 2 implements and assesses a demonstration system. Aim 3 assess the impact of the system on quality and intervention engagement. Studies are conducted in collaboration with the Multicenter Perioperative Outcomes Group (MPOG) which collects data on anesthesia quality from 50 hospitals and 5,000 providers. The proposed approach constructs personalized and tailored emails each month to each provider, and it tracks improvements in quality as a primary outcome, compared against a one-size-fitsall email in the control arm. Though this study is well designed, the limitation to a single application domain limits enthusiasm.

Significance: Strengths
• Identifies and seeks to correct an important problem in medical quality improvement, and advances here could have a large impact on standardizing and improve care. • Will be put into use at a national quality improvement consortium with about 5,000 different providers. • Principled design of quality improvement emails to providers is an achievable goal that could improve quality of measurable quality indicators. Weaknesses  • In this domain, performance is already high, which may limit impact if only focused on anesthesiology.

Investigator(s): Strengths
• PI has strong training and experience in this specific area, with a PhD in bioinformatics and a successful K-award. • Excellent inclusion of physicians in application domain Weaknesses • None noted.

Innovation: Strengths
• Designs a "precision" decision support system using a knowledge base to emphasize feedback with (1) "highest value in increasing performance" and (2) customized to the healthcare provider preferences. Weaknesses • Customizing to health care provider may be counterproductive in some cases, shielding providers from information they should see but elect not to see.
• The proposal claims to be "the first comprehensive study of a novel approach to A&F" but Table   1 states that there is "Medium" precedence, without many references. So it is difficult to assess what precisely is the innovation here.

Approach: Strengths
• Theory driven and guided approach, building off open source ontologies. • Aim 1. Study design to elicit requirements and preferences among providers is well conceived and designed.
• Good consideration of research questions in Aim 3. • Preliminary data mitigates risk, as does partnership with MPOG.

Weaknesses
• With only one application domain covered generalizability to other clinical domains is unstudied and unclear. Whether successful or not, it will be unclear what factors of the domain determine applicability. Proposal would be substantially improved by inclusion of an additional domain, at least in Aim 1, to assess generalizability of approach.
• How would this study determine cases where the user preference undermine the effectiveness of the system?
• No effort to consider how customization by health care providers might be counterproductive in some cases, shielding providers from information they should see but elect not to see.
Protections for Human Subjects: Acceptable Risks and/or Adequate Protections -This falls into a grey-zone, and may not be a clinical study, because it is only provider behavior being study and reported in a non-identified way. It is not very different than an email marketing campaign, which would not typically even engage with an IRB.

Authentication of Key Biological and/or Chemical Resources: Not Applicable
Budget and Period of Support: No issues raised.

CRITIQUE 2
Significance: 2 Investigator(s): 2 Innovation: 2 Approach: 3 Environment: 2 Overall Impact: This is a well written R01 application from an early stage new investigator focused on improving the quality and safety of health care through tailored mechanisms for audit and feedback (A&F). The proposed development of a precision A&F service is guided by formal theories, models, and frameworks. The approach is solid building on preliminary work, involving a national anesthesia quality improvement consortium, and involving clearly described aims. The research team and environment are also strong.

Significance: Strengths
• There is focus on improving quality and safety of health care through improved and tailored mechanisms for audit and feedback (A&F). For example, through clinical quality dashboards. Weaknesses • None noted.

Investigator(s): Strengths
• PI is an early stage new investigator and prior recipient of an NLM K01 award who provides expertise in biomedical informatics and implementation science. • Co-Is provide complementary expertise in informatics (Flynn) and anesthesiology quality improvement (Janda and Shah).

Weaknesses
• None noted

Innovation: Strengths
• Develop precision A&F service for health care providers guided by formal theories, models, and frameworks (e.g., information value chain theory and causal pathway models) Weaknesses • None noted.

Approach: Strengths
• Preliminary studies involving development and evaluation of relevant ontologies, knowledge bases, and message tailoring systems • Involvement of a national anesthesia quality improvement consortium with >50 hospitals and >5,000 providers • Clear aims focused on assessing requirements and preferences, implementing the precision A&F service, and studying the effects on the service on engagement and care. • Sufficient description provided for methods, evaluation, potential challenges, etc.

Weaknesses
• Some additional details could further strengthen the approach (e.g., for the live usability testing and number of stakeholders involved in qualitative interviews). Overall Impact: Investigators aim to demonstrate the utility of mass customization of audit and feedback to improve care quality at large scale. Based on their prior work, the proposed project will advance the creation of more general services for precision A&F, applying the service in anesthesia care. Investigators have extensive experience demonstrated through completed research projects and publications. The proposed project is the next step towards precision A&F. The infrastructure is already in place, and there is much support for quality improvement initiatives in the setting where the project will take place. Going from one size fits most to precision A&F. The plan is methodical and well-thought out, and they have considered potential problems, alternative approaches, and indicated benchmarks for success.

Significance: Strengths
• A precision approach to A&F would prioritize display of information for the single metric that, for each recipient, carries the highest value for improving performance. Well-established psychological principles, frameworks, and theories form a knowledge base to achieve precision. • The significance of the proposed work derives from the widespread, almost universal--but largely ineffective--current use of audit and feedback (A&F) as a strategy to improve the quality and safety of health care. Improving the usability of A&F requires recognition of multiple dimensions of "fit" for A&F, including the formatting of the message, the success of which depends on characteristics of the message recipient, their context, and the visual representation itself. Efforts to overcome this diversity has motivated precision interventions, which hold significant promise to improve their impact. • Instead of prioritizing negative feedback, which can demotivate care providers, the proposed A&F will focus on positive messages to recognize achievement and progress. • Investigators will incorporate UX techniques, adaptive conjoint analysis to develop group and individual level requirements and preferences. • Investigators propose to incorporate body of knowledge in the clinical decision support (CDS) framework (CDS Five Rights) into A&F. This has been difficult because of a lack of standardized terms in A&F that may have equivalent constructs in the CDS Five Rights. Investigators have already developed an ontology of visualized performance information in A&F reports that can be applied.

• Application of behavior change theories to feedback interventions Weaknesses
• Provider performance is high, which could blunt the impact of precision feedback on performance.

Investigator(s): Strengths
• PI and other key personnel have significant experience in the methods of knowledge representation, user-centered design focusing on audit and feedback to for healthcare quality improvement. They are affiliated with the Department of Learning Health Sciences and the Multicenter Perioperative Outcomes Group (MPOG) where they will be conducting their research. Landis-Lewis has developed a method for using feedback theory to specify the information content and form of feedback interventions. This work has yielded collaborations with feedback researchers in research network called the Audit and Feedback Metalab. • Flynn is an information scientist and health informatics researcher who is the research lead for the Knowledge Grid program and its technical software development team. He also contributes to work focusing on defining and developing metadata for describing computable biomedical 1 R01 LM013894-01 8 BLR LANDIS-LEWIS, Z knowledge (CBK) artifacts, including those needed for knowledge-based precision audit and feedback system and has 10 years prior experience working with EHR and clinical decision support systems. He also has completed related research in knowledge object and API development, as well as employing the USPSTF's Recommendations in computable form to achieve individualized precision prevention through scalable infrastructure, the Knowledge Grid. Janda is a physician researcher in cardiac anesthesiology and is affiliated with the MPOG and works on quality improvement issues for anesthesiology. She has experience with large-scale multicenter pragmatic clinical trials in anesthesia techniques. Shah has a background in clinical anesthesiology and experience building and implementing healthcare information technology to improve the workflow for clinicians. Shah has a unique background in EHR implementation, healthcare focused software development, and perioperative quality improvement. For the last 4 years, he has served as Program Director for the Anesthesiology Performance Improvement and Reporting Exchange, a quality improvement collaborative of 26 hospitals that leads efforts to improve anesthesia care in the state of Michigan. Shah is also the quality improvement director of the Multicenter Perioperative Outcomes Group (MPOG). Preliminary studies have already been conducted in knowledge representation, user-centered design, software development, and preference elicitation. These studies were conducted as part of 1) an NLM K01 award to develop a knowledge-based message tailoring system, 2) an A&F-based VA QUERI grant, and 3) a small grant from the University of Michigan Office of Research. Principal and Co-Investigators have significant publications that have contributed to the science relevant to this proposed project. Weaknesses • None noted by the reviewer

Innovation: Strengths
• The most significant innovation is the ability to provide precision audit and feedback at scale, going from one size fits most to messages customized for each clinician. According to the investigators: • This is the first comprehensive study of a novel approach to A&F for health care providers, in which each provider's individual performance data is analyzed across metrics to prioritize information that is most actionable and motivating.
• This study will employ for the first time an integrated representation of recipient requirements and preferences with theoretical constructs that direct the production of precision audit and feedback messages.
• The project proposes a novel approach, employing 3 customization strategies simultaneously, based on knowledge availability: a) theory-based customization using the characteristics of an individual's performance data, b) group-level segmentation and targeting based on usercentered design and preference data cluster analyses, and c) full tailoring using individual-level requirements and preferences obtained through user configuration of requirements and participation in a conjoint analysis survey.
• A scalable knowledge infrastructure for a precision A&F service: The proposed aims will collectively demonstrate a precision feedback service developed using an open-source technical computable biomedical knowledge management infrastructure. This infrastructure enables large-scale dissemination and management of knowledge that supports mass customization of audit and feedback. By implementing systems for deploying computable knowledge at large scale, the project team aims to create the potential to observe system-level learning about the conditions under which feedback interventions are effective at the individual, group, and universal level. Weaknesses • None noted by the reviewer 1 R01 LM013894-01 9 BLR LANDIS-LEWIS, Z

Approach: Strengths
• Aim 1: Systematically capture recipient requirements and preferences for precision A&F messages • Aim 2: Software service development and implementation study. Investigators will add an individualized message to the existing "one size fits most" A&F email sent monthly to 5,000+ providers and will evaluate the precision A&F service's performance using existing quality measurement data from 50+ hospitals and conduct usability testing with a diverse sample of providers and hospitals.
• Aim 3: Investigators will conduct an embedded, pragmatic cluster-randomized trial of precision A&F-enhanced email vs a standard "one size fits most" A&F email to anesthesia providers.
• Project extends previous research investigator completed in A&F messaging. • Study is theory based. The investigators addressed potential issues with bias and offered ways to prevent them.
• Primary and secondary outcomes are reasonable and measurable for Aim 3 trial and mixed methods process evaluation.
• Much preliminary work was done in prior research project.
• Although the first year is for developing requirements and preferences, implementing the service can begin the first year.
• The study will extend the MPOG infrastructure to use existing data and performance measurement processes Weaknesses • None noted by the reviewer

Environment: Strengths
• The University of Michigan has established academic programs in two areas relevant to this application: health informatics and learning health system infrastructure. These academic programs are complemented and supported by a comprehensive library system that leads in developing the digital libraries of the future. Besides, advanced computational and networking resources are available at Michigan, including all of the IT design, development, implementation, and evaluation resources needed for this proposal. Michigan Medicine uses an advanced Electronic Health Record (EHR) system in all of its hospitals and clinics. It has an outstanding record of developing and implementing Clinical Decision Support (CDS) solutions in many biomedical domains.
• Multicenter Perioperative Outcomes Group (MPOG). Based at the University of Michigan, MPOG maintains a quality improvement infrastructure that represents a large-scale platform for research in precision feedback. Currently monthly A&F emails reach approximately 5,000 anesthesia providers.