Health Care Providers and the Public Reporting of Nursing Home Quality in the United States Department of Veterans Affairs: Protocol for a Mixed Methods Pilot Study

Background In June 2018, the United States Department of Veterans Affairs (VA) began the public reporting of its 134 Community Living Centers’ (CLCs) overall quality by using a 5-star rating system based on data from the national quality measures captured in CLC Compare. Given the private sector’s positive experience with report cards, this is a seminal moment for stimulating measurable quality improvements in CLCs. However, the public reporting of CLC Compare data raises substantial and immediate implications for CLCs. The report cards, for example, facilitate comparisons between CLCs and community nursing homes in which CLCs generally fare worse. This may lead to staff anxiety and potentially unintended consequences. Additionally, CLC Compare is designed to spur improvement, yet the motivating aspects of the report cards are unknown. Understanding staff attitudes and early responses is a critical first step in building the capacity for public reporting to spur quality. Objective We will adapt an existing community nursing home public reporting survey to reveal important leverage points and support CLCs’ quality improvement efforts. Our work will be grounded in a conceptual framework of strategic orientation. We have 2 aims. First, we will qualitatively examine CLC staff reactions to CLC Compare. Second, we will adapt and expand upon an extant community nursing home survey to capture a broad range of responses and then pilot the adapted survey in CLCs. Methods We will conduct interviews with staff at 3 CLCs (1 1-star CLC, 1 3-star CLC, and 1 5-star CLC) to identify staff actions taken in response to their CLCs’ public data; staff’s commitment to or difficulties with using CLC Compare; and factors that motivate staff to improve CLC quality. We will integrate these findings with our conceptual framework to adapt and expand a community nursing home survey to the current CLC environment. We will conduct cognitive interviews with staff in 1 CLC to refine survey items. We will then pilot the survey in 6 CLCs (2 1-star CLCs, 2 3-star CLCs, and 2 5-star CLCs) to assess the survey’s feasibility, acceptability, and preliminary psychometric properties. Results We will develop a brief survey for use in a future national administration to identify system-wide responses to CLC Compare; evaluate the impact of CLC Compare on veterans’ clinical outcomes and satisfaction; and develop, test, and disseminate interventions to support the meaningful use of CLC Compare for quality improvement. Conclusions The knowledge gained from this pilot study and from future work will help VA refine how CLC Compare is used, ensure that CLC staff understand and are motivated to use its quality data, and implement concrete actions to improve clinical quality. The products from this pilot study will also facilitate studies on the effects of public reporting in other critical VA clinical areas. International Registered Report Identifier (IRRID) DERR1-10.2196/23516

This is a new application for a pilot study attempting to study a critically important topic for the VA -the effect of quality report cards in CLC. The application is well written, timely and would provide valuable information to VHA.
The investigators will utilize a mixed method approach to interview staff at 3 CLCs (a 1, 3 and 5-star facilities) to better understand the responses to public release of quality data. A second aim will use this information to expand a community nursing home survey to CLCs.

Innovation.
The project appears innovative in that it will produce information that will assist the VA in better using quality measurement in CLC.

Feasibility.
No concerns regarding feasibility.

Investigator Qualifications.
No concerns regarding investigator qualifications. Excellent team.

Resubmission Applications.
Not applicable.

Protection of Human Subjects from Research Risk.
No concerns regarding protection of human subjects from research risk.

Inclusion of Women and Minorities in Research.
No concerns regarding inclusion of women and minorities in research.

9.
Budget and Period of Support.
Well described budget and period of support.

Sharing Research Data (Data Management and Access Plan (DMAP)).
No concerns regarding the plan to share data.

Overall Impression.
Excellent proposal all the way around, with strong VACO support. The topic is timely and important to VHA. The Investigative team is first rate. 3. Well done methods section.

Key Weaknesses.
1. Perhaps a bit ambitious for a pilot project.
2. The team doesn't take into consideration the potential for fluctuations in the star ratings in the sampling.
In 2018, VHA introduced CLC quality ratings with report cards that facilitate comparisons between VA CLCs and community nursing homes. The investigators plan to collaborate with the VA Office of Geriatrics and Extended Care to evaluate and expand a brief survey (CLC Compare) to be used in future national administration to evaluate the impact of the quality ratings on CLC staff and Veterans. This study is timely. It responds to VHA Long-Term Care priorities. The project has the potential to contribute knowledge to guide quality improvement in CLCs.

Approach.
The research plan is well reasoned and feasible. A conceptual frame work guides the study. The approach includes interviews with staff at 1-star, 2-star and 5-star rated CLCs to identify actions taken by staff in response to CLC public data, staff commitment to/difficulties using CLC data for quality improvement and factors that motivate staff to improve CLC quality. Knowledge will be used to modify CLC Compare for use in the VA healthcare system.

Innovation.
The project takes advantage of a time sensitive priority-recent introduction of VHA CLC quality ratings that can be compared to community nursing homes. VHA doesn't have a survey to evaluate the impact of the report cards in which VA CLC often perform worse than community nursing homes.
Understanding staff attitudes and early responses is important step in building the capacity of public reporting to improve quality. The proposed study has the potential to have a substantial impact on advancing the health and health care of Veterans.

Feasibility.
The project is feasible. Power calculations are not included. Instead, a convenience sample of 12 subjects from 3 randomly selected CLCs will be used. The investigators propose to qualitatively examine responses to and perceptions of CLC Compare among staff in leadership roles at CLCs with different quality ratings (1-star, 2-stars and 5-stars) using semi structured interviews and to use this information to modify the survey. The exclusion of front line staff is a significant limitation as their 1 I21 HX002765-01 5 HSR6 PIMENTEL, C cooperation is needed for successful QI efforts. The sample size could be expanded to include additional staff.
Recruitment will be by email. Estimates are for a 50% response rate yielding 3 surveys per CLC. The investigators have used targeted emails with links to electronic studies of CLC staff in previous studies with response rates of 39-85%. Alternative recruitment strategies are not proposed.
The management plan appears to be effective. The team will communicate through regularly scheduled emails and monthly meetings and the quantitative teams will meet weekly.
The proposed study will be guided by an advisory group of VHA operations, clinical and research leaders in long-term care quality measurement. The advisory board does not include any line staff from CLCs. Representation from additional staff should be included as one of the goals of the study to study factors that motivate staff to improve CLC quality. Her role is to guide qualitative data collection, transcription and analysis. Additional time commitment from Dr. Zickmund is recommended to insure her full participation.

Resubmission Applications.
This item is not applicable.

Protection of Human Subjects from Research Risk.
No concerns regarding protection of human subjects from research risk.

Inclusion of Women and Minorities in Research.
No concerns regarding the participation of women and minorities in research.

Budget and Period of Support.
No concerns. The PI will commit 25% FTE to this project, which should be adequate. Additional funding should be allocated for Dr. Zickmund as she is the qualitative research expert for the study.

Sharing Research Data (Data Management and Access Plan (DMAP)).
No concerns regarding the plan to share data.

Overall Impression.
In 2018, VHA introduced CLC quality ratings with report cards that facilitate comparisons between VA CLCs and community nursing homes. The investigators plan to collaborate with the VA Office of Geriatrics and Extended Care to evaluate and expand a brief survey (CLC Compare) to be used in future national administration to evaluate the impact of the quality ratings on CLC staff and Veterans. This study is timely. It responds to VHA Long-Term Care priorities. The project has the potential to contribute knowledge to guide quality improvement in CLCs.
The research plan is well reasoned and feasible. The investigators propose to qualitatively examine responses to and perceptions of CLC Compare among staff in leadership roles at CLCs with different quality ratings (1-star, 2-stars and 5-stars) using semi structured interviews and to use this information in modifying the survey. However, the exclusion of front line staff from the interviews and from the Advisory Committee is a significant limitation as their cooperation and engagement is needed for successful QI efforts. The investigators and consultants are well qualified to conduct the study.

Key Strengths.
1. The project is timely and significant. In 2018, VHA introduced community living center (CLC) quality ratings with report cards that facilitate comparisons between VA CLCs and community nursing homes. The investigators plan to collaborate with the VA Office of Geriatrics and Extended Care to evaluate and expand a brief survey (CLC Compare) to be used in future national administration to evaluate the impact of the quality ratings on CLC staff and Veterans.
2. As CLCs often receive lower ratings than community nursing homes, the project has the potential to expand knowledge to guide quality improvement in CLCs and to improve Veterans' health care.
3. The research plan is well reasoned and feasible.
4. The investigators and consultants are well-qualified to conduct the study and to complete the research during the time allocated.