Bahamas National Implementation Project: Proposal for Sustainability of an Evidence-based HIV Prevention Intervention in a School Setting

Background Sustained implementation of school-based prevention programs is low. Effective strategies are needed to enhance both high-level implementation fidelity and sustainability of prevention programs. Objective This proposed study aims to determine if the provision of either biweekly monitoring and feedback and site-based assistance and mentorship or both to at-risk and moderate-performing teachers with monitoring through an enhanced decision-making platform by the Ministry of Education (MOE) and Ministry of Health (MOH) based on the real-time implementation data will increase national implementation fidelity and result in sustained implementation over time. Methods This study will target government schools including 200 grade 6 teachers in 80 primary schools and 100 junior/middle high school teachers (and their classes) on 12 Bahamian islands. Teacher and school coordinator training will be conducted by the MOE in year 1, followed by an optimization trial among teachers in the capital island. Informed by these results, an implementation intervention will be conducted to train using different levels of educational intensity all at-risk and moderate-performing teachers. Subsequently selected training and implementation strategies will be evaluated for the national implementation of Focus on Youth in the Caribbean and Caribbean Informed Parents and Children Together in years 2 to 5. Results It is hypothesized that a more intensive training and supervision program for at-risk and moderate-performing teachers will enhance their implementation fidelity to the average level of the high-performing group (85%), an HIV prevention program delivered at the national level can be implemented with fidelity in grade 6 and sustained over time (monitored annually), and student outcomes will continue to be highly correlated with implementation fidelity and be sustained over time (assessed annually through grade 9). The proposed study is funded by the National Institute of Child Health and Human Development from August 1, 2018, through May 31, 2023. Conclusions The study will explore several theory-driven implementation strategies to increase sustained teacher implementation fidelity and thereby increase the general public health impact of evidence-based interventions. The proposed project has potential to make significant contributions to advancing school-based HIV prevention research and implementation science and serve as a global model for the Fast Track strategy. International Registered Report Identifier (IRRID) PRR1-10.2196/14816

both FOYC and CImPACT consistent with that described by the developers) in grade-6; and, b) sustained over time (monitored annually); and, 3. Student outcomes (knowledge, reproductive health skills, perceptions and self-reported behaviors) will: a) continue to be highly correlated with implementation fidelity; and, b) be sustained over time (assessed annually through grade 9). We aim to determine if the provision of either or both a) biweekly monitoring, feedback and support through a "community of practice" to at-risk and moderate-performing teachers; and b) an enhanced decisionmaking platform by the MOE and MOH based on the real-time implementation data will: 1) increase national implementation fidelity; and 2) result in sustained implementation over time.

PUBLIC HEALTH RELEVANCE:
The global public health impact of the HIV/AIDS epidemic is unparalleled in the last half-century. Currently the global public health community is at a critical juncture, working towards the possibility of eradicating the global AIDS epidemic by the year 2030. However, this excitement is tempered by mounting concern that if this goal is not achieved, emerging resistant HIV strains and declining resources targeting the problem may result in a resurgence of the epidemic. The proposed research will support the efforts of the entire nation of The Commonwealth of The Bahamas and serve as a prototype for other nations in the quest for elimination of the HIV/AIDS epidemic by 2030. It employs a theory-based, practical, data-driven implementation process to deliver an evidence-based HIV prevention curriculum through the government school system to all students throughout the nation enrolled in grades-6 to 9 and their parents.

CRITIQUE 1
Significance: 3 Investigator(s): 1 Innovation: 1 Approach: 5 Environment: 2 Overall Impact: The focus of this application is to limit HIV infections by increasing adherence to an existing sexual health intervention for junior high youth, taught by teachers, in the Bahamas. The research team has a long history of collaboration with the Bahamian Ministries of Health and Education on HIV education with youth. The applicants utilize a multiphase optimization strategy (MOST) to inform their implementation approach and will utilize a full factorial design to assess the best implementation monitoring system to improve fidelity and sustainable implementation across time. The intervention has 2 components. The first is Focus on Youth Caribbean, (FOYC) which is a classroom based intervention consisting of 8 sessions occurring in the 6 th grade (with subsequent booster sessions in the 7 th and 8 th grades). The second is Caribbean Informed Parents and Children Together (CImPACT) that includes a single session with parents and youth to address communication and a condom demonstration. Previous research of FOYC + CImPACT by this group with Bahamian youth demonstrated effectiveness in attitude change and rates of condom use. A subsequent R01 focused on the national implementation of FOYC alone. Significant variation in dosage and curricula fidelity were noted. Drs. Wang and Stanton are co-PIs for this project. Dr. Wang is an associate professor and has a strong publication history in teen sexual health as well as the development and evaluation sexual education programs. Dr. Stanton, a professor and Dean, has mentored Dr. Wang for several years; they have collaborated extensively and have a clear multiple PI leadership plan. Additional strengths of the current application include using the Exploration, Preparation, Implementation, Sustainment (EPIS) model to guide the implementation process. Score driving weaknesses include minimal discussion of school leadership support, the new inclusion of CImPACT in this national implementation study, and issues regarding the site-based assistance/mentorship condition.

Strengths
 Focusing on partnering with teachers to implement the HIV risk reduction curricula is a significant need that warrants attention in the implementation research literature. Thus, there is a strong scientific premise for this project.

Weaknesses
 Minor: Evidence of sustainability is offered as pilot data. However, sustainability was described as improved student outcomes when teachers received additional FOYC training and that teachers who received training continued to teach elements of it after being placed in other schools. Thus, the pilot data related to sustainability is not relevant to the proposal.
 Moderate: School leadership organizational support is given short-shrift in the application.
School level factors (principal support) are listed in Table 1 as an issue impacted implementation and in Figure 2 in the SEM model. However, principal and school leadership support is not described further in the application nor is recruitment of principals described in the human subjects section.

Investigator(s):
Strengths  Drs. Wang and Stanton are productive investigators. They have collaborated successfully on previous projects with the same population.
 The collaborators in the research team bring additional research and clinical expertise in multiple areas.
 The extended research team also has strong publication histories in the relevant research literature specific to this project.
Weaknesses  None noted.

Innovation:
Strengths  A Type 3 hybrid design will be used on a national scale to improve fidelity to an effective HIV reduction intervention with early adolescents.

Weaknesses
 None noted.

Approach:
Strengths  The applicants have very strong pilot data that buttresses their current analysis plan. For instance, they are tested and validated a teacher fidelity scale that will identify teachers that need additional intervention to improve fidelity.
 A number of implementation fidelity measures will be included. For instance, a significant proportion of FOYC (10%) and CImPACT (20%) will be independently monitored.

Weaknesses
 Minor: A significant barrier to implementing CImPACT was teacher reimbursement for time to administer the intervention. As such, teachers will be reimbursed for conducting CImPACT with  Minor: In the previous expansion of FOYC across the nation, the Bahamian Ministry of Education conducted teacher workshops. Only 49% of teachers attended and 11% had attended a workshop in the original FOYC trial. As such, information regarding training will be placed on a flash drive for teachers. Nonetheless, it will not be possible to track utilization of the flash drive and only attendance at workshops can be measured by the applicants.
 Moderate: As a part of the site-based assistance/mentorship (SAM) condition of the full factorial design, schools will be randomized into 3 conditions (no SAM, SAM, and enhance SAM). Highperforming teachers, FOYC + CImPACT guidance counselors and retired HFLE teachers will serve as team leaders. For SAM sites, team leaders will meet weekly with 6th grade teachers and HFLE teachers to discuss progress. Team leaders for enhanced SAM will observe low performing teachers to improve fidelity. These are significant activities that are core aspects of the study. It does not appear that team leaders are incentivized for this additional work that is quite time-consuming.
 Moderate: In the previous national expansion of FOYC, the CImPACT portion of the intervention was not included as it was about a priority for the Bahamian Ministry of Education. In the current application, CImPACT is again included as a core feature of FOYC. This is a significant amendment to the previous national expansion that will warrant an additional focus for sustainability. Given that all decisions for implementation of FOYC + CImPACT will be determined by the Bahamian government agencies, the shifting focus is significant.

Environment:
Strengths  A research team has an exceptionally strong history of collaboration with Bahamian agencies, teachers and students.

Weaknesses
 Minor: Dr. Stanton is a recent founding Dean of a medical school and has several research projects that continue, including a center grant. She is devoting a relatively small portion of her time (10%) to this project considering she is a co-PI.

Budget and Period of Support:
Recommend as Requested

CRITIQUE 2
Significance: 1 Investigator(s): 1 Innovation: 2 Approach: 2 Environment: 1 Overall Impact: This new R01 application proposes a five-year study of national implementation of a school-based HIV prevention program in the Bahamas. The study will also investigate the effects of additional training with teachers who do not implement the intervention with high fidelity. The scientific premise is strong based on the argument for the need for enhanced teacher training and support. The environment is excellent, with previous buy-in from the Ministry of Health and Education in the Bahamas, and previous collaboration between the investigators and the Ministries. The investigative team is strong. Innovation is low to moderate. The approach is strong, with a weakness being the lack of attention to how to ensure buy in from low performing teachers. Overall this project could have major impact on public health practice and health outcomes in the Bahamas and provide an example of effective implementation for other countries.

Significance:
Strengths  Improving the implementation and effectiveness of HIV prevention within Bahamian schools addresses a significant problem.
 The scientific premise is strong based on the argument for the need for enhanced teacher training and support.

Investigator(s):
Strengths  The investigators appear to have the requisite skills and experience to conduct this five-year R01 project. The extensive experience and prior work on HIV prevention programs in the Bahamas sets the stage for this project.
Weaknesses  None noted.

Innovation:
Strengths  None noted.

Weaknesses
 Minor: The project is not particularly innovative.

Approach:
Strengths  The approach has many strengths. All aspects of the study are well described.
 The previous studies set the stage for the proposed work.
 The rationale for the project was very well described.
 The implementation aspect of the project is central and very well explained.

Weaknesses
 Moderate: The project focuses on improving implementation of the effective HIV risk reduction and sex education programs by identifying and targeting teachers who do not implement as well as they could/should. How will they ensure buy in and participation by low-participation teachers? The incentives for increased participation were not well described. If they cannot ensure these teachers will attempt to improve their performance, then the project will not be successful. However, I suppose that is part of the rationale of the project.

Strengths
 The environment at the home institution and in the Bahamas, is strong. The existing strong relationship with the schools is key to this project working well.
Weaknesses  None noted.

Protections for Human Subjects:
Acceptable Risks and/or Adequate Protections  There are minimal risks involved here, as the research involves improving the implementation of an existing school-based intervention.

Not Applicable (No Foreign Organizations)
Select Agents:

Recommend as Requested
Recommended budget modifications or possible overlap identified:  This is a lot packed into a 5-year project, but the investigators have had success in conducting similar large-scale work.

CRITIQUE 3
Significance: 1 Investigator(s): 1 Innovation: 3 Approach: 2 Environment: 1 1 R01 HD095765-01 9 DIRH WANG, B Overall Impact: The proposed study will examine the effect of two enhanced teacher training protocols (peer mentorship and an enhanced decision-making platform) on implementation fidelity and sustained implementation of an HIV/AIDS preventive intervention implemented nationally in the Bahamas. The scientific premise of the study is strong. Multiple rigorous trials demonstrate the effectiveness of Focus on Youth Caribbean (FOYC) and Caribbean Informed Parents and Children Together (CImPACT) on improved HIV/AIDS knowledge, condom use and self-efficacy among 6 th grade youth in the Bahamas. Youth outcomes were directly linked with the fidelity of curriculum implementation with students taught by high performing teachers demonstrating improved outcomes. Additional research showed that training in interactive teaching, completion of the FOYC training and comfort implementing the curriculum predicted higher implementation fidelity. The proposed study will build on these results by testing two teaching training protocols that directly target predictors of implementation fidelity previously identified. The study has a number of noteworthy strengths including its integration into a national implementation study, a strong partnership with the Ministries of Education and Health in the Bahamas, impressive preliminary research, and the use of MOST trial to identify an optimal package of training enhancements. Weaknesses included a moderate expansion in scope from the prior trial and the failure to include new HIV/AIDS cases as an indicator of program impact. Overall, the strengths outweigh the weaknesses and enthusiasm is moderate to high for this application.

Significance:
Strengths  The proposed study has the potential to reduce the risk of HIV/AIDS in emerging adults (< 25 years), the most vulnerable age group for new HIV/AIDS infections.
 If successful, the proposed study will identify scalable implementation strategies to support sustained implementation of evidence-based interventions.
 The proposed study has the potential to improve regional and national implementation of EBIs and subsequently their public health impact.

Weaknesses
 None noted.

Investigator(s):
Strengths  Co-PIs Wang and Stanton have published 43 papers together and have collaboratively implemented multiple studies in the Bahamas.
 The multi-PI plan clearly specifies the responsibilities of each PI.
 PI Wang has a background in implementation science, HIV/AIDS prevention and adolescent risk behaviors and has collaborated on two prior studies in the Bahamas.
 PI Stanton has over 25 years of experience in HIV prevention.
 Co-I Ghosh is a biostatistician and has experience utilizing the MOST design.
 The investigative team is highly qualified to implement the proposed study.
Weaknesses  None noted.

Innovation:
Strengths 1 R01 HD095765-01 10 DIRH WANG, B  The proposed study is the first national implementation study to integrate multiple strategies to promote implementation and sustainability.
 The integration of a MOST design is novel and permits identification of the optimal package of teacher support strategies.

Weaknesses
 Moderate: The proposed study moderately expands the prior national trial (a careful next step rather than an innovative leap).

Approach:
Strengths  An evidence-based intervention.
 The proposed study is built on an impressive series of preliminary studies.
 The Fidelity through Informed Technical Assistance and Training (FITT) implementation approach is utilized.  Multiple measures (e.g., Teachers Implementation Checklist, X) were developed and validated by the research team.
 Table 1 effectively summarizes differences between the previous national trial and the proposed study. In particular, the proposed study will develop and test fidelity maintenance strategies not considered in the original trial. Weaknesses  None noted.