Original Paper
Abstract
Background: The Extension for Community Healthcare Outcomes (ECHO) model was adapted for hypertension education of community health extension workers in the Federal Capital Territory, Nigeria, and delivered as a 7-part series.
Objective: This study aims to evaluate implementation outcomes of the hypertension ECHO series mapped to the first 3 levels of the Kirkpatrick model. Outcomes included reach, appropriateness (level 1), effectiveness (level 2), and penetration (level 3).
Methods: From August 2022 to April 2023, 7 hypertension ECHO sessions were delivered via Zoom (Zoom Video Communications, Inc) to a health care worker audience including targeted community health extension workers at 12 primary health centers (PHCs) in the Hypertension Treatment in Nigeria Program. Health care workers provided demographic information, engaged in pre- and postsession knowledge quizzes, and shared feedback during live sessions. Surveys were sent to health care workers at 12 PHCs approximately 1 month after each session to ask about the use of the presented material and focus group discussions were performed with these health care workers after the ECHO program concluded. Qualitative and quantitative results were evaluated using an explanatory sequential mixed methods design wherein qualitative data were used to help explain outcomes and variability among participants.
Results: Across 7 ECHO sessions, a total of 1407 live participants were documented. Participants largely found the program was acceptable, with more than 97% of respondents reporting that the session was useful. Postsession knowledge scores increased (range: 2.3%-10.5%) relative to presession scores demonstrating moderate effectiveness. Among 12 PHCs, most (more than 70%) health care workers applied information learned in each session to provide patient care. In 6 focus group discussions, with 31 health care workers (n=15; 48% community health extension workers), participants reported that network connectivity and clinical demands were barriers to live participation and expressed preferences for blended training and asynchronous resources.
Conclusions: Results show that a hypertension ECHO program adapted for community health extension workers effectively increased knowledge among participants and was useful to a majority. Insights gained may inform the scaling of remote hypertension education programs for community health extension workers in similar settings.
Trial Registration: ClinicalTrials.gov NCT04158154; https://clinicaltrials.gov/ct2/show/NCT04158154
doi:10.2196/66351
Keywords
Introduction
In Nigeria, where hypertension prevalence ranges from 29%-38%, health care workers face challenges in effectively managing this condition due to limited awareness, outdated practices, and inadequate training [,]. Building health care worker capacity in hypertension treatment is crucial for better care, enhanced patient outcomes, and reduced cardiovascular disease burden [,]. The University of New Mexico’s Extension for Community Healthcare Outcomes (ECHO) Project is an intervention aimed at empowering health care workers. Using telehealth technology, ECHO programs connect specialist medical providers with frontline health care workers, particularly in underserved and remote areas [,]. ECHO sessions incorporate case-based learning and bidirectional knowledge sharing with a community of practice model to facilitate longitudinal networking. The ECHO model has been used to enhance health care workers’ capacity across various areas, including hypertension [], diabetes [], pain management [-], dementia [-], mental health [,], women’s health [], and HIV or AIDs []. However, there is limited implementation and evaluation of the ECHO model among community health extension workers compared to the expanding scope of their work [-].
Various evaluation methodologies, including pre-post studies, control group comparisons, surveys, focus group discussions (FGDs), interviews, or a combination of these, have been used to assess the impact of ECHO programs []. Pre-post designs mainly gauge provider self-efficacy, confidence, and knowledge improvement before and after sessions [-,,]. Qualitative assessments, through postintervention surveys and interviews, evaluate satisfaction, overall experience, program impact, and perceived benefits [,-,,]. FGDs in some studies assessed self-efficacy, knowledge acquisition, perceived advantages, limitations, barriers, and facilitators [,,]. While a recent scoping review found moderate use of mixed methods to evaluate ECHO programs, we are aware of only 2 studies that have used mixed methods to evaluate ECHO programs implemented for community health extension workers [,]. Robust evaluation across multiple domains through mixed methods may allow a more comprehensive understanding of outcomes and provide further contextualization for others who may be interested in implementing similar programs.
In this study, the Hypertension Treatment in Nigeria (HTN) Program team partnered with mDoc Healthcare, a digital health social enterprise specializing in ECHO, to implement and evaluate a hypertension ECHO program for community health extension workers in the Federal Capital Territory, Nigeria. In this report, we evaluated the implementation outcomes of the ECHO program mapped to the Kirkpatrick model, a well-known framework for the evaluation of learning programs []. We evaluated selected implementation outcomes through an explanatory sequential mixed methods evaluation which allowed comprehensive evaluation and identification of specific contextual factors associated with our selected implementation outcomes [].
Methods
Study Setting, Design, and Sampling
Development of the hypertension ECHO program for community health extension workers in the Federal Capital Territory followed the ADAPT-ITT framework and has been previously described [,]. In our formative work, we found that the ECHO model was both feasible within the proposed setting and perceived as appropriate by health care workers for ongoing hypertension education. In this study, we report on the implementation of the ECHO program including the evaluation of implementation outcomes mapped to the Kirkpatrick model [,]. Selected implementation outcomes (Kirkpatrick level) included reach, appropriateness (level 1), effectiveness (level 2), and penetration (level 3; 
Table 1) []. The fourth level of the Kirkpatrick model, results, has been reported on separately through the evaluation of patient and clinic-level outcomes [].
| Implementation outcome | Reach | Appropriateness | Effectiveness | Penetration | ||||
| Kirkpatrick level | N/Ab | Level 1: reaction | Level 2: learning | Level 3: behavior | ||||
| Quantitative evaluation | ||||||||
| Methods |
| Reaction survey | Pre- or postsession knowledge quizzes | Behavior survey | ||||
| Sample |
| Participants from each live ECHO session | Participants from each live ECHO session | All ECHO participants at 12 HTNc Program PHCsd | ||||
| Outcomes |
| Appropriateness of the training in terms of timing, content, and structure. | Change in hypertension knowledge among hypertension ECHO program participants. | Use of the training content in clinical practice. | ||||
| Qualitative evaluation | ||||||||
| Methods |
| Focus group discussion | Focus group discussion |
| ||||
| Sample |
| ECHO participants at 12 HTN Program PHCs | ECHO participants at 12 HTN Program PHCs | ECHO participants at 12 HTN Program PHCs | ||||
| Outcomes |
| The perception among health care workers among PHCs in the Federal Capital Territory, Nigeria that the hypertension ECHO program is an appropriate intervention for hypertension education. | Effectiveness of the ECHO program as platform for hypertension training among health care workers at selected PHCs. | Integration of the hypertension ECHO program among health care workers at selected PHCs into clinical practice. | ||||
aECHO: Extension for Community Healthcare Outcomes.
bN/A: not applicable.
cHTN: Hypertension Treatment in Nigeria.
dPHC: Primary health care center.
Design
This study used an explanatory sequential mixed methods design to collect, analyze, and interpret quantitative and qualitative data from health care workers who participated in the hypertension ECHO series (
Figure 1). The mixed methods approach was selected to broadly survey and evaluate implementation outcomes among all ECHO session participants and to confirm and expand upon the results among our target audience of community health extension workers in the Federal Capital Territory, Nigeria [].
ECHO Participants
In central Nigeria, the Federal Capital Territory comprises 6 local government area (LGA) councils across which there are over 240 primary health care centers (PHCs). Sixty of these PHCs are part of the HTN Program (NCT04158154) []. Two PHCs from each LGA, selected as previously described, were invited to participate in the hypertension ECHO program.[] All health care workers at these 12 PHCs were empaneled in a WhatsApp group and invited to attend each ECHO session. Sessions were advertised broadly through multiple channels including email and social media (Instagram, Facebook, and Twitter), and were open to all registered participants.
FGDs With Health Care Workers
Six FGDs, 1 per LGA, were conducted with health care workers from the 12 targeted PHCs. The University of Abuja Teaching Hospital (UATH) research team recruited health care workers by phone including explaining the purpose and procedures of the FGD. Health care workers who participated were all adults (aged 18 years or older), able and willing to provide informed consent, and proficient in English. We prospectively developed an FGD guide to evaluate health care workers’ reactions and experiences with the hypertension ECHO series (). FGDs were conducted in English by team members trained and experienced in qualitative research (IAO and NRR). The interviewers (1 male and 1 female) were both academic researchers and primary care physicians and worked previously with the FGD participants. No translation was needed as all health care workers were comfortable speaking English. FGDs were centrally completed at UATH and each FGD comprised up to 60 minutes.
Hypertension ECHO Program
The ECHO sessions were held approximately monthly between August 2022 and April 2023, excluding December 2022 and January 2023 due to holidays and shortened clinical schedules (Figure 1 and ). Every ECHO session comprised a presession knowledge quiz, introduction, didactic presentation by domain experts, a case presentation by health care workers at an HTN Program site, a postsession knowledge quiz, and a reaction survey. The mDoc Healthcare team worked with a selected HTN Program Site before each session to select and present a case for each ECHO session. Prospective participants registered in advance for each session using an web-based form that included demographic characteristics. The registrant was also asked if multiple persons planned to share the device and the total number of individuals who planned to do so. A unique link for the live Zoom (Zoom Vidro Communications, Inc) session was sent to each registrant. Reminders were sent by email to registrants 3 days, 2 days, 1 day, and approximately 3 hours before each session. Reminders were also sent on the day of the session via WhatsApp channels.
Each unique login to a Zoom session was counted as a single live device and registration data were used to account for device sharing. Waiting rooms were used to manage entry and sessions were monitored to minimize disruptions. Sessions were also live-streamed on Facebook and recordings were uploaded for public access on YouTube. Slides and recordings were sent by email to all registrants following each ECHO session.
Outcomes
Implementation outcomes were evaluated through FGDs and survey responses, including the following.
- Reach: Registration surveys and in-Zoom tracking were used to quantify the number of ECHO program participants, the number of sessions attended among survey respondents, and the characteristics of participants including demographics, education, and experience. FGD participants provided further qualitative feedback on reasons for variation in participation in the ECHO program.
- Appropriateness (Kirkpatrick level 1): Surveys to quantitatively evaluate the appropriateness of the ECHO program in terms of timing, content, and structure were developed by the study team (). Reaction surveys were administered following each ECHO session through Zoom to all live participants. FGD participants provided further qualitative feedback on the appropriateness of the ECHO program for hypertension education.
- Effectiveness (Kirkpatrick level 2): Knowledge quizzes were uniquely developed for each ECHO session by the didactic lecturers (). Each knowledge quiz was administered twice through Zoom to all participants at the beginning and end of the ECHO session. The Zoom platform allowed tracking of pre- and postsession knowledge quizzes to individual devices, which allowed us to evaluate the quantitative change in hypertension knowledge among ECHO program participants. FGD participants provided further qualitative feedback on the effectiveness of the ECHO program for hypertension education.
- Penetration (Kirkpatrick level 3): Behavior surveys were developed by the study team and included questions about the application of learning within clinical practice (). Behavior surveys were sent approximately 30 days following each ECHO session as a REDCap (Research Electronic Data Capture; Vanderbilt University) link through a WhatsApp message to health care workers at the 12 targeted PHCs [,]. FGD participants provided further qualitative feedback on the use of the training content in clinical practice and integration of the hypertension ECHO program.
Data Management and Analysis
Survey data for reaction and learning were maintained through an enterprise Zoom account. Survey data for behavior were collected through REDCap. Quantitative data were summarized as descriptive statistics. Chi-square and Kruskal-Wallis tests were used to evaluate differences across ECHO sessions. Differences in pre-and postsession knowledge questions among respondents who participated in both sessions were compared using paired t tests (2-tailed) or Wilcoxon sign rank tests depending on the parametricity of the data. Statistical results were prepared using SAS (version 9.4; SAS Institute) and R (version 4.0; R Foundation for Statistical Computing). FGDs were audiotaped, transcribed verbatim, and read several times. A codebook was subsequently developed for deductive coding based on the interview guide and focused on responses related to selected implementation outcomes of reach, appropriateness, effectiveness, and penetration. The codebook was developed by a single team member (ASB) and reviewed with the full team. FGDs were analyzed thematically by a single reviewer (ASB) using Dedoose software (version 8; Socio-Cultural Research Consultants). Data integration comprised matching qualitative and quantitative results by implementation outcome through narrative connection []. The deidentified nature of survey and quiz responses did not allow for the direct merging of these data. Integration of the results focused on increasing the depth of understanding of the quantitative data and exploring variability among participants.
Ethical Considerations
The reporting of this study adheres to the Consolidated Criteria for Reporting Qualitative Research (COREQ) guidelines () []. The protocol for surveys and FGDs with ECHO participants was reviewed and approved by the ethics committee at UATH (UATH/HREC/PR/2021/011/015) and determined to be exempt by the Northwestern University institutional review board (STU00216041).
Informed consent was not required or collected for individuals participating in the ECHO program. Registration information collected before the ECHO sessions was inclusive of individual identifiers which were removed prior to analysis. All survey data collected during the Zoom sessions were deidentified prior to analysis. ECHO session participants were not compensated. All FGD participants provided written informed consent to participate and for audio recording. Data captured in REDCap and in FGDs were unlinked to individual identifiers. Each health care worker who participated in an FGD received a 
2000 honorarium (At the time, the was about US $4.5 per person) plus transportation costs.
Results
FGDs With Health Care Workers
Across 6 FGDs, 31 health care workers participated (range: 4-7 health care workers per FGD) and no prospective participants declined (). The median (range) age was 40 (26-57) years and over half (n=19; 63%) were female. Nearly half the participants (n=15; 48%) were community health extension workers and all were affiliated with the HTN Program and reported providing hypertension patient care.
Reach
Registration was required for each session. Across all 7 sessions, 1634 registrations were submitted of which 944 (58%) registrants participated live. Accounting for device-sharing, 1407 live participants were documented. Concurrent posts on Facebook or Instagram received 5231 impressions and reached 6341 people over the 7-part series. The same posts on LinkedIn received 2966 impressions.
The smallest reach was observed in session 7 (n=160 registrants; n=100 live devices; n=139 live participants; 
Figure 2). The greatest reach for registration, number of live devices, and number of live participants were observed in session 1 (n=278 registrants), session 2 (n=159 devices), and session 3 (n=252 participants) respectively. There were no significant trends observed over time for any indices of reach (all P values≥.05). Nearly two-thirds of registered participants were female (n=594, 63%) and most were between the ages of 30 and 49 years (n=521, 55%; Table 2). Physicians were the most frequent participants (n=284, 30%) followed by nurses (n=270, 29%). Registrant occupation did not differ (P=.35) across sessions.
In FGDs, the health care workers discussed both barriers and facilitators to participation, which limited reach. Barriers included a lack of electrical power, an unstable network, and competing clinical demands.
Sometimes when the sessions would be fixed, my battery would be low at that time. I don't have a power bank to recharge my battery and start attending too.
[FGD1, respondent 2]
The network poses a serious challenge in my area. Sometimes when I attempt to connect, I will just be seeing the video without any voice or sometimes, it will not connect at all. But the three times I was able to, we had a good network on those days.
[FGD3, respondent 6]
When you are on the program, a patient will come in and you won't be able to continue because you have to attend to the patient.
[FGD1, respondent 1]
Facilitators included frequent reminders through numerous channels—though not all health care workers received these. Health care workers emphasized that they preferred to receive reminders through phone-based means (ie, WhatsApp) as opposed to email. Health care workers also emphasized their internal drive for knowledge as facilitators of attendance.
When it is tomorrow, we would be reminded through our emails. When it is 30 minutes to the commencement, you would be sent a text message that there's 30 minutes remaining, and when it is 15 minutes. So, that really helps me personally to know that, yes, I need to do something.
[FGD1, respondent 1]
We want to know more… our patients usually ask us a lot of questions. Whenever there's an opportunity to learn, yes, you want to capitalize it, so you can attend to their questions.
[FGD2, respondent 3]
| Characteristica | Session (number of live devices)b | P valuec | |||||||
| 1 (n=132) | 2 (n=159) | 3 (n=147) | 4 (n=125) | 5 (n=136) | 6 (n=145) | 7 (n=100) | |||
| Sex, n (%) | .83 | ||||||||
| Female | 88 (66.7) | 97 (61.0) | 89 (60.5) | 82 (65.6) | 87 (64.0) | 86 (59.3) | 65 (65.0) | ||
| Male | 44 (33.3) | 62 (39.0) | 58 (39.5) | 43 (34.4) | 49 (36.0) | 59 (40.7) | 35 (35.0) | ||
| Age (years), n (%) | .79 | ||||||||
| 18-29 | 28 (21.2) | 46 (28.9) | 40 (27.2) | 35 (28.0) | 38 (27.9) | 38 (26.2) | 31 (31.0) | ||
| 30-49 | 80 (60.6) | 80 (50.3) | 86 (58.5) | 67 (53.6) | 75 (55.1) | 79 (54.5) | 54 (54.0) | ||
| 50-69 | 23 (17.4) | 33 (20.8) | 21 (14.3) | 23 (18.4) | 23 (16.9) | 28 (19.3) | 15 (15.0) | ||
| 70 and older | 1 (0.8) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | ||
| Occupation, n (%) | .35 | ||||||||
| Physician | 50 (37.9) | 58 (36.5) | 42 (28.6) | 37 (29.6) | 31 (22.8) | 40 (27.6) | 26 (26.0) | ||
| Nurse or midwife | 30 (22.7) | 48 (30.2) | 46 (31.3) | 33 (26.4) | 40 (29.4) | 46 (31.7) | 27 (27.0) | ||
| Pharmacist | 6 (4.5) | 9 (5.7) | 6 (4.1) | 3 (2.4) | 5 (3.7) | 10 (6.9) | 1 (1.0) | ||
| Laboratory scientist | 1 (0.8) | 1 (0.6) | 1 (0.7) | 3 (2.4) | 3 (2.2) | 3 (2.1) | 1 (1.0) | ||
| Community health extension worker | 7 (5.3) | 9 (5.7) | 12 (8.2) | 8 (6.4) | 13 (9.6) | 8 (5.5) | 7 (7.0) | ||
| Others | 38 (28.8) | 34 (21.4) | 40 (27.2) | 41 (32.8) | 44 (32.4) | 38 (26.2) | 38 (38.0) | ||
| Organizational setting, n (%) | .06 | ||||||||
| Primary care | 42 (31.8) | 53 (33.3) | 51 (34.7) | 45 (36.0) | 50 (36.8) | 41 (28.3) | 29 (29.0) | ||
| Secondary care | 15 (11.4) | 24 (15.1) | 24 (16.3) | 15 (12.0) | 19 (14.0) | 22 (15.2) | 7 (7.0) | ||
| Tertiary care | 18 (13.6) | 28 (17.6) | 24 (16.3) | 16 (12.8) | 9 (6.6) | 23 (15.9) | 12 (12.0) | ||
| NGOd or social enterprise | 48 (36.4) | 38 (23.9) | 41 (27.9) | 41 (32.8) | 53 (39.0) | 52 (35.9) | 47 (47.0) | ||
| Others | 9 (6.8) | 16 (10.1) | 7 (4.8) | 8 (6.4) | 5 (3.7) | 7 (4.8) | 5 (5.0) | ||
| In-Zoom evaluation responses, n (%) | |||||||||
| Reaction Survey | 49 (37.1) | 39 (24.5) | 37 (25.2) | 31 (24.8) | 43 (31.6) | 42 (29.0) | 30 (30.0) | N/Ae | |
| Preknowledge Quiz | 57 (43.2) | 48 (30.2) | 40 (27.2) | 30 (24.0) | 46 (33.8) | 50 (34.5) | 44 (44.0) | N/A | |
| Postknowledge Quiz | 43 (32.6) | 43 (27.0) | 36 (24.5) | 35 (28.0) | 55 (40.4) | 54 (37.2) | 33 (33.0) | N/A | |
aProportions are from among those who responded during registration. Participants selected only 1 response for each field.
bReported characteristics are based on the individual who registered. Multiple persons may have participated from each device.
cChi-square test.
dNGO: Non-governmental organization.
eN/A: Not applicable.
Appropriateness
Nearly every participant who completed the reaction survey reported that the session met their expectations and that the training would be useful to them in treating patients with high blood pressure (
Figure 3 and ). Most participants responded that the length of the session was appropriate (range: 72% [session 2] to 90% [session 4]). Among those who did not think the length was appropriate respondents generally felt the session was too long. In 6 out of 7 sessions, most respondents (more than 90%) reported that the didactic portion of the training was very helpful. In session 4, which focused on diabetes and pathogenesis, slightly fewer (n=22; 71% of respondents) reported that the information presented was very helpful. Respondents found case presentations slightly less helpful than didactic presentations though very few (less than 3%) reported that case presentations were unhelpful.
The qualitative results supported these results. FGD participants reported that the pedagogy and content of the training were appropriate.
The pattern of the lecture is also okay, in the sense that the lecturers will break it down. The slides are being shown, the lecturer is explaining, as well as there being a section for asking questions.
[FGD1, respondent 1]
It's just like the practicals we do everyday, we who see the patients. So, any time we attend the lecture, we learn from it, the challenges, and we will be picturing it, ‘Okay. Next time, if this patient comes, this is what I will do.’ That will make you want to learn more. That will make you concentrate because you are doing the practical aspect, so you want to learn.
[FGD1, respondent 3]
Mixed reactions were offered on the appropriateness of electronic versus in-person training. The recognized benefits of web-based training were primarily related to the convenience of location and costs.
[The ECHO Program] minimizes cost because many of us that are from far-reaching facilities, it normally takes time for us to attend training. The ECHO series has really done well by easing all that stress and the cost attached to attending meetings.
[FGD3, respondent 6]
Some health care workers reported a preference for in-person training, both individually and among their colleagues, citing advantages for engaging individuals’ interests, fostering interactions, and improving coverage of the workforce.
…this online stuff is not interesting…. They [colleagues] prefer a one-on-one class, instead of this ECHO stuff.
[FGD2, respondent 2]
Since it [training] is not an everyday thing, physical training is better because we interact with one another.
[FGD3, respondent 6]
Numerous health care workers advocated for electronic and physical resources to be made available before and after the sessions.
As much as we are in a digital world, there are still people who cannot operate phones like this. They cannot operate phones that would go on programs like that. So, if there is a way, maybe after the program, we can get hardcopies, that's in the form of [handouts in our facilities]. It will really help us. There are other colleagues that would want to go through them.
[FGD1, respondent 3]
Effectiveness
Knowledge scores (proportion of correct responses) ranged from 39% (presession 4) to 78% (postsession 1). Across all ECHO sessions, postsession knowledge quizzes were an average of 7.3% higher than presession scores (Figure 3). The largest knowledge gain was observed in session 3 (62%-73%; 10.5%) and the smallest was observed in session 7 (58%-60%; 2.3%). Results were similar when restricted to participants who responded to both the pre-and postquiz ( and ).
FGD participants largely reported that the training was effective. Health care workers noted confidence in the accuracy of the information they received because it was coming from experts.
I would say it [ECHO] helped positively, in the sense that you are discussing with those that specialize on the topic, like the consultants that manage the condition. So, it's giving you the assurance that, indeed, you are in the right hands.
[FGD2, respondent 2]
The mode of presentation and then the people doing the presentation themselves, they are experts in the field. That, in a way, gives a kind of confidence that whatever they tell you is actually the right thing you're getting.
[FGD4, respondent 1]
The FGD participants discussed how the training directly enabled them to provide better care for their patients.
One of the sessions I attended was Improving Medication Adherence. I think that has really helped me, of course, a lot in our facility because we were having many defaulters. … So, we were able to learn how we were going to follow them up. … Some of them have made up their minds to do the right thing, and that has even improved our facility attendance, in terms of hypertension treatment.
[FGD3, respondent 6]
There was variability in how well participants comprehended the lectures. Some health care workers reported that the terminology used during the training was difficult for them to follow while others felt the level of information was well tailored to the audience.
I think it was more or less like it loaded more than I wished I'd known, because the depth of the ECHO series itself is quite deep. Even at some point, we were like, ‘Ah, are we in a medical class, kind of?’ Because there was a lot of depth. So, I don't think there is any information I needed that I wasn't informed about then.
[FGD4, respondent 1, Data Manager]
The facilitators or should I say the lecturers there, they really tried. They explained it in a way that even a layman would understand what they are saying. So, the facilitators helped a lot, and the lectures go in a way that we understand.
[FGD6, respondent 6, Laboratory Technician]
Actually, the background of the ECHO is very okay. It makes me concentrate on whatever I'm learning. And also, the speakers, sometimes the way they speak is with very simple English that we can understand at our own level. So, it makes me really concentrate on the session. There are not many terminologies that will– And even if there were terminologies, they usually break it down to our own level.
[FGD1, respondent 2, community health extension worker]
FGD participants reported that live participation in the ECHO session was key for understanding the information but that they also benefited from having access to recordings to view at a later time which made the training more effective.
Participating in that ECHO session is very key because as the lecturer is explaining the slide, you'll understand what is there... Even after the program... I would still receive another email for the video, if I want to rewatch the presentation, as well as the slides, just the slides. I used to click on both. For the videos, I do go to YouTube because they usually upload it on YouTube, so I would download it on my phone.
[FDG1, respondent 1]
What's exactly encouraging is that, let me say you have a network problem, there is a video there. After the program, you can still click there and watch what was being discussed there. Honestly, that one encourages me to learn more from it.
[FGD5, respondent 4]
Penetration
Behavior survey results are reported only among health care workers working in the FCT, Nigeria, and were completed a median (IQR) of 28 (24-35) days after the ECHO session (). Individuals who provided patient care and participated in the last ECHO session were asked about their use of the information and gaps in their knowledge. Most respondents (more than 70%) in each survey reported that they used what they learned in the immediate prior ECHO session for patient care (Figure 3). Open-ended survey responses supported the application (ie, “I learnt on how to take care of a pregnant woman with hypertension [and] also the risk factors for developing hypertension disorder in pregnancy”).
A small number (range: 0 [0%] to 2 [29%]) of respondents reported knowledge gaps at each survey point. Gaps included the lack of understanding of risk factors for hypertension (ie, “Can obesity lead to hypertension?”) and duration of drug therapy (ie, “How long are we [the patient] going to continue taking the drugs?”).
FGD participants recommended that the hypertension ECHO program should be continuous and expanded to larger cohorts inclusive of both additional health facilities and health care workers.
There should be periodic training. It shouldn't be for just particular people or a particular set in the facility alone. By the time this set is trained, another set should be called, different faces from that facility should be trained as well, so that when this person is not around, the other person is around.
[FGD1, respondent 1]
[My suggestion] is extending to other facilities, because as the program is going on now, I can see that selected clinics are chosen from each of the area councils.
[FGD6, respondent 1]
We actually need training. Science is dynamic and we keep seeing new things every day … this program has really widened our knowledge in terms of hypertension treatment, and we will be happy to see more of this training.
[FGD3, respondent 6]
Discussion
Principal Findings
We evaluated the implementation of a seven-part hypertension ECHO series to train health care workers in Nigeria through selected implementation outcomes mapped to the first 3 levels of the Kirkpatrick model. To our knowledge, this study is one of the first to use the full Kirkpatrick model for the evaluation of an ECHO program. Prior evaluations, which largely focus on domains of reaction and learning, align with our findings of high satisfaction and moderate knowledge gains [-]. A large majority of FGD participants reported using information from ECHO sessions in their patient-related work, which is consistent with findings from a scoping review in which 40% of the reviewed studies found that ECHO participants either modified their practice behaviors or intended to do so as a result of participation [].
Our findings underscore the necessity for a supportive ecosystem that enables greater reach. FGD participants highlighted several barriers to live participation including busy work schedules and systemic challenges such as poor mobile networks and the unavailability of electricity to power their devices. In our implementation, we sent frequent reminders through an assortment of channels such as email, text messages, and social media platforms (WhatsApp, Facebook, Instagram, and LinkedIn). This strategy aimed to leverage health care workers’ intrinsic motivation for learning to enhance uptake, active participation, and adherence. Our approach of using multiple communication channels to promote ECHO sessions stands out from the usual practice of tailoring ECHO sessions to specific groups and enables us to reach a larger and more diverse audience.
FGD participants expressed a mixture of opinions on the appropriateness and effectiveness of the ECHO model specifically in how they thought training should be delivered. We observed preferences for a blend of in-person and web-based training with web-based training delivering both synchronously and asynchronously. In-person sensitization training could be partnered with a continuous ECHO program and a digital learning management system to promote a community of practice and facilitate access to recorded sessions and learning materials.
Limitations
The delivery and evaluation of this intervention were subject to several limitations. First, considerable time was dedicated to providing digital and in-person support to participants and limited access to mobile data and reliable internet connections restricted the number of people that could be supported by the intervention. Unique to this intervention was the backend infrastructure required to successfully host the entire series. It was run by a consortium of partners who focused separately on recruiting speakers, facilitating the sessions, publicizing the events, and collecting and analyzing study data. Second, the inability to track individual participants across sessions meant that outcomes and assessments were only evaluated per session instead of across the entire series. Third, the use and behavior surveys relied on self-reported data, which prevented us from objectively examining the impact of the sessions on participant behavior. Despite these limitations in data collection, our approach of including all comers and robust evaluation through mixed methods provides ample data for systematic evaluation of the ECHO series and may serve as a model for evaluations of other programs.
Conclusions
Our hypertension ECHO session resulted in substantial satisfaction and knowledge acquisition among participating health care workers. The program had wide reach despite challenges such as poor connectivity and competing clinical demands which hindered participation. Our findings underscore the importance of creating a supportive and flexible ecosystem for training and demonstrate that the ECHO program can be successfully implemented for hypertension education of health care workers in the FCT, Nigeria. This evaluation provides valuable insights for the potential scaling of hypertension education for health care workers in similar settings.
Acknowledgments
All FGD participants provided written informed consent prior to participation in the discussion. The authors are very appreciative to Blessing Akor, Godiya Bwala, and Chika Onwundijo for their support in facilitating focus group discussions. The authors are thankful to all of the participants in the FGD. Generative artificial intelligence was not used in any portion of the investigation, analysis, or writing of the manuscript. The study is supported by the National Heart, Lung, and Blood Institute (R01HL144708), Northwestern Bluhm Cardiovascular Institute, Northwestern Robert J Havey MD Institute for Global Health, and Resolve to Save Lives. The funders were not involved in the development of the study design; collection, management, analysis, and interpretation of data; writing of the report; and the decision to submit the report for publication. The Northwestern Robert J Havey MD Institute for Global Health and Resolve to Save Lives contributed support for projects embedded within the overall program.
Data Availability
The datasets analyzed during the current study are not publicly available because the data collection, as approved by the ethics committee and institutional review board, did not include having them become publicly available. The data can be made available to other researchers by contacting the corresponding author and with ethics committee and institutional review board approval.
Authors' Contributions
ASB, LRH, and NRK conceptualized the study, selected the methodology, and supervised the investigation. ASB, DBO, and MDH secured funding for the study. AO, ASB, IAO, NRR, and OB conducted the investigation, project administration, and provided resources. ASB and JA curated the data, sourced software for data management and analysis, conducted the formal analysis and visualization, and validated the findings. ASB wrote the first draft of the manuscript. AO, AO, DA, DBO, LNM, LRH, MDH, MM, NRK, OB, RCBO, and RM provided critical review and editing of the manuscript. All authors read and approved the final manuscript.
Conflicts of Interest
MDH has received travel support from the American Heart Association and World Heart Federation. MDH has an appointment at The George Institute for Global Health, which has a patent and license and has received investment funding with the intent to commercialize fixed-dose combination therapy through its social enterprise business, George Medicines. MDH has pending patents for heart failure polypills.
Multimedia Appendix 1
Semistructured guide for focus group discussions.
PDF File (Adobe PDF File), 96 KBMultimedia Appendix 2
Offerings, experts, and attendees of the hypertension ECHO (Extension for Community Healthcare Outcomes) training program.
DOCX File , 13 KBMultimedia Appendix 3
Postsession zoom survey to assess reaction (Kirkpatrick Level 1).
PDF File (Adobe PDF File), 33 KBMultimedia Appendix 4
Pre- and postsession zoom quizzes to assess knowledge (Kirkpatrick Level 2).
DOCX File , 37 KBMultimedia Appendix 5
Postsession REDCap (Research Electronic Data Capture) survey to assess behavior (Kirkpatrick Level 3).
PDF File (Adobe PDF File), 43 KBMultimedia Appendix 6
COREQ (Consolidated Criteria for Reporting Qualitative research) checklist.
PDF File (Adobe PDF File), 481 KBMultimedia Appendix 7
Characteristics of frontline health care worker focus group discussion participants from 12 Hypertension Treatment in Nigeria Program primary health centers.
DOCX File , 14 KBMultimedia Appendix 8
Reaction surveys from live participants in each hypertension ECHO session.
DOCX File , 17 KBMultimedia Appendix 9
Pre- and postsession knowledge quizzes from each hypertension ECHO (Extension for Community Healthcare Outcomes) session among all participants who responded.
DOCX File , 17 KBMultimedia Appendix 10
Pre- and postsession knowledge quizzes from each hypertension ECHO (Extension for Community Healthcare Outcomes) session among participants who responded to both the pre- and postsession quizzes.
DOCX File , 17 KBMultimedia Appendix 11
Participant feedback on use of information following each ECHO (Extension for Community Healthcare Outcomes) session among health care workers at 12 Hypertension Treatment in Nigeria Program primary health centers.
DOCX File , 17 KBReferences
- Mills KT, Stefanescu A, He J. The global epidemiology of hypertension. Nat Rev Nephrol. 2020;16(4):223-237. [FREE Full text] [CrossRef] [Medline]
- Mills KT, Bundy JD, Kelly TN, Reed JE, Kearney PM, Reynolds K, et al. Global disparities of hypertension prevalence and control: a systematic analysis of population-based studies from 90 countries. Circulation. 2016;134(6):441-450. [FREE Full text] [CrossRef] [Medline]
- Joshi R, Thrift AG, Smith C, Praveen D, Vedanthan R, Gyamfi J, et al. Task-shifting for cardiovascular risk factor management: lessons from the Global Alliance for Chronic Diseases. BMJ Glob Health. 2018;3(Suppl 3):e001092. [FREE Full text] [CrossRef] [Medline]
- Proia KK, Thota AB, Njie GJ, Finnie RKC, Hopkins DP, Mukhtar Q, et al. Team-based care and improved blood pressure control: a community guide systematic review. Am J Prev Med. 2014;47(1):86-99. [FREE Full text] [CrossRef] [Medline]
- Arora S, Thornton K, Jenkusky SM, Parish B, Scaletti JV. Project ECHO: Linking university specialists with rural and prison-based clinicians to improve care for people with chronic hepatitis C in New Mexico. Public Health Rep. 2007;122:74-77. [FREE Full text] [CrossRef] [Medline]
- Arora S, Geppert CMA, Kalishman S, Dion D, Pullara F, Bjeletich B, et al. Academic health center management of chronic diseases through knowledge networks: Project ECHO. Acad Med. 2007;82(2):154-160. [FREE Full text] [CrossRef] [Medline]
- Masi C, Hamlish T, Davis A, Bordenave K, Brown S, Perea B, et al. Using an established telehealth model to train urban primary care providers on hypertension management. J Clin Hypertens. 2012;14(1):45-50. [FREE Full text] [CrossRef] [Medline]
- Swigert TJ, True MW, Sauerwein TJ, Dai H. U.S. Air Force telehealth initiative to assist primary care providers in the management of diabetes. Clin Diabetes. 2014;32(2):78-80. [FREE Full text] [CrossRef] [Medline]
- Anderson D, Zlateva I, Davis B, Bifulco L, Giannotti T, Coman E, et al. Improving pain care with Project ECHO in community health centers. Pain Med. 2017;18(10):1882-1889. [FREE Full text] [CrossRef] [Medline]
- Ball S, Wilson B, Ober S, Mchaourab A. SCAN-ECHO for pain management: Implementing a regional telementoring training for primary care providers. Pain Med. 2018;19(2):262-268. [CrossRef] [Medline]
- Carlin L, Zhao J, Dubin R, Taenzer P, Sidrak H, Furlan A. Project ECHO telementoring intervention for managing chronic pain in primary care: insights from a qualitative study. Pain Med. 2018;19(6):1140-1146. [FREE Full text] [CrossRef] [Medline]
- Eaton LH, Godfrey DS, Langford DJ, Rue T, Tauben DJ, Doorenbos AZ. Telementoring for improving primary care provider knowledge and competence in managing chronic pain: a randomised controlled trial. J Telemed Telecare. 2020;26(1-2):21-27. [FREE Full text] [CrossRef] [Medline]
- Haozous E, Doorenbos AZ, Demiris G, Eaton LH, Towle C, Kundu A, et al. Role of telehealth/videoconferencing in managing cancer pain in rural American Indian communities. Psychooncology. 2012;21(2):219-223. [FREE Full text] [CrossRef] [Medline]
- De Witt Jansen B, Brazil K, Passmore P, Buchanan H, Maxwell D, McIlfatrick SJ, et al. Evaluation of the impact of telementoring using ECHO© technology on healthcare professionals' knowledge and self-efficacy in assessing and managing pain for people with advanced dementia nearing the end of life. BMC Health Serv Res. 2018;18(1):228. [FREE Full text] [CrossRef] [Medline]
- Meins AR, Doorenbos AZ, Eaton L, Gordon D, Theodore B, Tauben D. TelePain: A community of practice for pain management. J Pain Relief. 2015;4(2):177. [FREE Full text] [CrossRef] [Medline]
- Catic AG, Mattison MLP, Bakaev I, Morgan M, Monti SM, Lipsitz L. ECHO-AGE: An innovative model of geriatric care for long-term care residents with dementia and behavioral issues. J Am Med Dir Assoc. 2014;15(12):938-942. [CrossRef] [Medline]
- Farris G, Sircar M, Bortinger J, Moore A, Krupp JE, Marshall J, et al. Extension for Community Healthcare Outcomes–care transitions: Enhancing geriatric care transitions through a multidisciplinary videoconference. J Am Geriatr Soc. 2017;65(3):598-602. [CrossRef] [Medline]
- Fisher E, Hasselberg M, Conwell Y, Weiss L, Padrón NA, Tiernan E, et al. Telementoring primary care clinicians to improve geriatric mental health care. Popul Health Manag. 2017;20(5):342-347. [CrossRef] [Medline]
- Mehrotra K, Chand P, Bandawar M, Rao Sagi M, Kaur S, Raj A, et al. Effectiveness of NIMHANS ECHO blended tele-mentoring model on integrated mental health and addiction for counsellors in rural and underserved districts of Chhattisgarh, India. Asian J Psychiatr. 2018;36:123-127. [CrossRef] [Medline]
- Sockalingam S, Arena A, Serhal E, Mohri L, Alloo J, Crawford A. Building provincial mental health capacity in primary care: an evaluation of a Project ECHO mental health program. Acad Psychiatry. 2018;42(4):451-457. [CrossRef] [Medline]
- Cordasco KM, Zuchowski JL, Hamilton AB, Kirsh S, Veet L, Saavedra JO, et al. Early lessons learned in implementing a women's health educational and virtual consultation program in VA. Med Care. 2015;53(4):S88-S92. [FREE Full text] [CrossRef] [Medline]
- Wood BR, Mann MS, Martinez-Paz N, Unruh KT, Annese M, Spach DH, et al. Project ECHO: Telementoring to educate and support prescribing of HIV pre-exposure prophylaxis by community medical providers. Sex Health. 2018;15(6):601-605. [CrossRef] [Medline]
- Komaromy M, Ceballos V, Zurawski A, Bodenheimer T, Thom DH, Arora S. Extension for Community Healthcare Outcomes (ECHO): A new model for community health worker training and support. J Public Health Policy. 2018;39(2):203-216. [CrossRef] [Medline]
- Damian AJ, Robinson S, Manzoor F, Lamb M, Rojas A, Porto A, et al. A mixed methods evaluation of the feasibility, acceptability, and impact of a pilot Project ECHO for community health workers (CHWs). Pilot Feasibility Stud. 2020;6:132. [FREE Full text] [CrossRef] [Medline]
- Blecker S, Paul MM, Jones S, Billings J, Bouchonville MF, Hager B, et al. A Project ECHO and community health worker intervention for patients with diabetes. Am J Med. 2022;135(5):e95-e103. [CrossRef] [Medline]
- Thomas KT, Friedman SA, Larson MJ, Jorgensen TC, Sharma S, Smith A, et al. A cohort-based nutrition ECHO for community health workers. Health Educ Res. 2023;38(2):163-176. [CrossRef] [Medline]
- Maizel J, Filipp SL, Zori G, Yadav S, Avaiya K, Figg L, et al. Project Extension for Community Healthcare Outcomes intervention evaluations: a scoping review of research methods. J Contin Educ Health Prof. 2024. [CrossRef] [Medline]
- Kirkpatrick DL. Evaluating Training Programs: The Four Levels. San Francisco, CA. Berrett-Koehler Publishers; 1998:289.
- Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health. 2011;38(1):44-53. [FREE Full text] [CrossRef] [Medline]
- Wingood GM, DiClemente RJ. The ADAPT-ITT model: a novel method of adapting evidence-based HIV interventions. J Acquir Immune Defic Syndr. 2008;47:S40-S46. [CrossRef] [Medline]
- Baldridge AS, Goldstar N, Bellinger GC, DeNoma AT, Orji IA, Shedul GL, et al. Formative evaluation and adaptation of a hypertension Extension for Community Health Outcomes program for healthcare workers within the Federal Capital Territory, Nigeria. Glob Heart. 2023;18(1):64. [CrossRef] [Medline]
- Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65-76. [FREE Full text] [CrossRef] [Medline]
- Baldridge AS, Orji IA, Shedul GL, Iyer G, Jamro EL, Ye J, et al. Enhancing hypertension education of community health extension workers in Nigeria's Federal Capital Territory: the impact of the extension for community healthcare outcomes model on primary care, a quasi-experimental study. BMC Prim Care. 2024;25(1):334. [CrossRef] [Medline]
- Baldridge AS, Aluka-Omitiran K, Orji IA, Shedul GL, Ojo TM, Eze H, et al. Hypertension Treatment in Nigeria (HTN) Program: Rationale and design for a type 2 hybrid, effectiveness, and implementation interrupted time series trial. Implement Sci Commun. 2022;3(1):84. [FREE Full text] [CrossRef] [Medline]
- Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O'Neal L, et al. REDCap Consortium. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. 2019;95:103208. [FREE Full text] [CrossRef] [Medline]
- Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377-381. [FREE Full text] [CrossRef] [Medline]
- Tong A, Sainsbury P, Craig J. Consolidated Criteria for Reporting Qualitative research (COREQ): A 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349-357. [CrossRef] [Medline]
- Roy A, Das B, Garg P. Project ECHO (Extension for Community Health Care Outcomes), an online tool for residents' education: a pilot study. Indian J Ophthalmol. 2020;68(10):2318-2319. [FREE Full text] [CrossRef] [Medline]
- Moore PQ, Tilmon S, Chhabra N, McCabe DJ, Aks SE, Johnson D, et al. Applying Project ECHO (Extension for Community Health Care Outcomes) to improve addiction care in rural emergency departments. AEM Educ Train. 2022;6(5):e10804. [FREE Full text] [CrossRef] [Medline]
- Kamal AH, Bossley H, Blum R, Berman AJ, von Gunten C, Von Roenn J. Evaluation of a virtual learning collaborative to integrate palliative care into routine oncology practice. JCO Oncol Pract. 2020;16(11):e1371-e1377. [CrossRef] [Medline]
- Arora S, Brakey HR, Jones JL, Hood N, Fuentes JE, Cirolia L. Project ECHO for cancer care: a scoping review of provider outcome evaluations. J Cancer Educ. 2023;38(5):1509-1521. [CrossRef] [Medline]
Abbreviations
| COREQ: Consolidated Criteria for Reporting Qualitative research |
| ECHO: Extension for Community Healthcare Outcomes |
| FGD: Focus group discussion |
| HTN: Hypertension Treatment in Nigeria |
| LGA: local government area |
| PHC: primary health center |
| REDCap: Research Electronic Data Capture |
| UATH: University of Abuja Teaching Hospital |
Edited by X Ma; submitted 10.09.24; peer-reviewed by K Omitiran, QJ Zhao, S Otero; comments to author 02.01.25; revised version received 25.02.25; accepted 28.02.25; published 24.04.25.
Copyright©Abigail S Baldridge, Adaora Odukwe, Olabisi Dabiri, L Nneka Mobisson, Maria Moosa Munnee, Ayoposi Ogboye, Dorothy Naa Korkoi Aryee, Rodrick Mwale, Jonas Akpakli, Ikechukwu A Orji, Rosemary C B Okoli, Nanna R Ripiye, Dike B Ojji, Mark D Huffman, Namratha R Kandula, Lisa R Hirschhorn. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 24.04.2025.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.