Assessing the Response Results of an mHealth-Based Patient Experience Survey Among People Receiving HIV Care in Lusaka, Zambia: Cohort Study

Introduction

The increasing availability of mobile devices and mobile network coverage in sub-Saharan Africa presents a growing opportunity for mobile health (mHealth) interventions. By 2025, projections estimate that 50% of the sub-Saharan African population will own a mobile phone [1]. Zambia exemplifies this trend, with rapid mobile network expansion leading to an estimated 21.1 million mobile connections, reaching 91.4% of the population [2]. Mobile device usage in Zambia is shifting from voice calls to mobile internet, with users increasing from 9.1 million in 2019 to 12.5 million in 2023 [2,3]. In addition, the use of mobile money through local mobile network providers has risen rapidly, already 2.7 times higher in 2023 compared with 2021.

In much of sub-Saharan Africa, including Zambia, mobile network operators offer prepaid service plans. These plans allow users to purchase “talk-time,” providing finite minutes or data, as dedicated. With multiple competing mobile networks vying for market share, promotions and incentives for holding multiple phone numbers across the 3 main networks in Zambia are common. This can create challenges for mHealth engagement and response.

The rising level of accessibility to mobile devices allows for mHealth tools, such as mobile phone surveys, to capture important health-related feedback without substantially limiting participant eligibility. The use of the mHealth tools enables health care providers and health facility stakeholders to improve patient monitoring, enable mobile clinical decision support, and address contextual challenges in sub-Saharan Africa [4,5]. This technological advancement has the potential to greatly enhance health care outcomes by facilitating a better understanding of patient needs and improving service delivery in sub-Saharan Africa [6-8].

HIV care has now become chronic, with people making HIV care clinic visits every 3 months at the time of this research study in Zambia, providing an opportunity to regularly collect care experience measures. However, there is limited evidence in sub-Saharan Africa concerning the durability of an initial enrollment across multiple invitations and the role of incentives on survey response rates, especially in an era where mobile technology and mobile networks have become highly accessible. Previous work done in 4 countries in sub-Saharan Africa reports varying response rates to a general population SMS survey, with the highest response in Uganda (14.2%) and the lowest in Nigeria (0.3%) [9]. In 2016, a study in Kenya offering information on family planning methods, through SMS text messages, found that incentives varying between US $0.5 and US $1.0 were found to have no association with SMS survey response [10]. Another study in Zambia used SMS notifications as an option for linkage to care for individuals who are HIV positive [11]. Our study in Zambia is the first to examine the prospective outcomes of health-related surveys based on unstructured supplemental service data (USSD) at varying levels of talk-time incentives [12-14].

Using pilot data from a large stepped-wedge trial, we assess the pattern of responses across multiple rounds of invitation to participate in an mHealth model. Specifically, we investigated the association between talk-time incentive and rate of response to a brief, 4-question, USSD-based patient experience survey in Lusaka, Zambia. We collaborated with all the mobile network operators in Zambia to establish a unique USSD code (also called a short code) for the Person-Centered Public Health for HIV Treatment in Zambia (PCPH) study. We analyzed 3 levels of talk-time incentive, preferred language for survey administration, and clinic setting (periurban or urban) on the response rate of the USSD survey. Our goal was to evaluate the overall uptake, and response rate, to the survey after an HIV care visit in Lusaka, Zambia. Although this application was limited to HIV care, these findings may provide valuable insights for future initiatives involving mHealth in Zambia and sub-Saharan Africa.

Methods

Study Setting

This study was conducted as part of the pilot phase of the larger PCPH study, a cluster-randomized stepped-wedge trial to assess the impact of a patient-centeredness intervention on patient experiences of HIV care and outcomes, which ran from August 2018 to November 2021 in Lusaka Province (Pan African Clinical Trial Registry PACTR202101847907585) [15-17].

Data presented here were collected during the pilot phase, which was designed to develop and optimize a mobile phone–based survey for the main PCPH study, from November 1, 2018, to June 25, 2019. Pilot study clinics are run by the Zambian Ministry of Health (MoH) and receive various levels of technical support through the Centre for Infectious Disease Research in Zambia, an implementing partner of the MoH since 2011 and the primary implementer of the PCPH trial.

Recruitment Procedure

Potential survey participants were identified by research assistants at 1 of 2 (1 in an urban setting, 1 in a periurban setting) pilot study HIV care facilities using convenience sampling; participants were selected following reception and triage procedures, which included routine data collection (eg, temperature, weight, and blood pressure). Eligibility for the study participation was limited to individuals who were enrolled as a recipient of care at the study facility, were aged 18 years or older, self-reported as literate in 1 of 4 survey languages, had a mobile phone during enrollment, and voluntarily consented to participate in the study. We confirmed that the mobile phone number reported by the potential participant belonged to the individual through a registration process using the unique PCPH short code, *744#, on the potential participant’s phone at enrollment. The registration process confirmed the ability to initiate and complete a USSD session, collected the language preference, and recorded the facility where the enrolment took place.

Data Collection

Surveys

Survey participants were presented with 4 questions related to their experience at their most recent HIV care visit. The opening question was the same for all participants, the next 3 questions were drawn, using a systematic algorithm, from a bank of 9 questions (USSD survey question sets are shown in Multimedia Appendix 1). The survey questions were subject to a cognitive interviewing process to increase acceptability, understanding, and appropriateness before piloting. At enrollment, participants selected their preferred survey language from English, Bemba (spoken by 33.5% of the Zambian population), Nyanja (spoken by 14.8% of the Zambian population), or Tonga (spoken by 11.4% of the Zambian population). Although only 2% of the population have English as their first language, it is the most spoken second language [18].

Survey Administration

A total of 2 rounds of surveys were sent to those enrolled. The first round of survey invitations was sent through SMS text messages between November 1, 2018, and June 25, 2019, within 14 days of enrollment. The second round of invitations was sent 3 months after the first invitation, between February 1, 2019, and September 23, 2019. Survey spacing of 3 months was established through a review of median visit interval in the previous year at pilot study clinics. This second round was intended to capture the experience during the next routine HIV care visit, commonly scheduled quarterly (every 3 months). An incentive was systematically allocated to a participant once at registration for both survey rounds. The incentive allocation algorithm created a queue with values of 0 Zambian Kwacha (ZMW) or no incentives, 2 ZMW (US $0.16), and 5 ZMW (US $0.42), assigned in the order of enrollment. This queue was cycled by the incentive allocation algorithm, which allocated each incentive level before beginning at 0 at the end of each cycle. A currency exchange rate of 12.90 ZMW=US $1 was applicable and, at the time of the first round of the survey, 12.51 ZMW=US $1 was applicable [19]. In Zambia, talk-time is typically used to purchase bundles of minutes for voice calls, data, and sending and receiving SMS text messages. These bundles are valid for a day, 1 week, or 1 month. A 5 ZMW (US $0.42) talk-time can purchase a daily bundle with an average of 25 minutes for voice calls, 25 MB of data, and 127 SMS text messages. The survey invitation informed the survey participants of the incentive amount that they would receive upon survey completion. These incentive amounts were automatically sent to the participants through a third-party vendor immediately after the completion of all 5 survey questions. The money was sent as a talk-time top-up to study participants who all had a prepaid plan through their mobile network. Participants who did not start or complete the survey received a reminder through SMS text messages 24 hours after previous invitation and again 7 days after that invitation if the survey remained incomplete.

Data Analysis

We assessed the association between survey completion and incentive amount for a USSD-based survey using a mixed-effects Poisson regression model, allowing random effects at both the individual and facility levels. A probability model with 95% CIs was illustrated for incentive level and language chosen by the participant. In addition, we projected the cost for additional response by incentive level based on 1000 invitations to the survey, by comparing the cost of additional response between 2 ZMW (US $0.16) and no incentives and between 5 ZMW (US $0.42) and no incentives. All analyses were completed using Stata BE (StataCorp LLC).

Ethical Considerations

Ethical approval was granted by the University of Zambia Biomedical Research Ethics Committee (reference 008-03-19), the National Health Research Authority in Zambia, and the University of Alabama at Birmingham (300003282). The National Health Research Authority and the Zambian Ministry of Health provided authorization to conduct the research. The study participant data were deidentified, and a unique identifier was generated to identify participants’ responses across survey rounds. All the participants provided individual written informed consent before being enrolled in the study.

Results

A total of 1006 participants were enrolled, of which 986 (98%) received a second invitation to participate 3 months after the initial invitation, for a total of 1992 observations (Figure 1). At the beginning of the trial, the urban HIV care facility had a total of 7159 individuals receiving HIV care, while a periurban HIV care facility had a total of 3112 individuals receiving HIV care.

We found an overall response rate across both surveys to be 32% (637/1992) with significantly different response rates between the first (40.5%, 407/1006) and second round (23.3%, 230/986) surveys (χ²P<.001; Tables S1 and S2 in Multimedia Appendix 1). The majority of those enrolled visited the urban-based HIV care facility (727/1006, 72.3%) and the survey was most often requested in English (628/1006, 62.4%; Table 1).

The second round of survey invitation had a significantly lower adjusted prevalence (adjusted prevalence ratio [aPR] 0.49, 95% CI 0.44-0.53) of response compared with the first round. Implementing a 5 ZMW (US $0.42) incentive significantly increased the prevalence ratio for survey completion compared with those that received no incentives (aPR 1.35, 95% CI 1.11-1.63; Table 2). Those enrolled at a visit to the urban study clinic had a significantly higher aPR at 1.26 (95% CI 1.03-1.52) compared with those at the periurban study clinic (Table 2).

‎

We observed a significant decrease in survey completion between survey 1 and the follow-up survey sent 3 months later. The probability of completing the survey was lowest for those who received no incentive in round 2 for which 20.6% completed the survey (95% CI 17.1-25.2; Figure 2; Table S3 in Multimedia Appendix 1).

We found a statistically significant association between language and setting for response to both survey invitations compared with just the first invitations. Individuals completing at least 1 survey in Bemba have a significantly lower adjusted prevalence (aPR 0.87, 95% CI 0.87-0.87) of completing both survey invitations compared with those completing the survey in English. In addition, individuals in urban settings have a significantly lower prevalence (aPR: 0.90, 95% CI 0.87-0.94 of responding to both survey invitations compared with individuals in a rural setting (Table 3).

Using observed response rates, we projected cost per additional response to be highest, at the 5 ZMW (US $0.42) incentive level, at 72.8 ZMW (US $5.82) per additional response as compared with the 0 ZMW incentive level considering 1000 invitations (Table 4).

‎

Finally, we estimated the probability of survey completion by language and incentive level. English had a significantly higher completion rate than Bemba and Nyanja at all levels of the incentives with wide CIs around the Tonga point estimate (Figure 2). While not significant, survey participants who requested the survey in Tonga and were assigned 5 ZMW (US $0.42) had the highest estimated point probability of completing the survey, at 51.6% (95% CI 13.9-89.2), while those who spoke Bemba and received 0 ZMW had the lowest estimated point probability of completing the survey at 37.78% (95% CI 27.4-48.2; Figure 3; Table S4 in Multimedia Appendix 1). Incentive value was significantly associated with the response rate for those requesting the survey in English, while Bemba, Nyanja, and Tonga did not have a significant difference in survey completion probability by incentive (Figure 3).

‎

Discussion

Principal Findings

We found a significant reduction in survey response rate across different languages and incentive levels 3 months after enrollment. Interestingly, the significant association between incentive level and survey response holds only for the first round of survey invitations, while there is no significant difference across incentive levels for the second round. Overall, the response rate across both surveys is highest (32.1%) among participants who received a 5 ZMW (US $0.42) incentive.

In Australia, longitudinal data collection through SMS text messages, with presurvey notifications, has been shown to be effective for monitoring postoperative pediatric pain, providing hope that our mHealth model may be an efficient mechanism to collect data longitudinally in our setting as well [20,21]. However, in our population of adults in HIV care, we found that subsequent invitations to participants were significantly less successful than the initial invitation. This restriction does not totally exclude the tool’s usage as a longitudinal data collection instrument. Some contributors to this response attrition may include (1) a high rate of phone number turnover, (2) phone discontinuance (phone sold, lost, stolen, etc), (3) no interest in the survey, (4) do not need incentive or incentive insufficient, and (5) no recent care experience to report in the event the participant has not had a return visit for HIV care. We suspect that phone number turnover is a chief contributor to the lower response rate to the second invitation given 1.1 mobile numbers per person (107.6 mobile network activations per 100 people) in Zambia in 2023 [2]. Unlike more expensive smartphones in many well-resourced settings, where mobile devices increasingly store vital information, such as credit cards and identifiable information, mobile devices in resource-limited settings tend to be more affordable [3]. In these settings, obtaining phone numbers often through prepaid schemes is inexpensive and readily accessible, and sometimes even provided for free through promotions by mobile network operators. Poor network quality and high cost of services are 2 other reasons leading to mobile customer churn in urban and periurban settings [22]. Although subsequent invitation response rates declined, it may remain a valuable method for collecting data at multiple time points, remotely (a vital aspect during health shocks like the COVID-19 pandemic) [23,24]. It is also possible that subsequent invitation response rates could be improved, despite phone number turnover, through routine updating of participant contact information.

While similar incentive-for-participation projects have been conducted in Zambia, such as those related to voluntary medical male circumcision, which used a 2 ZMW (US $0.16) incentive provided upon SMS survey completion, However, the response rate was not reported [25]. In our study, the provision of incentives was significantly associated with the response rate. There was a dose-response relationship observed between the incentive level and response rate; however, only the 5 ZMW (US $0.42) level was significantly different compared with no incentive. This is consistent with previous literature in Uganda, where incentivization for a general population survey on demographics, socioeconomic status, and technology was found to be significantly associated with response rate; however, incentive level appears to play a stronger role in response rate in our study than in the Uganda work [10]. In South Africa, a survey targeting randomly selected phone numbers from 4 mobile networks found that over 50% of participants opted for a prize raffle draw as an incentive compared with talk-time [26]. This is also the case in South Africa, where an SMS-based data collecting tool is used as part of a national surveillance system [27]. Future research may consider offering different types of incentives to participants who may not find talk-time a strong enough incentive to complete a survey.

Monetary incentives have been found to increase response rates for both web-based and paper-based surveys compared with no incentives in the United States [28]. It has also been shown that both pre- and postsurvey incentive provisions can yield similar participation [29]. In the United Kingdom, offering unconditional incentives before survey completion was found to be more effective than providing an incentive after survey completion [30]. Then, it is possible that the completion rates observed in this study could be further improved through unconditional survey incentives; however, further research is needed to confirm that the unconditional incentive completion results in the United Kingdom hold true in Zambia.

The pilot was conducted in an urban area at a clinic located within the capital city of Lusaka with full network coverage. In this setting, each participant would likely have more than 1 SIM card to take advantage of promotions offered on other networks. Residents in this community typically work within the city limits and would be able to receive survey participation prompt texts shortly after they are sent. On the other hand, our study clinic in the periurban area may have limited network coverage, being outside the city limits. Typically, families in these communities often work in more agricultural industries, which may lead to movement in and out of network coverage as they search for and conduct work in varying places. In this setting, participants likely switch between networks, and each with a different phone number, to use the network with available reception depending on location which may, in part, drive the poorer response rates we observe in the periurban clinic compared with the urban clinic.

We observed a significant difference in response rate across periurban and urban settings, which may be related to mobile device availability and literacy where individuals in urban settings may have increased exposure to mobile technology compared with those in periurban settings. Periurban settings, especially in resource-limited countries, have the greatest potential to benefit from mHealth efforts. However, sensitization to such technologies may need to be tailored to this population, which may have reduced familiarity with or access to mobile phone technology [12,31].

One strategy that can be used to increase response rates to these types of surveys is to offer incentives to participants who do not initially respond to the survey. There is evidence that this can work in a longitudinal survey conducted in the United States to people exposed to the 9/11 terrorist attacks [32]. This approach would result in a reduction in the cost of survey implementation. Gift cards have also shown evidence of a nonresponse conversation strategy in the United States [33].

In Zambia, English has been the medium of instruction in schools at all levels from primary, secondary, and tertiary education. However, it was only in 2013 when the new language policy came into effect that local languages started being used as a medium of instruction from grades 1 to 4 [34]. This explains why most survey participants chose to receive the survey in English. Literacy in English often means one can read in their native language.

The survey was offered in the 4 most common languages in Zambia with the lowest response rate for the first survey observed among the group requesting the survey in Nyanja (aPR: 0.75, 95% CI 0.56-0.99); however, the prevalence ratio for completing both surveys was higher, although not significantly so, for those requesting the survey in Nyanja (aPR: 1.01, 95% CI 0.43-2.19) compared with English. Although the majority opted to complete the survey in English (62.4%), a large proportion of responses among Nyanja and Bemba languages suggests that this is a useful tool to collect information from those who cannot, or prefer not to, read English in Zambia.

Limitations

This study is subject to several limitations, the most substantial being that it is limited only to those with a mobile device. Although mobile technologies are becoming increasingly available in Zambia, there are still some that cannot yet be reached with this methodology. It is possible, because recruitment was based on convenience sampling, that the selection of participants was biased. However, given that we recruited 9.8% (1006/10,271) of the pilot clinic population and recruitment spanned almost 8 months (November 1, 2018, to June 25, 2019), we suspect the impact of potential selection bias to be minimal. Finally, we are limited in our model adjustment set as we did not collect participant demographic information, such as age, sex, and marital status, which may act as confounders and limit causal inference estimates. Our primary interest in the parent study was an ecological measure at the clinic level of patient experience and not the demographics of participants providing experience information.

Conclusions

The study findings demonstrate the waning response rates over time, indicating a potential benefit in regular contact updates, as well as the differential effectiveness of incentives in improving survey response rates in the context of a mHealth platform for individuals in HIV care. In addition, they suggest that it is important to accommodate varying language preferences. The cost analysis also highlights the need for careful evaluation of the cost-effectiveness of different incentive levels, especially when baseline response rates without incentives are relatively high. Future research and larger-scale studies are necessary to assess the long-term effectiveness of longitudinal, incentive-based approaches in mHealth platforms, aiming to optimize health outcomes for individuals in HIV care and other health care populations.