Review
Abstract
Background: Health systems and providers across America are increasingly employing telehealth technologies to better serve medically underserved low-income, minority, and rural populations at the highest risk for health disparities. The Patient-Centered Outcomes Research Institute (PCORI) has invested US $386 million in comparative effectiveness research in telehealth, yet little is known about the key early lessons garnered from this research regarding the best practices in using telehealth to address disparities.
Objective: This paper describes preliminary lessons from the body of research using study findings and case studies drawn from PCORI seminal patient-centered outcomes research (PCOR) initiatives. The primary purpose was to identify common barriers and facilitators to implementing telehealth technologies in populations at risk for disparities.
Methods: A systematic scoping review of telehealth studies addressing disparities was performed. It was guided by the Arksey and O’Malley Scoping Review Framework and focused on PCORI’s active portfolio of telehealth studies and key PCOR identified by study investigators. We drew on this broad literature using illustrative examples from early PCOR experience and published literature to assess barriers and facilitators to implementing telehealth in populations at risk for disparities, using the active implementation framework to extract data. Major themes regarding how telehealth interventions can overcome barriers to telehealth adoption and implementation were identified through this review using an iterative Delphi process to achieve consensus among the PCORI investigators participating in the study.
Results: PCORI has funded 89 comparative effectiveness studies in telehealth, of which 41 assessed the use of telehealth to improve outcomes for populations at risk for health disparities. These 41 studies employed various overlapping modalities including mobile devices (29/41, 71%), web-based interventions (30/41, 73%), real-time videoconferencing (15/41, 37%), remote patient monitoring (8/41, 20%), and store-and-forward (ie, asynchronous electronic transmission) interventions (4/41, 10%). The studies targeted one or more of PCORI’s priority populations, including racial and ethnic minorities (31/41, 41%), people living in rural areas, and those with low income/low socioeconomic status, low health literacy, or disabilities. Major themes identified across these studies included the importance of patient-centered design, cultural tailoring of telehealth solutions, delivering telehealth through trusted intermediaries, partnering with payers to expand telehealth reimbursement, and ensuring confidential sharing of private information.
Conclusions: Early PCOR evidence suggests that the most effective health system- and provider-level telehealth implementation solutions to address disparities employ patient-centered and culturally tailored telehealth solutions whose development is actively guided by the patients themselves to meet the needs of specific communities and populations. Further, this evidence shows that the best practices in telehealth implementation include delivery of telehealth through trusted intermediaries, close partnership with payers to facilitate reimbursement and sustainability, and safeguards to ensure patient-guided confidential sharing of personal health information.
doi:10.2196/28503
Keywords
Introduction
Background
People living in medically underserved, low-income, or rural areas; those from racial and ethnic minorities; members of the lesbian, gay, bisexual, and transgender (LGBT) community; and those with limited English proficiency or disabilities often face substantial barriers to accessing needed health care and are at high risk for health disparities [
, ]. About one-quarter of the US population experiences geographic and/or economic disparities. For example, 34 million Americans (10.5%) live in poverty [ ] and some 46 million Americans (15%) live in rural areas [ ]. These populations are more likely to report issues with access to primary and specialty care, longer travel time to providers, lower rates of insurance coverage, and higher rates of chronic conditions than urban resident [ , - ]. Racial and ethnic disparities are also common and are strongly linked to social determinants of health [ , , ].Other populations at particularly high risk for disparities include low-income African Americans in medically underserved urban and rural areas of the South [
- ], American Indian populations in rural areas [ , , ], the mostly Alaskan Native residents of Alaska [ ], and Latinx Americans, who face language, insurance, and other barriers to access to care [ ]. Additional groups identified by the Patient-Centered Outcomes Research Institute (PCORI) as priority populations include LGBT persons, those with low health literacy/numeracy and limited English proficiency, and those with disabilities [ - ]. All these populations face barriers related to the social determinants of health that prevent adequate access to care [ , - ], resulting in poor health outcomes across a multitude of domains.Potential for Using Telehealth to Address Disparities
Telehealth modalities have great potential to help overcome geographic, socioeconomic, cultural, and language barriers related to the social determinants of health and enhance access to essential health services for high-risk populations [
, ]. Telehealth has traditionally been a way to provide health care access in rural communities, and its adoption has accelerated for these populations during the COVID-19 pandemic [ , ]. While the potential for telehealth to address disparities, such as transportation and availability of providers, is high, preliminary data during the COVID-19 pandemic suggest that differences in internet and telehealth access may actually compound disparities in chronic disease outcomes [ ]. Telehealth can be used for multiple purposes, including direct delivery of care to patients and their caregivers, remote monitoring of health outcomes, education of patients and caregivers, support for self-care and health behavior change, assistance with health care decision-making, and communication of test results [ ]. Likewise, telehealth can employ multiple communication modalities, including telephone, video teleconference, text messaging, mobile apps, and wearable monitors, that allow providers to remotely communicate, monitor, and share information with patients [ - ]. Previous research has documented the effectiveness of these telehealth modalities in reaching rural populations to provide care, prescreening evaluations, and patient education [ , , ], yet data on the effectiveness of telehealth in other populations is more limited [ - ]. Moreover, while many health care delivery systems have implemented telehealth solutions to address barriers to access and better serve populations at risk for disparities, outcomes and sustainability have been variable [ , , ].Study Aims
The primary purpose of this scoping review was to identify key barriers to telehealth implementation and describe how barriers can be addressed in populations at risk for disparities. PCORI has invested US $386 million in comparative effectiveness research in telehealth, yet little is known about the key early lessons of this research for using telehealth to address disparities. This review describes preliminary lessons from early patient-centered outcomes research (PCOR) experience and literature, using study findings and illustrative case studies drawn from these seminal PCOR initiatives and literature.
Methods
Overview
A systematic scoping review of telehealth studies addressing disparities was performed and was guided by the Arksey and O’Malley Scoping Review Framework [
], with a focus on PCORI’s active portfolio of telehealth studies and key supporting PCOR identified by several of the investigators.Identifying the Research Question
The purpose of this scoping review was to describe barriers and facilitators to implementing telehealth or remote interventions in populations at risk of health disparities (eg, racial and ethnic minority groups or people living in rural areas) within comparative effectiveness research projects funded by PCORI. The specific question addressed was as follows: “How have PCORI-funded investigators overcome barriers to implementing telehealth interventions in populations at risk for disparities?”
Identifying Relevant Studies
Between December 2012 and March 2019, PCORI funded 84 comparative effective research projects, in which at least one of the study’s comparison arms used telehealth to improve health outcomes. PCORI defines telehealth as the delivery of health services via remote telecommunication modalities, such as telephonic communication, remote monitoring devices, real-time videoconferencing, and mobile devices. For this scoping review, PCORI staff first reviewed the telehealth portfolio to identify studies aiming to improve health outcomes for populations at risk of or facing disparities. Studies included those that targeted at least one of the following PCORI priority populations at risk for disparities: people living in medically underserved, low-income, or rural areas; those representing racial and ethnic minorities; members of the LGBT community; those with low health literacy/numeracy and limited English proficiency; and those with disabilities [
]. Other PCORI populations of interest, including older adults, women, children, individuals with multiple chronic diseases, individuals with rare diseases, individuals whose genetic makeup affects their medical outcomes, veterans, and members of the armed forces and their families, were not specifically included in this scoping review.Of the 84 PCORI-funded studies in this portfolio, PCORI staff identified 41 studies focused on improving health outcomes for populations at risk of or facing disparities by confirming the target population for each study. Qualifying studies were further reviewed by PCORI staff to determine if any barriers to implementation of the planned telehealth intervention were experienced and whether the investigators were able to mitigate these barriers to successfully implement the study telehealth intervention. A convenience sample of 8 studies was selected based on their representativeness (ie, to ensure that all major targeted populations at risk for disparities were included), study phase (ie, studies either completed or nearing completion), and perceived principal investigator availability to provide illustrative examples of successful approaches for working through obstacles in implementing telehealth interventions for diverse populations.
Identifying Barriers and Facilitators
We drew on these 8 illustrative examples from early PCOR experience and evidence from the literature to assess barriers and facilitators to implementing telehealth in populations at risk for disparities [
]. Key informant interviews of principal investigators were conducted for the convenience sample of 8 PCORI studies to identify common challenges in the implementation of telehealth in populations at risk for disparities. Among the 8 studies in the convenience sample, 6 principal investigators were contacted for phone interviews and 4 interviews were completed. PCORI then hosted a webinar attended by investigators from all 8 of the studies to facilitate discussion about the study challenges investigators faced and the solutions they identified and implemented. After the initial information gathering from the interviews and webinar, PCORI staff invited investigators to participate in this scoping review and share the lessons learned.Extracting the Data
Following identification of all PCORI studies meeting the criteria for the scoping review (n=41), PCORI staff extracted descriptive data for all these studies, including telehealth purpose and modality, population, health condition, budget amount, principal investigator, and institution, using published project descriptions and study papers. PCORI staff and study investigators then extracted additional data from these same sources and interim progress reports (where available) in order to categorize documented barriers and facilitators using a published framework of identified barriers (eg, limitations) and facilitators (eg, solutions) to telehealth [
]. We then used prespecified data definitions from the Fixsen active implementation framework to further categorize barriers and facilitators to implementing telehealth within 3 major domains, including competency drivers (ie, participant selection, training, and supervision), organizational drivers (ie, decision support, administrative support, and system intervention), and leadership drivers [ , ]. The reporting process used the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) extension for scoping reviews ( ) [ , ].Collating, Summarizing, and Reporting the Data
A table including the core descriptive data regarding all PCORI-funded telehealth studies addressing populations at risk for disparities was created. In addition, figures and charts were created to summarize project details about this subset of studies. An assessment of the quality of studies included in this review was not performed. A table was created summarizing the barriers and facilitators identified within each major implementation domain. Finally, major themes regarding how telehealth interventions can overcome barriers to telehealth adoption and implementation were identified through review of both the extracted data and illustrative examples using an iterative Delphi process among the 8 PCORI principal investigators participating in the study to achieve consensus. The Delphi process was facilitated by PCORI staff and a lead principal investigator for the scoping review, and was conducted through teleconference and email communication. PCORI staff and the lead principal investigator initially categorized extracted data and illustrative examples by barrier and facilitator types based on previously published categories [
]. Studies were then recategorized and category labels were revised based on feedback from participating investigators until consensus on the barrier and facilitator types was achieved.Results
Identifying Relevant Studies
As shown in
, 41 PCORI-funded studies were identified that assessed the use of telehealth to improve outcomes for populations at risk for health or health care disparities in this analysis. Of note, all 41 studies employed a randomized controlled trial methodology, and all were pragmatic or “real-world” comparative effectiveness studies rather than efficacy studies.Study Characteristics
The descriptions, study goals, telehealth interventions studied, settings, number of participants, and targeted study populations for these 41 PCORI-funded studies are detailed in
[ - ]. The studies ranged in size from 59 to 84,828 participants; all were conducted in ambulatory settings; and the majority (33/41, 80.5%) were conducted in primary care and/or community settings.As shown in
, all 41 studies targeted at least one of PCORI’s priority populations, including racial and ethnic minorities, people living in rural areas, people with low income or low socioeconomic status, people with low health literacy, members of the LGBT community, and people with disabilities.PCORI-funded studies employed multiple telehealth modalities. The majority of these 41 studies used a mobile device (n=29, 71%) or were web-based (n=30, 73%). Fewer studies used real-time videoconferencing (n=15, 37%), remote patient monitoring (n=8, 20%), or store-and-forward capability (ie, asynchronous electronic transmission; n=4, 10%). It is important to note that a similar telehealth modality does not indicate similar studies. For example, studies employing mobile device interventions were diverse and included web-based interventions requiring full internet access, as well as lower-tech interventions that only employed SMS or simple text messaging. All the web-based interventions referenced below required reliable broadband internet access. Real-time videoconferencing was used in 15 studies, and 11 of these studies utilized telehealth to improve access to primary or specialty care. Health conditions involving videoconferencing were cardiovascular diseases (n=2), kidney diseases (n=2), mental/behavioral health (n=4), nutritional and metabolic disorders (n=3), allergies and immune disorders, multiple chronic conditions, neurological disorders, respiratory diseases, and trauma/injury. Remote patient monitoring (eg, blood pressure or blood glucose levels) was used in 8 studies. Remote monitoring varied from a wearable device (eg, Fitbit) to mobile app monitoring to an in-home device (eg, scale, blood pressure monitor, or video monitoring). Health conditions incorporating remote monitoring included cardiovascular diseases (n=3), mental/behavioral health (n=2), nutritional and metabolic disorders, and respiratory diseases. Store-and-forward technology was employed in 5 studies that included the following conditions: functional limitations and disabilities, mental/behavioral health, rare diseases, hearing loss, and skin diseases.
Furthermore, the purpose of the 41 PCORI telehealth interventions also varied substantially, with 8 (20%) focused on enabling health monitoring, 20 (49%) focused on improving access to specialty care, 22 (54%) focused on health education, and 27 (66%) focused on promoting chronic condition self-management. The spectrum of chronic conditions studied also varied substantially. The largest proportion of studies focused on behavioral health (n=9, 22%), and nutritional and metabolic disorders (n=6, 15%), including obesity and diabetes. In addition, PCORI has funded telehealth projects addressing disparities focused on cardiovascular conditions (n=3), chronic kidney disease (CKD; n=2), neurological conditions (n=2), reproductive and perinatal health (n=2), rare diseases (n=2), allergies and immune disorders (n=1), cancer (n=1), hearing loss and ear diseases (n=1), functional limitations and disabilities (n=1), infectious diseases (n=1), multiple chronic conditions (n=1), trauma (n=1), and skin diseases (n=1). Although only 1 study focused on patients with multiple chronic conditions, most of the studies included patients with multiple chronic conditions and some additional studies used such presence as an inclusion criterion.
Identifying Barriers and Facilitators
Barriers and facilitators of telehealth implementation in PCORI-funded studies using telehealth to address disparities are detailed in
and . Barriers and facilitators are described for only 35 of the 41 studies as 6 were in the early stages of implementation and lacked sufficient data for evaluation. Review of this data revealed that 20 (57%) studies actively engaged patients to assist investigators in building patient-centered design with active patient participation in each development phase, 13 (37%) studies culturally tailored their telehealth intervention, 4 (11%) studies employed partnership with payers to expand telehealth reimbursement, and 11 (31%) studies delivered telehealth care through or with the assistance of trusted intermediaries. Of the 11 studies, 5 (14%) had clinical support team members available to support their telehealth interventions with a human interaction component.Barriers (ie, limitations or challenges) | Barriers identified, n | Example | ||
Competency drivers | ||||
Participant engagement barriers (eg, social determinants of health presenting barriers to telehealth adoption) | ||||
Inadequate access to telecommunication technology | 9 | Patients with many competing social needs and low levels of social support and resources that may have prevented individuals who would otherwise be eligible and/or interested in study participation (eg, inadequate access to technology). This barrier was removed by adding “access to a phone or iPad” to the inclusion criteria. | ||
Patient and stakeholder feedback | 1 | Patient and stakeholder partners were integrated into studies to provide feedback on the usability and feasibility of technology. | ||
Staff training barriers | 3 | Providers and health care staff may not receive training on technology use or telehealth etiquette. Inadequate communication among patients, multiple providers, primary care providers, and specialists exacerbates the technology divide. | ||
Inadequate program supervision/management | 3 | Lack of personnel structure to oversee administrators of technology platforms. | ||
Organizational drivers | ||||
Inadequate decision support | ||||
Technology problems | 0 | Not reported. | ||
Inadequate administrative support | ||||
Insufficient administrative support staff to deliver or support telehealth solutions | 5 | Lack of support staff available to help access and utilize the different app platforms. | ||
Health systems workflow | 1 | Poor integration of telehealth intervention and existing clinical care workflows. | ||
Inadequate clinical support | ||||
Insufficient staff to deliver telehealth solutions | 5 | Lack of clinical support, such as providers or lay health workers, available to assist with the delivery of telehealth platforms. | ||
System-level barriers | ||||
Legal (eg, inadequate safeguards to ensure patient-guided confidential protected health information sharing, state licensure laws, need for credentialing at multiple sites, and liability concerns) | 2 | Lack of protection for study participants, specifically those who were undocumented immigrants. | ||
Financial (eg, limited insurance coverage for telehealth) | 5 | Inconsistent reimbursement coverage for providers. State-based licensure laws exist. | ||
Leadership drivers | ||||
Inadequate administrative leadership | 0 | Not reported. | ||
Inadequate clinical leadership (eg, concerns regarding lower quality patient–physician relationship or poor physician buy-in) | 0 | Not reported. | ||
Partnership | 1 | Partnership with national organizations to help design and implement the intervention. |
Facilitators (ie, solutions or improvements) | Facilitators identified, n | Example | ||
Competency drivers | ||||
Improvements in participant engagement (eg, approaches to address social needs and facilitate telehealth adoption) | ||||
Provision of access to telecommunication technology | 4 | Case managers assisted patients with addressing telehealth access barriers. | ||
Patient-centered design (eg, telehealth solutions) actively guided by patients to meet community needs | 20 | Most studies engaged patients in the design of the intervention and obtained feedback from the target population. | ||
Cultural tailoring of telehealth solutions to meet the needs of specific communities and populations | 13 | Culturally tailored interventions for target populations. | ||
Improvements in staff training (eg, training in patient empowerment techniques such as motivational interviewing to facilitate telehealth adoption) | 4 | Technology training for providers and community health workers. Help solving workflow issues. | ||
Organizational drivers | ||||
Decision support solutions | ||||
Technology solutions | 0 | Not reported. | ||
Administrative solutions | ||||
Strengthening of administrative support staff to support telehealth solution | 11 | Support staff to provide ongoing support and engagement for technical issues. | ||
Process evaluation | 1 | Feasibility assessments to determine the fidelity of the technology. | ||
Clinical support team solutions | ||||
Strengthening clinical support staff to deliver or support telehealth solutions (eg, delivering telehealth through trusted intermediaries) | 5 | Support staff, such as nurses, providers, and lay health workers, are available to provide human interaction, which strengthens trust with technology. | ||
System-level improvements | ||||
Legal (eg, safeguards to ensure patient-guided confidential sharing of protected health information) | 2 | Reassurance for securing data using appropriate computer technology. | ||
Financial (eg, partnering with payers to expand insurance coverage for telehealth and telehealth reimbursement) | 4 | Working with communities to provide coverage for community health aides to help implement technology and to better understand telehealth insurance claims. | ||
Legislative policies | 1 | Experience working with state legislature/licensure to help lower barriers to telehealth. | ||
Leadership drivers | ||||
Improvements in administrative leadership | 0 | Not reported. | ||
Improvements in clinical leadership (eg, engagement of strong physician champions to lead change) | 0 | Not reported. |
Major themes identified across these studies included the importance of patient-centered design, cultural tailoring of telehealth solutions, delivering telehealth through trusted intermediaries, partnering with payers to expand telehealth reimbursement, and ensuring confidential sharing of private information.
Illustrative examples of these key themes are reviewed in the sections below, and key system- and policy-level changes needed to address barriers to telehealth adoption and implementation are identified.
Patient-Centered Design
Patient engagement is a core component of PCORI studies. Some studies go beyond incorporating patient input and feedback on the research and actively engage patients into the study design and overall design of the intervention.
The VIGoROUS (Video Game Rehabilitation for Outpatient Stroke) study illustrates how user-friendly consumer-driven design serves as a critical component of sustainable telehealth initiatives [
, ]. It employed a video gaming system to deliver therapeutic exercises and track adherence/progress remotely for individuals with motor disability. The gaming system was co-designed by therapists and people with motor disability. Artificial intelligence automatically adjusted the difficulty of the exercises to match the ability of the patient, allowing for a simple user interface and less than 1-minute setup.Similarly, the study by Williams et al comparing high-touch and high-tech care in patients with complex care needs utilized a smartphone-based remote care management platform and digital health innovations to support self-directed care management [
, ]. While the participant technology comfort level was assessed prerandomization and was expected to be variable among the study population, the study team found that participants often requested assistance in understanding basic phone functionality. Thus, the investigators worked to devise a modality flexible enough to support all levels of technical knowledge, which was easy to access, was user friendly, and addressed a broad range of skills from simple to difficult. The investigators developed a high-tech care strategy user guide with input from the stakeholder partners, patients, and providers. The need for more tailored technical support also resulted in concierge-style in-person technical visits to perform functional assessments that could not be completed over the phone, leading to a new workforce supporting additional technical initiatives across the care delivery continuum.Cultural Tailoring of Telehealth Solutions
When developing telehealth interventions, it is particularly important to consider the need for culturally competent care, defined as care that respects the diverse characteristics and values of the patients, which can ultimately affect their overall health and health care [
]. This review of PCORI-funded studies showed that patient study engagement improved with the use of culturally competent messaging delivered via telehealth interventions.The MODEL (Management of Diabetes in Everyday Life) study assessed the effectiveness of tailored motivational text messaging for improving diabetes self-care activities in medically underserved African American adults in the Mid-South, and illustrated the importance of cultural tailoring of telehealth solutions [
, ]. This study incorporated feedback from a patient advisory council into the message development process to more competently culturally tailor message content to the target population, an approach that has demonstrated positive impacts in other populations with diabetes [ , ]. The MODEL study’s registry-based text message system employed electronic medical record and survey data, as well as ongoing message input from patients themselves to personalize and deliver culturally tailored content to patients with uncontrolled diabetes [ ]. Preliminary program results indicated high satisfaction with the MODEL text message program, with greater than 90% participant retention in this group of the study.Similarly, the Health-E You/SaludiTu study demonstrated how cultural tailoring of telehealth interventions can improve their acceptance by populations at risk of disparities [
, ]. This study aimed to assess the effectiveness of a patient-centered computer-based clinical intervention to reduce health disparities in unintended pregnancies among Latina adolescent girls in 18 school-based health centers in Los Angeles, California. Latina adolescents and local community representatives were key stakeholders in the design and implementation of this app, which may have helped the adoption and implementation of this intervention into clinic workflows at the school-based health centers. The culturally tailored tool was available in English and Spanish, and adapted language that resonated with Latina adolescents (eg, the use of the more colloquial term condones versus preservativos to refer to condoms). Preliminary findings showed that both providers and study participants reported high satisfaction with the app, as it was easily integrated into the clinic flow, and more importantly, participants showed an increase in knowledge of contraceptive use from baseline to their 6-month follow-up.Likewise, early implementation experience from a PCORI-funded study of underserved Asian Americans having chronic hepatitis B suggested that culturally and linguistically competent interventions improve participant retention [
]. This study employed patient-designed culturally tailored text messaging along with patient navigator–led educational sessions to improve adherence to medication therapy for hepatitis B. Patient partners and community stakeholder groups provided extensive input on the appropriateness and framing of the messages. These messages were delivered in Chinese, Vietnamese, or Korean. Early implementation findings showed evidence that this culturally and linguistically appropriate intervention was effective in improving follow-up in patients living with hepatitis B.Delivering Telehealth Through Trusted Intermediaries
Many of the studied PCORI telehealth projects successfully employed trusted intermediaries to deliver telehealth solutions and help support people at high risk for health disparities in crossing the digital divide. These projects provided confirmatory evidence to extensive literature suggesting that lay health worker and paraprofessional intermediaries can help build trust among patients and providers in new telehealth modalities. New technologies, including telehealth, that enable the delivery of interventions and diagnostics remotely are emerging, but patients commonly struggle to use these technologies. Patients struggle with both simple technical issues, such as neglecting to ensure that an electronic system is plugged in, and more complex barriers related to social determinants of health (eg, poverty, unpredictable work schedules, unreliable transportation, and lack of internet access) when using advanced technologies such as secure videoconferencing.
PCORI telehealth projects consistently demonstrate high patient acceptance of telehealth modalities that connect them to trusted providers, but even with careful attention to patient-centered design, investigators commonly found availability of in-person technological support to be essential. For example, when 1 participant in the aforementioned VIGoROUS video gaming intervention reported that her gaming system would not turn on, a support call revealed that she had not plugged it into an electrical outlet [
, ]. This early experience suggested that widescale implementation of telehealth technologies requires funding of responsive technical support infrastructure to avoid technology abandonment by patients in greatest need.Similarly, a study of Native Americans addressing gaps in self-management through home-based CKD care found that continuous care coordination using trained and trusted tribal community health representatives (CHRs) was essential to effective program implementation [
- ]. This program used videoconferencing with patients via internet-connected portable tablets using Verizon wireless Jetpack mobile hotspots. The program worked with other local health and wellness programs (eg, Indian Health Services and other programs operating within tribal communities) and conducted community-wide health fairs. Telehealth adoption was also encouraged through distribution of culturally sensitive newsletters and brochures to all these health programs and individual community members. The secondary data analysis of the study demonstrated that behavioral and lifestyle educational reinforcement through motivational text messaging and alternate weekly home visits by the CHRs with quarterly group sessions was an effective means of providing care to patients with type 2 diabetes mellitus and CKD who may otherwise avoid diagnosis and treatment due to stigmatization. The CHR-led home-based care model provided the additional care necessary to bolster patient levels of disease-specific knowledge, self-efficacy, and diabetes and CKD self-management, enabling patients to more effectively carry out the recommendations that they received during the home visits, as compared to patients who received clinic-based usual care [ ].A study addressing childhood hearing loss in an Alaska Native population similarly found that using trusted lay health workers based in the communities served was essential to telehealth program deployment [
- ]. In the remote rural communities served by this program, health care is provided by community health aides/practitioners who are commonly from the community they serve and are selected by their community to receive training. This care is supported by specialist triage using the statewide telemedicine network. As a result, ear and hearing problems in isolated communities are routinely managed remotely by audiologists and surgeons through telemedicine, including provision of local care (such as for ear infections), preoperative planning, and postoperative follow-up [ , ]. This requires community health aides/practitioners performing telemedicine consults to be proficient with telemedicine equipment, including basic testing, such as tympanometry, to assess eardrum movement and digital otoscopy for photos of the eardrum. Training with telemedicine equipment is built into each session of the standardized community health aide/practitioner program and includes continued on-the-job training and direct observation of practical skills. Expanding the use of telemedicine equipment for prevention, such as follow-up for referred school hearing screenings, utilizes community health aide/practitioner knowledge and training on the telemedicine infrastructure already in place. However, Emmett et al also provided additional training on the expedited telemedicine consult form to facilitate the application of modified workflows for school hearing screening referrals [ , ].Partnering With Payers to Expand Telehealth Reimbursement
Reimbursement for telehealth encounters is increasing. However, lack of adequate reimbursement continues to present a major challenge to telehealth adoption, implementation, and sustainability. Reimbursement barriers to telehealth were acknowledged by nearly all the PCORI-funded study investigators who participated in this study. Despite recent expansions in telehealth reimbursement during the COVID-19 pandemic, variable state and federal reimbursement policies and low levels of reimbursement for services delivered via telehealth continue to challenge telehealth implementation efforts. In response to these challenges, many studies found it essential to partner with payers to expand their support and/or reimbursement for telehealth services.
For example, the TEAM UP (Treatment Efforts Addressing Child Weight Management by Unifying Patients, Parents, and Providers) study assembled a Payer Advisory Board that explored how military medical sites use telehealth for services such as teleradiology and telebehavioral health [
]. The Board noted how telehealth could facilitate access to qualified and trained health care professionals in rural areas, as well as foster treatment engagement and reduce program attrition by increasing flexibility for patients with challenging schedules. The Payer Advisory Board stressed the viability of telehealth and the need for these types of visits to be considered billable encounters to help overcome the challenges facing populations with barriers to care.In the rural Alaska study addressing childhood hearing loss, the research team developed a streamlined telemedicine consultation process in order to make it financially sustainable in the school setting [
, , ]. Translating a clinically oriented technology to a preventive service raises inherent logistical challenges [ ]. The team shortened the telemedicine consult time from 60-90 minutes to 10-15 minutes, making it feasible to perform 10 to 15 telemedicine referrals for school hearing screening in a single day. This process took several iterations, with feedback sought from multiple stakeholders including community health aides/practitioners. Maintaining the integrity of billing requirements, while shortening the telemedicine consultation process, allowed the preventive service to remain reimbursable and financially sustainable after the conclusion of the randomized trial.Similarly, in the Mid-South MODEL study, investigators worked with regional health systems to provide long-term support for a regional registry-based tailored text message system that can assist both large health systems and small independent primary care practices in sending regular motivational messages to interested patients [
, ].Ensuring Confidential Sharing of Private Information
Early implementation experience from numerous PCORI telehealth studies demonstrates the importance of ensuring confidential sharing of private information in all telehealth interventions considered. The application of telehealth often involves paperless collection and transmission of data at each touch point with participants and with communication between staff at the regional and local levels. This requires careful attention to compliance with local, regional, and federal legal requirements for protection of privacy.
For example, the TEAM UP study of alternative childhood obesity treatments for Medicaid-insured individuals employed a secure Health Insurance Portability and Accountability Act (HIPAA)-compliant videoconferencing system to enable remote communication with children and families that protected privacy [
]. Working with a low-income population presents an inherent set of barriers, including unpredictable schedules and unreliable transportation, both of which could negatively impact the fidelity of family-based behavioral treatment delivery. In anticipation of these potential barriers, the study team contracted with a fully HIPAA-compliant videoconferencing system to allow for telehealth delivery of family-based behavioral treatment sessions as needed. Secure video was found to be an ideal technology solution since it is accessible on virtually all devices, including cell phones, and provides necessary flexibility to support adherence with demanding treatment schedules. Use of secure HIPAA-compliant videoconferencing ensured the confidentiality and security of participants’ protected health information in compliance with part 11 of Title 21 of the Code of Federal Regulations pertaining to Electronic Records and Electronic Signatures (21 CFR Part 11) [ ]. Likewise, the study employed an electronic consent process that allows for the use of alternative consent forms for each participating site.The VIGoROUS study also had to address challenges related to information security [
]. First, it was necessary to find a HIPAA-compliant videoconferencing platform for secure video chat between participants and providers. While there are numerous platforms available, some of which are free, the lead site’s hospital system was relatively unfamiliar with telehealth and thus required new technology platforms to undergo an extensive information security vetting process (eg, conducting risk assessments and obtaining administrative approvals), a process that ultimately spanned 3 years. To proceed with the study, investigators were able to identify and pivot to a user-friendly videoconferencing platform (Bluejeans) that had already been vetted by the university health system. The second challenge was how to make large amounts of remote-monitoring data (adherence data, quality of movement data generated continuously during rehabilitation, and behavioral assessment data) accessible to a clinician providing care remotely. This was addressed by streaming deidentified data to a secure cloud server. The data were stored under a unique machine identifier that could only be obtained through physically accessing the rehabilitation gaming system (located within participants’ homes). During video consultations with a therapist, the participant provided the therapist with the machine identifier to enable the therapist to track his/her progress [ ].Discussion
Principal Findings
This scoping review of early PCOR evidence suggests that the most effective health system- and provider-level telehealth implementation solutions to address disparities employ patient-centered and culturally tailored telehealth solutions. We found that the development of the most effective telehealth solutions was actively guided by patients themselves in order to meet the needs of specific communities and populations. Early PCOR evidence demonstrates that the best practices in telehealth implementation include delivering telehealth through trusted intermediaries, close partnership with payers to facilitate reimbursement and sustainability, and safeguards to ensure patient-guided confidential sharing of personal health information.
The COVID-19 pandemic has given new urgency to these questions regarding telehealth adoption among people at the highest risk for disparities, including those living in medically underserved, low-income, or rural areas; those from racial and ethnic minorities; LGBT persons; and those with limited English proficiency or disabilities. Telehealth modalities have great potential to help overcome these geographic, socioeconomic, cultural, and language barriers and to give populations at risk of disparities enhanced access to essential health services. Although many PCORI-funded studies had to put their research on hold and transfer their institution’s focus on the needs of patients experiencing COVID-19 [
], the pandemic generally helped to accelerate telehealth adoption for vulnerable populations [ , ]. However, since early evidence indicates that differences in internet and telehealth access may worsen disparities in chronic disease and COVID-19 outcomes [ ], it is clear that telehealth solutions must deliberately target and prioritize populations at the highest risk for disparities.Toward that end, this review places strong emphasis on the importance of deploying patient-centered and culturally tailored telehealth solutions that address the social determinants of health faced by populations at risk for disparities [
- ]. Even though low-income, minority, and rural populations face heightened barriers to effective self-care related to the social determinants of health, most telehealth behavioral interventions for chronic conditions are tailored for individuals with higher incomes. Thus, low-income, minority, and rural groups need specific culturally tailored solutions that specifically address the social determinants of health they face [ , ]. Preliminary evidence from PCORI-funded studies consistently demonstrates that intensive personalization and cultural tailoring of intervention components [ ] can help enable vulnerable patients to address critical social determinants of health [ ]. Effective telehealth interventions need to address the key social determinants of health at the root of entrenched health behaviors in systematic ways [ ]. Most of the reviewed PCOR initiatives employed component interventions that were extensively culturally tailored during the initial program planning phases and on an ongoing basis to ensure that they were culturally congruent and appropriate based on the subjective culture (ie, norms and attitudes), behavioral preferences, and cultural values and expectations of the population served [ , ]. Thus, although cultural tailoring alone is not sufficient, the early PCOR literature indicates that it is an essential component of effective telehealth solutions to address health disparities.Furthermore, this review revealed that telehealth solutions can take advantage of and expand on existing technological capacities accessible to low-income and other populations at risk for health disparities, and can often be deployed at low cost. The existing PCORI-funded telehealth research suggests that low-cost strategies that employ existing telehealth capacities using standard mobile phones and smartphones may be particularly effective in reaching patients where they are and engaging them in needed care. Although more than 90% of Americans overall carry cell phones and 80% have smartphones [
], among people with household incomes less than US $30,000, only 71% own a smartphone, 54% own some type of computer, and 56% have home broadband [ ]. As demonstrated by several of our case studies above, early evidence indicates that text messaging and other culturally tailored mobile health (mHealth) interventions can provide effective low-cost approaches for engaging patients in self-care [ - ]. Some systematic reviews have identified text messaging as among the most effective low-cost technological strategies for engaging patients in behavior change and have highlighted the importance of such technology-supported behavioral interventions [ - ]. Thus, health systems seeking to target telehealth solutions to address disparities should actively seek to employ existing technologies and devices like traditional cell phones to which populations at risk for disparities already have access.This review also emphasizes the importance of telehealth solutions that work through existing trusted care networks and providers, and that employ trusted lay health worker and paraprofessional intermediaries to introduce and support the use of these technologies by people from the community. Many of the most effective PCORI telehealth initiatives considered in this paper focused their interventions around engaging, training, and deploying trusted intermediaries from the community and/or the population at risk for disparities to be served. This evidence also suggests that personalized human interaction is essential. For example, text messaging interactions were found to be most effective when delivered in real-time, tailored to participant interests and needs, and originated from a trusted known source. The PCORI-funded telehealth projects reviewed suggested that face-to-face videoconferencing and telephone interventions with trusted caregivers are particularly effective.
Our review further suggests that substantive reimbursement and regulatory changes are immediately needed to enable low-cost and efficient telehealth solutions. We found that the best practices in telehealth implementation include close partnership with payers to facilitate reimbursement and sustainability. However, ultimately, experience from the existing portfolio of PCORI telehealth projects indicates that permanent changes are needed in national reimbursement policies and regulations to facilitate broader telehealth adoption and implementation. Policies and reimbursement for telehealth were rapidly put into place to allow health care to continue during the severe restrictions early in the COVID-19 public health emergency after March 2020, in order to allow care to continue during the COVID-19 pandemic. However, these changes clearly need to be extended. Needed regulatory changes include expanded reimbursement as well as relaxations of regulations around where the patient or practitioner is located, licensure, and reciprocity across states. For example, since populations at risk for disparities generally have less access to the internet and technology, regulatory solutions need to facilitate and enable the use of existing lower-cost technologies, such as telephone visits using traditional cell phones, rather than requiring video teleconferencing for medical billing. As the pandemic eases, it will be essential to carefully think about how to preserve the progress that was made in expanding telehealth in ways that improve patient care at the federal (eg, Medicare), state, and individual insurance company levels [
- ].Strengths and Limitations
The existing PCORI telehealth research targeting populations at risk for health disparities is of very high quality and has high applicability to real-world clinical and community settings. Of note, 100% of the PCORI-funded studies in this category employed a randomized controlled trial design and 100% were pragmatic rather than being conducted in highly controlled university settings among highly selected patients. This is a major strength of both the research reviewed in this scoping review and the scoping review itself. Furthermore, the studies included in this scoping review are also notable for their generalizability, given their pragmatic real-world approach. Rather than excluding patients with multiple chronic conditions as is often done in randomized clinical trials, many of the PCORI-funded studies specifically included patients with multiple morbidities given their representativeness of patients most commonly seen in real-world clinical settings.
This review is limited by the small numbers of studies that were considered, and its main findings are subject to numerous potential biases. First, since PCORI staff chose studies for an in-depth review based on their perception regarding which study teams overcame notable obstacles to telehealth, selection bias could have occurred. Our expectation is that this bias would most likely lead to the overemphasis of facilitators and solutions as opposed to barriers and challenges. Second, because PCORI staff and investigators comprised the study team, it is possible that there was some bias toward reporting of positive study findings. However, the study team and methods actively sought to address this potential limitation by identifying and categorizing barriers first prior to the discussion of solutions. In addition, some of the telehealth services provided in these studies may be difficult to consistently replicate outside of structured evaluations.
The review is also limited because final study results are not available for many of the included studies, and even fewer have detailed analyses of implementation factors and lessons learned available in the peer-reviewed literature. However, the review was facilitated by the detailed project knowledge of PCORI staff and the investigators themselves.
Future Directions
Over the past 5 years, PCORI has supported many comparative effectiveness studies of adopting virtual care solutions to manage health outcomes across a wide range of populations at risk for disparities. Through these research initiatives, health care delivery systems are learning to leverage low-cost telehealth solutions to promote better health outcomes by increasing access to care, making care more effective, and continuously engaging patients. PCORI-funded comparative effectiveness research is demonstrating how we can reduce health disparities across the spectrum of disease conditions and populations through low-cost telehealth solutions. This research is identifying key barriers and limitations to implementing telehealth in vulnerable populations and ways to overcome these barriers to help improve health and health care outcomes.
In this new era of the COVID-19 pandemic, there has been rapid growth in the use of technology across all industries, but it is proving to be particularly important in the areas of health and health care. Future efforts should focus on fostering collaborations between researchers, payers, and administrators to facilitate widespread adoption of new treatment models that have already demonstrated success through definitive clinical trials. Our review indicates that key system- and policy-level changes are desperately needed to address barriers to telehealth reimbursement. Moreover, further pragmatic research is needed to demonstrate the best approaches for disseminating and scaling these early telehealth findings in broader community practice to promote population health.
Conclusions
This scoping review gives strong guidance to health systems seeking to target evidence-based telehealth modalities for health disparities. Results from the studies demonstrate that systems can do so in innovative ways, and in fact, some of the studies identify methods for systems to effectively reach underserved populations in their own communities. These case studies also highlight the critical importance of both supportive infrastructure, and regulatory and reimbursement policies to facilitate telehealth modalities and make them sustainable.
However, despite growing evidence that telehealth can deliver care to people at risk for disparities, many major obstacles exist for its implementation. Early PCOR evidence demonstrates that the most effective health system- and provider-level solutions for at-risk populations use patient-centered and culturally tailored telehealth solutions whose development is actively guided by the patients themselves.
Acknowledgments
The authors gratefully acknowledge the many Patient-Centered Outcomes Research Institute (PCORI) study participant patient advisors who contributed their substantial expertise on patient experience to this scoping review. Without their participation and insights, this study would not have been possible. We are also very thankful for the contributions of the numerous PCORI-funded investigators and teams whose studies are cited in this scoping review. Lastly, we specifically acknowledge the editorial contributions of James Schuster, MD, at the University of Pittsburgh Medical Center, who also serves on the Board of Governors of the PCORI.
Conflicts of Interest
All listed authors are either investigators associated with Patient-Centered Outcomes Research Institute (PCORI)-funded studies in its specific telehealth portfolio, PCORI employees, or PCORI contractors/consultants. All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of PCORI, or its Board of Governors or Methodology Committee.
Patient-Centered Outcomes Research Institute–funded studies assessing the use of telehealth to improve outcomes for populations at risk for health or health care disparities.
DOCX File , 38 KBReferences
- James CV, Moonesinghe R, Wilson-Frederick SM, Hall JE, Penman-Aguilar A, Bouye K. Racial/Ethnic Health Disparities Among Rural Adults - United States, 2012-2015. MMWR Surveill Summ 2017 Nov 17;66(23):1-9 [FREE Full text] [CrossRef] [Medline]
- Bach PB. Book Review Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care Edited by Brian D. Smedley, Adrienne Y. Stith, and Alan R. Nelson. 764 pp. Washington, D.C., National Academies Press, 2003. $79.95. 0-309-08532-2. N Engl J Med 2003 Sep 25;349(13):1296-1297. [CrossRef] [Medline]
- Semega J, Kollar M, Shrider E, Creamer J. Income and Poverty in the United States: 2019. United States Census Bureau. URL: https://www.census.gov/library/publications/2020/demo/p60-270.html [accessed 2020-09-24]
- Preventing Chronic Diseases and Promoting Health in Rural Communities. Centers for Disease Control and Prevention. URL: https://www.cdc.gov/chronicdisease/resources/publications/factsheets/rural-health.htm [accessed 2020-12-15]
- Bolin JN, Bellamy GR, Ferdinand AO, Vuong AM, Kash BA, Schulze A, et al. Rural Healthy People 2020: New Decade, Same Challenges. J Rural Health 2015;31(3):326-333. [CrossRef] [Medline]
- Blewett LA, Johnson PJ, Lee B, Scal PB. When a usual source of care and usual provider matter: adult prevention and screening services. J Gen Intern Med 2008 Sep 28;23(9):1354-1360 [FREE Full text] [CrossRef] [Medline]
- Kim MY, Kim JH, Choi I, Hwang IH, Kim SY. Effects of having usual source of care on preventive services and chronic disease control: a systematic review. Korean J Fam Med 2012 Nov;33(6):336-345 [FREE Full text] [CrossRef] [Medline]
- Keppel KG. Ten largest racial and ethnic health disparities in the United States based on Healthy People 2010 Objectives. Am J Epidemiol 2007 Jul 01;166(1):97-103. [CrossRef] [Medline]
- Conway BN, Han X, Munro HM, Gross AL, Shu X, Hargreaves MK, et al. The obesity epidemic and rising diabetes incidence in a low-income racially diverse southern US cohort. PLoS One 2018;13(1):e0190993 [FREE Full text] [CrossRef] [Medline]
- Ogden CL, Carroll MD, Fakhouri TH, Hales CM, Fryar CD, Li X, et al. Prevalence of Obesity Among Youths by Household Income and Education Level of Head of Household - United States 2011-2014. MMWR Morb Mortal Wkly Rep 2018 Feb 16;67(6):186-189 [FREE Full text] [CrossRef] [Medline]
- Hales CM, Carroll MD, Fryar CD, Ogden CL. Prevalence of Obesity Among Adults and Youth: United States, 2015-2016. NCHS Data Brief 2017 Oct(288):1-8 [FREE Full text] [Medline]
- Dewees S, Marks B. Twice Invisible: Understanding Rural Native America. First Nations Development Institute Research Note. 2017. URL: https://www.usetinc.org/wp-content/uploads/bvenuti/WWS/2017/May%202017/May%208/Twice%20Invisible%20-%20Research%20Note.pdf [accessed 2021-10-10]
- Marrone S. Understanding barriers to health care: a review of disparities in health care services among indigenous populations. Int J Circumpolar Health 2007 Jun 15;66(3):188-198. [CrossRef] [Medline]
- Disparities. Indian Health Services. URL: https://www.ihs.gov/newsroom/factsheets/disparities/ [accessed 2021-02-22]
- Hafeez H, Zeshan M, Tahir MA, Jahan N, Naveed S. Health Care Disparities Among Lesbian, Gay, Bisexual, and Transgender Youth: A Literature Review. Cureus 2017 Apr 20;9(4):e1184 [FREE Full text] [CrossRef] [Medline]
- Hickey KT, Masterson Creber RM, Reading M, Sciacca RR, Riga TC, Frulla AP, et al. Low health literacy: Implications for managing cardiac patients in practice. Nurse Pract 2018 Aug;43(8):49-55 [FREE Full text] [CrossRef] [Medline]
- Proctor K, Wilson-Frederick SM, Haffer SC. The Limited English Proficient Population: Describing Medicare, Medicaid, and Dual Beneficiaries. Health Equity 2018 Jun;2(1):82-89 [FREE Full text] [CrossRef] [Medline]
- Sentell T, Braun KL. Low health literacy, limited English proficiency, and health status in Asians, Latinos, and other racial/ethnic groups in California. J Health Commun 2012;17 Suppl 3:82-99 [FREE Full text] [CrossRef] [Medline]
- Butkus R, Rapp K, Cooney TG, Engel LS. Envisioning a Better U.S. Health Care System for All: Reducing Barriers to Care and Addressing Social Determinants of Health. Ann Intern Med 2020 Jan 21;172(2_Supplement):S50-S59. [CrossRef]
- Bierman AS, Dunn JR. Swimming upstream. Access, health outcomes, and the social determinants of health. J Gen Intern Med 2006 Jan;21(1):99-100 [FREE Full text] [CrossRef] [Medline]
- Hill-Briggs F, Adler NE, Berkowitz SA, Chin MH, Gary-Webb TL, Navas-Acien A, et al. Social Determinants of Health and Diabetes: A Scientific Review. Diabetes Care 2020 Nov 02;44(1):258-279. [CrossRef] [Medline]
- Syed ST, Gerber BS, Sharp LK. Traveling towards disease: transportation barriers to health care access. J Community Health 2013 Oct;38(5):976-993 [FREE Full text] [CrossRef] [Medline]
- Syed ST, Sharp LK, Kim Y, Jentleson A, Lora CM, Touchette DR, et al. Relationship Between Medication Adherence and Distance to Dispensing Pharmacies and Prescribers Among an Urban Medicaid Population with Diabetes Mellitus. Pharmacotherapy 2016 Jun 28;36(6):590-597 [FREE Full text] [CrossRef] [Medline]
- Huilgol Y, Joshi A, Carr B, Hollander J. Giving Urban Health Care Access Issues The Attention They Deserve In Telemedicine Reimbursement Policies. Health Affairs Blog. 2017. URL: https://www.healthaffairs.org/do/10.1377/hblog20171022.713615/full/ [accessed 2021-11-12]
- Dorsey ER, Topol EJ. State of Telehealth. N Engl J Med 2016 Jul 14;375(2):154-161 [FREE Full text] [CrossRef] [Medline]
- Demeke HB, Merali S, Marks S, Pao LZ, Romero L, Sandhu P, et al. Trends in Use of Telehealth Among Health Centers During the COVID-19 Pandemic - United States, June 26-November 6, 2020. MMWR Morb Mortal Wkly Rep 2021 Feb 19;70(7):240-244 [FREE Full text] [CrossRef] [Medline]
- Rodriguez-Guzman E, Aktaruzzaman U, Baker A, DiBitetto K, Lima A, Lindros J, et al. Engaging key stakeholders in a pragmatic, family-centered childhood obesity study in primary care practices. Pediatrics 2020;146(Issue 1_MeetingAbstract):84-85. [CrossRef]
- Rodriguez JA, Betancourt JR, Sequist TD, Ganguli I. Differences in the use of telephone and video telemedicine visits during the COVID-19 pandemic. Am J Manag Care 2021 Jan 14;27(1):21-26 [FREE Full text] [CrossRef] [Medline]
- Kvedar J, Coye MJ, Everett W. Connected health: a review of technologies and strategies to improve patient care with telemedicine and telehealth. Health Aff (Millwood) 2014 Feb;33(2):194-199. [CrossRef] [Medline]
- Dellifraine JL, Dansky KH. Home-based telehealth: a review and meta-analysis. J Telemed Telecare 2008;14(2):62-66. [CrossRef] [Medline]
- Polisena J, Tran K, Cimon K, Hutton B, McGill S, Palmer K. Home telehealth for diabetes management: a systematic review and meta-analysis. Diabetes Obes Metab 2009 Oct;11(10):913-930. [CrossRef] [Medline]
- Chi N, Demiris G. A systematic review of telehealth tools and interventions to support family caregivers. J Telemed Telecare 2015 Jan;21(1):37-44 [FREE Full text] [CrossRef] [Medline]
- Vadheim LM, Patch K, Brokaw SM, Carpenedo D, Butcher MK, Helgerson SD, et al. Telehealth delivery of the diabetes prevention program to rural communities. Transl Behav Med 2017 Jun;7(2):286-291 [FREE Full text] [CrossRef] [Medline]
- Xu T, Pujara S, Sutton S, Rhee M. Telemedicine in the Management of Type 1 Diabetes. Prev Chronic Dis 2018 Jan 25;15:E13 [FREE Full text] [CrossRef] [Medline]
- Bhatt J, Bathija P. Ensuring Access to Quality Health Care in Vulnerable Communities. Acad Med 2018 Sep;93(9):1271-1275 [FREE Full text] [CrossRef] [Medline]
- Arksey H, O'Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology 2005 Feb;8(1):19-32. [CrossRef]
- What and Who We Fund. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/funding-opportunities/what-who-we-fund [accessed 2020-09-23]
- Fixsen D, Blase K, Naoom S, Duda M. Implementation Drivers: Assessing Best Practices. Chapel Hill, NC: National Implementation Science Network; 2013.
- Fixsen D, Naoom S, Blase K, Friedman R, Wallace F. Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network; 2005.
- Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med 2018 Oct 02;169(7):467-473 [FREE Full text] [CrossRef] [Medline]
- Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009 Jul 21;339:b2535 [FREE Full text] [CrossRef] [Medline]
- Armstrong AW, Chambers CJ, Maverakis E, Cheng MY, Dunnick CA, Chren M, et al. Effectiveness of Online vs In-Person Care for Adults With Psoriasis: A Randomized Clinical Trial. JAMA Netw Open 2018 Oct 05;1(6):e183062 [FREE Full text] [CrossRef] [Medline]
- Ford AR, Gibbons CM, Torres J, Kornmehl HA, Singh S, Young PM, et al. Access to Dermatological Care with an Innovative Online Model for Psoriasis Management: Results from a Randomized Controlled Trial. Telemed J E Health 2019 Jul;25(7):619-627 [FREE Full text] [CrossRef] [Medline]
- Bailey JE, Surbhi S, Gatwood J, Butterworth S, Coday M, Shuvo SA, et al. The management of diabetes in everyday life study: Design and methods for a pragmatic randomized controlled trial comparing the effectiveness of text messaging versus health coaching. Contemp Clin Trials 2020 Sep;96:106080. [CrossRef] [Medline]
- Gatwood J, Shuvo S, Ross A, Riordan C, Smith P, Gutierrez ML, et al. The Management of Diabetes in Everyday Life (MODEL) program: development of a tailored text message intervention to improve diabetes self-care activities among underserved African-American adults. Transl Behav Med 2020 Feb 03;10(1):204-212. [CrossRef] [Medline]
- Blalock SJ, Solow EB, Reyna VF, Keebler M, Carpenter D, Hunt C, et al. Enhancing Patient Understanding of Medication Risks and Benefits. Arthritis Care Res (Hoboken) 2020 Aug 16:1-26. [CrossRef] [Medline]
- Remote Cognitive Behavior Therapy for Major Depression in Primary Care. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2018/remote-cognitive-behavior-therapy-major-depression-primary-care [accessed 2020-10-25]
- Comparing Two Ways to Improve Depression among African-American and/or Latino LGBTQ Adults. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2016/comparing-two-ways-improve-depression-among-african-american-andor-latino [accessed 2020-10-25]
- Coker TR, Porras-Javier L, Zhang L, Soares N, Park C, Patel A, et al. A Telehealth-Enhanced Referral Process in Pediatric Primary Care: A Cluster Randomized Trial. Pediatrics 2019 Mar 15;143(3):e20182738. [CrossRef] [Medline]
- Contreras S, Porras-Javier L, Zima BT, Soares N, Park C, Patel A, et al. Development of a Telehealth-Coordinated Intervention to Improve Access to Community-Based Mental Health. Ethn Dis 2018 Sep 06;28(Supp):457-466. [CrossRef]
- Comparing Ways to Help Sexual and Gender Minority Male Survivors of Sexual Abuse Reduce Distress and Engage in Treatment. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2018/comparing-ways-help-sexual-and-gender-minority-male-survivors-sexual-abuse [accessed 2020-12-15]
- Comparing Two Methods to Deliver a Health Education Program for Pregnant Women Treated for Opioid Use Disorder in Rural Kentucky -- The PATH Home Trial. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2018/comparing-two-methods-deliver-health-education-program-pregnant-women-treated [accessed 2020-10-25]
- Dehlendorf C, Fitzpatrick J, Fox E, Holt K, Vittinghoff E, Reed R, et al. Cluster randomized trial of a patient-centered contraceptive decision support tool, My Birth Control. Am J Obstet Gynecol 2019 Jun;220(6):565.e1-565.e12. [CrossRef] [Medline]
- Comparing Three Ways to Obtain Hearing Aids for People with Age-Related Hearing Loss. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2019/comparing-three-ways-obtain-hearing-aids-people-age-related-hearing-loss [accessed 2020-12-15]
- Beck CA, Beran DB, Biglan KM, Boyd CM, Dorsey ER, Schmidt PN, Connect.Parkinson Investigators. National randomized controlled trial of virtual house calls for Parkinson disease. Neurology 2017 Sep 12;89(11):1152-1161 [FREE Full text] [CrossRef] [Medline]
- Dorsey ER, Achey MA, Beck CA, Beran DB, Biglan KM, Boyd CM, et al. National Randomized Controlled Trial of Virtual House Calls for People with Parkinson's Disease: Interest and Barriers. Telemed J E Health 2016 Jul;22(7):590-598 [FREE Full text] [CrossRef] [Medline]
- Achey MA, Beck CA, Beran DB, Boyd CM, Schmidt PN, Willis AW, et al. Virtual house calls for Parkinson disease (Connect.Parkinson): study protocol for a randomized, controlled trial. Trials 2014 Nov 27;15:465 [FREE Full text] [CrossRef] [Medline]
- Carroll JK, Tobin JN, Luque A, Farah S, Sanders M, Cassells A, et al. "Get Ready and Empowered About Treatment" (GREAT) Study: a Pragmatic Randomized Controlled Trial of Activation in Persons Living with HIV. J Gen Intern Med 2019 Sep 25;34(9):1782-1789 [FREE Full text] [CrossRef] [Medline]
- Fiscella K, Boyd M, Brown J, Carroll J, Cassells A, Corales R, et al. Activation of persons living with HIV for treatment, the great study. BMC Public Health 2015 Oct 16;15(1):1056 [FREE Full text] [CrossRef] [Medline]
- Fortney JC, Heagerty PJ, Bauer AM, Cerimele JM, Kaysen D, Pfeiffer PN, et al. Study to promote innovation in rural integrated telepsychiatry (SPIRIT): Rationale and design of a randomized comparative effectiveness trial of managing complex psychiatric disorders in rural primary care clinics. Contemp Clin Trials 2020 Mar;90:105873. [CrossRef] [Medline]
- Cerimele JM, LePoire E, Fortney JC, Hawrilenko M, Unützer J, Bauer AM. Bipolar disorder and PTSD screening and telepsychiatry diagnoses in primary care. Gen Hosp Psychiatry 2020 Jul;65:28-32. [CrossRef] [Medline]
- Comparing Online and In-Person Cognitive Behavioral Therapy in Youth with Anxiety Disorders -- The Kids FACE FEARS Study. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2018/comparing-online-and-person-cognitive-behavioral-therapy-youth-anxiety [accessed 2020-10-30]
- Pekmezaris R, Kozikowski A, Pascarelli B, Wolf-Klein G, Boye-Codjoe E, Jacome S, et al. A Telehealth-Delivered Pulmonary Rehabilitation Intervention in Underserved Hispanic and African American Patients With Chronic Obstructive Pulmonary Disease: A Community-Based Participatory Research Approach. JMIR Form Res 2020 Jan 31;4(1):e13197 [FREE Full text] [CrossRef] [Medline]
- Comparing telehealth to outpatient rehabilitation for Hispanic and African-American patients with chronic obstructive pulmonary disease (COPD). Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2016/testing-whether-referral-telehealth-program-helps-african-american-and [accessed 2020-10-29]
- Comparing Ways to Promote Quitting Smoking for People Referred for Lung Cancer Screenings. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2018/comparing-ways-promote-quitting-smoking-people-referred-lung-cancer-screenings [accessed 2020-12-15]
- Comparing Two Ways to Help Patients with Sickle Cell Disease Manage Pain -- CaRISMA. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2019/comparing-two-ways-help-patients-sickle-cell-disease-manage-pain [accessed 2020-12-15]
- Comparing Three Methods to Help Patients Manage Type 2 Diabetes. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2013/comparing-three-methods-help-patients-manage-type-2-diabetes [accessed 2020-10-30]
- Emmett SD, Robler SK, Wang N, Labrique A, Gallo JJ, Hofstetter P. Hearing Norton Sound: a community randomised trial protocol to address childhood hearing loss in rural Alaska. BMJ Open 2019 Jan 15;9(1):e023078 [FREE Full text] [CrossRef] [Medline]
- Emmett SD, Robler SK, Gallo JJ, Wang N, Labrique A, Hofstetter P. Hearing Norton Sound: mixed methods protocol of a community randomised trial to address childhood hearing loss in rural Alaska. BMJ Open 2019 Jan 22;9(1):e023081 [FREE Full text] [CrossRef] [Medline]
- Gauthier LV, Kane C, Borstad A, Strahl N, Uswatte G, Taub E, et al. Video Game Rehabilitation for Outpatient Stroke (VIGoROUS): protocol for a multi-center comparative effectiveness trial of in-home gamified constraint-induced movement therapy for rehabilitation of chronic upper extremity hemiparesis. BMC Neurol 2017 Jun 08;17(1):109 [FREE Full text] [CrossRef] [Medline]
- Comparing Ways to Treat Arm Weakness Due to Stroke. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2015/comparing-ways-treat-arm-weakness-due-stroke [accessed 2021-02-24]
- Testing a Program to Help Monitor Chronic Hepatitis B among Asian-American Patients. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2014/testing-program-help-monitor-chronic-hepatitis-b-among-asian-american-patients [accessed 2020-10-30]
- Improving Family-Centered Pediatric Trauma Care -- The Standard of Care versus the Virtual Pediatric Trauma Center. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2020/improving-family-centered-pediatric-trauma-care-standard-care-versus-virtual [accessed 2020-12-15]
- Margolis KL, Crain AL, Bergdall AR, Beran M, Anderson JP, Solberg LI, et al. Design of a pragmatic cluster-randomized trial comparing telehealth care and best practice clinic-based care for uncontrolled high blood pressure. Contemp Clin Trials 2020 May;92:105939. [CrossRef] [Medline]
- Comparing Transitional Care for Teens and Young Adults with Sickle Cell Disease with and without Peer Mentoring Sign Up for Updates. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2017/comparing-transitional-care-teens-and-young-adults-sickle-cell-disease-and [accessed 2020-11-11]
- Comparing Two Ways to Help African-American and Latino Patients with HIV to Receive and Retain Care. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2018/comparing-two-ways-help-african-american-and-latino-patients-hiv-receive-and [accessed 2020-12-15]
- Pekmezaris R, Nouryan CN, Schwartz R, Castillo S, Makaryus AN, Ahern D, et al. A Randomized Controlled Trial Comparing Telehealth Self-Management to Standard Outpatient Management in Underserved Black and Hispanic Patients Living with Heart Failure. Telemed J E Health 2019 Oct 01;25(10):917-925 [FREE Full text] [CrossRef] [Medline]
- Pekmezaris R, Schwartz RM, Taylor TN, DiMarzio P, Nouryan CN, Murray L, et al. A qualitative analysis to optimize a telemonitoring intervention for heart failure patients from disparity communities. BMC Med Inform Decis Mak 2016 Jun 24;16:75 [FREE Full text] [CrossRef] [Medline]
- Comparing Two Ways to Help Hispanic Patients Manage Type 2 Diabetes. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2018/comparing-two-ways-help-hispanic-patients-manage-type-2-diabetes [accessed 2020-11-11]
- The HOMBRE Trial: Comparing Two Innovative Approaches to Reduce Chronic Disease Risk among Latino Men. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2016/hombre-trial-comparing-two-innovative-approaches-reduce-chronic-disease-risk [accessed 2020-11-11]
- Comparing Ways to Treat Insomnia among Black Women. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2017/comparing-ways-treat-insomnia-among-black-women [accessed 2020-11-11]
- Comparing Two Ways to Promote Healthy Weight Gain and Prevent Obesity in Early Childhood. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2018/comparing-two-ways-promote-healthy-weight-gain-and-prevent-obesity-early [accessed 2020-11-13]
- Community Health Worker and Mobile Health Programs to Help Young Adults with Sickle Cell Disease Transition to Using Adult Healthcare Services -- The COMETS Study. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2017/community-health-worker-and-mobile-health-programs-help-young-adults-sickle [accessed 2020-11-11]
- Faro JM, Orvek EA, Blok AC, Nagawa CS, McDonald AJ, Seward G, et al. Dissemination and Effectiveness of the Peer Marketing and Messaging of a Web-Assisted Tobacco Intervention: Protocol for a Hybrid Effectiveness Trial. JMIR Res Protoc 2019 Jul 23;8(7):e14814 [FREE Full text] [CrossRef] [Medline]
- Comparing Three Ways to Help Patients Manage Multiple Chronic Conditions. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2017/comparing-three-ways-help-patients-manage-multiple-chronic-conditions [accessed 2020-12-15]
- Nelson RG, Pankratz VS, Ghahate DM, Bobelu J, Faber T, Shah VO. Home-Based Kidney Care, Patient Activation, and Risk Factors for CKD Progression in Zuni Indians. CJASN 2018 Nov 15;13(12):1801-1809. [CrossRef]
- Shah VO, Carroll C, Mals R, Ghahate D, Bobelu J, Sandy P, et al. A Home-Based Educational Intervention Improves Patient Activation Measures and Diabetes Health Indicators among Zuni Indians. PLoS One 2015;10(5):e0125820 [FREE Full text] [CrossRef] [Medline]
- Providing at-Home Kidney Disease Care for Native Americans in New Mexico. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2016/providing-home-kidney-disease-care-native-americans-new-mexico [accessed 2020-12-15]
- Chung J, Donelan K, Macklin EA, Schwartz A, Elsharkawi I, Torres A, et al. A randomized controlled trial of an online health tool about Down syndrome. Genet Med 2021 Jan;23(1):163-173. [CrossRef] [Medline]
- Comparing Three Ways to Treat Insomnia in Adults Living in Rural Areas. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2019/comparative-effectiveness-zolpidem-and-cognitive-behavioral-therapy-insomnia [accessed 2020-12-15]
- Tebb KP, Rodriguez F, Pollack LM, Trieu SL, Hwang L, Puffer M, et al. Assessing the effectiveness of a patient-centred computer-based clinic intervention, to reduce health disparities in unintended pregnancies among Hispanic adolescents: study protocol for a cluster randomised control trial. BMJ Open 2018 Jan 10;8(1):e018201 [FREE Full text] [CrossRef] [Medline]
- Tebb KP, Leng Trieu S, Rico R, Renteria R, Rodriguez F, Puffer M. A Mobile Health Contraception Decision Support Intervention for Latina Adolescents: Implementation Evaluation for Use in School-Based Health Centers. JMIR Mhealth Uhealth 2019 Mar 14;7(3):e11163 [FREE Full text] [CrossRef] [Medline]
- Comparing Two Depression Prevention Programs for Teens -- The Path 2 Purpose Study. Patient-Centered Outcomes Research Institute. 2020. URL: https://www.pcori.org/research-results/2018/comparing-two-depression-prevention-programs-teens-path-2-purpose-study [accessed 2020-12-15]
- Fernandez ME, Schlechter CR, Del Fiol G, Gibson B, Kawamoto K, Siaperas T, et al. QuitSMART Utah: an implementation study protocol for a cluster-randomized, multi-level Sequential Multiple Assignment Randomized Trial to increase Reach and Impact of tobacco cessation treatment in Community Health Centers. Implement Sci 2020 Jan 30;15(1):9-13 [FREE Full text] [CrossRef] [Medline]
- Connecting Low-Income Tobacco Users with a Telephone Quitline to Help Them Quit. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2018/connecting-low-income-tobacco-users-telephone-quitline-help-them-quit [accessed 2021-01-07]
- Comparing Two Ways to Treat Childhood Obesity. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/research-results/2018/comparing-two-ways-treat-childhood-obesity [accessed 2020-11-13]
- Williams K, Markwardt S, Kearney S, Karp J, Kraemer K, Park M, et al. Addressing Implementation Challenges to Digital Care Delivery for Adults With Multiple Chronic Conditions: Stakeholder Feedback in a Randomized Controlled Trial. JMIR Mhealth Uhealth 2021 Feb 1;9(2):e23498. [CrossRef]
- Improving Cultural Competence to Reduce Health Disparities for Priority Populations. Agency for Healthcare Research and Quality. 2014. URL: https://effectivehealthcare.ahrq.gov/products/cultural-competence/research-protocol [accessed 2020-12-15]
- Pankratz VS, Choi EE, Qeadan F, Ghahate D, Bobelu J, Nelson RG, et al. Diabetes status modifies the efficacy of home-based kidney care for Zuni Indians in a randomized controlled trial. J Diabetes Complications 2021 Feb;35(2):107753. [CrossRef] [Medline]
- Golnick C, Asay E, Provost E, Van Liere D, Bosshart C, Rounds-Riley J, et al. Innovative primary care delivery in rural Alaska: a review of patient encounters seen by community health aides. Int J Circumpolar Health 2012 Jun 29;71:18543 [FREE Full text] [CrossRef] [Medline]
- Overview of the Alaska community health aide program. Alaska Community Health Aide Program. 2005. URL: https://www.akchap.org/resources/chap_library/Referral_Physician/CHAM_CHAP_Overview.pdf [accessed 2019-09-12]
- Kokesh J, Ferguson AS, Patricoski C. Telehealth in Alaska: delivery of health care services from a specialist's perspective. Int J Circumpolar Health 2004 Dec;63(4):387-400. [CrossRef] [Medline]
- Kokesh J, Ferguson AS, Patricoski C. The Alaska experience using store-and-forward telemedicine for ENT care in Alaska. Otolaryngol Clin North Am 2011 Dec;44(6):1359-74, ix. [CrossRef] [Medline]
- Suen JJ, Bhatnagar K, Emmett SD, Marrone N, Kleindienst Robler S, Swanepoel DW, et al. Hearing care across the life course provided in the community. Bull. World Health Organ 2019 Aug 20;97(10):681-690. [CrossRef]
- PCORI Researchers Are Answering Rapidly Emerging Questions about Telehealth. Patient-Centered Outcomes Research Institute. URL: https://www.pcori.org/blog/pcori-researchers-are-answering-rapidly-emerging-questions-about-telehealth [accessed 2020-12-15]
- Digital divide persists even as Americans with lower incomes make gains in tech adoption. Pew Research Center. 2019. URL: https://www.pewresearch.org/fact-tank/2021/06/22/digital-divide-persists-even-as-americans-with-lower-incomes-make-gains-in-tech-adoption/ [accessed 2019-02-26]
- Mobile Fact Sheet. Pew Research Center. URL: https://www.pewresearch.org/internet/fact-sheet/mobile/ [accessed 2020-12-26]
- Arora S, Peters AL, Agy C, Menchine M. A mobile health intervention for inner city patients with poorly controlled diabetes: proof-of-concept of the TExT-MED program. Diabetes Technol Ther 2012 Jun;14(6):492-496. [CrossRef] [Medline]
- Arora S, Peters AL, Burner E, Lam CN, Menchine M. Trial to examine text message-based mHealth in emergency department patients with diabetes (TExT-MED): a randomized controlled trial. Ann Emerg Med 2014 Jun;63(6):745-54.e6. [CrossRef] [Medline]
- Franklin VL, Waller A, Pagliari C, Greene SA. A randomized controlled trial of Sweet Talk, a text-messaging system to support young people with diabetes. Diabet Med 2006 Dec;23(12):1332-1338. [CrossRef] [Medline]
- Mulvaney SA, Anders S, Smith AK, Pittel EJ, Johnson KB. A pilot test of a tailored mobile and web-based diabetes messaging system for adolescents. J Telemed Telecare 2012 Mar;18(2):115-118 [FREE Full text] [CrossRef] [Medline]
- van Olmen J, Ku GM, van Pelt M, Kalobu JC, Hen H, Darras C, et al. The effectiveness of text messages support for diabetes self-management: protocol of the TEXT4DSM study in the democratic Republic of Congo, Cambodia and the Philippines. BMC Public Health 2013 May 01;13:423 [FREE Full text] [CrossRef] [Medline]
- Head KJ, Noar SM, Iannarino NT, Grant Harrington N. Efficacy of text messaging-based interventions for health promotion: a meta-analysis. Soc Sci Med 2013 Nov;97:41-48. [CrossRef] [Medline]
- Sieverdes JC, Treiber F, Jenkins C, Hermayer K. Improving diabetes management with mobile health technology. Am J Med Sci 2013 Apr;345(4):289-295. [CrossRef] [Medline]
- Walker RJ, Gebregziabher M, Martin-Harris B, Egede LE. Relationship between social determinants of health and processes and outcomes in adults with type 2 diabetes: validation of a conceptual framework. BMC Endocr Disord 2014 Oct 09;14(1):82 [FREE Full text] [CrossRef] [Medline]
- Mehrotra A, Wang B, Snyder G. Telemedicine: What Should the Post-Pandemic Regulatory and Payment Landscape Look Like? The Commonwealth Fund. 2020. URL: https://www.commonwealthfund.org/publications/issue-briefs/2020/aug/telemedicine-post-pandemic-regulation [accessed 2021-03-09]
- The Future Of State Telehealth Policy. National Governors Association. 2020. URL: https://www.nga.org/center/publications/the-future-of-state-telehealth-policy/ [accessed 2020-11-13]
- Leveraging telehealth to support aging Americans. America's Health Insurance Plan. 2018. URL: https://www.ahip.org/wp-content/uploads/2018/10/AHIP-CTAC_Report.pdf [accessed 2021-03-05]
Abbreviations
CHR: community health representative |
CKD: chronic kidney disease |
HIPAA: Health Insurance Portability and Accountability Act |
LGBT: lesbian, gay, bisexual, and transgender |
MODEL: Management of Diabetes In Everyday Life |
PCOR: patient-centered outcomes research |
PCORI: Patient-Centered Outcomes Research Institute |
PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses |
TEAM UP: Treatment Efforts Addressing Child Weight Management by Unifying Patients, Parents, and Providers |
VIGoROUS: Video Game Rehabilitation for Outpatient Stroke |
Edited by R Kukafka, G Eysenbach; submitted 10.03.21; peer-reviewed by MA Bahrami, H Mehdizadeh; comments to author 23.04.21; revised version received 04.06.21; accepted 03.10.21; published 07.12.21
Copyright©James E Bailey, Cathy Gurgol, Eric Pan, Shirilyn Njie, Susan Emmett, Justin Gatwood, Lynne Gauthier, Lisa G Rosas, Shannon M Kearney, Samantha Kleindienst Robler, Raymona H Lawrence, Karen L Margolis, Ifeyinwa Osunkwo, Denise Wilfley, Vallabh O Shah. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 07.12.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.