This paper is in the following e-collection/theme issue:

Viewpoints and Perspectives (484) Quality Evaluation and Descriptive Analysis/Reviews of Multiple Existing Mobile Apps (362) Mobile Health (mhealth) (2769) Quality/Credibility of eHealth Information and Trust Issues (489) Evaluation and Research Methodology for mHealth (194)

Published on in Vol 23, No 6 (2021): June

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/27105, first published .
Goldilocks and the Three Bears: A Just-Right Hybrid Model to Synthesize the Growing Landscape of Publicly Available Health-Related Mobile Apps

Goldilocks and the Three Bears: A Just-Right Hybrid Model to Synthesize the Growing Landscape of Publicly Available Health-Related Mobile Apps

Goldilocks and the Three Bears: A Just-Right Hybrid Model to Synthesize the Growing Landscape of Publicly Available Health-Related Mobile Apps

Authors of this article:

Nancy Lau1, 2, 3 Author Orcid Image ;   Alison O'Daffer1 Author Orcid Image ;   Joyce Yi-Frazier1 Author Orcid Image ;   Abby R Rosenberg1, 3, 4 Author Orcid Image
Article Authors Cited by (1) Tweetations (20) Metrics

    Viewpoint

    1Palliative Care and Resilience Lab, Center for Clinical and Translational Research, Seattle Children’s Research Institute, Seattle, WA, United States

    2Department of Psychiatry and Behavioral Sciences, University of Washington School of Medicine, Seattle, WA, United States

    3Cambia Palliative Care Center of Excellence, University of Washington, Seattle, WA, United States

    4Department of Pediatrics, University of Washington School of Medicine, WA, United States

    Corresponding Author:

    Nancy Lau, PhD

    Palliative Care and Resilience Lab

    Center for Clinical and Translational Research

    Seattle Children’s Research Institute

    1920 Terry Ave

    Seattle, WA, 98101

    United States

    Phone: 1 206 884 0569

    Email: nancy.lau@seattlechildrens.org


    Mobile health (mHealth) technologies have provided an innovative platform for the deployment of health care diagnostics, symptom monitoring, and prevention and intervention programs. Such health-related smartphone apps are universally accepted by patients and providers with over 50 million users worldwide. Despite the rise in popularity and accessibility among consumers, the evidence base in support of health-related apps has fallen well behind the rapid pace of industry development. To bridge this evidence gap, researchers are beginning to consider how to best apply evidence-based research standards to the systematic synthesis of the mHealth consumer market. In this viewpoint, we argue for the adoption of a “hybrid model” that combines a traditional systematic review with a systematic search of mobile app download platforms for health sciences researchers interested in synthesizing the state of the science of consumer apps. This approach, which we have successfully executed in a recent review, maximizes the benefits of traditional and novel approaches to address the essential question of whether popular consumer mHealth apps work.

    J Med Internet Res 2021;23(6):e27105

    doi:10.2196/27105

    Keywords

    telemedicine; smartphone; mobile phones; mHealth; mobile apps; health services 


    In the past decade, smartphones have become ubiquitous across personal, social, and vocational domains [1], regardless of gender, race, ethnicity, and socioeconomic status [2]. There are 3.5 billion smartphone users worldwide [3]. Nearly 75% of Americans own a smartphone and 83% of smartphone owners never leave home without it [1,4]. Mobile health (mHealth) technologies may improve access to health care by overcoming financial constraints and geographical barriers; 73% of families living below the poverty line have 1 or more smartphones even if they lack access to other resources, and telehealth enables expanded access to services in rural communities [5,6]. Over 50 million people use apps for health monitoring and diagnostic purposes worldwide [7]. Smartphone app-based tools for diagnosis, symptom monitoring, behavioral change, provider–patient communication, and disease-related education have become increasingly popular and have the potential to improve health and behavioral outcomes [8-10].

    Previous research suggests that both patients and providers have a strong interest in utilizing mHealth technologies as part of health care practices, particularly apps that are supported by research evidence [11-13]. However, the research evidence lags far behind the exponential growth of publicly available apps for consumer download. As of 2020, there were approximately 101,000 mHealth apps available in major app stores with 3.7 billion annual downloads, and the market is forecasted to reach US $312 billion by 2027 [14-17]. This presents a significant research–practice divide with the widespread adoption of app-based health care tools and interventions that may not be backed by science [18].


    This viewpoint is structured according to the main points of the “design science” framework for information systems research: problem identification, objectives, design and development, and demonstration [19].

    Problem

    Further fracturing the research–practice divide into a chasm, there are no gold-standard methods for evaluating the evidence in support of the efficacy of publicly available consumer apps within the growing mHealth industry.

    Objectives

    To address the research–practice chasm, we endeavored to develop methodologically rigorous and reproducible standards for evaluating whether publicly available mHealth tools and interventions work. In this viewpoint, our primary objective is to provide a narrative description of the lessons we learned from the process of designing a recent study evaluating the evidence in support of popular stress management and psychosocial wellness apps [20].

    Design and Development

    We considered methodological approaches such as a traditional systematic review and a novel search of mobile app download platforms. A systematic review is particularly well-suited for research questions pertaining to feasibility and efficacy of apps developed in research settings. Novel searches of mobile app download platforms allow researchers to examine the functionality and usability of popular apps. Neither was sufficient alone for novel research questions pertaining to the state of the science of popular apps available for consumer download. Ultimately, we decided on a “hybrid model” combining a systematic review with a systematic search of mobile app download platforms, a methodological approach that was “not too hot, not too cold, but just right.” We recommend that researchers interested in the review and synthesis of publicly available consumer mHealth apps in their respective disciplines utilize a “hybrid model” such as this to guide research design conceptualization.

    Demonstration

    We present 2 illustrative examples of studies following a “hybrid model” design, while providing additional citations of other successful studies.

    Porridge Bowl #1: Systematic Review

    First, we explored the possibility of utilizing a systematic review to synthesize the consumer app landscape. This traditional approach benefits from gold standards that have been extensively detailed in references such as the Cochrane Handbook for Systematic Reviews of Interventions [21] which covers all aspects of review planning from idea inception to data collection and analysis. Health science researchers interested in summarizing the state of the science in mHealth-related topics have conducted traditional systematic reviews utilizing databases of references such as Ovid MEDLINE, Embase, Cochrane Central Register of Controlled Trials, Web of Science, Scopus, and PsycINFO using the PICO framework to inform the search, where prespecified parameters include Patient problem/Population, Intervention, Comparator, and Outcomes of interest [22-31]. Previous smartphone app systematic reviews have spanned topics such as health behavior change interventions, medication management, and cognitive behavioral therapy and behavioral activation apps for depression [22-30].

    Despite the advantage of rigorous well-defined methods with reproducible results, there are fatal flaws in the application of this approach specifically for those interested in asking and answering research questions pertaining to the state of the science of the mHealth industry space. First, the majority of apps developed in traditional laboratory-based and research settings are not available for public download and require private access codes provided to research participants only [26]. Second, less than 1% of mHealth apps across a range of previously explored health domains had corresponding scientific publications describing their efficacy [20,24,32]. Taken together, there is little to no overlap between apps evaluated in traditional systematic reviews (which query the existing literature in extensive library databases) and the types of consumer apps available for public download (which query the existing apps available for your mobile devices). For example, a systematic review of mHealth psychological interventions for anxiety which showed small to medium effect sizes can only tell us about the efficacy of apps that have been formally tested in clinical trials [26]; it does not provide information regarding the efficacy of the types of apps we all download directly onto our smartphones based on popularity metrics such as Top 100 lists, Editor’s Picks, media buzz, and consumer ratings. Apps developed in industry and research settings are siloed tracks. Thus, conclusions drawn from traditional systematic reviews are limited for informing the types of novel research questions endeavoring to synthesize the landscape of 101,000+ mHealth apps currently available to smartphone users.

    Porridge Bowl #2: Systematic Search of Mobile App Platforms

    Second, we explored the possibility of utilizing a systematic search of mobile app platforms to synthesize the state of the science of the mHealth industry. In the past decade, mHealth researchers have conducted mobile app download platform searches as an alternative method to traditional systematic reviews befitting the consumer app space [23,33-40]. (Editorial note: Note that JMIR journals do not apply the term "systematic review" to these kinds of studies, but calls them "Systematic Searches on App Stores" or similar; the term "systematic review" is reserved for literature reviews. Other publishers/journals may not distinguish these different study types). This user-centered approach prioritizes broad applicability of findings to day-to-day mobile phone users seeking digital health-management tools and interventions. Mobile app download platforms are utilized as the equivalent of library databases for data extraction in order to identify, screen, and review apps for inclusion and exclusion. In addition, the Mobile Application Rating Scale (MARS) is a commonly used tool for assessing the quality of mHealth apps and provides objective classifications (eg, price, platform, aspects of health targeted), and subjective subscale ratings in the domains of engagement, functionality, aesthetics, and information quality along with a composite app quality rating [41]. The 23-item scale has demonstrated high internal consistency and fair interrater reliability with independent coders [41]. Thus, researchers may compare and gauge the potential impact or value of mHealth programs that happen to be available for consumer download. In the United States, 54.4% of smartphone owners use Android devices and 44.3% use Apple devices [42]. Thus, the majority of existing systematic app searches (and thus app content and quality assessments) span Apple and Android platforms only [33,34,38-40]. Previous systematic searches of mobile app download platforms have spanned a wide range of topics such as smoking cessation, mindfulness, physical activity promotion, and pharmacology education [33,35,36,38,40].

    Recent methodological innovations are as follows: (1) The European Innovation and Knowledge mHealth Hub is a project established in 2020 by the International Telecommunication Union (ITU) in collaboration with the World Health Organization (WHO) [43]. The mHealth Hub offers an overview of 24 health app assessment frameworks evaluating domains including privacy, transparency, safety, and technical stability; this provides additional resources and tools for researchers to systematically synthesize app features and content. (2) Big data innovations have focused on developing automated methods to extract information on app features and components from the web using natural language processing and text analytics [44].

    Despite the advantages of “real-world” representativeness of apps, research questions are limited to a synthesis of user-centered metrics such as mHealth app usability, functionality, engagement, consumer appeal, and content [45-47]. Although such scientific inquiries are important, a systematic search of mobile app platforms does not provide information on whether and how an app works, which are the scientific merits that providers and researchers rely on for establishing evidence-based standards of care and treatment recommendations. In addition, FDA-approved apps that are validated medical devices “for diagnosis of disease or other conditions, or the cure, mitigation, treatment, or prevention of a disease” are not separately listed from nonvalidated health apps; this further limits the ability of users to make informed choices about which health apps are certified tools subject to regulatory oversight [48,49].

    Porridge Bowl #3: “Just Right” Hybrid Designs

    To recap, systematic review methodology confers the benefit of providing information on the scientific merit of apps developed in research settings but do not represent the “real-world” consumer apps that we all download to our smartphones. App download platform search methodology, by contrast, confers the benefit of “real-world” app quality, content, and representation but not of scientific merit. Recent research has utilized hybrid design methodology by combining traditional literature review methods with systematic searches of mobile app platforms, bridging well-established traditional and novel methodologies (Figure 1) [20,24,25,32]. Next, we describe 2 illustrative examples.

    ‎
    Figure 1. Hybrid design model.
    View this figure

    Illustrative Example #1: Systematic App Search Followed by Literature Review

    We utilized a hybrid design to evaluate the following research aims: (1) What are the therapeutic contents and features of popular stress management and psychosocial wellness apps, and (2) Which apps, if any, are supported by peer-reviewed original research publications? This study was conducted in 3 stages [20].

    Step 1 (User-Centered Approach)

    We identified conventional self-help search terms from the background literature and refined the list in consensus conversations among our interdisciplinary team of intervention science researchers, health services researchers, physicians, social workers, and psychologists. Then, we input search terms directly into Android and Apple iOS mobile app search engines to identify consumer apps meeting inclusion criteria. Two authors (NL and AO) independently reviewed all apps. Independent raters created a comprehensive database with content categories representing all intervention ingredients identified, and abstracted relevant data from product pages.

    Step 2 (Traditional Review Approach)

    A literature review was conducted via Google Scholar, Medline, and PsycINFO databases of all commercially available apps identified in Step 1 using the search terms “[app name]” AND smartphone. Data on feasibility and efficacy outcomes were abstracted from the journal articles, and risk of bias was coded by 2 independent raters (NL and AO).

    Step 3 (Synthesize Findings)

    Using this hybrid design, we answered these complementary questions relevant to the state of the science of the mHealth market: Which everyday consumer apps are popular and what is their treatment content? What is the evidence in support of everyday consumer apps available to the general public?

    Illustrative Example #2: Cyclical Approach

    de la Vega and colleagues [32] conducted a systematic review of pain-related apps for pain symptom assessment and education. Utilizing a cyclical model, they conducted independent parallel searches, in one case starting with a systematic search of mobile app download platforms followed by a traditional literature review, and in the other case starting with a traditional literature review followed by a systematic search of mobile app download platforms.

    Search 1/Step 1 (Traditional Review Approach)

    A literature review was conducted across 17 relevant scientific databases (Medline, PsycINFO, Web of Science, etc.) utilizing Boolean-operator pain “AND” mobile app search terms.

    Search 1/Step 2 (User-Centered Approach)

    Apps meeting inclusion criteria identified in Search 1/Step 1 were input directly into 5 mobile app download platforms (Apple Store, Google Play, Blackberry App World, Nokia Store, and Windows Play Store) to determine if research-supported apps were available for public download.

    Search 2/Step 1 (User-Centered Approach)

    User-friendly search terms related to pain management were input directly into mobile app download platforms (Apple Store, Google Play, Blackberry App World, Nokia Store, and Windows Play Store).

    Search 2/Step 2 (Traditional Review Approach)

    A literature review was conducted via the aforementioned library databases of all commercially available apps identified in Search 2/Step 1 by app name. Authors also searched Google and MyHealthApps.net to learn about app origins and creators.

    Step 3 (Synthesize Findings)

    Using this hybrid design, de la Vega and colleagues [32] answered these complementary questions relevant to the state of the science of the mHealth market: Which research-supported apps identified for inclusion via a traditional literature review are commercially available? How many commercially available apps identified for inclusion via a systematic search of app download platforms are supported by peer-reviewed publications?

    Strengths of the “hybrid model” included the cumulative advantages of systematic review and systematic search methodologies, and a comprehensive holistic analysis of the subset of “real-world” consumer apps that are research based. A hybrid design was the only approach that would allow researchers, clinicians, and patients/consumers alike to answer the essential question of whether consumer app-based health care tools and interventions that have been increasingly adopted worldwide actually work and what therapeutic content and features are incorporated in their design. In our previous study, we found further evidence to support the conclusion that apps developed in industry and research settings are siloed (ie, consumer apps developed in the mHealth industry and available for public download rarely have corresponding research publications). The “hybrid” approach allows health sciences researchers to identify the subset of apps that are both research based and publicly available despite the fact that there is no existing database/repository for consumers seeking evidence-based care.


    Smartphone apps to address a diverse array of health care needs are being developed at a rapid rate and are widely adopted worldwide. However, the scientific merit of “real-world” apps remains largely understudied and unknown. This is due, in part, to the absence of well-established methods for the evaluation of the efficacy of consumer apps. Of the 2 more common approaches to evaluation, neither is “just right” to determine both evidence base and quality for the consumer app space. Traditional literature reviews are valuable for gathering and synthesizing information regarding the scientific backing of popular apps in the form of feasibility and efficacy study data. Review of mobile app search engines allows for a direct synthesis of popular consumer apps and user-centered metrics such as usability, engagement, functionality, and app content. The “hybrid model” described in this viewpoint allows researchers to address novel research questions leveraging the complementary strengths of a systematic review and app search engine review. Although not without its limitations, hybrid approaches provide a unique opportunity to develop and iteratively refine methodologies for synthesizing the state of the science of the quickly evolving consumer mHealth market. Future research should focus on the standardization of mobile app download platform searches and systematic blending of traditional and novel methodological approaches via “just-right” hybrid designs. Such research endeavors will help bridge the research–practice chasm by rigorously evaluating digital health industry solutions to health care problems. Ultimately, this will help us understand whether popular apps work and inform mHealth clinical practice guidelines.

    Acknowledgments

    NL is funded as an Implementation Science Scholar through the National Heart, Lung, and Blood Institute of the National Institutes of Health (Grant number 5K12 HL137940-02). The opinions herein represent those of the authors and not necessarily the funders.

    Conflicts of Interest

    None declared.

    1. Bakker D, Kazantzis N, Rickwood D, Rickard N. Mental Health Smartphone Apps: Review and Evidence-Based Recommendations for Future Developments. JMIR Ment Health 2016;3(1):e7 [FREE Full text] [CrossRef] [Medline]
    2. Anderson M, Jiang J. Pew Research Center. 2018.   URL: https://www.pewresearch.org/internet/2018/05/31/teens-social-media-technology-2018/ [accessed 2021-05-26]
    3. O'Dea S. Number of smartphone users worldwide from 2016 to 2021 Published 2020. 2020.   URL: https://www.statista.com [accessed 2021-05-26]
    4. Mobile FS. Mobile Fact Sheet. Pew Research Center. 2018.   URL: https://www.pewinternet.org/fact-sheet/mobile/ [accessed 2021-05-26]
    5. Rideout V, Katz V. Opportunity for All? Technology and Learning in Lower-Income Families. 2016.   URL: https://www.joanganzcooneycenter.org/wp-content/uploads/2016/01/jgcc_opportunityforall.pdf [accessed 2021-05-26]
    6. Myers CR. Using Telehealth to Remediate Rural Mental Health and Healthcare Disparities. Issues Ment Health Nurs 2019 Mar;40(3):233-239. [CrossRef] [Medline]
    7. Millenson ML, Baldwin JL, Zipperer L, Singh H. Beyond Dr. Google: the evidence on consumer-facing digital tools for diagnosis. Diagnosis (Berl) 2018 Dec 25;5(3):95-105. [CrossRef] [Medline]
    8. Torous J, Chan SR, Yee-Marie TS, Behrens J, Mathew I, Conrad EJ, et al. Patient Smartphone Ownership and Interest in Mobile Apps to Monitor Symptoms of Mental Health Conditions: A Survey in Four Geographically Distinct Psychiatric Clinics. JMIR Ment Health 2014;1(1):e5 [FREE Full text] [CrossRef] [Medline]
    9. Featherall J, Lapin B, Chaitoff A, Havele SA, Thompson N, Katzan I. Characterization of Patient Interest in Provider-Based Consumer Health Information Technology: Survey Study. J Med Internet Res 2018 Apr 19;20(4):e128 [FREE Full text] [CrossRef] [Medline]
    10. Rowland SP, Fitzgerald JE, Holme T, Powell J, McGregor A. What is the clinical value of mHealth for patients? NPJ Digit Med 2020;3:4 [FREE Full text] [CrossRef] [Medline]
    11. Apolinário-Hagen J, Hennemann S, Fritsche L, Drüge M, Breil B. Determinant Factors of Public Acceptance of Stress Management Apps: Survey Study. JMIR Ment Health 2019 Nov 07;6(11):e15373 [FREE Full text] [CrossRef] [Medline]
    12. Schueller SM, Washburn JJ, Price M. Exploring Mental Health Providers' Interest in Using Web and Mobile-Based Tools in their Practices. Internet Interv 2016 May;4(2):145-151. [CrossRef] [Medline]
    13. Berkowitz CM, Zullig LL, Koontz BF, Smith SK. Prescribing an App? Oncology Providers' Views on Mobile Health Apps for Cancer Care. JCO Clin Cancer Inform 2017 Dec;1:1-7 [FREE Full text] [CrossRef] [Medline]
    14. Globe Newswire. Mobile Health (mHealth) Market To Reach USD 311.98 Billion By 2027. 2020 Apr 28.   URL: https://tinyurl.com/yjr62d8d [accessed 2021-05-26]
    15. Pohl M. mHealth App Economics 2017/2018: Current Status and Future Trends in Mobile Health. 2017.   URL: https:/​/research2guidance.​com/​wp-content/​uploads/​2017/​11/​R2G-mHealth-DeveloperEconomics-2017-Status-And-Trends.​pdf [accessed 2021-05-26]
    16. Statista. Number of mHealth Apps Available at Google Play From 1st Quarter 2015 to 4th Quarter 2020. 2021.   URL: https://www.statista.com/statistics/779919/health-apps-available-google-play-worldwide/ [accessed 2021-05-26]
    17. Statista. Number of mHealth Apps Available in the Apple App Store from 1st Quarter 2015 to 4th Quarter 2020. 2021.   URL: https://www.statista.com/statistics/779910/health-apps-available-ios-worldwide/ [accessed 2021-04-26]
    18. Aitken M. Patient Adoption of mHealth: Use, Evidence, and Remaining Barriers to Mainstream Acceptance. Minneapolis, MN: Institute for Health Informatics; 2015.
    19. Peffers K, Tuunanen T, Rothenberger MA, Chatterjee S. A Design Science Research Methodology for Information Systems Research. Journal of Management Information Systems 2014 Dec 08;24(3):45-77. [CrossRef]
    20. Lau N, O'Daffer A, Colt S, Yi-Frazier JP, Palermo TM, McCauley E, et al. Android and iPhone Mobile Apps for Psychosocial Wellness and Stress Management: Systematic Search in App Stores and Literature Review. JMIR Mhealth Uhealth 2020 May 22;8(5):e17798 [FREE Full text] [CrossRef] [Medline]
    21. Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane Handbook for Systematic Reviews of Interventions. London, UK: The Cochrane Collaboration and John Wiley & Sons; 2019.
    22. Chib A, Lin SH. Theoretical Advancements in mHealth: A Systematic Review of Mobile Apps. J Health Commun 2018;23(10-11):909-955 [FREE Full text] [CrossRef] [Medline]
    23. Franko OI. Smartphone apps for orthopaedic surgeons. Clin Orthop Relat Res 2011 Jul;469(7):2042-2048 [FREE Full text] [CrossRef] [Medline]
    24. MacKenzie Greenle M, Morgan B, Sayani S, Meghani SH. Identifying Mobile Apps Targeting Palliative Care Patients and Family Members. J Palliat Med 2018 Oct;21(10):1380-1385. [CrossRef] [Medline]
    25. Martínez-Pérez B, de la Torre-Díez I, López-Coronado M, Herreros-González J. Mobile apps in cardiology: review. JMIR Mhealth Uhealth 2013;1(2):e15 [FREE Full text] [CrossRef] [Medline]
    26. Firth J, Torous J, Nicholas J, Carney R, Rosenbaum S, Sarris J. Can smartphone mental health interventions reduce symptoms of anxiety? A meta-analysis of randomized controlled trials. J Affect Disord 2017 Aug 15;218:15-22 [FREE Full text] [CrossRef] [Medline]
    27. Han M, Lee E. Effectiveness of Mobile Health Application Use to Improve Health Behavior Changes: A Systematic Review of Randomized Controlled Trials. Healthc Inform Res 2018 Jul;24(3):207-226. [CrossRef] [Medline]
    28. Hieftje K, Edelman EJ, Camenga DR, Fiellin LE. Electronic media-based health interventions promoting behavior change in youth: a systematic review. JAMA Pediatr 2013 Jun;167(6):574-580 [FREE Full text] [CrossRef] [Medline]
    29. Huguet A, Rao S, McGrath PJ, Wozney L, Wheaton M, Conrod J, et al. A Systematic Review of Cognitive Behavioral Therapy and Behavioral Activation Apps for Depression. PLoS One 2016;11(5):e0154248 [FREE Full text] [CrossRef] [Medline]
    30. Wang K, Varma DS, Prosperi M. A systematic review of the effectiveness of mobile apps for monitoring and management of mental health symptoms or disorders. J Psychiatr Res 2018 Dec;107:73-78. [CrossRef] [Medline]
    31. Schardt C, Adams MB, Owens T, Keitz S, Fontelo P. Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Med Inform Decis Mak 2007;7:16 [FREE Full text] [CrossRef] [Medline]
    32. de la Vega R, Miró J. mHealth: a strategic field without a solid scientific soul. a systematic review of pain-related apps. PLoS One 2014;9(7):e101312 [FREE Full text] [CrossRef] [Medline]
    33. Abroms LC, Lee WJ, Bontemps-Jones J, Ramani R, Mellerson J. A content analysis of popular smartphone apps for smoking cessation. Am J Prev Med 2013 Dec;45(6):732-736 [FREE Full text] [CrossRef] [Medline]
    34. Blázquez Martín D, De La Torre I, Garcia-Zapirain B, Lopez-Coronado M, Rodrigues J. Managing and Controlling Stress Using mHealth: Systematic Search in App Stores. JMIR Mhealth Uhealth 2018 May 09;6(5):e111 [FREE Full text] [CrossRef] [Medline]
    35. Coulon SM, Monroe CM, West DS. A Systematic, Multi-domain Review of Mobile Smartphone Apps for Evidence-Based Stress Management. Am J Prev Med 2016 Jul;51(1):95-105. [CrossRef] [Medline]
    36. Haffey F, Brady RRW, Maxwell S. Smartphone apps to support hospital prescribing and pharmacology education: a review of current provision. Br J Clin Pharmacol 2014 Jan;77(1):31-38. [CrossRef] [Medline]
    37. Mani M, Kavanagh DJ, Hides L, Stoyanov SR. Review and Evaluation of Mindfulness-Based iPhone Apps. JMIR Mhealth Uhealth 2015;3(3):e82 [FREE Full text] [CrossRef] [Medline]
    38. Middelweerd A, Mollee JS, van der Wal CN, Brug J, Te Velde SJ. Apps to promote physical activity among adults: a review and content analysis. Int J Behav Nutr Phys Act 2014;11:97 [FREE Full text] [CrossRef] [Medline]
    39. Nicholas J, Larsen ME, Proudfoot J, Christensen H. Mobile Apps for Bipolar Disorder: A Systematic Review of Features and Content Quality. J Med Internet Res 2015;17(8):e198 [FREE Full text] [CrossRef] [Medline]
    40. Weekly T, Walker N, Beck J, Akers S, Weaver M. A Review of Apps for Calming, Relaxation, and Mindfulness Interventions for Pediatric Palliative Care Patients. Children (Basel) 2018 Jan 26;5(2):16 [FREE Full text] [CrossRef] [Medline]
    41. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015;3(1):e27 [FREE Full text] [CrossRef] [Medline]
    42. Statista. Subscriber Share Held by Smartphone Operating Systems in the United States from 2012 to 2018. 2019.   URL: https:/​/www.​statista.com/​statistics/​266572/​market-share-held-by-smartphone-platforms-in-the-united-states/​ [accessed 2021-05-26]
    43. Statistica. Knowledge Tool 1: Assessment Frameworks in mHealth. 2020.   URL: https:/​/mhealth-hub.​org/​download/​d2-1-knowledge-tool-1-health-apps-assessment-frameworks-pending-ec-approval [accessed 2021-05-26]
    44. Paglialonga A, Schiavo M, Caiani EG. Automated Characterization of Mobile Health Apps' Features by Extracting Information From the Web: An Exploratory Study. Am J Audiol 2018 Nov 19;27(3S):482-492 [FREE Full text] [CrossRef] [Medline]
    45. Romero RL, Kates F, Hart M, Ojeda A, Meirom I, Hardy S. Quality of Deaf and Hard-of-Hearing Mobile Apps: Evaluation Using the Mobile App Rating Scale (MARS) With Additional Criteria From a Content Expert. JMIR Mhealth Uhealth 2019 Oct 30;7(10):e14198 [FREE Full text] [CrossRef] [Medline]
    46. Jamaladin H, van de Belt TH, Luijpers LC, de Graaff FR, Bredie SJ, Roeleveld N, et al. Mobile Apps for Blood Pressure Monitoring: Systematic Search in App Stores and Content Analysis. JMIR Mhealth Uhealth 2018 Nov 14;6(11):e187 [FREE Full text] [CrossRef] [Medline]
    47. Kim BY, Sharafoddini A, Tran N, Wen EY, Lee J. Consumer Mobile Apps for Potential Drug-Drug Interaction Check: Systematic Review and Content Analysis Using the Mobile App Rating Scale (MARS). JMIR Mhealth Uhealth 2018 Mar 28;6(3):e74 [FREE Full text] [CrossRef] [Medline]
    48. Athanas AJ, McCorrison JM, Smalley S, Price J, Grady J, Campistron J, et al. Association Between Improvement in Baseline Mood and Long-Term Use of a Mindfulness and Meditation App: Observational Study. JMIR Ment Health 2019 May 08;6(5):e12617 [FREE Full text] [CrossRef] [Medline]
    49. Gordon WJ, Landman A, Zhang H, Bates DW. Beyond validation: getting health apps into clinical practice. NPJ Digit Med 2020;3:14 [FREE Full text] [CrossRef] [Medline]

    ‎
    ITU: International Telecommunication Union
    MARS: Mobile Application Rating Scale
    mHealth: mobile health
    WHO: World Health Organization

    Edited by R Kukafka; submitted 11.01.21; peer-reviewed by E Caiani, O El-Gayar; comments to author 17.03.21; revised version received 09.04.21; accepted 06.05.21; published 07.06.21

    Copyright

    ©Nancy Lau, Alison O'Daffer, Joyce Yi-Frazier, Abby R Rosenberg. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 07.06.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.