Original Paper
Abstract
Background: The digital health sector has experienced rapid growth over the past decade. However, health care technology stakeholders lack a comprehensive understanding of clinical robustness and claims across the industry.
Objective: This analysis aimed to examine the clinical robustness and public claims made by digital health companies.
Methods: A cross-sectional observational analysis was conducted using company data from the Rock Health Digital Health Venture Funding Database, the US Food and Drug Administration, and the US National Library of Medicine. Companies were included if they sell products targeting the prevention, diagnosis, or treatment phases of the care continuum. Clinical robustness was defined using regulatory filings and clinical trials completed by each company. Public claims data included clinical, economic, and engagement claims regarding product outcomes made by each company on its website.
Results: A total of 224 digital health companies with an average age of 7.7 years were included in our cohort. Average clinical robustness was 2.5 (1.8 clinical trials and 0.8 regulatory filings) with a median score of 1. Ninety-eight (44%) companies had a clinical robustness score of 0, while 45 (20%) companies had a clinical robustness score of 5 or more. The average number of public claims was 1.3 (0.5 clinical, 0.4 economic, and 0.4 engagement); the median number of claims was 1. No correlation was observed between clinical robustness and number of clinical claims (r2=0.02), clinical robustness and total funding (r2=0.08), or clinical robustness and company age (r2=0.18).
Conclusions: Many digital health companies have a low level of clinical robustness and do not make many claims as measured by regulatory filings, clinical trials, and public data shared online. Companies and customers may benefit from investing in greater clinical validation efforts.
doi:10.2196/37677
Keywords
Introduction
The digital health sector has grown rapidly over the past decade [
, ]. There are more than 1900 digital health startup companies in the United States that have raised more than US $2 million in venture funding, which in total have raised more than US $77 billion in venture capital funding since 2011 [ ].Although growth is apparent, the ability to measure impact is not. Several studies have highlighted the need for greater clinical validation [
, ] and found that many solutions were not supported by robust clinical evidence [ , ] and demonstrated mixed results on cost savings and cost-effectiveness [ , ]. In addition, there is evidence that some claims made by digital health companies have been misleading [ - ], with a few highly publicized cases resulting in legal action by the Federal Trade Commission and state attorneys general [ - ]. Most studies focusing on clinical impact are narrowly defined to specific clinical therapeutic areas (eg, diabetes, cardiac arrhythmia), making it difficult to extrapolate findings to the broader field of digital health [ - ]. Additional limitations include a small sample of companies and the use of publications as a proxy for clinical impact. Other studies in digital health have examined larger clinical trends, such as growth in clinical trials, but these often lack data on clinical and customer focus [ , ].As a result, the literature is often too narrow or too broad to enable an understanding of clinical impact. In addition, no studies have examined both clinical rigor and public claims made by companies. To address these limitations, we sought to comprehensively examine the topic of clinical robustness in digital health companies by using a more comprehensive definition of clinical rigor and examining companies’ public claims across the most in-depth database of US-based digital health companies. These findings provide additional context for all stakeholders in health technology that rely on a more accurate characterization of digital health solutions.
Methods
Population
Companies were identified using the Digital Health Venture Funding Database maintained by Rock Health Inc, a digital health venture fund and advisory firm, which has been used in prior studies [
- ]. The database includes all digital health companies with headquarters in the United States that have raised at least one venture funding round of US $2 million or more since 2011. Our analysis included companies that sell products targeting the prevention, diagnosis, or treatment phases of the care continuum, which raised at least one round of funding between 2011 and 2020. Digital health companies are defined as those that build and sell digital technologies in health care [ , ].Company Variables
Total venture funding, clinical area(s) of focus, care continuum phase(s), and customer data were collected for each company. Clinical areas represented 1 of 20 specific clinical domains (eg, cardiovascular, nephrology). Care continuum phase was defined according to the following: prevention, diagnosis, or treatment. Customer type referred to the category of buyer for a company’s products, such as payer (ie, health insurance companies), biopharma (ie, pharmaceutical or biopharmaceutical companies), and medical devices (ie, companies that manufacture medical devices). Companies can be categorized into multiple categories (ie, companies may address multiple phases of the care continuum or multiple clinical areas). Company variables were gathered from the Digital Health Venture Funding Database, which Rock Health maintains using publicly available information such as company websites, press releases, and US Securities and Exchange Commission filings. Data for companies were collected through August 3, 2021.
Claims Variables
The number and type of claims, defined as unique quantitative statements about product outcomes, made on a company’s website were collected in the following categories: clinical, economic, and engagement. The definitions of these claim types are detailed in
. Data were obtained by reviewing all pages of a company’s website, excluding links to external pages such as press releases, between May 3, 2021, and August 3, 2021.Claim type | Definition | Claim subtype |
Engagement | A quantitative statement on how engaged users are with the technology or that it provides a better patient experience |
|
Economic | A quantitative statement about a product’s impact on health-related expenses or revenue for the buyer or end user of the product |
|
Clinical | A quantitative statement about a product’s impact on patient health or well-being |
|
Clinical Robustness Variables
We collected regulatory data from the Food and Drug Administration (FDA), including the number of 510(k), De Novo, and premarket approval filings (where the company was listed as the “Requester” on fda.gov). In addition, we collected the number and type of clinical trials by searching ClinicalTrials.gov where the company was listed as “Sponsor / Collaborator.” Data on both FDA filings and clinical trials were collected between July 1, 2021, and September 2, 2021, through a combination of web scraping and manual searching. Data collection on claims was completed by at least 3 authors, with blinded cross-review to ensure consistency in data collection.
A “clinical robustness” score was calculated for each company, defined as the sum of the number of regulatory filings and clinical trials. Each regulatory filing and clinical trial was weighted equally in the calculation.
Data and Statistical Analysis
All data were stored in Microsoft Excel (Microsoft Corp), where all descriptive and statistical analyses including coefficient of correlation between variables were calculated.
Results
Population Characteristics
There were 224 companies in the cohort, with an average age of 7.7 years. Collectively, these companies have raised a total of US $8.2 billion in venture capital funding since 2011. The companies spanned 3 phases of the care continuum, with 25 offering solutions for prevention, 106 offering solutions for diagnosis, and 110 companies offering solutions for treatment (
). Some companies offered products across multiple phases of the care continuum and were therefore counted in multiple categories. Average company funding was similar across phases of care—prevention companies raised US $35.3 million; diagnosis companies raised US $37.8 million; and treatment companies raised US $37.9 million.Clinical Robustness
The average clinical robustness score for all companies was 2.5 (clinical trials: 1.8; regulatory filings: 0.8). The median clinical robustness score was 1, with 98 companies (44%) having a score of 0 and 34 companies (15%) having a score of 1. Diagnosis companies had the highest average clinical robustness scores (2.8), followed by treatment companies (2.2), and then prevention companies (1.9).
The average clinical robustness score of companies that sold to employers was 3.1, compared to 2.0, 2.2, and 2.7 for those companies that sold to payers, consumers, and providers, respectively (
). Fifteen of the 18 clinical areas had a higher average number of clinical trials than regulatory filings. The distribution of companies by clinical robustness score can be found in .Clinical robustness score | Companies, n (%) |
0 | 98 (44) |
1 | 34 (15) |
2 | 24 (11) |
3 | 15 (7) |
4 | 8 (4) |
5 | 13 (6) |
6 | 7 (3) |
7 | 4 (2) |
8 | 7 (3) |
9 | 1 (0) |
≥10 | 13 (6) |
Claims
The average number of total claims for all companies was 1.3 (0.5 clinical, 0.4 economic, and 0.4 engagement) with the median number of total claims equal to 1 (0 clinical, 0 economic, 0 engagement) (
). The median number of claims of any type was zero for companies that sold to consumers and providers. The median number of economic and engagement claims for companies that sold to payers was also zero. Companies that sold to employers made more clinical, economic, and engagement claims than companies that sold to all other customer types ( ). Many companies (43%) made zero claims.Clinical Robustness and Claims
There was no correlation between clinical robustness and number of clinical claims (r2=0.02), clinical robustness and total funding (r2=0.08), or clinical robustness and company age (r2=0.18). In addition, there was no strong correlation between clinical areas that had higher clinical robustness scores and clinical areas where companies have received higher average funding (r2=0.07).
Discussion
Principal Findings
Our findings indicate that many venture-backed startups in digital health have limited clinical robustness as measured by regulatory filings and clinical trials. There was, however, a sizable minority (20%) that had a score of 5 or more, suggesting a small population of rigorously tested solutions (
). Although this subpopulation may portend progress, the lack of meaningful clinical validation for nearly half of digital health companies (44% had a clinical robustness score of 0) highlighted a major gap in health care technology today. The lack of overall correlation between a company’s total venture funding and its clinical robustness score similarly highlighted a significant asymmetry in how companies are potentially valued in today’s marketplace (ie, no correlation between clinical impact and funding) ( ). However, it is possible that funding amounts reflect future anticipated value rather than current value [ ].Although average clinical robustness was quite low across the population, there was significant variation across clinical areas (eg, high in cardiovascular and nephrology and low in oncology and primary care) (
). This may reflect the varying levels of technological maturity of digital health solutions across clinical disciplines. Prior literature points to significant differences in technological maturity between well-funded clinical areas such as diabetes [ ] and less well-funded areas such as reproductive and maternal health [ ].The average and median number of all claims was also low. These findings indicated that digital health companies largely did not share outcomes publicly. This may have reflected a desire to keep this data private, but more likely represented a lack of meaningful analyses of any impact (clinical, economic, or engagement) since this data could be used as a competitive differentiator if shared publicly. Separately, we identified 32 companies that had one or more clinical claims and a clinical robustness score of 0. These findings may suggest a disconnect between marketing and evidence. Additional future research could examine the links between public claims and clinical trials or regulatory filings (ie, are individual claims directly supported by clinical trials or regulatory filings?).
Companies that sold to employers made more claims and had higher clinical robustness scores compared to other customer types (
). This may suggest that this customer base is currently the most competitive and/or has the highest entry standards, despite accounting for a relatively small percentage (<12%) of companies in the sample. A competitive employer market for digital health tools is unsurprising given the rising cost burden of health care on employers [ ] and evidence that employers are increasingly offering digital health benefits to improve health outcomes [ , ] and contain health care costs [ , ].Limitations
Our definition of clinical robustness was limited to clinical trials and regulatory findings, which remain proxies for effectiveness and require companies to register their activities. However, we believe these elements offer a better estimation of clinical rigor than publications, which can lag in timing and may not always relate to clinical outcomes. We chose to focus on clinical trials and regulatory filings because they were publicly available data and are generally undertaken to demonstrate an impact on clinical outcomes. Our approach to measuring clinical robustness equally accounts for clinical trials that demonstrate a technology does and does not work. Additionally, our data collection methodology could have missed clinical trials that were not registered on ClinicalTrials.gov or regulatory filings that were submitted to the FDA under an individual’s name rather than the company name (both scenarios are atypical). Future research could incorporate condition- or disease-specific metrics of effectiveness that are then standardized across clinical areas to provide a more accurate measure of clinical impact.
In addition, our cohort only included venture-backed companies above US $2 million in funding. This may have resulted in selection bias by excluding both ends of the spectrum (ie, not including earlier stage companies or large conglomerate technology companies). Although our population excludes these companies, we believe that our sample represents the most comprehensive assessment of outcomes or claims across the digital health landscape and encompasses the majority of funded companies and activity in this space. As a point of reference, the average seed stage deal size in 2021 was US $3.5 million, 75% higher than our minimum funding threshold [
].Conclusions
Despite the hundreds of digital health companies targeting the myriad of needs across the care continuum, clinical robustness and public communication of claims remains low across much of the sector. These results highlight a significant opportunity for companies to differentiate themselves and for customers to demand greater validation for the products and services they purchase.
Acknowledgments
The authors would like to thank Jasmine DeSilva, Adriana Krasniansky, and Pavan Shah, who are members of the Rock Health research team and manage the Digital Health Venture Funding Database.
Conflicts of Interest
SD, VS, and SK are employees of Rock Health, which supplied data for this analysis. Rock Health is an investor in digital health companies, but no individuals involved in investments were part of this analysis.
Digital health company characteristics across the care continuum, clinical areas, and customer types.
PDF File (Adobe PDF File), 116 KB
Clinical robustness scores, clinical trials, and regulatory filings.
PDF File (Adobe PDF File), 143 KB
Claims made by digital health companies.
PDF File (Adobe PDF File), 33 KBReferences
- Topol EJ. A decade of digital medicine innovation. Sci Transl Med 2019 Jun 26;11(498):eaaw7610. [CrossRef] [Medline]
- Cohen AB, Dorsey ER, Mathews SC, Bates DW, Safavi K. A digital health industry cohort across the health continuum. NPJ Digit Med 2020 May 12;3(1):68 [FREE Full text] [CrossRef] [Medline]
- Rock Health Digital Health Venture Funding Database. URL: https://rockhealth.com/insights/a-new-research-tool-for-a-new-phase-of-digital-health-the-analyses-we-now-support/ [accessed 2020-01-01]
- Mathews SC, McShea MJ, Hanley CL, Ravitz A, Labrique AB, Cohen AB. Digital health: a path to validation. NPJ Digit Med 2019 May 13;2(1):38 [FREE Full text] [CrossRef] [Medline]
- Sedhom R, McShea MJ, Cohen AB, Webster JA, Mathews SC. Mobile app validation: a digital health scorecard approach. NPJ Digit Med 2021 Jul 15;4(1):111 [FREE Full text] [CrossRef] [Medline]
- Safavi K, Mathews SC, Bates DW, Dorsey ER, Cohen AB. Top-Funded Digital Health Companies And Their Impact On High-Burden, High-Cost Conditions. Health Aff (Millwood) 2019 Jan;38(1):115-123. [CrossRef] [Medline]
- Cristea IA, Cahan EM, Ioannidis JPA. Stealth research: Lack of peer-reviewed evidence from healthcare unicorns. Eur J Clin Invest 2019 Apr 13;49(4):e13072. [CrossRef] [Medline]
- Wolff J, Pauling J, Keck A, Baumbach J. The economic impact of artificial intelligence in health care: systematic review. J Med Internet Res 2020 Feb 20;22(2):e16866 [FREE Full text] [CrossRef] [Medline]
- Jiang X, Ming W, You JH. The cost-effectiveness of digital health interventions on the management of cardiovascular diseases: systematic review. J Med Internet Res 2019 Jun 17;21(6):e13166 [FREE Full text] [CrossRef] [Medline]
- Larsen ME, Huckvale K, Nicholas J, Torous J, Birrell L, Li E, et al. Using science to sell apps: Evaluation of mental health app store quality claims. NPJ Digit Med 2019 Mar 22;2(1):18 [FREE Full text] [CrossRef] [Medline]
- Wisniewski H, Liu G, Henson P, Vaidyam A, Hajratalli NK, Onnela J, et al. Understanding the quality, effectiveness and attributes of top-rated smartphone health apps. Evid Based Ment Health 2019 Feb 11;22(1):4-9 [FREE Full text] [CrossRef] [Medline]
- Fraser H, Coiera E, Wong D. Safety of patient-facing digital symptom checkers. Lancet 2018 Nov 24;392(10161):2263-2264. [CrossRef] [Medline]
- Lumosity to pay $2 million to settle FTC deceptive advertising charges for its 'brain training' program. Federal Trade Commission. 2016. URL: https://www.ftc.gov/news-events/news/press-releases/2016/01/lumosity-pay-2-million-settle-ftc-deceptive-advertising-charges-its-brain-training-program [accessed 2022-06-09]
- FTC Charges Marketers of ‘Vision Improvement’ App With Deceptive Claims. Federal Trade Commission. 2015. URL: https://www.ftc.gov/news-events/news/press-releases/2015/09/ftc-charges-marketers-vision-improvement-app-deceptive-claims [accessed 2022-06-09]
- A.G. Schneiderman Announces Settlements With Three Mobile Health Application Developers For Misleading Marketing And Privacy Practices. New York State Office of the Attorney General. 2017. URL: https://ag.ny.gov/press-release/2017/ag-schneiderman-announces-settlements-three-mobile-health-application-developers [accessed 2022-06-09]
- Kitsiou S, Paré G, Jaana M, Gerber B. Effectiveness of mHealth interventions for patients with diabetes: An overview of systematic reviews. PLoS One 2017 Mar 1;12(3):e0173160 [FREE Full text] [CrossRef] [Medline]
- Wongvibulsin S, Habeos EE, Huynh PP, Xun H, Shan R, Porosnicu Rodriguez KA, et al. Digital Health Interventions for Cardiac Rehabilitation: Systematic Literature Review. J Med Internet Res 2021 Feb 08;23(2):e18773 [FREE Full text] [CrossRef] [Medline]
- Widmer RJ, Collins NM, Collins CS, West CP, Lerman LO, Lerman A. Digital health interventions for the prevention of cardiovascular disease: a systematic review and meta-analysis. Mayo Clin Proc 2015 Apr;90(4):469-480 [FREE Full text] [CrossRef] [Medline]
- Hewitt S, Sephton R, Yeowell G. The effectiveness of digital health interventions in the management of musculoskeletal conditions: systematic literature review. J Med Internet Res 2020 Jun 05;22(6):e15617 [FREE Full text] [CrossRef] [Medline]
- Osborn J, Ajakaiye A, Cooksley T, Subbe CP. Do mHealth applications improve clinical outcomes of patients with cancer? A critical appraisal of the peer-reviewed literature. Support Care Cancer 2020 Mar 4;28(3):1469-1479 [FREE Full text] [CrossRef] [Medline]
- Hilty DM, Ferrer DC, Parish MB, Johnston B, Callahan EJ, Yellowlees PM. The effectiveness of telemental health: a 2013 review. Telemed J E Health 2013 Jun;19(6):444-454 [FREE Full text] [CrossRef] [Medline]
- Feldman N, Back D, Boland R, Torous J. A systematic review of mHealth application interventions for peripartum mood disorders: trends and evidence in academia and industry. Arch Womens Ment Health 2021 Dec 30;24(6):881-892 [FREE Full text] [CrossRef] [Medline]
- Chen CE, Harrington RA, Desai SA, Mahaffey KW, Turakhia MP. Characteristics of Digital Health Studies Registered in ClinicalTrials.gov. JAMA Intern Med 2019 Jun 01;179(6):838-840 [FREE Full text] [CrossRef] [Medline]
- Pathak A, Bain S, Pacifici E. 4230 An app a day: examining clinical evidence for safety and efficacy of diabetes mobile health apps. J Clin Trans Sci 2020 Jul 29;4(s1):53-54. [CrossRef]
- Safavi K, Mathews SC, Bates DW, Dorsey ER, Cohen AB. Top-funded digital health companies and their impact on high-burden, high-cost conditions. Health Aff (Millwood) 2019 Jan;38(1):115-123. [CrossRef] [Medline]
- Cohen AB, Dorsey ER, Mathews SC, Bates DW, Safavi K. A digital health industry cohort across the health continuum. NPJ Digit Med 2020 May 12;3(1):68 [FREE Full text] [CrossRef] [Medline]
- Klonoff DC, Evans B, Zweig M, Day S, Kerr D. Is digital health for diabetes in an investment bubble? J Diabetes Sci Technol 2020 Jan 30;14(1):165-169 [FREE Full text] [CrossRef] [Medline]
- What digital health is (and isn’t). Rock Health. 2013. URL: https://rockhealth.com/what-digital-health-is-and-isnt/ [accessed 2021-12-22]
- What is digital health? U.S. Food and Drug Administration. 2020. URL: https://www.fda.gov/medical-devices/digital-health-center-excellence/what-digital-health [accessed 2021-12-22]
- Damodaran A. Valuing young, start-up and growth companies: estimation issues and valuation challenges. SSRN Journal 2009:1-67. [CrossRef]
- Wiederhold BK. Femtech: digital help for women's health care across the life span. Cyberpsychol Behav Soc Netw 2021 Nov 01;24(11):697-698. [CrossRef] [Medline]
- Keehan S, Cuckler G, Poisal J, Sisko A, Smith S, Madison A, et al. National health expenditure projections, 2019-28: expected rebound in prices drives rising spending growth. Health Aff (Millwood) 2020 Apr;39(4):704-714. [CrossRef] [Medline]
- Howarth A, Quesada J, Silva J, Judycki S, Mills PR. The impact of digital health interventions on health-related outcomes in the workplace: A systematic review. Digit Health 2018 May 10;4:2055207618770861 [FREE Full text] [CrossRef] [Medline]
- Schaetz L, Rimner T, Pathak P, Fang J, Chandrasekhar D, Mueller J, et al. Employee and employer benefits from a migraine management program: disease outcomes and cost analysis. Headache 2020 Oct 16;60(9):1947-1960 [FREE Full text] [CrossRef] [Medline]
- Horstman CM, Ryan DH, Aronne LJ, Apovian CM, Foreyt JP, Tuttle HM, et al. Return on investment: medical savings of an employer-sponsored digital intensive lifestyle intervention for weight loss. Obesity (Silver Spring) 2021 Apr 23;29(4):654-661 [FREE Full text] [CrossRef] [Medline]
- Cunningham-Hill M, Dodge-Rice Z, Wilson-Myers C, Sherman C, Neary M, Schueller S. Digital tools and solutions for mental health: an employer's guide. Northeast Business Group on Health and One Mind PsyberGuide. 2020 May 04. URL: https://online.flippingbook.com/view/911432/ [accessed 2022-06-13]
- Pitchbook-NVCA venture monitor. Pitchbook. URL: https://nvca.org/research/pitchbook-nvca-venture-monitor/ [accessed 2020-05-02]
Abbreviations
FDA: Food and Drug Administration |
Edited by A Mavragani; submitted 02.03.22; peer-reviewed by J Andrews, J Segel, A Meyers; comments to author 05.04.22; revised version received 25.05.22; accepted 07.06.22; published 20.06.22
Copyright©Sean Day, Veeraj Shah, Sari Kaganoff, Shannon Powelson, Simon C Mathews. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 20.06.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.