This paper is in the following e-collection/theme issue:

Digital Health Reviews (1289) E-Health Policy and Health Systems Innovation (230) Health Services Research (482)

Published on in Vol 22, No 3 (2020): March

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/17026, first published .
An Analysis of the Learning Health System in Its First Decade in Practice: Scoping Review

An Analysis of the Learning Health System in Its First Decade in Practice: Scoping Review

An Analysis of the Learning Health System in Its First Decade in Practice: Scoping Review

Authors of this article:

Jodyn E Platt1 Author Orcid Image ;   Minakshi Raj2 Author Orcid Image ;   Matthias Wienroth3 Author Orcid Image
Article Authors Cited by (74) Tweetations (15) Metrics

    Review

    1Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, MI, United States

    2Department of Health Management and Policy, University of Michigan School of Public Health, Ann Arbor, MI, United States

    3School of Geography, Politics & Sociology, Newcastle University, Newcastle upon Tyne, United Kingdom

    *all authors contributed equally

    Corresponding Author:

    Minakshi Raj, MPH

    Department of Health Management and Policy

    University of Michigan School of Public Health

    1420 Washington Heights

    Ann Arbor, MI, 48109

    United States

    Phone: 1 3155593112

    Email: miraj@umich.edu


    Background: In the past decade, Lynn Etheredge presented a vision for the Learning Health System (LHS) as an opportunity for increasing the value of health care via rapid learning from data and immediate translation to practice and policy. An LHS is defined in the literature as a system that seeks to continuously generate and apply evidence, innovation, quality, and value in health care.

    Objective: This review aimed to examine themes in the literature and rhetoric on the LHS in the past decade to understand efforts to realize the LHS in practice and to identify gaps and opportunities to continue to take the LHS forward.

    Methods: We conducted a thematic analysis in 2018 to analyze progress and opportunities over time as compared with the initial Knowledge Gaps and Uncertainties proposed in 2007.

    Results: We found that the literature on the LHS has increased over the past decade, with most articles focused on theory and implementation; articles have been increasingly concerned with policy.

    Conclusions: There is a need for attention to understanding the ethical and social implications of the LHS and for exploring opportunities to ensure that these implications are salient in implementation, practice, and policy efforts.

    J Med Internet Res 2020;22(3):e17026

    doi:10.2196/17026

    Keywords

    learning health system; review; knowledge management; bioethics; health information exchange 


    Background

    In 2007, Lynn Etheredge [1] envisioned the Learning Health System (LHS) in which he described a system aimed at increasing the value of health care without draconian cost cutting. He encouraged facilitating what he called rapid learning from new evidence for practice and policy. The publication of this first article to explicitly use the language of the LHS and urge the further consideration of such a system coincided with the adoption of electronic health records (EHRs) in clinical settings aimed at integrating clinical, financial, and administrative data [1]. Etheredge [1] described several opportunities for answering key questions about population health and health care delivery. Addressing these questions would demand a change in institutional (eg, Medicaid) and organizational (eg, in hospitals) leadership and in funding structures to advance the use of health information to improve health [1]. Etheredge’s proposals envisioned competitive markets led by health plans and providers who use EHRs; payment linked to evidence-based protocols; and Medicaid and the State Children’s Health Insurance Program as national leaders in EHR adoption and in the use of EHR research databases. National computer-searchable clinical trial databases and national assessments of new technologies would support these efforts. Here, we considered Etheredge’s article [1] as an initial conceptualization of gaps and opportunities for future examination and development of the LHS. Many subsequent articles on the LHS topic have cited the guiding principles presented in Etheredge’s article [1,2].

    Over a decade has passed since this early vision. Health care provision generates significant amounts of patient and experiential data, and health records, laboratory results, population health surveillance, and patient-generated data, that can be agglomerated and analyzed within health systems. These activities are the result of (1) an increase in health data availability within growing information technology systems via the widening use of patients’ EHRs; (2) efforts to increase the volume of clinical research with patients undertaken at health care facilities; and (3) considerable research and data generation by government-funded and commercial enterprises [3-8]. Bringing together diverse actors in the health and life sciences context, LHSs aim to gather and analyze differently sourced data to create useful knowledge that is disseminated to all stakeholders, put into practice, and then evaluated [9].

    The LHS framework marks a departure from data practices that are governed by the intended use for data, be it research, quality improvement (QI), clinical care, or public health. The aim of LHS is to enable continuously and rapidly operating virtuous cycles of study, feedback, and practice change, regardless of the original intention for data collection; its vision has come to shape the goals of initiatives in the United States, the United Kingdom, and globally [1,2]. Examples include the American Society of Clinical Oncology’s CancerLinQ initiative, which offers an emerging large-scale database for an oncology learning community to improve the quality of cancer care [10], the CommonWell Health Alliance, which enables querying of treatment data for over 17 million unique individuals [11], and the National Institutes of Health All of Us research initiative, which seeks to connect genomic, health, and social media data from over 1 million individuals [12]. Similar avenues toward large-scale agglomeration, analysis, and sharing of patient health information exist in the United Kingdom, such as the National Health Service–supported data platform, Lambeth DataNet, which provides general physicians and academic researchers with access to data from approximately 350,000 patients [13] and the Connected Health Cities initiative [14], which is building data stores to support direct care across the North of England region.

    Objectives

    In this review, we aimed to examine the literature and rhetoric on the LHS, identify trends and themes in published efforts undertaken to moving this idea from concept to reality, and assess gaps in the literature that suggest areas for future research and policy considerations that are necessary for moving the LHS forward (Figure 1).

    ‎
    Figure 1. Areas critical to the realization of a learning health system.
    View this figure

    Search Strategy

    We conducted a scoping review using search terms “learning health system(s),” “learning health care system(s),” and “learning healthcare system(s)” on PubMed, Web of Science, and Scopus databases to identify peer-reviewed publications. The search was limited to peer-reviewed articles published in the English language between January 2007 and December 2017.

    We identified and included articles published in a variety of clinical contexts, with most articles focusing on the United States and written by US-based authors and some focused on the United Kingdom, India, Sweden, Kenya, and China. Articles describing or examining the LHS from clinical, technical, and ethical perspectives were included. Our initial search yielded 542 articles for review, including USA Institute of Medicine (now USA National Academy of Medicine) proceedings and chapters that informed our search but were not included in the final full analyses.

    Data Extraction and Article Selection

    Following a title and abstract review, we excluded 222 articles. Articles were excluded if they were duplicates, conference proceedings or posters, or not peer reviewed. Results were compared among members of the study team before exclusion. We created a Microsoft Excel spreadsheet charting information from the 320 remaining articles, including (1) article title; (2) year; (3) country; (4) category; (5) concern or focus; (6) field; and (7) number of papers citing the article of interest (Table 1).

    Next, we sorted articles by year, identified the top quarter of most cited papers within each year. We used citation data provided by Web of Science and Scopus to determine the number of citations for each article, sorted all included articles in the sample to identify the top quarter, and then compiled a list of these articles. We chose to focus on the most cited papers as representatives of those with the most salient themes to the discourse and scholarship on the LHS. This sorting resulted in 85 articles for our final review, which represents just over 15.7% (85/542) of the total number of articles identified in our initial search (see Figure 2). The 85 articles were read in full to conduct a more thorough discourse (thematic mapping) analysis that could be compared with the vision laid out in Etheredge’s [1] manuscript in 2007, which is described in the section, Analytical Strategy.

    Table 1. Summary of abstracted information.
    Information documentedDescription
    Article titleTitle of publication
    YearYear of publication
    CountryGlobal context of publication
    CategoryPreset categories for thematic understanding, including the following:
    • Policy (eg, relating to health reform)
      • ‎
    • Advocacy (eg, encouraging implementation into organizations)
      • ‎
    • Theory (ie, generating or enhancing LHSa frameworks)
      • ‎
    • Empirical (eg, studies testing infrastructures or hypotheses)
      • ‎
    • Implementation (ie, evaluating implementation into clinical contexts)
      • ‎
    • Ethics (ie, introducing or describing ethical perspectives or limitations of the LHS)
      • ‎
    • General commentary
      • ‎
    Concern and/or FocusInformation on the primary focus or concern of the article, such as the following:
    • Quality improvement
      • ‎
    • Personalized health care
      • ‎
    • Evidence-based medicine
      • ‎
    • Electronic health record
      • ‎
    • Ethical oversight
      • ‎
    FieldInformation on perspective through which article discusses the LHS, such as the following:
    • Clinical context (eg, oncology and surgery)
      • ‎
    • Professional context (eg, nursing)
      • ‎
    • Discourse context (eg, medical informatics, research, and ethics)
      • ‎
    • Health care system (ie, articles discussing the LHS across a larger scope)
      • ‎
    Number of citationsNumber of times each article has been cited according to Web of Science or Scopus

    aLHS: Learning Health System.

    ‎
    Figure 2. Summary of search strategy and themes.
    View this figure

    Analytical Strategy

    The analysis presented in this paper draws on both quantitative and qualitative approaches. First, we identified historical trends in the LHS discourse by quantitatively assessing the number of articles published on the LHS over time. We then evaluated article categories over time and assessed the variety of clinical domains that have integrated the LHS discourse for the 85 articles selected for review.

    Next, we performed a thematic mapping analysis to qualitatively examine the variety of ways in which articles define the LHS. This is a method for extracting, analyzing, and reporting themes in data, such as from interviews or texts [15]. Themes describe textual data that are grouped around a main issue [16] and are recurrent and systematic [17]. Moreover, themes emerge from the data and can be refined into levels or subthemes that reflect on the theme [18]. In this review, we initially identified broad themes that emerged from the articles. These themes were then refined into subthemes. Finally, we compared the qualitative findings from our literature review with the vision developed by Etheredge [1] in 2007 to assess the extent to which these initiatives have been undertaken and discussed according to the literature over the past decade and to identify areas for continued effort. One member of the study team coded the 85 articles using the MAXQDA software and shared codes and code relation matrices with the other members to ensure agreement. The study team held meetings every 2 weeks to discuss emerging themes and subthemes, thus arriving at the final list iteratively and collaboratively.


    Defining the Learning Health System

    Articles in our final sample most commonly draw upon the Institute of Medicine’s (now the National Academy for Medicine) ambitious and encompassing definition describing LHSs as those that: generate and apply the best evidence for the collaborative health carechoices of each patient and provider;…drive the process of discovery as a natural outgrowth of patient care; and ensure innovation, quality, safety, and value in health care [19].

    These articles describe the LHS as an avenue for delivering more targeted, safe, and effective health care by using information from the experiences and treatment of patients to inform decision making and subsequent care in real time [20,21]. A central ambition, as well as a vehicle for the aspirations of the LHS, is the notion of learning. Importantly, learning itself is considered the transfer of knowledge through formal curricula (eg, during medical training) and a transfer of culture, attitudes, and beliefs in ways that can be implemented in research as well as practice when combined with QI methods [21,22]. For example, Faden et al [23] describe learning as a process including research, information gleaned from QI efforts, and comparative effectiveness research, culminating in improved practice [24]. In this way, QI is described as a critical stepping stone to learning, which depends on research and also has the potential to bridge the traditional gap between research and practice to inform the goals of an LHS [24].

    Some articles refer to the LHS more specifically, as a rapid learning health system to emphasize the celerity—eg, real-time delivery—with which evidence-based medicine can be used to determine health care decisions [1,25]. A system, in this context, is characterized as requiring a coherent, flexible organizational structure, with data maintained in a repository until needed for a particular purpose [26]. In addition, mechanisms are in place to ensure that data are usable by all entities within the system and that they are transferred appropriately, safely, and ethically. Data originate from clinical practice, research, participation, and inquiry provided by organizational leaders, physicians, researchers, patients, and research participants [26]. Although some publications in our sample offer a discussion of the LHS as a large-scale system spanning borders, others implement, evaluate, or present challenges on the system at a local or regional scale.

    Quantitative Trends in the Learning Health System Discourse

    Our review indicated historical trends in the LHS discourse, with a particular surge in articles published between 2013 and 2017. This trend, we noticed in the broader literature search, is consistent with the trends in articles we selected for review (Figure 3).

    Our review resulted in the selection of 85 articles published between 2007 and 2017. Most of the 85 articles included in our review were published in 2014, and the articles cited most were published in 2010 (132 citations); 2007 (116 citations); and 2013 (102 citations).

    ‎
    Figure 3. Frequency of articles published by year (n=542).
    View this figure

    Trends in Categories of Articles

    Following the categorization of each paper according to its primary concerns, we found that a striking majority of articles are concerned with theory (n=33), followed by implementation (n=23), commentary (n=18), advocacy, ie, promoting the idea or driving the demand for an LHS (n=12), empirical data (n=11), ethics (n=8), and policy (n=7). Articles published in 2007 were exclusively theory and/or advocacy; in 2010, articles were published across a broader variety of categories. However, articles predominantly discussing ethics in our sample were only published in 2013; similarly, policy articles were only published in 2007 and 2014 (Figure 4).

    ‎
    Figure 4. Trends in article categories among frequently cited articles based on review of frequently cited articles (n=85).
    View this figure

    Most of the implementation and empirical articles discuss the LHS in the context of oncology (n=10) [27-36]; however, other contexts include primary care (n=3) [37,38], surgery (n=3) [20,39], and pediatrics (n=7) as well as others. Some articles, such as commentary articles, exemplify clinical contexts such as pediatrics (n=3) and oncology (n=3) but are not exclusively concerned with a particular clinical context. In Table 2, we summarize the clinical contexts of articles included in our review, including the initiative of interest in each article. Articles were classified based on the objective of the literature—ie, (1) whether the article discusses or is concerned with the use of the initiative for QI, (2) whether the article discusses or is concerned with the use of the initiative for research, and/or (3) whether the article specifically evaluates the initiative with regard to improving quality or research. Most of the articles are classified as being concerned with QI, eg, around EHR point-of-care diagnostic capabilities or in clinical guidelines and subsequent process measures via networks through which information is shared. Some of these articles frame research as a stepping stone for improving point-of-care quality; however, their ultimate objective is not to improve research itself. Some studies evaluate the efficacy of QI initiatives. Fewer articles (n=9) are concerned with initiatives seeking to improve research, eg, through the development of clinical trials, or by considering the limitations in the processes through which data are collected in research used to inform practice (eg, informed consent).

    Table 2. Clinical context of articles.
    Article type, clinical context, sourceName or focus of initiativeClassification
    Implementationa

    		Oncology


    		Buetow and Niederhuber [29]Cancer Biomedical Informatics GridQIb


    		Abernathy et al [28]Rapid learning health careQI


    		Elson et al [27]Athena Breast Health NetworkQI


    		Sledge et al [30]CancerLinQQI


    		Schilsky et al [31]CancerLinQQI


    		Abernathy et al [32]Innovation in oncologyQI

    		Primary care


    		Delaney et al [37]Electronic Primary Care Research NetworkEc


    		Peterson et al [38]Practice-based Research NetworksE

    		Surgery


    		Kwon et al [39]Surgical Care and Outcomes Assessment ProgramQI/E


    		Flum et al [20]Comparative Effectiveness Research Translation NetworkQI/E

    		Pediatrics


    		Forrest et al [40]PEDSnet consortium of 8 children’s hospitalsQI


    		Forrest et al [41]ICNdQI/E

    		Endocrinology



    		Fiore et al [42]Point-of-care clinical trialRe/E

    		Cardiology


    		Califf and Sugarman [43]

    		Pragmatic clinical trials

    		R



    		Maddox et al [44]Veterans Administration Clinical Assessment, Reporting, and Tracking programQI/E

    		Intensive Care Unit



    		Warner et al [45]Temporal phenotype data via an EHRfQI/E

    		Gastro-enterology



    		Forrest et al [41]ICNQI/E

    		Public Health Departments



    		Klann et al [46]Query HealthQI/E

    		Health systems


    		Greene et al [47]Rapid learning health systemQI


    		Harper [48]Clinical Demand IndexQI/E


    		Weng et al [49]Integrated Model for Patient Care and Clinical TrialsR/E


    		Forrest et al [40]PEDSnet consortium of 8 children’s hospitalsQI


    		McGlynn et al [50]Patient Outcomes Research to Advance Learning networkR


    		Mandl et al [51]Scalable Collaborative Infrastructure for a Learning Healthcare SystemR/E
    Empirical

    		Pediatrics


    		Lannon and Peterson [52]Pediatric Collaborative Improvement NetworksQI/E


    		Kelley et al [53]Examining attitudes toward researchR

    		Oncology



    		Spinks et al [33]Not applicableNot applicable

    		Surgery



    		Pingleton et al [21]Evaluating quality and patient safety curriculaQI/E

    		Nephrology



    		Kelley et al [53]Not applicableNot applicable

    		Primary care and/or internal medicine



    		Pingleton et al [21]Evaluating quality and patient safety curriculaQI/E
    General

    		Pediatrics


    		Clancy et al [22]Collaborative Improvement NetworksQI/R


    		Gardner and Kelleher [54]LHSg for PediatricsQI


    		Kahn et al [55]Common Pediatric Research TerminologyR/E

    		Oncology


    		Feeley et al [34]Health Information Technology to improve quality of cancer careQI


    		Shaikh et al [35]Collaborative BiomedicineQI


    		Shah et al [36]CancerLinQR/E

    aItalicized categories describe concern or focus of article.

    bQI: quality improvement.

    cE: evaluation.

    dICN: ImproveCareNow.

    eR: research.

    fEHR: electronic health record.

    gLHS: Learning Health System

    Themes

    Our thematic analysis revealed five broad themes across the 85 articles we reviewed: (1) culture, ie, the environment or environmental change required to support the LHS; (2) innovations, including new tools or ideas needed or being developed to realize the vision of an LHS; (3) data infrastructure; and (4) ethical considerations generally framed as moral imperatives or in terms of principles such as privacy and efficiency [15]. Although some articles present evidence or discussion on multiple themes, others tend to focus exclusively on one theme, eg, articles commenting on ethical aspects. Furthermore, articles focusing on technical or research aspects rarely comment on social or ethical considerations of the LHS.

    Culture

    Most articles (n=81) set clear expectations and priorities for advancing the LHS and achieving its goals. Culture, describing the way in which research and clinical care are considered and how they contribute to the vision or mission of the LHS, is central to enhancing the system. For example, academic health centers are discussed for balancing high-quality teaching with attention to increasing customer service, productivity, and research missions and reducing knowledge gaps for evidence-based medicine [56]. In the reviewed papers, aspects of culture included (1) organizational culture; (2) research and practice; and (3) establishing a framework for health, ie, establishing the boundaries of research, practice, and QI while ensuring appropriate oversight of risks and benefits. These aspects of culture align with the initial priorities set by Etheredge in 2007, including and extending beyond leadership and collaboration, to include actionable steps such as professional education [1].

    Innovations, Tools, and Ideas

    Most of the reviewed articles (n=77) propose ideas and tools or evaluate innovations that are anticipated to contribute to achieving an LHS by (1) enhancing existing research and (2) envisioning, developing, and enhancing care methodologies to ultimately be implemented in care. For example, the Patient Outcomes Research to Advance Learning network is a research network studying the effectiveness of various approaches to diagnosis, treatment, and management, which could create cohorts of patients with common diagnoses to conduct large-scale comparative effectiveness research to accomplish its goals of assisting patients, caregivers, and physicians in making informed decisions [50]. Other tools, in contrast, are concerned with enhancing methodologies for clinical practice. For example, ImproveCareNow (ICN) is a network aiming to transform the health of children with Crohn disease and ulcerative colitis through a collaboration of pediatric gastroenterology practices working together to develop and enhance care methodologies [57]. The innovations, tools, and ideas that have been developed since 2007 seem particularly responsive to Etheredge’s envisioning of moving toward an LHS via the EHR, predictive modeling, and software development [1].

    Data Infrastructure

    Data infrastructure emerges as a key theme in articles (n=67), with authors discussing a variety of technical avenues for pursuing and/or achieving the LHS. Given that the LHS framework is heavily reliant on an emerging technical infrastructure, the significance of this theme is not surprising. Publications are concerned with (1) networks and platforms; (2) expanding study and data reliability and/or validity; and (3), particularly, the EHR as a valuable resource. In 2007, Etheredge [1] discussed the Cancer Research Network, the Vaccine Safety Datalink network, and the American Medical Group Association as exemplars of data infrastructure progressing toward an LHS. A decade later, the literature is discussing myriad networks and platforms such as the National Patient-Centered Clinical Research Network and the data-sharing platform, PopMedNet [46,50].

    The literature argues that these networks and platforms rely on robust participation to build reliability and validity; however, reported participation is limited owing to inadequate sampling, limited availability of clinical information in datasets, lack of rigorous inclusion and/or exclusion criteria resulting in confounders, and resulting issues around generalizability of findings [58-60]. Furthermore, the quality of data suffers because of issues such as inconsistent terminology and the use of statistical techniques that are not advanced enough for complex observational data aggregated from the EHR [37,61]. However, articles regard the EHR as a valuable resource with the capacity to capture, communicate, aggregate, store, and analyze large pools of data for real-time clinician decision support and, broadly, for establishing a nationwide LHS [26,30,48]. Indeed, articles address limitations of the EHR spanning from the practice of individual teams building their own data repositories to the need for manual data aggregation [25,39]. However, articles urge enhancing data interoperability and tailoring the technology to specific clinical contexts such as oncology to ensure effective use of the EHR [28].

    Ethics

    According to Faden et al [23], the development of an LHS relies heavily on altruism and the understanding that although participating in research may not offer personal therapeutic or curative benefits, what is inherent to clinical research is the potential for large-scale benefits to society by filling knowledge gaps and enhancing care methods. Only eight papers focus on ethics, and most of these are part of the same special issue of the Hastings Center Report in 2013 [24,60,62]. Yet, many other papers—just under half of the reviewed papers (n=39)—discuss at least some ethical and/or social aspects of the LHS, including what is often posited as challenges in combining research and clinical frameworks and cultures. Specifically, articles address (1) efficacy and patient safety in research and practice, (2) the moral imperative on patients to participate in research, (3) effects of participation, eg, on privacy and the lack of guaranteed therapeutic benefits in research studies, and (4) questions of social justice (ie, fair subject selection and a just distribution of research benefits and burdens) [23,24,43,63]. For example, patients who are actively undergoing treatment may be more willing to participate in research as a result of their current experience of benefiting from treatment and scientific knowledge. However, as participants in an LHS, EHRs and other data sources will be used during, and after, specific instances of being a patient. Notions of benefit and risk are likely to change over time, both for a specific individual and within the context of a given institution. Ongoing conversations about the risks and benefits of participation, and what the idiom research refers to, will be critical to achieving adequate engagement and ensuring that willingness to participate is not confounded with a need for care [64,65]. Furthermore, potential users of health care services may anticipate receiving further and/or advanced treatment for long-term and rare diseases. For example, in the context of the United States, health care is not free of charge and features some of the highest costs globally for health services and long-term care. A question of social justice would be how data from patients are used for the health system to ultimately give back to patients who are charged for services and whose health and associated data may be turned into monetizable assets by health care providers.

    It is a positive sign that such a significant number of articles in the sample engage with ethical and social aspects of the LHS. However, this engagement still focuses primarily on efficacy and safety and on espousing a moral duty for physicians, clinicians, and patients to participate in the LHS. In his 2007 article, Etheredge [1] discusses the ethical implications of an LHS, particularly in terms of responsibility, gaps in our understanding of comparative benefits and risks of clinical research, and prescriptions across minority and special needs groups, and raises issues of patient confidentiality. The literature, to date, concerned with ethical implications of an LHS, though sparse, continues to prioritize these gaps. Furthermore, it urges the consideration of participation and recognizes some of these issues as relating to social justice more broadly.

    Progress and Opportunities Over the Past Decade

    Over the past decade, there has been tremendous movement with regard to the technical side of LHS infrastructure including the EHR, which is discussed in much of the literature as a valuable resource for research, clinical care, and the translation from one to the other [66,67]. Much of the literature in our review focuses on using EHRs for QI, signaling progress in the use and impact of health information technology on moving toward the vision of the LHS. However, our findings suggested a lack of attention to ethical considerations, eg, social justice issues, to intersections of different types of data that might enter potentially large-scale data systems such as the LHS (health, welfare, education, criminal justice, etc), and to the role, recruitment, and retention of participants in research, and equitable distribution of benefits. These aspects of health care and research have been discussed in other bodies of health literature but do not seem to have been adapted widely for and by the LHS discourse. There is also a need for further consideration of leadership, collaboration, and responsibilities of various entities within the health care system. Etheredge [1] cites Medicaid as a key leader of the LHS; for leaders to emerge, it is essential to identify, understand, and map the responsibilities of LHS leaders and identify how other stakeholders such as providers, patients, and hospitals conceptualize the LHS as a sociotechnical system.


    Principal Findings

    Since Etheredge’s paper [1] and the US Institute of Medicine’s roundtable report in 2007 [2], more than 500 academic papers have been published discussing the LHS, with most of these emerging since 2014. This work predominantly cites empirical research describing technical developments and analytical capacity. However, LHS infrastructure concepts navigate complex systems of policy, ethics, networks, and processes in local clinical care and research settings as well as in the global consortia. Although the literature is expanding with the rate of technological innovation, an empirically based ethical analysis of large health data systems, including the deliberation of their ethical management and sociotechnical engagements, further requires considerable attention.

    Articles in our review generally suggest an engagement with the LHS discourse by exploring or evaluating tools and innovations to bring the LHS framework to fruition in research and practice; by examining technical infrastructures to test the process of aggregating data from research to implement findings into practice; and by evaluating challenges and subsequent priorities to facilitate the LHS across teams and institutions. Our sample suggests that the LHS framework is considered suitable for, and is in parts already utilized across, a variety of institutions and clinical contexts, albeit not anywhere near the capacity of what the LHS framework suggests might be possible. The publication numbers themselves are indicative of the desire and effort to present, develop, and apply the LHS in action as research, technical, and clinical teams collaborate and propose new models. However, while some articles test the LHS framework as an avenue to pursuing QI, others consider the LHS a framework for implementable research. There is a need to examine whether, in applying an LHS model, a boundary exists between QI and research, and, further, the terms and implications of such a boundary. At present, the expectations and requirements for informed consent, notification, return of research results, risk assessment, privacy, and confidentiality are vastly different for research vs QI.

    Future studies should examine stakeholder perceptions of the boundaries between QI and research, engaging patients, healthy participants, researchers, clinicians, and technical designers involved in LHS teams to understand the extent to which they distinguish between the two. Importantly, the ways that the LHS might be utilized need to be clarified to develop appropriate and effective governance frameworks.

    Our review revealed a number of trends and themes in the LHS discourse that may inform avenues for further deliberation to contribute to a more robust discussion of the LHS. For example, the relatively small number of articles comprehensively engaging with its social, ethical, and governance aspects is surprising. This may suggest limited discussion of the LHS as a concept in the social sciences and humanities and an opportunity for further examination of aspects and implications of the LHS from these perspectives—ongoing discussions in bodies of social science scholarship around health and data would provide fruitful platforms for further discussion. The LHS relies on the collaboration of various stakeholders in research, clinical, and technical arenas to utilize technology for the improvement of patient health, in fact forming an ecosystem of interrelated and linked activities. One might expect that these interpersonal interactions as well as interactions between stakeholders and technologies would prompt further discussion of the social and ethical concerns that might arise or of the social and societal enablers, drivers, and impacts of the LHS. For example, the intersection between the patient duty to participate and the technical challenge of maintaining privacy, including data security, could be examined further to understand whether there are nuances to when patients may be more or less willing to share data across a network and to understand what the processes for decision making and data sharing need to look like to be compatible with data subjects’ needs [23].

    To date, the LHS has primarily focused on the technical access between different users (via EHRs) and across local networks; this is reflected in the articles in our sample as well, suggesting a need for tests of implementation at a larger scale. ICN is an example of a regional-level network; there is the possibility of testing similar networks at a global level. It is possible that owing to our criteria of including only articles written in English, our sample excluded global-level examples of LHSs; however, it is unclear whether there are current collaborations between global settings and whether this is a part of the current LHS vision. We do know that aspects of types of LHSs—such as data governance in health care research—are being discussed in other bodies of the literature but that scholarship either does not refer to the LHS or does not take a systemic approach to health research and care.

    Implications for Future Research, Practice, and Policy

    Our review suggests that although some knowledge gaps and uncertainties presented by Etheredge [1] have received consideration over the past decade, significant gaps persist, presenting opportunities for future research and policy (Table 3).

    Table 3. Progress over the past decade in addressing knowledge gaps and uncertainties.
    Knowledge gaps and uncertainties (Etheredge, 2007) [1]Perspectives from the literature (2007-2017)Areas for considering the way forward
    Diffused responsibilityRobust discussion of the need for leadership and coordination across various stakeholders, research, policy, and practice. Tools exist for development of new technologies; however, they appear to be largely segregated into those for enhancing research vs practice, leading to disparate regulatory oversightFurther examination is needed of stakeholder perspectives of funding and responsibility for supporting research. Need for a better understanding of the roles and responsibilities of those involved in research from collection to sharing and implementation
    Concerns about clinical trialsThe past decade has seen a variety of innovations for enhancing research as well as clinical care, along with the emergence of networks and platforms for trials and studies across various clinical contexts. The literature sees a continued need for clinical trials to support precision medicine but asserts the need for ensuring that participants are well-informed about risks and benefits of participating, and that sampling and distribution of benefits are equitableContinued examination of public needs, attitudes, beliefs, and knowledge about participating in research and clinical trials, especially at a systemic level of health research and care data agglomeration and learning cycles
    Consequences of underinvestmentThe literature suggests that the emergence of networks and platforms, as well as the increasing use of the EHRa has facilitated investment in evidence-based research. However, there is a need for ensuring that data are interoperable and meaningful, that studies and data are reliable and valid, and that the public is better able to envision its role in researchNeed for an enhanced design of the EHR to increase interoperability, standardization, and quality of data to ensure that findings are translatable and generalizable. Ensuring adequate regulation of research and informed consent for participants
    Genetics and genomics-based medicineData quality, reliability, and validity are critical to genetics and genomics-based medicine. The literature primarily discusses a need for ensuring the just and equitable distribution of benefits and minimization of risksSampling must include patient populations beyond the socioeconomic environment of a hospital or region, and rigorous inclusion and exclusion criteria
    Overall system performanceThe literature extensively discusses the EHR as a resource for improving quality; several articles urge for a consideration of research vs quality improvement. In addition, there is a need for improving and developing curricula in academic medical centers on research and clinical safety and qualityMoving the LHSb forward requires further theoretical consideration of research vs quality improvement. The literature is mixed and defining boundaries will be beneficial to examining the policy and practice implications of rapid learning

    aEHR: electronic health record.

    bLHS: Learning Health System.

    Several authors have theorized and advocated the need and potential for the LHS to enhance patient safety and social justice. However, few empirical studies have actually examined the effects of immersing in an LHS culture on the patient-doctor relationship, on professional relationships in health systems, or on public accountability and governance. Generally, only a small subset of publications is acutely engaged with key ethical concerns such as trust, solidarity, equity, or privacy. Those that are engaged with these ethical concerns primarily do so from the perspective of calling attention to these concepts as potential rather than actual concerns. Given the ideal of a technical infrastructure that supports and is supported by a social and cultural learning system, it is important for studies to examine whether and how, eg, relationships between providers and administrative leaders may be different in institutions that have implemented an LHS-based technical framework. As such, we suggest that rather than considering concepts such as trust as a challenge, burden, or barrier necessary to master or overcome, the LHS discourse requires studies on how trust affects and is affected by the LHS context to understand its potential to enhance the LHS, if strengthened.

    Many cited challenges and needs in our review align with governance concerns—eg, limitations of data safety and reliability, validity of data, or validity of the algorithmic outputs of data in an LHS. While this literature appears to focus on overcoming these hurdles through consideration of the moral obligation of researchers and participants, there was little indication of future research specifically examining the ethical and social aspects of health care and clinical research, eg, through the lens of dignity, equity, equality, and social justice. We suggest that future research needs to examine recruitment and participation in research with a particular focus on the moral imperative, to better understand implications of various approaches to enhancing participation in research on the LHS. Furthermore, the discourse around the LHS needs to attend more to the ethical implications of such systems on patients, participants, and their data and those who may share patient or participant data from both technical and ethical perspectives (eg, physicians, hospitals, public health departments, and payers). For example, it would be important to clearly understand and present practical (eg, economic and therapeutic), social, and ethical rationales for patient participation in clinical research at a systemic level of data agglomeration and analysis, outputs of which may be subject to commercial interests.

    The EHR is considered a valuable resource in publications in our review, a number of articles also point specifically to this resource as requiring improvements to effectively and ethically carry through the LHS vision. Notably, EHRs are financially incentivized in the United States, making the local context ideal for implementing the LHS nationally. At the time Etheredge [1] made his call for the LHS, the Health Information Technology for Economic and Clinical Health (HITECH) Act and the US $35 billion investment in the adoption and meaningful use of EHRs had yet to materialize [68]. Since the HITECH Act, adoption and use of the EHR has continued to increase with evidence of better health outcomes after systems have had time to mature [67,69]. However, studies have also suggested consequences of EHR use on user (patient and provider) satisfaction, calling to attention the need for considering and understanding implications of the LHS on patients, and the ways in which their embeddedness in the LHS is influenced by the systemic pressures driving health information technology [70-72]. In addition, there is a need to improve ethical governance, and that continues to include necessary progress on enabling data security. Importantly, improving security does not necessarily mean restricting data from leaving the environment of a screen in an office room at a particular institution; however, there is a need to better understand the boundaries of data as understood by stakeholders including patients, clinicians, and researchers, and their rights and views on how data are retained and used. Furthermore, the information that is stored on the EHR is not always in a standardized format—eg, some data may be in the form of narrative text, whereas others may be more easily coded. For information to be usable, it is necessary to identify ways to make EHRs interoperable and, further, the information on them interpretable in a consistent and standardized way, eg, by using natural language processing techniques [73,74].

    Although our review primarily focused on papers based in the United States, there is a need to examine the role and implications of an LHS in other health care system contexts. For example, the expectations and incentives for EHR use may vary across contexts. The procedures for recruitment and enrollment in research may also vary, and the principles governing use and sharing of data and health information may be different. In the US context, historical injustices in the research domain have brought to the forefront regulatory principles such as the Belmont Report describing the ethical foundations of research [75]. In addition, the influx of new actors within the LHS and big data practices expands questions related to who is doing research and innovation, including sources of funding and commercial interests. These evolutions are likely to guide the direction of work related to developing, implementing, and attending to the ethical and social considerations of an LHS [76]. As other countries and health care contexts envision and implement the LHS, it will be important to study how the histories and policies of these countries influence the LHS and how key stakeholders perceive and have expectations for what an LHS can deliver.

    Finally, our sample of publications is indicative of the successes or anticipated challenges with the LHS; however, learning also results from an understanding of what goes wrong. Attempts at implementing an LHS along with evaluations of the process should be disseminated as well, for a more robust understanding of what might be optimal conditions for an LHS and what contexts may need more malleability to adopt an LHS framework and culture. For example, future studies may seek to identify the conditions under which the use of emerging technologies and delivery models achieve the goals of an LHS. For example, are the adoption of telemedicine and the use of patient-generated data via health apps and wearable devices contributing to systems that are also engaged in learning and/or feedback? Furthermore, how are the technical, social, and ethical implications of learning from clinical care delivered through telemedicine different from learning and knowledge generation that occurs through traditional, face-to-face care delivery?

    Conclusions

    The past decade has observed tremendous progress in the technical infrastructure and innovative spirit of advancing the framework and implementation of the LHS. However, there are opportunities to continue to examine and develop policies ensuring the cultural and ethical imperatives for an LHS. Since Etheredge [1] presented an initial vision of the LHS in 2007, the growing implementation of the EHR and emergence of networks and platforms have contributed to the initial stages of understanding the LHS framework. However, to continue to define and realize the aims of an LHS, there is a need to consider the boundaries of research, practice, and QI. Furthermore, there is a need for engaging with various stakeholders including patients, participants, providers, and organizational leaders in clinical, technical, administrative, and research domains to understand the broader implications of an LHS.

    Acknowledgments

    The authors thank M Grace Trinidad for developing Figure 1 that illustrates the areas critical to the realization of an LHS.

    Conflicts of Interest

    None declared.

    1. Etheredge LM. A rapid-learning health system. Health Aff (Millwood) 2007;26(2):w107-w118. [CrossRef] [Medline]
    2. Olsen L, Aisner D, McGinnis JM, Roundtable on Evidence-Based Medicine, Institute of Medicine. The Learning Healthcare System: Workshop Summary. Washington, DC: National Academies Press; 2007.
    3. Dean BB, Lam J, Natoli JL, Butler Q, Aguilar D, Nordyke RJ. Review: use of electronic medical records for health outcomes research: a literature review. Med Care Res Rev 2009 Dec;66(6):611-638. [CrossRef] [Medline]
    4. Treweek S, Mitchell E, Pitkethly M, Cook J, Kjeldstrøm M, Taskila T, et al. Strategies to improve recruitment to randomised controlled trials. Cochrane Database Syst Rev 2010 Jan 20(1):MR000013. [CrossRef] [Medline]
    5. Swan M. Emerging patient-driven health care models: an examination of health social networks, consumer personalized medicine and quantified self-tracking. Int J Environ Res Public Health 2009 Feb;6(2):492-525 [FREE Full text] [CrossRef] [Medline]
    6. Bourne PE, Lorsch JR, Green ED. Perspective: sustaining the big-data ecosystem. Nature 2015 Nov 5;527(7576):S16-S17. [CrossRef] [Medline]
    7. Hafen E, Kossmann D, Brand A. Health data cooperatives - citizen empowerment. Methods Inf Med 2014;53(2):82-86. [CrossRef] [Medline]
    8. Platt J, Kardia S. Public trust in health information sharing: implications for biobanking and electronic health record systems. J Pers Med 2015 Feb 3;5(1):3-21 [FREE Full text] [CrossRef] [Medline]
    9. Flynn AJ, Friedman CP, Boisvert P, Landis-Lewis Z, Lagoze C. The Knowledge Object Reference Ontology (KORO): a formalism to support management and sharing of computable biomedical knowledge for learning health systems. Learn Health Syst 2018 Apr;2(2):e10054 [FREE Full text] [CrossRef] [Medline]
    10. ASCO CancerLinQ.   URL: https://www.cancerlinq.org/ [accessed 2020-01-23]
    11. Asnaani J. HealthIT. 2015 Aug. CommonWell Health Alliance: Interoperability for the Common Good   URL: https://www.healthit.gov/sites/default/files/facas/ITF_CommonWell_POV_2015-08-14_v2.pdf [accessed 2020-01-23]
    12. Collins FS, Varmus H. A new initiative on precision medicine. N Engl J Med 2015 Feb 26;372(9):793-795 [FREE Full text] [CrossRef] [Medline]
    13. NHS Lambeth CCG. Lambeth DataNet   URL: https:/​/www.​lambethccg.nhs.uk/​your-health/​Information-for-patients/​lambeth-datanet/​Pages/​default.​aspx [accessed 2020-01-23]
    14. Connected Health Cities. CHC HUB   URL: https://www.connectedhealthcities.org/chc-hub/ [accessed 2020-01-23]
    15. Boyatzis RE. Transforming Qualitative Information: Thematic Analysis and Code Development. Thousand Oaks, CA: Sage; 1998.
    16. Brink P, Wood MJ, Marilynn JW, Pamela JB. Advanced Design in Nursing Research. Thousand Oaks, CA: Sage; 1997.
    17. Polit DF, Hungler BP, Beck CT. Essentials of Nursing Research: Methods, Appraisal, and Utilization. Philadelphia: Lippincott Williams & Wilkins; 1994.
    18. DeSantis L, Ugarriza DN. The concept of theme as used in qualitative nursing research. West J Nurs Res 2000 Apr;22(3):351-372. [CrossRef] [Medline]
    19. Olsen L, Saunders RS, McGinnis JM, Institute of Medicine, Roundtable on Value & Science-Driven Health Care, The Learning Health System Series. Patients Charting the Course: Citizen Engagement and the Learning Health System: Workshop Summary. Washington, DC: National Academies Press; 2011.
    20. Flum DR, Alfonso-Cristancho R, Devine EB, Devlin A, Farrokhi E, Tarczy-Hornoch P, CERTAIN Collaborative. Implementation of a 'real-world' learning health care system: Washington State's Comparative Effectiveness Research Translation Network (CERTAIN). Surgery 2014 May;155(5):860-866. [CrossRef] [Medline]
    21. Pingleton SK, Davis DA, Dickler RM. Characteristics of quality and patient safety curricula in major teaching hospitals. Am J Med Qual 2010;25(4):305-311. [CrossRef] [Medline]
    22. Clancy CM, Margolis PA, Miller M. Collaborative networks for both improvement and research. Pediatrics 2013 Jun;131(Suppl 4):S210-S214. [CrossRef] [Medline]
    23. Faden RR, Beauchamp T, Kass N. Learning health care systems and justice. Hastings Cent Rep 2011;41(4):3. [CrossRef] [Medline]
    24. Faden RR, Kass NE, Goodman SN, Pronovost P, Tunis S, Beauchamp TL. An ethics framework for a learning health care system: a departure from traditional research ethics and clinical ethics. Hastings Cent Rep 2013;43(s1):S16-S27. [CrossRef] [Medline]
    25. Etheredge LM. Rapid learning: a breakthrough agenda. Health Aff (Millwood) 2014 Jul;33(7):1155-1162. [CrossRef] [Medline]
    26. Friedman CP, Wong AK, Blumenthal D. Achieving a nationwide learning health system. Sci Transl Med 2010 Nov 10;2(57):57cm29. [CrossRef] [Medline]
    27. Elson SL, Hiatt RA, Anton-Culver H, Howell LP, Naeim A, Parker BA, Athena Breast Health Network. The Athena Breast Health Network: developing a rapid learning system in breast cancer prevention, screening, treatment, and care. Breast Cancer Res Treat 2013 Jul;140(2):417-425. [CrossRef] [Medline]
    28. Abernethy AP, Etheredge LM, Ganz PA, Wallace P, German RR, Neti C, et al. Rapid-learning system for cancer care. J Clin Oncol 2010 Sep 20;28(27):4268-4274 [FREE Full text] [CrossRef] [Medline]
    29. Buetow KH, Niederhuber J. Infrastructure for a learning health care system: CaBIG. Health Aff (Millwood) 2009;28(3):923-4; author reply 924. [CrossRef] [Medline]
    30. Sledge GW, Hudis CA, Swain SM, Yu PM, Mann JT, Hauser RS, et al. ASCO's approach to a learning health care system in oncology. J Oncol Pract 2013 May;9(3):145-148 [FREE Full text] [CrossRef] [Medline]
    31. Schilsky RL, Michels DL, Kearbey AH, Yu PP, Hudis CA. Building a rapid learning health care system for oncology: the regulatory framework of CancerLinQ. J Clin Oncol 2014 Aug 1;32(22):2373-2379. [CrossRef] [Medline]
    32. Abernethy A, Abrahams E, Barker A, Buetow K, Burkholder R, Dalton WS, et al. Turning the tide against cancer through sustained medical innovation: the pathway to progress. Clin Cancer Res 2014 Mar 1;20(5):1081-1086 [FREE Full text] [CrossRef] [Medline]
    33. Spinks TE, Ganz PA, Sledge GW, Levit L, Hayman JA, Eberlein TJ, et al. Delivering high-quality cancer care: the critical role of quality measurement. Healthc (Amst) 2014 Mar 1;2(1):53-62 [FREE Full text] [CrossRef] [Medline]
    34. Feeley TW, Sledge GW, Levit L, Ganz PA. Improving the quality of cancer care in America through health information technology. J Am Med Inform Assoc 2014;21(5):772-775 [FREE Full text] [CrossRef] [Medline]
    35. Shaikh AR, Butte AJ, Schully SD, Dalton WS, Khoury MJ, Hesse BW. Collaborative biomedicine in the age of big data: the case of cancer. J Med Internet Res 2014 Apr 7;16(4):e101 [FREE Full text] [CrossRef] [Medline]
    36. Shah A, Stewart AK, Kolacevski A, Michels D, Miller R. Building a rapid learning health care system for oncology: why CancerLinQ collects identifiable health information to achieve its vision. J Clin Oncol 2016 Mar 1;34(7):756-763. [CrossRef] [Medline]
    37. Delaney BC, Peterson KA, Speedie S, Taweel A, Arvanitis TN, Hobbs FD. Envisioning a learning health care system: the electronic primary care research network, a case study. Ann Fam Med 2012;10(1):54-59 [FREE Full text] [CrossRef] [Medline]
    38. Peterson KA, Lipman PD, Lange CJ, Cohen RA, Durako S. Supporting better science in primary care: a description of practice-based research networks (PBRNs) in 2011. J Am Board Fam Med 2012;25(5):565-571 [FREE Full text] [CrossRef] [Medline]
    39. SCOAP Collaborative‚ Writing Group for the SCOAP Collaborative, Kwon S, Florence M, Grigas P, Horton M, Horvath K, et al. Creating a learning healthcare system in surgery: Washington State's Surgical Care and Outcomes Assessment Program (SCOAP) at 5 years. Surgery 2012 Feb;151(2):146-152 [FREE Full text] [CrossRef] [Medline]
    40. Forrest CB, Margolis PA, Bailey LC, Marsolo K, del Beccaro MA, Finkelstein JA, et al. PEDSnet: a national pediatric learning health system. J Am Med Inform Assoc 2014;21(4):602-606 [FREE Full text] [CrossRef] [Medline]
    41. Forrest CB, Crandall WV, Bailey LC, Zhang P, Joffe MM, Colletti RB, et al. Effectiveness of anti-TNFα for Crohn disease: research in a pediatric learning health system. Pediatrics 2014 Jul;134(1):37-44 [FREE Full text] [CrossRef] [Medline]
    42. Fiore LD, Brophy M, Ferguson RE, D'Avolio L, Hermos JA, Lew RA, et al. A point-of-care clinical trial comparing insulin administered using a sliding scale versus a weight-based regimen. Clin Trials 2011 Apr;8(2):183-195 [FREE Full text] [CrossRef] [Medline]
    43. Califf RM, Sugarman J. Exploring the ethical and regulatory issues in pragmatic clinical trials. Clin Trials 2015 Oct;12(5):436-441 [FREE Full text] [CrossRef] [Medline]
    44. Maddox TM, Plomondon ME, Petrich M, Tsai TT, Gethoffer H, Noonan G, et al. A national clinical quality program for Veterans Affairs catheterization laboratories (from the Veterans Affairs clinical assessment, reporting, and tracking program). Am J Cardiol 2014 Dec 1;114(11):1750-1757. [CrossRef] [Medline]
    45. Warner JL, Zollanvari A, Ding Q, Zhang P, Snyder GM, Alterovitz G. Temporal phenome analysis of a large electronic health record cohort enables identification of hospital-acquired complications. J Am Med Inform Assoc 2013 Dec;20(e2):e281-e287 [FREE Full text] [CrossRef] [Medline]
    46. Klann JG, Buck MD, Brown J, Hadley M, Elmore R, Weber GM, et al. Query Health: standards-based, cross-platform population health surveillance. J Am Med Inform Assoc 2014;21(4):650-656 [FREE Full text] [CrossRef] [Medline]
    47. Greene SM, Reid RJ, Larson EB. Implementing the learning health system: from concept to action. Ann Intern Med 2012 Aug 7;157(3):207-210. [CrossRef] [Medline]
    48. Harper EM. The economic value of health care data. Nurs Adm Q 2013;37(2):105-108. [CrossRef] [Medline]
    49. Weng C, Li Y, Berhe S, Boland MR, Gao J, Hruby GW, et al. An Integrated Model for Patient Care and Clinical Trials (IMPACT) to support clinical research visit scheduling workflow for future learning health systems. J Biomed Inform 2013 Aug;46(4):642-652 [FREE Full text] [CrossRef] [Medline]
    50. McGlynn EA, Lieu TA, Durham ML, Bauck A, Laws R, Go AS, et al. Developing a data infrastructure for a learning health system: the PORTAL network. J Am Med Inform Assoc 2014;21(4):596-601 [FREE Full text] [CrossRef] [Medline]
    51. Mandl KD, Kohane IS, McFadden D, Weber GM, Natter M, Mandel J, et al. Scalable Collaborative Infrastructure for a Learning Healthcare System (SCILHS): architecture. J Am Med Inform Assoc 2014;21(4):615-620 [FREE Full text] [CrossRef] [Medline]
    52. Lannon CM, Peterson LE. Pediatric collaborative improvement networks: background and overview. Pediatrics 2013 Jun;131(Suppl 4):S189-S195. [CrossRef] [Medline]
    53. Kelley M, James C, Kraft SA, Korngiebel D, Wijangco I, Rosenthal E, et al. Patient perspectives on the learning health system: the importance of trust and shared decision making. Am J Bioeth 2015;15(9):4-17 [FREE Full text] [CrossRef] [Medline]
    54. Gardner W, Kelleher K. A learning health care system for pediatrics. JAMA Pediatr 2014 Apr;168(4):303-304. [CrossRef] [Medline]
    55. Kahn MG, Bailey LC, Forrest CB, Padula MA, Hirschfeld S. Building a common pediatric research terminology for accelerating child health research. Pediatrics 2014 Mar;133(3):516-525 [FREE Full text] [CrossRef] [Medline]
    56. Grumbach K, Lucey CR, Johnston SC. Transforming from centers of learning to learning health systems: the challenge for academic health centers. J Am Med Assoc 2014 Mar 19;311(11):1109-1110. [CrossRef] [Medline]
    57. Seid M, Margolis PA, Opipari-Arrigan L. Engagement, peer production, and the learning healthcare system. JAMA Pediatr 2014 Mar;168(3):201-202 [FREE Full text] [CrossRef] [Medline]
    58. Katzan IL, Rudick RA. Time to integrate clinical and research informatics. Sci Transl Med 2012 Nov 28;4(162):162fs41. [CrossRef] [Medline]
    59. Mullins CD, Vandigo J, Zheng Z, Wicks P. Patient-centeredness in the design of clinical trials. Value Health 2014 Jun;17(4):471-475 [FREE Full text] [CrossRef] [Medline]
    60. Brody H, Miller FG. The research-clinical practice distinction, learning health systems, and relationships. Hastings Cent Rep 2013;43(5):41-47. [CrossRef] [Medline]
    61. Wallace PJ, Shah ND, Dennen T, Bleicher PA, Bleicher PD, Crown WH. Optum Labs: building a novel node in the learning health care system. Health Aff (Millwood) 2014 Jul;33(7):1187-1194. [CrossRef] [Medline]
    62. Largent EA, Miller FG, Joffe S. A prescription for ethical learning. Commentary. Hastings Cent Rep 2013;43(s1):S28-S29. [CrossRef] [Medline]
    63. Aronson SJ, Rehm HL. Building the foundation for genomics in precision medicine. Nature 2015 Oct 15;526(7573):336-342 [FREE Full text] [CrossRef] [Medline]
    64. Wienroth M, Caffrey L, Wolfe C, McKevitt C. Patient-initiated recruitment for clinical research: evaluation of an outpatient letter research statement. Health Expect 2018 Apr;21(2):494-500 [FREE Full text] [CrossRef] [Medline]
    65. Wienroth M, Pearce C, McKevitt C. Research campaigns in the UK National Health Service: patient recruitment and questions of valuation. Sociol Health Illn 2019 Sep;41(7):1444-1461. [CrossRef] [Medline]
    66. Furukawa MF, King J, Patel V, Hsiao C, Adler-Milstein J, Jha AK. Despite substantial progress In EHR adoption, health information exchange and patient engagement remain low in office settings. Health Aff (Millwood) 2014 Sep;33(9):1672-1679. [CrossRef] [Medline]
    67. Adler-Milstein J, DesRoches CM, Kralovec P, Foster G, Worzala C, Charles D, et al. Electronic health record adoption in US hospitals: Progress continues, but challenges persist. Health Aff (Millwood) 2015 Dec;34(12):2174-2180. [CrossRef] [Medline]
    68. Thune J, Alexander L, Roberts P, Burr R, Enzi M. Health Affairs. 2015 Mar 4. Where Is HITECH’s $35 Billion Dollar Investment Going?   URL: https://www.healthaffairs.org/do/10.1377/hblog20150304.045199/full/ [accessed 2020-01-23]
    69. Lin SC, Jha AK, Adler-Milstein J. Electronic health records associated with lower hospital mortality after systems have time to mature. Health Aff (Millwood) 2018 Jul;37(7):1128-1135. [CrossRef] [Medline]
    70. Meyerhoefer CD, Sherer S, Deily M, Chou SY, Guo X, Chen J, et al. Provider and patient satisfaction with the integration of ambulatory and hospital EHR systems. J Am Med Inform Assoc 2018 Aug 1;25(8):1054-1063. [CrossRef] [Medline]
    71. Rose D, Richter LT, Kapustin J. Patient experiences with electronic medical records: lessons learned. J Am Assoc Nurse Pract 2014 Dec;26(12):674-680 [FREE Full text] [CrossRef] [Medline]
    72. Street RL, Liu L, Farber NJ, Chen Y, Calvitti A, Zuest D, et al. Provider interaction with the electronic health record: the effects on patient-centered communication in medical encounters. Patient Educ Couns 2014 Sep;96(3):315-319 [FREE Full text] [CrossRef] [Medline]
    73. Krumholz HM. Big data and new knowledge in medicine: the thinking, training, and tools needed for a learning health system. Health Aff (Millwood) 2014 Jul;33(7):1163-1170 [FREE Full text] [CrossRef] [Medline]
    74. Haendel MA, Chute CG, Robinson PN. Classification, ontology, and precision medicine. N Engl J Med 2018 Oct 11;379(15):1452-1462 [FREE Full text] [CrossRef] [Medline]
    75. The National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. The US Department of Health and Human Services. 1979. The Belmont Report   URL: https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/index.html [accessed 2020-01-23]
    76. Sharon T. The Googlization of health research: from disruptive innovation to disruptive ethics. Per Med 2016 Nov;13(6):563-574. [CrossRef] [Medline]

    ‎
    EHR: electronic health record
    HITECH: Health Information Technology for Economic and Clinical Health
    ICN: ImproveCareNow
    LHS: Learning Health System
    QI: quality improvement

    Edited by G Eysenbach; submitted 12.11.19; peer-reviewed by D Wyatt, S Choi; comments to author 21.12.19; revised version received 30.12.19; accepted 31.12.19; published 19.03.20

    Copyright

    ©Jodyn E Platt, Minakshi Raj, Matthias Wienroth. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 19.03.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.