This paper is in the following e-collection/theme issue:

Clinical Information and Decision Making (1314) Digital Science, Open Science (69) Knowledge Translation and Implementation Science (372) Adoption and Change Management of eHealth Systems (670) Clinical Informatics (904) Electronic Health Records (961) eHealth Infrastructures (306) Precision Medicine (63)

Published on in Vol 26 (2024)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/54265, first published .
Making Science Computable Using Evidence-Based Medicine on Fast Healthcare Interoperability Resources: Standards Development Project

Making Science Computable Using Evidence-Based Medicine on Fast Healthcare Interoperability Resources: Standards Development Project

Making Science Computable Using Evidence-Based Medicine on Fast Healthcare Interoperability Resources: Standards Development Project

Authors of this article:

Andrey Soares1 Author Orcid Image ;   Lisa M Schilling1 Author Orcid Image ;   Joshua Richardson2 Author Orcid Image ;   Bhagvan Kommadi3, 4 Author Orcid Image ;   Vignesh Subbian4, 5 Author Orcid Image ;   Joanne Dehnbostel4, 6 Author Orcid Image ;   Khalid Shahin4, 6 Author Orcid Image ;   Karen A Robinson4, 7 Author Orcid Image ;   Muhammad Afzal4, 8 Author Orcid Image ;   Harold P Lehmann4, 7 Author Orcid Image ;   Ilkka Kunnamo4, 9 Author Orcid Image ;   Brian S Alper4, 6 Author Orcid Image
Article Authors Cited by Tweetations Metrics

    Original Paper

    1Department of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, United States

    2Center for Informatics, Research Triangle Institute International, Berkeley, CA, United States

    3Quantica Computacao, Hyderabad, India

    4Scientific Knowledge Accelerator Foundation, Franklin, NC, United States

    5College of Public Health, Department of Epidemiology and Biostatistics, University of Arizona, Tucson, AZ, United States

    6Computable Publishing LLC, Franklin, NC, United States

    7Department of Medicine, Johns Hopkins School of Medicine, Baltimore, MD, United States

    8Department of Computing and Data Science, Birmingham City University, England, United Kingdom

    9Duodecim Publishing Company Ltd, Helsinki, Finland

    Corresponding Author:

    Andrey Soares, PhD

    Department of Medicine

    University of Colorado Anschutz Medical Campus

    1890 North Revere Court

    Mailstop F443

    Aurora, CO, 80045

    United States

    Phone: 1 3037242825

    Email: andrey.soares@cuanschutz.edu


    Background: Evidence-based medicine (EBM) has the potential to improve health outcomes, but EBM has not been widely integrated into the systems used for research or clinical decision-making. There has not been a scalable and reusable computer-readable standard for distributing research results and synthesized evidence among creators, implementers, and the ultimate users of that evidence. Evidence that is more rapidly updated, synthesized, disseminated, and implemented would improve both the delivery of EBM and evidence-based health care policy.

    Objective: This study aimed to introduce the EBM on Fast Healthcare Interoperability Resources (FHIR) project (EBMonFHIR), which is extending the methods and infrastructure of Health Level Seven (HL7) FHIR to provide an interoperability standard for the electronic exchange of health-related scientific knowledge.

    Methods: As an ongoing process, the project creates and refines FHIR resources to represent evidence from clinical studies and syntheses of those studies and develops tools to assist with the creation and visualization of FHIR resources.

    Results: The EBMonFHIR project created FHIR resources (ie, ArtifactAssessment, Citation, Evidence, EvidenceReport, and EvidenceVariable) for representing evidence. The COVID-19 Knowledge Accelerator (COKA) project, now Health Evidence Knowledge Accelerator (HEvKA), took this work further and created FHIR resources that express EvidenceReport, Citation, and ArtifactAssessment concepts. The group is (1) continually refining FHIR resources to support the representation of EBM; (2) developing controlled terminology related to EBM (ie, study design, statistic type, statistical model, and risk of bias); and (3) developing tools to facilitate the visualization and data entry of EBM information into FHIR resources, including human-readable interfaces and JSON viewers.

    Conclusions: EBMonFHIR resources in conjunction with other FHIR resources can support relaying EBM components in a manner that is interoperable and consumable by downstream tools and health information technology systems to support the users of evidence.

    J Med Internet Res 2024;26:e54265

    doi:10.2196/54265

    Keywords

    evidence-based medicine; FHIR; Fast Healthcare Interoperability Resources; computable evidence; EBMonFHIR; evidence-based medicine on Fast Healthcare Interoperability Resources 


    Background and Significance

    Timely and relevant biomedical evidence is essential to provide high-quality health care. Decision makers rely on “synthesized evidence” from systematic reviews (SRs) and clinical practice guidelines (CPGs) to inform clinical care decisions and policies at multiple levels [1-7]. Therefore, actionable evidence is key for optimizing health care delivery and outcomes. Yet, with an estimated 75 clinical trials and 11 SRs published every day [8], efforts to synthesize, disseminate, and implement biomedical evidence throughout the evidence ecosystem to inform decision-making are unsustainable [9].

    Evidence-based medicine (EBM) is increasingly recognized as critical in the decision-making process related to patient care and policy development. EBM is “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients” [10]. The practice of EBM depends on comprehensive and up-to-date synthesized research evidence, which requires continuous updating and reconciliation of new scientific results with previous results. For example, early trial results on prenatal steroids for preterm births were initially inconclusive, but evidence syntheses justified steroidal therapy as the best practice [11-14]. However, EBM practice is laborious, methodologically complex, and often error prone.

    Multiple barriers hinder the effective and efficient use of evidence syntheses. First, the time and effort required to synthesize evidence often leads to SRs and CPGs being out of date by the time of their publication [15,16]. Even when synthesized evidence is available in digital form, it is typically in narrative form and accessed via a bibliographical database that requires proactive searching (eg, PubMed or a publisher), an email newsletter (eg, Journal of the American Medical Association Internal Medicine Newsletter), or social media notification (eg, X, formerly known as Twitter, or Doximity). These “pull” methods of synthesized evidence are helpful means of dissemination, but they do not facilitate more efficient “push” methods that promote use, implementation, and action at scale. Second, the common mode of evidence dissemination, which is human-readable text, is not standardized in ways that support technical solutions to scalable dissemination and implementation [15-19]. Despite notable past efforts for structuring and disseminating guidance, such as the GuideLine Interchange Format (GLIF) [20,21] and Standards-Based Sharable Active Guideline Environment (SAGE) [22] standards, there are no existing scalable and reusable computer-readable standards for distributing research results and synthesized evidence among creators, implementers, and the ultimate users of that evidence.

    The current state is an ecosystem of researchers, informaticians, statisticians, policy makers, epidemiologists, librarians, and other stakeholders in the biomedical research community who synthesize evidence by manually searching for relevant studies, assessing the studies for quality and risk of bias, and compiling the results in labor-intensive ways. Evidence implementers, including CPG and clinical decision support (CDS) creators, whether they are creating or implementing local or third-party tools, must similarly review and verify the evidence before use. The challenges with finding and delivering evidence are amplified by needs due to the COVID-19 pandemic [23-28], where “scientists [have] published well over 100,000 articles about the coronavirus pandemic in 2020” [29] and for which there are currently over 2.5 million publications [30]. Today, these mostly manual, redundant, and disjointed processes seem to be the acknowledged status quo, even though new evidence continues to be generated at a rapid pace. The evidence synthesis ecosystem is, therefore, rife with duplication and uncoordinated efforts [31,32] for identifying, appraising, synthesizing, and disseminating evidence, requiring considerable resources, expertise, and time [33,34]. Evidence that is more rapidly updated, synthesized, disseminated, and implementable would improve both the delivery of EBM and evidence-based health care policy [4,35].

    Computable Evidence

    Solutions to improve the evidence-to-practice lifecycle [36] should begin with the transformation of study results into “computable evidence” or “knowledge artifacts” that could be consumed by software-based information systems. Key to computable evidence is knowledge representation in machine-interpretable formats that enable findable, accessible, interoperable, and reusable [37] information across the evidence ecosystem. Figure 1 (adapted from the “Digital and Trustworthy Evidence Ecosystem” [38]) depicts this vision where research results at many stages throughout analysis, publication, and synthesis (the EBM on Fast Healthcare Interoperability Resources [EBMonFHIR] area) can be extracted and transformed from numerous disparate sources (such as registry reports, gray literature, preprints, and peer-reviewed literature databases) and stored as machine-readable evidence in an interoperable standard format that could then be used by a wide variety of evidence users and developers from biomedical knowledge bases (ie, evidence repositories) to SRs, CPGs, and CDS systems. Figure 1 also shows where related interoperability solutions [39] applied to CPG and CDS representation (the CPGonFHIR area), such as CDS Connect (a platform with a repository and authoring tool for CDS artifacts) [40] and CDS Hooks (a Health Level Seven [HL7] specification that provides a way to embed CDS services within the clinician workflow of an electronic health record) [41], can complete the evidence-to-practice implementation within the evidence ecosystem.

    ‎
    Figure 1. Application of Fast Healthcare Interoperability Resources (FHIR) resources to the evidence ecosystem. EBMonFHIR denotes application to results of scientific analysis, synthesis, and publication. CPGonFHIR denotes application to guidance and decision support. CSV, TXT, XLS, PDF, DOC, HTML, JSON, and XML denote file formats. CDMS: clinical data management system; CDS: clinical decision support; CPG: clinical practice guideline; CQL: Clinical Quality Language; EBM: evidence-based medicine; EBMonFHIR: Evidence-Based Medicine on Fast Healthcare Interoperability Resources; EHR: electronic health record; REDCap: Research Electronic Data Capture. Adapted from the “Digital and Trustworthy Evidence Ecosystem” [38].

    The EBMonFHIR Project

    Built upon the success of Fast Healthcare Interoperability Resources (FHIR) [42-44] to promote interoperability and standards for data exchange, we initiated the EBMonFHIR project on May 16, 2018. The aim was to extend the methods and infrastructure of HL7 FHIR to provide an interoperability standard for the electronic exchange of biomedical knowledge from and about clinical research and recommendations [45]. We sought to stand on the shoulders of those who have previously made strides in structuring evidence and standardizing the means to share that evidence (eg, GLIF) by leveraging FHIR’s growing popularity and acceptance. The project solicits the participation of known experts and gathers input from broad communities in the evidence ecosystem to determine the data exchange needs for interoperable knowledge assets [45] and defines the FHIR resources related to the domain of clinical research evidence. We have begun showing that FHIR can deliver not just clinical data but also synthesized evidence and knowledge to end users.

    Historical Context

    The project started with implementers from EBSCO Health, Duodecim Medical Publications Ltd, HarmoniQ, and MAGIC Evidence Ecosystem Foundation and expanded to a multisector project with participation from academia, industry, government, and nonprofit organizations [46].

    The COVID-19 pandemic introduced a new impetus in the quest to make biomedical evidence computable and interoperable. In light of the pandemic, where scientists and clinicians urgently needed timely results and evidence, the EBMonFHIR project leaders organized a larger group named the “COVID-19 Knowledge Accelerator (COKA) Initiative” and shifted their focus to the biomedical evidence regarding SARS-CoV-2 [47]. COKA is a nonprofit unincorporated organization with global collaboration to develop and advance interoperability standards for COVID-19 knowledge and to enhance the evidence exchange standards. As of January 2023, the COKA initiative was renamed to Health Evidence Knowledge Accelerator (HEvKA) to better represent the group’s now broader scope that is inclusive of clinical, public health, and environmental health domains.

    Objectives

    In this paper, we introduce the EBMonFHIR project that aims to produce an HL7 FHIR schema to express biomedical evidence as computable evidence—as well as FHIR resource instances, terminologies, and tools—and to promote an effective and efficient evidence ecosystem. We describe the participants involved in this effort, the process to develop the EBMonFHIR standards, and the progress made to represent evidence findings as FHIR resources. We also provide information on initial impacts, limitations, and next steps for stakeholders involved in standards and controlled terminology development, EBM implementation, and evidence used for both clinical studies and syntheses of those studies. We finally describe how others can get involved in the EBMonFHIR effort by way of the HEvKA initiative.


    HEvKA is advancing EBMonFHIR according to the five aspects of the HL7 standards development process [48], and they are (1) foster consensus, (2) ensure content is fit for purpose, (3) ensure content is implementable, (4) establish an appropriate implementer community, and (5) ensure ongoing maintenance of the standard. Through up to 15 online meetings each week and up to 3 HL7 FHIR Connectathons [49] each year, HEvKA creates and refines FHIR resources to represent evidence from clinical studies and syntheses of those studies and develops tools to assist with the creation and visualization of FHIR resources.

    Updates and improvements for any FHIR specification can be developed, proposed, reviewed, improved, voted on, and released within a documented environment in accordance with the American National Standards Institute (ANSI)-sanctioned HL7 ballot process, and ultimately published as part of the official HL7 FHIR specification. [50]

    The FHIR resources that HEvKA develops are continuously revised and adapted to reflect the best representation of the knowledge from the community. The HL7 CDS Work Group [51] is responsible for evaluating and approving the additions and changes to these FHIR resources.

    To support a culture of transparency and openness about the process of developing the FHIR resources and support tools, HEvKA makes use of numerous digital approaches to document and keep track of its activities and disseminate its progress, including the HL7 Confluence Web Page (content management) [45], Google Drive (content repository) [52], and Microsoft Teams (videoconference) [53]. The content produced is open-source and freely available to the community, with examples published on the Fast Evidence Interoperability Resources (FEvIR) Platform [54]. HEvKA also coordinates input and dissemination across many communities in the evidence ecosystem.

    HEvKA has 14 active working groups that meet weekly to address different aspects of the project (ie, Communications; Computable EBM Tools Development; CQL Development; EBM Implementation Guide; Eligibility Criteria; Funding the Ecosystem Infrastructure; GRADE [Grading of Recommendations Assessment, Development and Evaluation] Ontology; Measuring the Rate of Scientific Knowledge Transfer; Ontology Management; Project Management; Risk of Bias Terminology; Setting the Scientific Record on FHIR; Statistic Terminology; and StatisticsOnFHIR). For details, visit the Confluence web page for information about HEvKA [55].


    EBM Representation With FHIR Resources

    HEvKA first created FHIR resources for representing research results (Evidence) and variable definitions (EvidenceVariable); and after March 2020, HEvKA further created FHIR resources that express compositions (EvidenceReport), citations (Citation), and judgments about knowledge (ArtifactAssessment). To ease the readability of this article, we will broadly refer to these FHIR resources as the “EBMonFHIR Resources.” Table 1 shows the list of EBMonFHIR resources the project developed. From here on, we list FHIR resources, FHIR elements, and data types [56] in italics.

    An Evidence resource [57] provides an expression of the most granular components of evidence. Evidence is often represented by values and parameters of statistical measures (eg, mean, confidence interval, relative risk, and hazard ratio) and expressions of certainty or classifications of these statistical findings. These statistics are about a particular combination of variables from a particular study, so the Evidence resource can refer to EvidenceVariable resources for definitions of the observed or intended variables. However, to support interoperability across systems, the Group resource [58] may be used instead of an EvidenceVariable resource [59], especially when referring to a group of people such as a population, sample, or subgroup.

    The Evidence resource refers to statistical measures and their values with a machine-interpretable expression of a statistic, including the quantity; unit of measure; classification of statistic type; sample size; attribute estimates such as confidence intervals, P values, and heterogeneity estimates; and statistical model characteristics (Figure 2). The statisticType element has, as of November 2, 2023, a suggested set of 22 possible codes [60], assembled by the HEvKA team, that represent types of statistics (eg, median, relative risk, and incidence rate ratio). HEvKA has drafted 139 terms for statistic types in total (these will eventually replace the Statistic Type Value Set).

    Biomedical evidence is comprised of facts and interpretations derived from an analysis of observations of a selective sample. Certainty about any evidence may change due to methodological factors, statistical factors, and contextual factors, and for the relatedness between the sample, the evidence was derived from the population to which the evidence is applied. The certainty element provides a machine-interpretable expression of confidence in, or certainty or quality of, the evidence. The type subelement can express the aspect of the certainty being rated, using codable concepts from a suggested value set (eg, including the overall certainty, risk of bias, inconsistency, indirectness, imprecision, publication bias, dose-response gradient, plausible confounding, and large effect size [61]).

    An EvidenceVariable resource [59] provides an expression of a single evidence variable (eg, a single exposure or a single outcome or measured variable).

    The Citation resource “enables reference to any knowledge artifact for purposes of identification and attribution” [62], including “location, authorship, and contributorship to a journal article, report, document, resource, or other knowledge artifact” [62]. For instance, the Citation resource [63] is a reference to the Evidence resource [64], representing the primary outcome from an article reporting the results of a randomized clinical trial [65].

    The EvidenceReport resource represents a “container for a collection of resources and codable concepts, adapted to support compositions of Evidence, EvidenceVariable, and Citation resources and related concepts” [66]. This resource may bundle knowledge from 1 or multiple studies [66]. The report can be represented in sections of different forms including text, references to codable concepts, or other FHIR resources and sections. The EvidenceReport resource is “suited for communicating reports about research and data analysis not specific to individual persons” [66], distinct from the FHIR Composition resource commonly used for reports specific to individual persons. However, the EvidenceReport resource will be deprecated as the Composition resource has been modified to include an EvidenceReport profile to support EBMonFHIR use.

    The ArtifactAssessment resource “represents one or more assessments of another record or resource” [67]. This resource covers assessments about clinical records, health care provision, and records related to community knowledge (eg, evidence), and may include comments, corrections, classifications, ratings, questions, and responses.

    Table 1. List of FHIRa resources developed by the EBMonFHIRb and HEvKAc projects (FHIR version 6.0.0 current build, as of November 2, 2023).
    FHIRa resourceDescriptionReference to URL
    ArtifactAssessmentThe ArtifactAssessment resource provides 1 or more comments, classifiers, or ratings about a resource and supports attribution and rights management metadata for the added content.[67]
    CitationThe Citation resource enables reference to any knowledge artifact for purposes of identification and attribution. The Citation resource supports existing reference structures and developing publication practices such as versioning, expressing complex contributorship roles, and referencing computable resources.[62]
    EvidenceThe Evidence resource provides a machine-interpretable expression of an evidence concept including the evidence variables (eg, population, exposures or interventions, comparators, outcomes, measured variables, confounding variables), the statistics, and the certainty of this evidence.[57]
    EvidenceReportThe EvidenceReport resource is a specialized container for a collection of resources and codable concepts, adapted to support compositions of Evidence, EvidenceVariable, and Citation resources and related concepts.[66]
    EvidenceVariableThe EvidenceVariable resource describes an element that knowledge (Evidence) is about.[59]

    aFHIR: Fast Healthcare Interoperability Resources.

    bEBMonFHIR: Evidence-Based Medicine on Fast Healthcare Interoperability Resources.

    cHEvKA: Health Evidence Knowledge Accelerator.

    ‎
    Figure 2. Evidence resource backbone elements are represented with an abbreviated unified modeling language diagram. 0..1 denotes the element is optional and can only have 1 instance. 0..* denotes the element is optional and can have any number of instances. 1..* denotes the element is required and can have any number of instances.

    Terminologies

    The use of controlled terminologies supports the interoperable representation of multiple concepts. In FHIR, controlled terminologies [68] are represented in code systems and value sets. Reporting a specific term in FHIR uses a Coding data type. The Coding data type includes a system element to identify the terminology system, a code element for the precise code for machine use, and a display element for human-readable expression. A CodeableConcept data type is “a value that is usually supplied by providing a reference to one or more terminologies or ontologies but may also be defined by the provision of text” [56]. A CodeableConcept element is composed of a coding element (with none, 1, or more instances) and a text element (with none or 1 instance).

    The coding element of a CodeableConcept element can use a variety of terminologies, such as Systematized Nomenclature of Medicine—Clinical Terms (SNOMED CT), Logical Observation Identifiers Names and Codes (LOINC), RxNORM, and Identification of Medicinal Products (IDMP). FHIR has a noncomprehensive registry of external code systems [68]. New items can be proposed by the community, and implementers can choose to use other code systems not listed in the registry. Figure 3 shows a sample CodeableConcept element for the “Disease caused by severe acute respiratory syndrome coronavirus 2.”

    HEvKA has used, developed, or extended over 40 terminologies for use with Evidence, EvidenceVariable, EvidenceReport, Citation, and ArtifactAssessment resources. Consolidation of several of these terminologies has resulted in HEvKA leading the effort to create a Scientific Evidence Code System (SEVCO) with nearly 600 terms for study design, risk of bias, and statistics [50]. Multimedia Appendix 1 presents a list of FHIR data elements with reference to controlled terminologies.

    ‎
    Figure 3. Sample valueCodeableConcept element showing multiple coding elements. In this example, the coding element represents the COVID-19 disease with both SNOMED CT and ICD-10 terminologies. ICD-10: International Classification of Diseases, Tenth Revision; SNOMED CT: Systematized Nomenclature of Medicine—Clinical Terms.

    Walkthrough: Sample Study Result Represented With EBMonFHIR Resources

    HEvKA tested the EBMonFHIR standard by conducting a successful proof-of-concept exercise that used the FHIR Evidence resource to represent a critically appraised summary of the primary outcome of a multi-platform randomized controlled trial (RCT) of anticoagulation for hospitalized noncritically ill patients with COVID-19 [69]. The primary outcome was “organ support–free days, evaluated on an ordinal scale that combined in-hospital death and the number of days free of cardiovascular or respiratory organ support up to day 21 among patients who survived to hospital discharge” [69]. The primary result for the overall group was reported as a median adjusted odds ratio of 1.27 (95% credible interval 1.03-1.58), with 939 (80.2%) out of 1171 in the therapeutic dose anticoagulation group and 801 (76.4%) out of 1048 in the usual-care thromboprophylaxis group. An overview of FHIR resources used to represent the appraisal of this outcome is shown in Table 2. See the EBMonFHIR walkthrough [70] of this study with details about how the resources are used to represent the study including samples of JSON code to illustrate the EBM representation with FHIR resources.

    Table 2. List of EBMonFHIRa resources that represent the result for the primary outcome of the sample study.
    ReferenceFHIR resource typeDescription
    [71]CitationCitation for the article in the NEJMb
    [63]CitationCitation for evidence resource with FEvIRc Object Identifier (FOI) 7637
    [64]EvidenceSummary of 1 unit of evidence (statistical findings for 1 set of variables) from the study
    [72]Evidence variableExposure in the intervention arm of the study
    [73]Evidence variableExposure in the comparator arm of the study
    [74]Evidence variableMeasured variable for the primary outcome of the study
    [75]GroupGroup description for the intended population for evidence interpretation
    [76]GroupGroup description for the observed population in the study

    aEBMonFHIR: Evidence-Based Medicine on Fast Healthcare Interoperability Resources.

    bNEJM: New England Journal of Medicine.

    cFEvIR: Fast Evidence Interoperability Resources.


    Overview

    HEvKA is spearheading transformations and advances of EBMonFHIR on several fronts to support the process of exchanging biomedical evidence in a machine-readable format. As a continuously evolving effort, HEvKA has coordinated efforts and collaborated with industry, academia, government, and nonprofit organizations to develop EBMonFHIR resources and related tools, and has presented progress reports and results of the project to both EBM and informatics communities at several national and international events such as American Medical Informatics Association (AMIA) Annual Symposium, American Medical Informatics Association Informatics Summit, Guidelines International Network (GIN) Conference, Mobilizing Computable Biomedical Knowledge (MCBK), and Cochrane Colloquium.

    Status and Maturity of Evidence-Related Standards

    The EBMonFHIR standard contains five FHIR resources (ie, Evidence, EvidenceVariable, Citation, EvidenceReport, and ArtifactAssessment resources) and HEvKA is continually revising and refining the resources with support from members of the EBM community. Currently, the ArtifactAssessment, Citation, Evidence, and EvidenceVariable resources are at FHIR Maturity Level 1, “the artifact produces no warnings during the build process and the responsible WG has indicated that they consider the artifact substantially complete and ready for implementation” [77], whereas the EvidenceReport resource is at Maturity Level 0 (ie, Draft) and will be deprecated. Dedicated HEvKA working groups are developing functional examples to confirm and demonstrate the use of EBMonFHIR resources and data elements in various use cases. The examples are published on the FEvIR Platform [54]. In addition, following efforts for developing an implementation guide for representing evidence-based CPG recommendations [78], the EBMonFHIR project is producing an EBMonFHIR Implementation Guide [79,80] with 73 profiles of 12 resources to fully represent clinical research and evidence-based guidelines.

    The Role of Controlled Terminologies

    Communicating evidence in coded and structured forms requires controlled terminologies (or code systems) to uniquely and accurately express essential concepts. HEvKA created a Code System Development protocol to enable standardized terminologies for the exchange of scientific evidence [50]. This protocol is being executed to develop four code sets related to EBM: study design, statistic type, statistical model, and risk of bias.

    Relatedly, the SEVCO Expert Working Group (an offshoot of HEvKA) has 39 members from 18 countries as of November 3, 2023. SEVCO has identified 23 commonly used tools and methods for what the code system will support, such as the ROBINS-I tool for risk of bias assessment [81]. As of this writing, there are 602 prospective terms (253 for risk of bias, 76 for study design, and 273 for statistics) to support all recognized commonly used tools and systems. Coordination with the GRADE Working Group, a collaboration to develop a common and transparent approach to grading quality (or certainty) of evidence and strength of recommendations, is underway to support a GRADE Ontology.

    Tools for End Users

    HEvKA participants have developed tools to facilitate the visualization and data entry of evidence into EBMonFHIR resources, including human-readable interfaces and JSON viewers. The tools are integrated into the FEvIR Platform [54], and include intuitive forms, purposefully created to not require manual JSON coding or working knowledge of FHIR, for viewing and building Citation, Evidence, EvidenceVariable, and Group resources; for viewing and building multiple profiles of Composition resource (Guideline, Recommendation, SummaryOfFindings); for viewing and building multiple profiles of ArtifactAssessment resource (Classification, Rating, RiskOfBiasAssessment, and RecommendationJustification); and automated converters to translate data from MEDLINE, RIS, ClinicalTrials.gov, and MAGICapp into the FHIR specification. The viewing of resources created with these tools is open without an account, but an account (at no cost) is required to create content.

    This environment may reduce the time spent on SR evidence gathering. The laborious process of searching for articles and screening them by hand could be automated because the search query would be expressed in the “language” of EBMonFHIR and judgments made by previous searchers (such as population classification) can be recorded for reuse. Similarly, automated tools could be used to update evidence within SRs rather than relying on manual updates.

    With data available in the “language” of EBMonFHIR, it will become easier to develop tools to analyze and process scientific knowledge. Automated meta-analysis tools could be developed based on the recognition of Evidence resources with matching variableDefinition element content and processing algorithms mapped to the structured statistic content. Tools to accelerate original research will also be developed, such as tools to match clinical trials with potentially eligible patients [82].

    The ArtifactAssessment resource can also be used to represent the quality or certainty of the evidence itself. The EBMonFHIR Implementation Guide includes a CertaintyOfEvidence element profile for this purpose. There is also a DatasetCitation element Profile of the Citation resource, and an ArtifactAssessment resource referencing a DatasetCitation element can be used to represent the data quality.

    Conclusions

    Continuously identifying, synthesizing, and incorporating evidence into care are the key tasks of medical knowledge management, yet they are also prohibitively labor-intensive. HEvKA has taken approaches to standardize and eventually automate these tasks through its efforts around EBMonFHIR, an HL7 standard for making biomedical evidence computable. EBMonFHIR will be used for enabling seamless data flow between published evidence reports, repositories, SR authoring tools, and guideline development tools; automating the searching and matching of evidence with any subgroups of patients; connecting individual patient data with medical knowledge for computerized CDS; and providing individualized effect estimates for different outcomes to facilitate shared decision-making. EBMonFHIR resources in conjunction with other FHIR resources can support relaying EBM components in a manner that is interoperable and consumable by downstream tools and health information systems to support evidence users (eg, creators of biomedical knowledge bases, CPGs, CDS artifacts, and SRs). Anyone may join HEvKA to engage a community of FHIR users and committed volunteers to accelerate the development and implementation of standards for evidence exchange.

    Acknowledgments

    Health Level Seven (HL7) and Fast Healthcare Interoperability Resources (FHIR) are the registered trademarks of Health Level Seven International and their use of these trademarks does not constitute an endorsement by HL7. Evidence-Based Medicine on FHIR (EBMonFHIR) is an HL7 project. We would like to thank all members of the EBMonFHIR and COVID-19 Knowledge Accelerator (COKA) or Health Evidence Knowledge Accelerator (HEvKA) projects who contributed to the design and development of the FHIR resources to support EBM, and Janice Tufte for reviewing from a lay perspective. We would like to acknowledge the Data Science to Patient Value (D2V) initiative at the University of Colorado School of Medicine. This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. This paper was written on behalf of Health Evidence Knowledge Accelerator (HEvKA).

    Authors' Contributions

    All authors contributed to the conceptualization, writing—review and editing, and approval of the submitted paper and will be accountable for its contents. AS and BA performed supervision. AS, JD, KS, and BSA performed data curation. AS, LMS, BK, VS, JD, KS, MA, and BSA performed investigation and methodology. HPL contributed to methodology. AS, JD, and BSA handled project administration. KS and BSA managed resource and software. JD managed software. BK, JD, and BSA conducted validation. AS, BK, VS, JD, KS, and BSA performed visualization. AS, LMS, JR, BK, JD, MA, and BSA wrote the original draft. Classification based on the CRediT taxonomy.

    Conflicts of Interest

    BSA owns Computable Publishing LLC and leads the Health Evidence Knowledge Accelerator (HEvKA) and Evidence-Based Medicine on Fast Healthcare Interoperability Resources (EBMonFHIR) projects. JD and KS are employees of Computable Publishing LLC. BSA, KS, JD, MA, BK, HPL, KAR, VS, and IK are board members of the Scientific Knowledge Accelerator Foundation (SKAF). AS, LMS, and JR have no disclosures to report.

    Multimedia Appendix 1

    List of Fast Healthcare Interoperability Resources (FHIR) data elements with reference to controlled terminologies introduced by the Evidence-Based Medicine on FHIR (EBMonFHIR) project.

    DOCX File , 27 KB
    1. Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics. 2010;11:55. [FREE Full text] [CrossRef] [Medline]
    2. Miwa M, Thomas J, O'Mara-Eves A, Ananiadou S. Reducing systematic review workload through certainty-based screening. J Biomed Inform. 2014;51:242-253. [FREE Full text] [CrossRef] [Medline]
    3. Becker M, Jaschinski T, Eikermann M, Mathes T, Bühn S, Koppert W, et al. A systematic decision-making process on the need for updating clinical practice guidelines proved to be feasible in a pilot study. J Clin Epidemiol. 2018;96:101-109. [FREE Full text] [CrossRef] [Medline]
    4. Middleton B, Sittig DF, Wright A. Clinical decision support: a 25 year retrospective and a 25 year vision. Yearb Med Inform. 2016;25(Suppl 1):S103-S116. [FREE Full text] [CrossRef] [Medline]
    5. Rosenfeld RM, Shiffman RN. Clinical practice guideline development manual: a quality-driven approach for translating evidence into action. Otolaryngol Head Neck Surg. 2009;140(6 Suppl 1):S1-S43. [FREE Full text] [CrossRef] [Medline]
    6. Murad MH. Clinical practice guidelines: a primer on development and dissemination. Mayo Clin Proc. 2017;92(3):423-433. [FREE Full text] [CrossRef] [Medline]
    7. Therasse P. From clinical trials to evidence-based medicine: how to build the evidence! Eur J Cancer Suppl. 2003;1(6):55-65. [FREE Full text] [CrossRef]
    8. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326. [FREE Full text] [CrossRef] [Medline]
    9. Boutron I, Créquit P, Williams H, Meerpohl J, Craig JC, Ravaud P. Future of evidence ecosystem series: 1. introduction evidence synthesis ecosystem needs dramatic change. J Clin Epidemiol. 2020;123:135-142. [FREE Full text] [CrossRef] [Medline]
    10. Sackett DL. Evidence-based medicine. Semin Perinatol. 1997;21(1):3-5. [FREE Full text] [CrossRef] [Medline]
    11. Wapner RJ, Waters T. Introduction and historical perspective. Clin Obstet Gynecol. 2003;46(1):125-131. [FREE Full text] [CrossRef] [Medline]
    12. Crowley P, Chalmers I, Keirse MJ. The effects of corticosteroid administration before preterm delivery: an overview of the evidence from controlled trials. Br J Obstet Gynaecol. 1990;97(1):11-25. [FREE Full text] [CrossRef] [Medline]
    13. Crowley PA. Antenatal corticosteroid therapy: a meta-analysis of the randomized trials, 1972 to 1994. Am J Obstet Gynecol. 1995;173(1):322-335. [FREE Full text] [CrossRef] [Medline]
    14. Briceño-Pérez C, Reyna-Villasmil E, Vigil-De-Gracia P. Antenatal corticosteroid therapy: historical and scientific basis to improve preterm birth management. Eur J Obstet Gynecol Reprod Biol. 2019;234:32-37. [FREE Full text] [CrossRef] [Medline]
    15. Shekelle PG, Ortiz E, Rhodes S, Morton SC, Eccles MP, Grimshaw JM, et al. Validity of the agency for healthcare research and quality clinical practice guidelines: how quickly do guidelines become outdated? JAMA. 2001;286(12):1461-1467. [FREE Full text] [CrossRef] [Medline]
    16. Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med. 2007;147(4):224-233. [FREE Full text] [CrossRef] [Medline]
    17. Winters BD, Bharmal A, Wilson RF, Zhang A, Engineer L, Defoe D, et al. Validity of the agency for health care research and quality patient safety indicators and the centers for medicare and medicaid hospital-acquired conditions: a systematic review and meta-analysis. Med Care. 2016;54(12):1105-1111. [FREE Full text] [CrossRef] [Medline]
    18. Hulse NC, Rocha RA, Del Fiol G, Bradshaw RL, Hanna TP, Roemer LK. KAT: a flexible XML-based knowledge authoring environment. J Am Med Inform Assoc. 2005;12(4):418-430. [FREE Full text] [CrossRef] [Medline]
    19. Akl EA, Meerpohl JJ, Elliott J, Kahale LA, Schünemann HJ, Living Systematic Review Network. Living systematic reviews: 4. living guideline recommendations. J Clin Epidemiol. 2017;91:47-53. [FREE Full text] [CrossRef] [Medline]
    20. Boxwala AA, Peleg M, Tu S, Ogunyemi O, Zeng QT, Wang D, et al. GLIF3: a representation format for sharable computer-interpretable clinical practice guidelines. J Biomed Inform. 2004;37(3):147-161. [FREE Full text] [CrossRef] [Medline]
    21. Peleg M. Computer-interpretable clinical guidelines: a methodological review. J Biomed Inform. 2013;46(4):744-763. [FREE Full text] [CrossRef] [Medline]
    22. Tu SW, Campbell JR, Glasgow J, Nyman MA, McClure R, McClay J, et al. The SAGE guideline model: achievements and overview. J Am Med Inform Assoc. 2007;14(5):589-598. [FREE Full text] [CrossRef] [Medline]
    23. Tse EG, Klug DM, Todd MH. Open science approaches to COVID-19. F1000Res. 2020;9:1043. [FREE Full text] [CrossRef] [Medline]
    24. McCartney M. COVID-19: has EBM been replaced by hype-based medicine? Drug Ther Bull. 2020;58(7):99-100. [FREE Full text] [CrossRef] [Medline]
    25. Romiti GF, Corica B, Cangemi R, Basili S, Raparelli V. Need for innovative and timely synthesis of evidence during COVID-19 outbreak. Eur J Intern Med. 2020;77:165-166. [FREE Full text] [CrossRef] [Medline]
    26. Atallah ÁN. COVID-19: better trustworthiness of clinical evidence through clinical trials and systematic reviews. Sao Paulo Med J. 2020;138(3):171-173. [FREE Full text] [CrossRef] [Medline]
    27. Bero LA. Producing independent, systematic review evidence: cochrane's response to COVID-19. Am J Public Health. 2020;110(7):952-953. [FREE Full text] [CrossRef] [Medline]
    28. Laine C, Taichman DB, Guallar E, Mulrow CD. Keeping up with emerging evidence in (Almost) real time. Ann Intern Med. 2020;173(2):153-154. [FREE Full text] [CrossRef] [Medline]
    29. Else H. How a torrent of COVID science changed research publishing—in seven charts. Nature. 2020;588(7839):553. [FREE Full text] [CrossRef] [Medline]
    30. Covid-19 report. Google Data Studio. URL: https:/​/lookerstudio.​google.com/​reporting/​70fdce85-1b53-4605-8698-839b66c 8010b/​page/​JwhJB?feature=opengraph [accessed 2024-06-11]
    31. Dunn AG, Bourgeois FT. Is it time for computable evidence synthesis? J Am Med Inform Assoc. 2020;27(6):972-975. [FREE Full text] [CrossRef] [Medline]
    32. Alper BS, Richardson JE, Lehmann HP, Subbian V. It is time for computable evidence synthesis: the COVID-19 knowledge accelerator initiative. J Am Med Inform Assoc. 2020;27(8):1338-1339. [FREE Full text] [CrossRef] [Medline]
    33. Thomas J, Noel-Storr A, Marshall I, Wallace B, McDonald S, Mavergames C, et al. Living systematic reviews: 2. combining human and machine effort. J Clin Epidemiol. 2017;91:31-37. [FREE Full text] [CrossRef] [Medline]
    34. O'Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4(1):5. [FREE Full text] [CrossRef] [Medline]
    35. Elliott J, Lawrence R, Minx JC, Oladapo OT, Ravaud P, Jeppesen BT, et al. Decision makers need constantly updated evidence synthesis. Nature. 2021;600(7889):383-385. [FREE Full text] [CrossRef] [Medline]
    36. Lang ES, Wyer PC, Haynes RB. Knowledge translation: closing the evidence-to-practice gap. Ann Emerg Med. 2007;49(3):355-363. [FREE Full text] [CrossRef] [Medline]
    37. Wilkinson MD, Dumontier M, Aalbersberg IJJ, Appleton G, Axton M, Baak A, et al. The FAIR guiding principles for scientific data management and stewardship. Sci Data. 2016;3(1):160018. [FREE Full text] [CrossRef] [Medline]
    38. Evidence Ecosystem. MAGIC Evidence Ecosystem Foundation. URL: https://www.magicevidence.org/evidence-ecosystem [accessed 2024-06-11]
    39. Taber P, Radloff C, Del Fiol G, Staes C, Kawamoto K. New standards for clinical decision support: a survey of the state of implementation. Yearb Med Inform. 2021;30(1):159-171. [FREE Full text] [CrossRef] [Medline]
    40. CDS connect. Agency for Healthcare Research and Quality. URL: https://cds.ahrq.gov/cdsconnect [accessed 2023-01-16]
    41. CDS hooks. HL7 and Boston Children's Hospital. URL: https://cds-hooks.hl7.org/ [accessed 2023-01-16]
    42. Lehne M, Luijten S, Imbusch PVFG, Thun S. The use of FHIR in digital health—a review of the scientific literature. Stud Health Technol Inform. 2019;267:52-58. [CrossRef] [Medline]
    43. Jaffe C, Mandel J, Sartin S, Posnack S, Overhage JM, Tripathi M. W09: FHIR -- implementing the HL7 interoperability platform: a community of implementers for research, patient care, and value-based care. 2019. Presented at: AMIA 2019 Annual Symposium; November 16-20, 2019; Washington DC. URL: https://symposium2019.zerista.com/event/member/602305
    44. FHIR v4.0.1. HL7® FHIR® Release 4. URL: http://hl7.org/fhir/R4/index.html [accessed 2024-05-17]
    45. EBMonFHIR—clinical decision support—confluence. Atlassian Confluence Community License. URL: https://confluence.hl7.org/display/cds/ebmonfhir [accessed 2021-03-28]
    46. HL7 searchable project index—FHIR resources for evidence-based medicine knowledge assets (EBMonFHIR). HL7 International. URL: https:/​/www.​hl7.org/​special/​Committees/​projman/​searchableProjectIndex.​cfm?action=edit&ProjectNum ber=1422 [accessed 2021-04-05]
    47. COVID-19 Knowledge Accelerator (COKA)—clinical decision support—confluence. Atlassian Confluence Community License. URL: https://confluence.hl7.org/pages/viewpage.action?pageId=97468919 [accessed 2021-03-29]
    48. Understanding the standards process—HL7—confluence. Atlassian Confluence Community License. URL: https://confluence.hl7.org/display/HL7/Understanding+the+Standards+Process [accessed 2022-08-10]
    49. FHIR Connectathons. Atlassian Confluence Community License. URL: https://confluence.hl7.org/display/FHIR/Connecta thons [accessed 2023-01-16]
    50. Alper BS, Dehnbostel J, Afzal M, Subbian V, Soares A, Kunnamo I, et al. Making science computable: developing code systems for statistics, study design, and risk of bias. J Biomed Inform. 2021;115:103685. [FREE Full text] [CrossRef] [Medline]
    51. Clinical decision support workgroup—confluence. Atlassian Confluence Community License. URL: https://confluence.hl7.org/display/CDS/WorkGroup+Home [accessed 2021-06-29]
    52. Personal cloud storage and file sharing platform—Google. Google Drive. URL: https://www.google.com/drive/ [accessed 2023-01-16]
    53. Video conferencing, meetings, calling. Microsoft Teams. URL: https://www.microsoft.com/en-us/microsoft-teams/group -chat-software [accessed 2023-01-16]
    54. FEvIR platform. FEvIR Platf. URL: https://fevir.net/ [accessed 2024-03-27]
    55. HEvKA-Health Evidence Knowledge Accelerator—clinical decision support—confluence. Atlassian Confluence Community License. URL: https://confluence.hl7.org/display/CDS/HEvKA-Health+Evidence+Knowledge+Accelerator [accessed 2023-01-16]
    56. Datatypes—FHIR v6.0.0-cibuild. HL7®FHIR® FHIR CI-Build. URL: https://build.fhir.org/datatypes.html [accessed 2023-11-02]
    57. Evidence. FHIR v6.0.0-cibuild. URL: https://build.fhir.org/evidence.html [accessed 2023-11-02]
    58. Group. FHIR v6.0.0-cibuild. URL: https://build.fhir.org/group.html [accessed 2023-11-02]
    59. EvidenceVariable. FHIR v6.0.0-cibuild. URL: https://build.fhir.org/evidencevariable.html [accessed 2023-11-02]
    60. Valueset-statistic-type. FHIR v6.0.0-cibuild. URL: https://build.fhir.org/valueset-statistic-type.html [accessed 2023-11-02]
    61. Valueset-certainty-type. FHIR v6.0.0-cibuild. URL: https://build.fhir.org/valueset-certainty-type.html [accessed 2023-11-02]
    62. Citation. FHIR v6.0.0-cibuild. URL: https://build.fhir.org/citation.html [accessed 2023-11-02]
    63. FEvIR citation—citation for FEvIR evidence 7637: critically appraised summary of primary outcome of multi-platform RCT of anticoagulation for non-critically ill COVID-19. FEvIR Platf. URL: https://fevir.net/resources/Citation/7638 [accessed 2023-11-02]
    64. FEvIR evidence—critically appraised summary of primary outcome of multi-platform RCT of anticoagulation for non-critically ill patients with COVID-19. FEvIR Platform. URL: https://fevir.net/resources/Evidence/7637 [accessed 2023-11-02]
    65. RECOVERY Collaborative Group, Horby P, Lim WS, Emberson JR, Mafham M, Bell JL, et al. Dexamethasone in hospitalized patients with COVID-19. N Engl J Med. 2021;384(8):693-704. [FREE Full text] [CrossRef] [Medline]
    66. EvidenceReport. FHIR v6.0.0-cibuild. URL: https://build.fhir.org/evidencereport.html [accessed 2023-11-02]
    67. ArtifactAssessment. FHIR v6.0.0-cibuild. URL: https://build.fhir.org/artifactassessment.html [accessed 2023-11-02]
    68. Terminologies-systems. FHIR v6.0.0-cibuild. URL: https://build.fhir.org/terminologies-systems.html [accessed 2023-11-02]
    69. ATTACC Investigators, ACTIV-4a Investigators, REMAP-CAP Investigators, Lawler PR, Goligher EC, Berger JS, et al. Therapeutic anticoagulation with heparin in noncritically ill patients with COVID-19. N Engl J Med. 2021;385(9):790-802. [FREE Full text] [CrossRef] [Medline]
    70. EBMonFHIR walkthrough. Atlassian Confluence Community License. URL: https://confluence.hl7.org/display/CDS/EBM onFHIR+Walkthrough [accessed 2023-11-03]
    71. FEvIR citation—anticoagulation for COVID-19 combined RCTs in NEJM. FEvIR Platf. URL: https://fevir.net/resources/Citation/7636 [accessed 2023-11-02]
    72. FEvIR EvidenceVariable—therapeutic-dose anticoagulation with heparin. FEvIR Platf. URL: https://fevir.net/resources/EvidenceVariable/7751 [accessed 2023-11-02]
    73. FEvIR EvidenceVariable—usual-care pharmacologic thromboprophylaxis. FEvIR Platf. URL: https://fevir.net/resources/EvidenceVariable/7752 [accessed 2023-11-02]
    74. FEvIR EvidenceVariable—organ support-free days. FEvIR Platf. URL: https://fevir.net/resources/EvidenceVariable/7753 [accessed 2023-11-02]
    75. FEvIR Group—patients who are hospitalized for COVID-19 and who are not critically ill. FEvIR Platf. URL: https://fevir.net/resources/Group/7749 [accessed 2023-11-02]
    76. FEvIR Group—participants in anticoagulation for COVID-19 combined (ATTACC, ACTIV-4a, and REMAP-CAP) RCT (hospitalized, not critically ill). FEvIR Platf. URL: https://fevir.net/resources/Group/7750 [accessed 2023-11-02]
    77. FHIR maturity model. Health Level Seven International. URL: https://wiki.hl7.org/FHIR_Maturity_Model [accessed 2021-07-30]
    78. Lichtner G, Alper BS, Jurth C, Spies C, Boeker M, Meerpohl JJ, et al. Representation of evidence-based clinical practice guideline recommendations on FHIR. J Biomed Inform. 2023;139:104305. [FREE Full text] [CrossRef] [Medline]
    79. FEvIR project—evidence based medicine implementation guide. FEvIR Platf. URL: https://fevir.net/resources/Project/29736 [accessed 2024-03-27]
    80. Evidence based medicine on FHIR implementation guide v1.0.0-ballot. HL7 International. URL: https://build.fhir.org/ig/HL7/ebm/index.html [accessed 2024-03-27]
    81. Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919. [FREE Full text] [CrossRef] [Medline]
    82. Alper BS, Dehnbostel J, Shahin K, Ojha N, Khanna G, Tignanelli CJ. Striking a match between FHIR-based patient data and FHIR-based eligibility criteria. Learn Health Syst. 2023;7(4):e10368. [FREE Full text] [CrossRef] [Medline]

    ‎
    CDS: clinical decision support
    COKA: COVID-19 Knowledge Accelerator
    CPG: clinical practice guideline
    EBM: evidence-based medicine
    EBMonFHIR: Evidence-Based Medicine on Fast Healthcare Interoperability Resources
    FEvIR: Fast Evidence Interoperability Resources
    FHIR: Fast Healthcare Interoperability Resources
    GRADE: Grading of Recommendations Assessment, Development and Evaluation
    GIN: Guidelines International Network
    GLIF: GuideLine Interchange Format
    HEvKA: Health Evidence Knowledge Accelerator
    HL7: Health Level Seven
    IDMP: Identification of Medicinal Products
    LOINC: Logical Observation Identifiers Names and Codes
    MCBK: Mobilizing Computable Biomedical Knowledge
    RCT: randomized controlled trial
    SAGE: Standards-Based Sharable Active Guideline Environment
    SEVCO: Scientific Evidence Code System
    SNOMED CT: Systematized Nomenclature of Medicine—Clinical Terms
    SR: systematic review
    UML: unified modeling language

    Edited by G Tsafnat; submitted 03.11.23; peer-reviewed by S Thun, C Vorisek; comments to author 22.03.24; revised version received 04.04.24; accepted 06.04.24; published 25.06.24.

    Copyright

    ©Andrey Soares, Lisa M Schilling, Joshua Richardson, Bhagvan Kommadi, Vignesh Subbian, Joanne Dehnbostel, Khalid Shahin, Karen A Robinson, Muhammad Afzal, Harold P Lehmann, Ilkka Kunnamo, Brian S Alper. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 25.06.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.