Provenance Information for Biomedical Data and Workflows: Scoping Review

Introduction

Background

Both the use and reuse of electronic medical and patient-related data offers enormous potential for clinical research [1,2]. National programs such as the German Medical Informatics Initiatives support knowledge discovery and data sharing using adequate computational infrastructure and secure processes [3]. In this context, provenance information (Textbox 1) offers access to quality assured, traceable, and credible shared data. These and other advantages of data provenance have been demonstrated, for instance, in the EU Horizon 2020 TRANSFORM project [4] or in the MeDaX Knowledge Graph Prototype [5]. Schröder et al [6] provided an “Electronic Laboratory Notebook” use case in the wet lab to show how provenance supports the understanding and reproducibility of research investigations. At the same time, researchers not considering the origin of data run into the hazard of systematically incomplete or wrong data [7].

Notably, the FAIR (findable, accessible, interoperable, reusable) guiding principles for data stewardship [9] explicitly mention provenance [10,11]. A provenance-oriented approach requires thorough planning, execution, and evaluation of data management processes in the respective application domain [2]. In the scientific context, adherence to criteria such as consistency, interoperability, and confidentiality are generally required across all software tools [1,12,13].

The concept and implementation of provenance are essential for most scientific domains, such as environmental fields (geoprocessing workflows or climate assessments), in nuclear fusion engineering, or material sciences [14,15]. In particular, the biomedical domains demand comprehensive investigation and information about their data management scenarios, including extract, transform, load jobs for data transfer and integration. Reliable data and data pipelines both require provenance data to be embedded in concepts for traceability to understand the relationships between results and source data.

Objectives

Our work reviews approaches and criteria for provenance tracking in the biomedical domain and discloses current knowledge gaps. This comprises modeling aspects and metadata frameworks for meaningful and usable provenance information during the creation, collection, and processing of scientific biomedical data. The review also covers the examination of quality aspects relating to provenance.

Methods

Overview

We followed the scoping methodological framework developed by Arksey and O’Malley [16] for conducting a scoping review with the following stages: (1) stage 1—identification of the research questions (RQs); (2) stage 2—identification of relevant studies; (3) stage 3—study selection; (4) stage 4—data extraction and charting; and (5) stage 5—collating, summarizing, and reporting the results.

Change From the Original Protocol

The protocol of this scoping review has been published in JMIR Research Protocols (international registered report identifier DERR1-10.2196/31750) [17]. In accordance with the original protocol developed for this scoping review, the search period was initially planned to include studies published from January 2006 to March 2021. However, due to the extensive nature of the data extraction and write-up process, it became apparent that additional time was necessary to ensure a comprehensive analysis of the relevant literature. As a result, we extended the search period from 2006 to the end of 2022.

No other changes were made to the original protocol. Thematic analysis methods were applied to analyze the extracted data by organizing themes according to the RQs [18]. In line with the framework developed by Arksey and O’Malley [16], the review does not attempt to assess the quality of studies or the risk of bias. It also does not assess the generalizability of the results.

Stage 1: Identifying RQs

The main objective of this review was to investigate existing evidence regarding approaches and criteria for provenance tracking and disclosing current knowledge gaps in the biomedical domain. The objective led to the following RQs:

• RQ1: Which potential (methodological) approaches exist for the classification and tracking of provenance criteria and methods in a biomedical or domain-independent context?
• RQ2: How can the potential value of provenance information be harnessed and by whom? How can usability be provided?
• RQ3: What are the challenges and potential problems or bottlenecks for the accomplishment of provenance?
• RQ4: Which guidelines or demands for the consideration of provenance criteria in a biomedical or domain-independent context have to be followed?
• RQ5: How completely can provenance be mapped in the data life cycle or during data management?

Stage 2: Identifying Relevant Studies

Concepts and matching keywords were categorized into 4 groups (Table 1): target domain refers to the context of the research topic and includes studies with a biomedical, health care, clinical, or scientific background. In this work, scientific background is limited to domain-independent studies and excludes all other domain-specific studies. Provenance concerns the information about the genesis of a given object. Provenance properties cover specific requirements tied to the term provenance; they also describe selected characteristics in this context. Objective includes the purposes or intention of provenance capture. The comprehensive search strategy is recorded in the study protocol [17], and search strings combined with Boolean operators are attached (Multimedia Appendix 1).

Stage 3: Study Selection

The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart depicts the selection process. First, we identified all relevant studies in the PubMed and Web of Science databases based on our search strategy. After deduplication, we launched a transparent screening process by importing all relevant studies into Rayyan [19], a systematic review supporting solution. The studies were then reviewed by at least 2 independent researchers. In the case of vote agreement, the study was either included in the next review phase or excluded from the review. A third independent reviewer was consulted to solve the conflict if no consensus could be reached. The study screening phase started with a title and abstract evaluation for eligibility. Included studies were submitted to a full-text screening, while performing a thorough investigation on the study report. Reviewers voted for inclusion or exclusion considering the inclusion and exclusion criteria. Finally, the residing set of qualified studies was moved into the data extraction pipeline. A description of the study selection is provided in the protocol [17].

Studies were included if they (1) were focused on the biomedical domain or were domain independent, (2) described provenance-tracking approaches, and (3) were written in English. Studies were excluded if they (1) were not specific to the biomedical or general domain, (2) were gray literature, and (3) did not focus on provenance-tracking approaches.

Stage 4: Charting the Data

We followed a collaborative and iterative process to define a charting table for data extraction. Individual reviewers (KG, FK, FH, SG, AAZ, and DW) then scrutinized all studies and extracted central textual occurrences into the data extraction sheet. The variables in the data extraction sheet correspond with the RQs. General characteristics of the studies, approaches for classification and tracking of provenance, and their associated challenges along with the significance and completeness of provenance information in the given context were part of the investigational charting. The reviewers independently charted the data in a structured and consistent way and discussed the results.

Stage 5: Collating, Summarizing, and Reporting the Results

The extracted data were analyzed using summary statistics by calculating the total number and percentages of all studies per category, if applicable. Charts were presented for the distribution of the individual data elements where applicable. The data analysis was partially supported with scripts in Python (version 3.10.0) [20]. Plots were generated with R version 4.0.4 (R Core Team) [21] and version 1.3.0 of the tidyverse package [22].

Further analysis was performed using qualitative evaluation. The reporting of the results and outcomes was structured according to the RQs. On the basis of the analysis of the review results, we have developed a road map for a customized provenance framework that considers the life cycle of the software framework (Provenance-Software Framework Life Cycle [SFL]). Implications for future research, practice, and policy makers were outlined. Our reporting adheres to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) reporting guidelines [23].

Results

Literature Search

The search in the PubMed and Web of Science databases resulted in 764 hits and included papers from January 1, 2006, to December 31, 2022. Afterward, 140 duplicates were removed. The remaining 624 papers were subjected to title-abstract screening in an interactive selection process, leaving 118 eligible papers for the full-text review. The full-text papers were further screened to identify papers eligible for the subsequent step of data charting. During this step, additional 52 papers were excluded (see the Stage 4: Charting the Data section). These papers either did not meet the study design context (31/52, 60%) or they lacked the domain concept (15/52, 29%). Four papers reported the same study or contained parts of it, and 2 were not a full paper. A total of 66 articles were included in the data extraction phase (Table 2). The paper selection followed the PRISMA [24] approach (Figure 1), and the PRISMA-ScR checklist is presented in Multimedia Appendix 2.

‎

Characteristics of the Included Studies

All papers (n=66) were first published between 2006 and 2022 (Table 2). More than half of the reviewed studies were published within the past 5 years, which we selected for our scoping review. Predominantly, studies originated from the biomedical or health care domain (45/66, 68%), followed by the domain-independent studies (21/66, 32%). Document characteristics of the study corpus are provided in Multimedia Appendix 3.

All studies in this review were screened with regard to the 5 RQs described in the Methods section. The following subsections describe our findings for RQ1 to RQ5.

RQ1: Classification and Tracking of Provenance Criteria in Biomedical Workflows and Data

RQ1.1: Characteristics of Provenance Framework Types

Heterogeneous approaches for classifying and tracking provenance criteria have been reported in the selected literature (n=66). We propose to categorize them by their focus (Multimedia Appendix 4). Most articles (58/66, 88%) focused explicitly on practical provenance management approaches. The remaining theoretical frameworks (8/66, 12%) provided recommendations or reviews. They can be classified into the following subcategories.

Semantics and Models, Ontologies, and Metadata (27/58, 47%)

This comprises provenance-tracking approaches on different granularity, ontology, and model abstraction levels. The semantic Provenance Context Entity approach [63] was developed to track provenance in Resource Description Framework–based semantic web applications. An example of an annotation mechanism was introduced with collection-oriented modeling and design [55]. The Provenance Metadata Model (ProvCaRe S3), supporting scientific reproducibility, was represented with the Web Ontology Language and provenance triples served as a basis for the provenance graph [38]. Later, the NeuroBridge ontology extended ProvCaRe, combined with a deep learning model [84]. Further ontologies include the REPRODUCE-ME ontology, integrated in the CAESAR project [75], or the BioMedical Study–Lifecycle Management core ontology [80]. application programming interfaces for visualization [71] or querying purposes [28] and web services for user access to provenance data [2] were reported.

Scientific Workflows and Workflow Executions (18/58, 31%)

These are mainly Open Provenance Model (OPM)–oriented workflows [86] on different semantic levels, like in the BioWorkbench [27], OpenPREDICT [29], or Web Ontology Language projects. Provenance data were stored in relational databases, like in OPMProv [73] or in graph databases [53,77]. Querying possibilities were offered via a web service or with specific querying languages at the graph level [73].

Privacy Aspects (5/58, 9%)

Decentralized management and General Data Protection Regulation requirements led to the use of blockchain technologies [48] in combination with the PROV model standard. Another scenario incorporated blockchain in a proof-of-concept study [69] to enable an audit trail mechanism for a trusted artificial intelligence model.

Visualization Aspect (4/58, 7%)

The complexity of representing provenance information at different levels of aggregation was examined in the AVOCADO project [62]. The NeuroProv project [61] shows how visualization supports clinicians in information tracking and reproducibility analysis.

General Data Managing Tools (4/58, 7%)

Frameworks provide different modules for data and workflow provenance capture, representation, storage, comparison, and visualization [75] or automatic recomputation of arbitrary data-processing results [82].

RQ1.2: Provenance Model Characteristics

At all, 58 papers reported about provenance model characteristics. The dominant provenance models refer to the PROV [8] specification (25/58, 43%), established by the World Wide Web Consortium (W3C) as the de facto standard for provenance modeling, and the frequently used OPM [86] (17/58, 29%; Multimedia Appendix 5). Other models cite specific solutions (11/58, 19%), are concerned with metadata provision (5/58, 9%), or do not provide any information on the provenance model (8/58, 14%).

OPM is the result of 3 provenance challenges (since 2011 until today). OPM (version 1.1) is exchangeable across systems and supports a process-oriented and dataflow-oriented view. It is based on the notion of the annotated causality graph with nodes as artifacts, processes, and agents. OPM was further developed into a provenance data model. PROV [8] comprises a family of specifications for provenance, designed to promote the publication of provenance information on the web. It offers interoperability across systems and is quite generic.

Figure 2 displays the temporal evolution of the characterized frameworks depending on the applied models. We observed an increased number of papers relating to these provenance management frameworks between 2016 and 2022. At this time, the OPM and W3C PROV standards were extended. The onset of the FAIR principles [29] and the Fast Healthcare Interoperability Resources framework [40] furthermore set new requirements for modeling and implementation projects.

‎

RQ1.3: Validation Status

Most of the studies (n=52) report a successful validation of their provenance solution. The use cases were mostly domain-specific (eg, as part of the AVOCADO [62] project). Other authors chose classical semantic evaluation schemes that demonstrated feasibility by responding to competency questions. Examples are the provenance challenges or proof-of-concept frameworks [13,29,56,69,72].

To pass the provenance challenges, participants needed to solve predefined provenance queries [47,55,73]. Can and Yilmazer [56] evaluated their domain-independent model with an infectious disease use case and implementing the Healthcare Provenance Information System. Sometimes, more than one approach had been taken for evaluation (eg, use cases and factors, ontology validation, and a user-based study were applied to evaluate the CAESAR approach for scientific experiments [75]). A prototype visualization tool for provenance information used satisfaction surveys to assess the usability of the system [76].

Curcin et al [57] emphasized that the setup of provenance data needs to be modeled and verified separately from the software implementation. Precise validation methods for provenance services focus on usability, performance, scalability, fault tolerance, and functionality [64]. Moreover, they demanded more formal engineering techniques to foster provenance implementation across a broad range of software tools in the biomedical domain and beyond [1,74]. In that sense, formal validation as part of the software engineering process contributes to increased software quality, and formal validation requires testing efforts and testing evidence. However, accurate alignment of testing procedures against predefined requirements in the software life cycle could not be identified.

RQ1.4: Provenance Characteristics

The term provenance is subjected to an evolutionary and technical process with multifaceted meanings and roles in the selected papers (n=66). There is agreement that provenance is a piece of history. However, the focus of provenance work ranges from abstract workflow descriptions to summaries of workflow executions to more general knowledge about data sources and result dependencies [2,51,65,72,73]. For example, provenance as semantic metadata was specified in several works between 2007 and 2019. Monnin et al [65] required the encoding of provenance of pharmacogenomics knowledge units. Other works refer to data provenance as knowledge about data sources [45] or as a piece of analytic software [39], as machine-interpretable provenance of data sets, as software and computations, as metadata for all computed results [81], or as the description of the data and its original context, and tracing data history from their creation to their sharing [80].

Sahoo et al [38] stated that the provenance data model together with the PROV Ontology define the minimal categories of provenance metadata terms. Other studies discussed the combined provenance of data and workflows and introduce the terms prospective, retrospective, and domain provenance [1,42,59]. While prospective provenance expresses future abstract workflow information, retrospective provenance gathers past workflow execution and data derivation information. Domain-specific provenance can be defined as an extension to the PROV Ontology. Workflow provenance has repeatedly been mentioned in the context of workflow execution [27,31,34]. Wittner et al [74] introduced the term provenance backbone, which covers coarse granularity representation of traceable object artifacts, whereas Mitchell et al [83] expressed provenance in relation to intrinsic and extrinsic metadata. The “FAIRly big framework” [82] demonstrates how records of process provenance are captured and stored in a machine-readable, automatically re-executable way. The 7 Ws (who, what, where, why, when, which, and how) characterize provenance in the study by Ruiz-Olazar et al [85].

RQ1.5: Requirements for Provenance Frameworks

Out of 66 reviewed papers, 44 (67%) papers mentioned ≥1 functional and nonfunctional requirements for the referenced framework type. However, 33% (22/66) of the papers did not identify any specific requirements. For those studies that did, we identified 9 different word fields, matched them (Figure 3), and explained the citations (Multimedia Appendix 6).

Intensive interdisciplinary work on requirements analysis has been undertaken [75,83]. As such, a workshop with scientists from multiple disciplines (biology, computer science, ecology, and chemistry) and an exploratory study [75] contribute to requirements collection in the epidemiological field [83]. Another way to identify requirements is based on exhaustive literature research and interviews with domain specialists [85]. Figure 3 visualizes the reported provenance requirements. The most popular requirements refer to the word fields integrity (16/44, 36%) and reproducibility (13/44, 30%), followed by interoperability, traceability, and performance or scalability related topics (each 9/44, 20%). Others were related to the word fields organizational and security (each 8/44, 18%). Only a few studies reported on trust (5/44, 11%) and usability (3/44, 7%) linked approaches.

‎

RQ1.6: Domain-Specific Conditions Including Guidelines

Some papers reported domain-specific standards for provenance (n=17). Beyond the W3C standards, we identified the Open Archival Information System Functional Model as a basis for the development of a research object concept [32]. Another example is the International Organization for Standardization (ISO) 15489-1, which defines the term metadata [51] or the Technical Committee 276 “Biotechnology” ISO, which standardizes provenance information for the biotechnology domain [74]. The National Institutes of Health guideline “Rigor and Reproducibility” [49] addresses topics impeding the study replicability. The FAIR principles explicitly guided data management practices in studies [77,83]. The FAIRSCAPE framework presented standards and interoperability such as JSON-LD, W3C PROV or CAT, DOI, and EVI’s formal model as extension to W3C as evidence chains that support or challenge a result [81].

RQ2: Potential Value of Provenance Information

RQ2.1: Impact of Provenance Information

In our review, a total of 42 papers reported about various impacts (n=99) on different stakeholders (Multimedia Appendix 7). The availability of provenance data impacts the scientific and biomedical communities (Figure 4). With regard to the work of researchers, scientists, academia, investigators, and clinicians (n=64), most papers reported guidance benefits (19/64, 30%) and reproducibility-related effects (12/64, 19%). Considerably fewer papers observed validity (4/64, 6%), managing influence (4/64, 6%), reusability (5/64, 8%) and confidence effects (6/64, 9%). Other studies reported that provenance information impacts the willingness to share knowledge (6/64, 9%), for example, by providing a unified repository for the experimental data for the research group [82,85]. Interestingly, only 13% (8/64) of the studies discussed implications on the quality of research (eg, [46,49,52,72,77,78,84]).

Other involved team or staff members (n=22) such as developers, data managers, or domain experts were also affected by the availability of provenance information. The majority recognizes benefits in validity (5/22, 23%) [26,40,49,60,61] and managing benefits (8/22, 36%) [27,33,41,56,71,76,80,83] followed by guidance benefits (5/22, 23%) [33,53,60,67,83]. In addition, reproducibility (3/22, 14%) [26,60,79] and reusability (1/22, 4%) impacts were mentioned.

Only low impact on patients (n=7) was described, mostly referring to the consent of their data (5/7, 71%) [26,45,48,56,69] to an improved measurable patient outcome and trust in evidence for clinical recommendations (each 1/7, 14%) [57].

Only a few effects on other third parties (n=6) such as data privacy officers, authorities, government, or industry were reported. Related implications concerned mainly the evidence for data validity or sensitive data-processing solutions [26,45,48,56,69,83].

‎

RQ2.2: Data Sources

A large number of papers (n=42) reported studies processed different types of data sources to generate provenance information, for example, neurological data [1,28,84,85], with large-scale data from the brain imaging component of the UK Biobank project [82], electronic health record data [27], study data [62], animals data [80], pathology data [74], omics data [52], biomedical or medical data [64,77,78], computational data [72], time-series analysis of neonatal intensive care unit data [81], and data from hybrid methods [69].

RQ3: Challenges, Problems, and Bottlenecks

Overall, 47 papers reported 74 distinct challenges impeding the implementation of provenance. We categorized these challenges into organizational and technical groups, provided details (Multimedia Appendix 8) and presented a temporal overview of the challenges (Figure 5). In summary, issues are related to data annotation, metadata, and modeling of provenance, as well as performance-related challenges. However, the need for more detailed provenance information; consideration of compliance managing topics (eg, security-related conditions); and adherence to quality and software engineering principles such as exchange, discovery, and interoperability emerged later in the course. Furthermore, usability and scalability questions emerged very early in context with provenance consumption.

More than three-quarters of the reported challenges are technical (64/74, 86%). Thereof, approximately one-fourth is associated with provenance granularity issues (15/64, 23%). Curcin et al [1] pointed out that a granular tracking of relevant human interactions, automated processes, or logging is needed and emphasized the difficulty of choosing a proper level of granularity of provenance and associated with this, the right semantic complexity [4,57]. Beyond that, a balanced trade-off between fast execution and provenance granularity must be found [42]. In fact, a fine-granular provenance level impacts the computing and storage resources [57,58]. Furthermore, managing sensitive data restriction requires the integration of adequate security level granularity into the provenance model [56].

Approximately one-third of the reported challenges (20/64, 31%) either mention the insufficient availability of metadata—which subsequently leads to incomplete provenance models—that claims the terminological heterogeneity in the metadata terms associated with study data sets [84] or does not conform to explicit annotation standards [82]. An improved availability of provenance metadata and FAIR enrichment of the data was demanded [29,38,80]. Furthermore, stakeholders should be involved in the semantic enrichment of provenance data [4,51]. However, during this metadata annotation phase, a lack of semiautomated procedures for ontology selection, semantic modeling, or mapping techniques was reported [2,29,51]. Although the use of existing models is encouraged [59], as it improves semantic interoperability [29], the reuse of vocabularies to represent provenance information remains an extensive task [29]. Cheng et al [70] noted that it was necessary to properly integrate domain-specific demands into the provenance model. Ruiz-Olazar et al [85] claimed that a unified data model for handling metadata is still missing.

One-fifth of the studies (14/64, 22%) reported performance problems during the acquisition of provenance data, such as workflow overhead [54,73] and scalability [13,85] issues. Increasing data volumes hamper the processing of provenance visualization and stream handling [76]. One proposal with respect to the cost-intensive visualization was to reduce the size of large provenance graphs [39]. Other authors reported challenges with quality [26,29,36,41,52] and usability [54,64,69,70,73,75]. According to the literature, data quality and reuse are lacking due to the deficit in provenance deployment, particularly for observational and administrative studies [26]. Furthermore, the lack of information about experimental origins in genomics data and their related systematic quality control assessment reduce the quality of provenance and the level of creditability [52]. In particular, the low uptake of high-quality semantic models [9] and the unavailability of provenance in general [36] cause information loss and data quality issues. A minor concern is the usability because provenance is recognized to be still in infancy [73]. The challenge of applying more software engineering techniques (4/64, 6%) [4,32,42,67] was reported to facilitate provenance implementation across a broad range of software tools in the biomedical domain and beyond [1].

Significantly fewer organizational challenges (10/74, 13%) [1,4,38,56-58,64,69,73] were reported, partly attributable to a basic unawareness of provenance benefits and less exchange between stakeholders. Khan et al [58] stressed that provenance capture must be established as a standard practice, not as an afterthought. McClatchey et al [64] also recommended working toward gaining the stakeholder’s acceptance and confidence in the infrastructure. In the same vein, it is recommended to integrate developers already in the design phase [1]. However, financial challenges were reported due to the necessary investments in provenance-enabled tooling and capabilities [4]. The upcoming relevance of patient-mediated data handling raised new challenges and requirements, especially with respect to policy and governance topics [69] and rigorous validation approaches [74].

‎

RQ4: Demands

Because of the extensive information obtained from RQ1, we extended the RQs to gain more insights into the provenance tracing and classification requirements identified in RQ1.

Interestingly, most of the 15 papers referred to claims relating to quality aspects.

For example, a more robust assessment of data quality is required [36], and clearer and more consistent policies and policy ontologies are requested to prevent disclosure of sensitive data [56] and more trained staff is required [50], including data managers, software architects, or semantic web specialists. User-friendly interfaces should help scientists in the provenance querying process [54]. Developers should not only recognize technologies but also data model recommendations during the design phase [1]. Performance of provenance reasoning needs to be improved [68] and the further development of ontologies needs to be automated [4,49]. The term “intelligible machines” rather than “intelligent machines” was suggested to better respect the specific aspects of big data technologies in medical research [57]. Integrating the Healthcare Enterprise standards, health care legacy protocols, interoperability, and legacy issues are furthermore mentioned [48], and mappings between entities of various provenance models should be completed [65]. Future integration into a recognized ISO standard similar to BioCompute was proposed [60].

RQ5: Completeness of Provenance Information

The literature predominantly reports on a qualitative evaluation of completeness during the data management processes. However, we found one study describing a data management process dealing with metadata for traceability in clinical studies, which delivered complete provenance in this respect [12]. Curcin [4] proposed to incorporate provenance information in the validation against standards.

One study applied data from 6 clinical research studies and >100 variables to evaluate the coverage of the provenance ontology in the semantic annotation of the study descriptions [49]. Two other documents invoked the need for minimal information elements to ensure sufficient process specification [28] and the existence of rich provenance information for reconstructing and rerunning pipelines [29].

A visualization of provenance data in neuroimaging took a semiqualitative approach for measuring the coverage. They mapped the metrics to use cases for the traceability of results and concluded that there is no absolute measure possible to verify the visualization approach [61]. Arshad et al [61] tested 15% of their workflows for verifiability of results, comparability of workflows, progression of the data for the analysis and origin of results, and evolution to see how data products evolved during an experiment.

Furthermore, Sahoo et al [38] examined the proportion of provenance metadata information across research articles using a qualitative hypothesis method. The method also provides a provenance ranking algorithm for the computation of a reproducibility rank for each article. The outcome of the self-contained DataLad data set presented valid, machine-actionable provenance information for every single result file of the performed data processing [82].

Numerical indication of completeness was not achieved in any of the other papers. However, the papers pointed out the advantages of provenance capture, for example, related to the longevity and accessibility of data after years [40].

Road Map for a Tailor-Made Provenance Framework

On the basis of the insights obtained from the literature review, we developed a road map for the implementation of a tailor-made provenance framework (Provenance-SFL). This approach is based on the SFL for the development, provision, and management of software [87]. The heterogeneous tracking approaches, their artifacts, and varying degrees of fulfillment of the RQs are depicted in Figure 6 and determine our main discussion points.

‎

Discussion

Principal Findings

This scoping review presents various approaches and criteria for provenance tracking as a crucial aspect of research, especially in the biomedical domain. Our holistic view leads to an extensive summary of pooled research results, provides possible answers to the 5 RQs and discloses current knowledge gaps.

Following the previously published scoping review protocol and the described deviation led us to include 66 full-text papers from initially 764 papers found in the PubMed and Web of Science databases. Using a structured and pretested data extraction sheet, contextual, but detailed, results were extracted to answer the 5 RQs defined in the protocol. The results of the scoping review led us to propose a Provenance-SFL road map, which distinguishes between the framework types and model characteristics, the validation status, and the requirement and provenance characteristics (Figure 6).

Most models in this review referred to the W3C PROV and OPM standards. As shown in Figure 2, an increased number of papers relate to the implementation frameworks published between 2016 and 2022. One reason for the increase in implementations might be the substantiation to extend W3C PROV and OPM [58]. Another reason might be the increasing awareness of data management practices. At present, heterogeneous data sources, dynamic infrastructures, data exchange across boundaries, and a lack of standards for quality measures characterize the state of electronic health record data sets [48].

A provenance framework must acknowledge the semantic complexity of the domain and its relevant facets and requirements [58] and the importance of good data management tooling and practices (Figure 2). In addition to requirements analysis, a thorough strategy is necessary to plan the typical data management steps such as collecting, managing, and analyzing data [30]. According to Curcin [4], validation readiness can be achieved by separating modeling and verification of provenance data from the software implementation. We agree that precise requirements analysis, as part of the software life cycle, and the subsequent individual life cycle steps, such as testing and maintenance procedures, support the consequent temporal evolution and hence improve the quality of provenance frameworks and applications.

When incorporated in an inspection, provenance information must be sufficient for a content-related validation against applicable and accepted standards [4]. Therefore, precise validation methods for provenance services regarding usability and performance, scalability, fault tolerance, and functionality are needed [64]. We saw that validation approaches are linked to the evolution of provenance modeling and subsequent implementation attempts. Curcin et al [1] argued that it was necessary to launch more formal software engineering techniques to foster provenance implementation across a broad range of software tools in the biomedical domain and beyond. In that sense, formal validation as part of the software engineering process contributes to increased software and data quality. Formal validation requires testing efforts and testing evidence. Accurate alignment of testing procedures against predefined requirements in the software life cycle could not be identified in the included papers.

Provenance information is of high value for the scientific and biomedical community (eg, researchers); support staff (eg, developers); patients and other third parties (eg, data privacy officer or authority; Figure 4). It is interesting to see that despite the high impact of provenance (Multimedia Appendix 7), only some stakeholders provide sufficient provenance information. Rather, it appears that responsibility for overall provenance management is being shifted to the support staff [88]. We argue that available technology, IT knowledge, and data management skills need to be paired with both domain-specific knowledge and combined with constraints of legal nature or guidance [4,50]. This complexity indeed results in a very time-consuming business. However, automation and metadata collection can support this process [4,6]. As a matter of fact, good provenance information strengthens the credibility of the data and proves that data have not been intentionally or unintentionally changed throughout the data life cycle [89].

We believe that the persisting disagreement on the interpretation of the term provenance hampers the uptake of existing frameworks. A unique understanding of the concepts surrounding provenance should be developed, followed by engineering efforts for modeling, implementation, and validation interventions. The ISO 8000-2:2022 standard [90] defines the term data quality and clearly recommends defining degrees of requirements. This definition should be considered for use in provenance systems.

With regard to the implementation of provenance systems, we observed that increasing legal and scientific demands require research projects to be implemented more transparently. However, the granularity of provenance could not yet be resolved and so-called knowledge bottlenecks [50,65] persist. It is important to understand that appropriate modeling of provenance information and effective provenance management techniques are required to protect sensitive provenance data.

It furthermore remains unclear how to scale provenance systems for high amounts of data [2,58] (eg, how to store and represent provenance information in an aggregated and efficient manner or how to assist users in sophisticated provenance queries [13]). Without doubt, automated and scalable solutions become impelling due to new challenges arising from the disposal and use of permanently increasing computing power [40]. Growing focus is on the usability of the interface, particularly when provenance systems are implemented in the broad medical community, including patients, physicians, and researchers [73].

The lack of mandatory specifications or guidelines for provenance capture might be the reason why other papers only mention partial completeness. We strongly recommend doing more research on completeness checks as part of provenance tracing. The level of completeness and accuracy of provenance information of core data elements, especially in real-world data, could reveal data integrity issues and thus, affect the overall validity of the study results. Furthermore, reproducibility significantly depends on the accuracy of provenance information. For example, Mondelli et al [27] delivered a tool for better scientific and longitudinal data management, which supports users, reproducibility by provenance, and reproduction through docker containers.

Interestingly, the concept of “quality of provenance” is not clearly defined in any of the papers included for this review. We believe that data quality issues need to be addressed to reach completeness, accuracy, and timeliness of the data and to create trust in it.

Another direction reveals the importance of good and systematic data management practices [51] and the coordination with relevant stakeholders through the data life cycle. As such, first approaches toward interdisciplinary collaboration started within the MIRAPIE community project [91] and were also recognized in the CAESAR project [75].

In summary, our review collects and structures the challenges during the accomplishment of provenance capture (Figure 5). Challenges that address missing or lacking organizational and technical capabilities were triangulated into more specific subcategories such as organizational (eg, investment and training and administrative) and technical (eg, granularity, performance and modeling and metadata annotation, delimitation reproducibility, and replicability) challenges.

Comparison With Prior Work

In the realm of similar studies, a recent systematic literature review by Sembay et al [92] delved into provenance data management in health information systems. This review is a valuable resource if interested in the technical aspects, encompassing various methods, models, methodologies, and technologies associated with provenance data management in health information systems. It followed the guidelines proposed by Kitchenham and Charters [93] for performing systematic literature reviews in software engineering. Another systematic literature review by Ahmed et al [94] studied the impacts of data provenance in health care and General Data Protection Regulation compliance–based data provenance from a technological perspective. Although our RQ1 shares conceptual and domain-related similarities with the compared studies, there are notable differences in scope, RQs, methodological approaches, and thematic analysis focus.

More than a year after we published our protocol and concluded the thematic analysis of our review findings, a scoping review on biomedical data provenance was published [95]. The authors acknowledged and referenced our protocol. As they had anticipated, our review comprised 5 broad RQs, with the primary overlap occurring in RQ1. However, our scoping review offers more comprehensive results concerning the practical application of provenance and the associated challenges, including aspects such as completeness and validation. In addition, we provide in-depth descriptions of thematic areas and their subtopics, supported by tables, figures, and, most notably, a development of a tailor-made provenance framework road map. These aspects can be considered as the added value of our findings.

Our scoping review shares only few references with the other 2 (in detail: 11 out of 66 references are identical with references used in Johns et al [95] and 2 references are identical with those used in Sembay et al [92]). Therefore, all 4 reviews examine different research results and focus on other aspects. Ahmed et al [94] did not provide the list of the selected 59 articles.

Strengths and Limitations

This work applied a rigorous scoping review methodology using the framework developed by Arksey and O’Malley [16]. All screening stages were carried out by at least 2 independent reviews of 4 members. A previously published protocol [17] guided our review. The fact that the scoping review includes comprehensive results for the 5 related RQs and a road map for a tailor-made Provenance-SFL framework with many additional results as supplements can be considered a strength of this review. The criteria for categorization of provenance impact, requirements, and challenge classes were identified qualitatively in peer review.

One limitation may be because we excluded gray literature from our analysis.

Conclusions

In this paper, we highlighted various approaches and criteria for provenance tracking together with their referenced artifacts, and we developed a road map for a tailor-made Provenance-SFL framework.

Provenance capture benefits all stakeholders involved in data processing, but it is associated with manifold and individual challenges during design, implementation, and the active use scenario phase.

Sophisticated data management planning, documentation, metadata expression, and automation along the sensitive data-processing pipelines need to be scrutinized and implemented throughout the data life cycle and in adherence to the underlying infrastructure condition. With rising data volumes and the legal and scientific demands, there is an urgent need for greater transparency in implementing provenance systems in research projects, despite the challenges of unresolved granularity and knowledge bottlenecks. In addition, the roles and responsibilities of a data stewardship escorting the data should be expressed in this context and intensive training and education measures should be put in place. Guidance and recommendations are requested to provide the systematic measurement of provenance and calls for defining a minimal or gold standard. Governance for good data management and scale-up of good data management capabilities matter in this regard.

The mentioned artifacts, particularly those related to quality aspects, can be seen as transition points resulting from incomplete preliminary work. Therefore, harmonized engineering efforts are now necessary to overcoming the existing hurdles. Awareness of these challenges can facilitate an easier qualified and accurate provenance construction and auditable consumption while enforcing FAIR principles and interoperability standards for data sharing. The effect of provenance for data quality monitoring and the impact of expressive metadata on provenance quality can be considered as open RQs for future work.