Challenges and Recommendations for Electronic Health Records Data Extraction and Preparation for Dynamic Prediction Modeling in Hospitalized Patients: Practical Guide and Tutorial

Background

Predictive modeling using electronic health record (EHR) data has become increasingly important in enhancing patient outcomes through real-time risk detection and timely interventions. However, the effectiveness of these models is heavily reliant on the quality and structure of the underlying data, which are influenced by the processes of data extraction and preparation. While recent advancements in artificial intelligence and machine learning [1,2] have shown promise in this area [3-5], significant challenges remain in ensuring that the data used for model development are representative and reliable. The use case that inspired this work involves a recent project on the dynamic prediction of central line-associated bloodstream infections (CLABSI) using hospital-wide data from the University Hospitals Leuven (Belgium). Multimedia Appendix 1 [6-8] (use case: CLABSI prediction using data from a single-center university hospital) summarizes the main challenges we encountered together with corresponding solutions.

Data collection is part of the routine hospital workflow. Data extraction entails retrieving and structuring raw EHR data from hospital databases using an extract, transform, load (ETL) process, with standards such as Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) facilitating structure and terminology consistency (Multimedia Appendix 1. Glossary of technical terms). Publicly available intensive care unit (ICU) and emergency admission datasets [9-12], such as MIMIC [13,14], support reproducible research, while hospitals also extract private, not publicly shared data for predictive modeling. Data preparation transforms extracted data into a structured format for predictive modeling, often using R (R Foundation) or Python (Python Software Foundation) pipelines. Open-source frameworks, whether generalizable [15,16] or specific to MIMIC [17,18], aim to establish reproducible workflows for EHR data preparation. Data quality issues, such as inconsistencies and artifacts introduced during extraction and preparation, can compromise the performance of predictive models. Understanding the intricacies of data extraction and preparation challenges is essential for developing robust predictive models that can be reliably integrated into clinical practice.

Structured data quality assessments [19-22] are facilitated by frameworks such as by Weiskopf and Weng [23] and METRIC (Measurement Process, Timeliness, Representativeness, Informativeness, Consistency) [24]. However, researchers often conduct unstructured assessments, documenting “lessons learned” from their projects [25-30]. These assessments tend to focus primarily on data cleaning rather than the entire data extraction and preparation process, including cohort definition, outcome definition, and feature engineering. While literature on data quality assessments exists, mitigation strategies remain largely undocumented [31].

Objective

This paper provides a comprehensive list of challenges encountered during data extraction and preparation using EHR data for developing dynamic prediction models for clinical use. It further proposes recommendations with actionable insights, with the intention to enhance data quality and improve the practical applicability of (dynamic) prediction models in clinical settings. Our insights are drawn from a selective literature review, as well as our experience with various EHR data extractions. This list is intended as a hands-on resource for data extraction engineers (who perform the data extraction) and researchers (who prepare the data for model building) to consult during the extraction and preparation process. By addressing these challenges, we hope to contribute to the development of more reliable and effective predictive modeling frameworks that can ultimately benefit patient care.

We focus on single-hospital structured data, covering both ICU-specific and hospital-wide extractions. We focus on medium to large-scale extractions, that is, generated through structured ETL processes for a large number of extracted items spanning diverse clinical domains (laboratory results, medications, demographics, comorbidities, etc), and we cover standardized (eg, OMOP CMD) and nonstandard, private extractions. We do not address combined data registries, such as national registries (integrating surveys, general practice, insurance data, and EHR extractions), multicenter data extractions, hospital data collected for clinical trials, or the extraction and processing of unstructured data (eg, text notes or images). We cover the broader application of dynamic models (or continuous prediction) for offering the most comprehensive framework, although many of the recommendations are applicable to static prediction models (eg, a single prediction per admission at 24 hours after admission).

Recognizing that EHR software may contain bugs, human errors will occur, and hospital processes will evolve and generally improve over time, we do not provide recommendations for EHR vendors or for modifying hospital workflows, clinical practices, or data recording procedures within EHR systems. When data quality issues, such as those arising from data recording procedures, render the extracted data inadequate for a specific prediction task, we leave it to researchers to assess adequacy, without offering guidance on what constitutes suitable data for a given prediction task.

The section “EHR Data Flow from Data Collection to Model Building” explains the typical trajectory of data, for both model building and clinical implementation, setting the stage and providing context for the detailed list of challenges that follows. The section “Challenges and Recommendations” lists the challenges and actionable recommendations, which represent the core of the paper. These are categorized into 4 groups: cohort definition, outcome definition, feature engineering, and data cleaning. This is one possible categorization, inspired by the stages of the data preparation process. The “Discussion and Conclusion” section summarizes the recommendations and reflects on their broader implications.

EHR Data Flow From Data Collection to Model Building

The data flow from the patient’s bed to the prediction model building typically follows three stages: (1) data collection, (2) data extraction, and (3) data preparation (Figure 1A). (1) Patient data are either manually entered in EHR software modules or collected by devices and stored in one or multiple databases. The data collected serve multiple purposes, for example, daily bedside clinical work, national benchmarking [32], or reimbursing the care delivered. (2) EHR data are then extracted from relational databases in a simplified (denormalized) format and typically stored in a data warehouse as multiple “base tables” per category, each capturing different aspects of patient data and health care events. Examples of base tables naming as per OMOP CDM are PERSON, DRUG_EXPOSURE, DEVICE_EXPOSURE, CONDITION_OCCURRENCE, MEASUREMENT, NOTE, OBSERVATION, but naming and granularity of extraction can vary for nonstandard extractions. (3) Researchers building a prediction model either have access to the data warehouse or get a copy of all tables or, under specific ethical and legal considerations, a subset of the base tables or a subset of the patients in the data warehouse (eg, admissions during a specified period of interest for patients undergoing mechanical ventilation). Further, they process the extracted data and bring it in a format on which a prediction model can be built.

At model implementation time in clinical practice in the EHR software (Figure 1B), a trigger (eg, update of laboratory results) or a scheduled task (eg, every 24 hours) will initiate the request for a prediction. Data are already collected (1) for the patient in the EHR databases. The same logic as for extraction (2) is reproduced (typically by reusing the queries or code used for extraction). Using the data packed in a specific format (eg, JSON and XML), the prediction service (typically using a RESTful (Representational State Transfer) application programming interface for communication) is invoked. Data exchange between the EHR and the prediction service is generally performed using the Fast Healthcare Interoperability Resources standard. The prediction service will further prepare the data (3), invoke the model, and return a prediction (and additional information, if foreseen) to the EHR software, which will present it to users in the form of alerts or patient flags within the patient’s chart. The process is typically logged in the EHR system for monitoring purposes. In summary, at real-time prediction, the (1) data collection happens implicitly and is part of the normal clinical flow; (2) data extraction and (3) data preparation are identically reproduced as for model building.

We provide additional background on some particularities for each of the processes of collection, extraction, and preparation, which represent transition phases from a data format to another.

‎

Data Collection

The EHR database will not reflect with maximum accuracy the “true state” of the patient. First, it will suffer from incompleteness, as not all possible markers and observations can be collected for all patients at all times. The decisions with regards to what data are collected (eg, which laboratory tests are ordered and performed) are highly dependent on the patient’s conditions and on the hospital procedures. Data collected during routine clinical practice are generally documented more carefully when also used for national reporting of quality of care indicators [32]. Sufficient data are collected during clinical practice by doctors, nurses, assistants, etc, in the system to support the patient’s clinical follow-up and treatment. From this perspective, EHR data differ vastly from data collected for clinical trials, where researchers specify the measurements, measurement methods, and collection procedure. Second, nurses and clinicians might have slightly more information than the data collected in the system, either from patient conversations or organizational knowledge. Considering that we do not cover the extraction of text notes and reports, tabular data only will always suffer from a level of incompleteness. Nevertheless, tabular data might prove sufficient for specific prediction tasks. Third, both manual data entry and data collected by devices can be, at times, error-prone, software can have bugs, and data recording procedures in the system will affect the granularity of observed data and will change over time. These considerations, which can be summarized as data completeness, correctness, and currency, as defined by Weiskopf and Weng [23], have to be carefully considered by researchers to assess if EHR data are fit for the prediction goal [21].

Data Extraction

The extracted data might not correctly reflect the EHR database. Although data extraction generally aims for completeness of all clinically relevant data, the extraction process can introduce undesired artifacts or can have its own limitations. Whenever extraction logic bugs are detected, these should be corrected and safeguarded through unit or integration tests. Mature extraction platforms, tested through repeated use and proven reliability, will generally be less error-prone, and researchers can consider the maturity of the extraction platform as a factor influencing the time they will spend on data preparation. The extraction process is typically carried out by a data extraction engineer, data integration developer, data warehouse engineer, or ETL developer representing the EHR vendor or the hospital information technology department. Hereafter, we refer to this role as the data extraction engineer. They work closely with EHR software developers, hospital information technology staff, and clinical personnel to understand database structures and data recording procedures. They perform clinical concept and terminology mapping and document the extracted data.

Data Preparation

The prepared data might not correctly reflect the extracted data. Undesired artifacts can be introduced by feature engineering or data cleaning. Good documentation of the extraction format and close collaboration between researchers, data engineers, and clinical experts will ensure that the data and the features are not misinterpreted. Coding errors can always occur. Time-sensitive data poses an additional challenge to ensure no temporal leaks (using future data to predict past events) are introduced by mistake. Outcome leaks (including outcome information in predictors) and test-train leaks (including information from test patients in the training datasets) can also occur if data are not preprocessed carefully. A good practice is to separate the train and test sets first and apply the data preprocessing separately. A modular organization of the code will facilitate unit testing (testing individual functions of a program in isolation to ensure they work as expected) and easy modifications without introducing new errors. The more complex the data preparation, the more error-prone it becomes. The data preparation is performed by a researcher, data scientist, statistician, or machine learning engineer, whom we will refer to as a researcher for the remainder of this paper.

Each stage of processing the data has its own challenges and can introduce new problems, widening the gap between the patient’s state and the data used by the prediction model and ultimately impacting the model building (suboptimal model), model evaluation (misleading performance metrics), or model implementation in clinical practice.

Challenges and Recommendations

We list common problems originating in the (1) data collection process, the (2) data extraction process, and the (3) data preparation process (Figure 1A). We provide recommendations for mitigation strategies that can be implemented during the (2) data extraction or (3) data preparation. We also focus on problems that can impede the identical reproduction of the extraction and preparation at clinical implementation time. We have categorized the challenges and recommendations into four groups: (1) cohort definition (and inclusion or exclusion criteria), (2) outcome definition, (3) feature engineering, and (4) data cleaning, and each group contains problems originating in the collection, extraction, or preparation process.

We provide mapping of the items to both the Weiskopf and Weng framework [23] and the METRIC framework [24], whenever applicable. Weiskopf and Weng do not include dimensions covering the cohort representativeness and completeness of the extracted features, which are important in prediction settings. We will use the completeness dimension to refer to completeness of data values, as in the original definition [23], as well as completeness of the cohort and of the extracted features (our extension).

While we aimed to make these challenges as generic as possible, we recognize that specific issues are often unique to individual projects and may not apply to all prediction tasks. We advise users to assess the impact of each listed challenge in their specific context. Similarly, the recommendations might not always be universally applicable and depend on the project context. Our guidance remains pragmatic; at times, the best approach may be to “leave as is” to avoid the risk of overcorrection, which can backfire. While certain corrections may improve alignment with clinical meaning for a specific patient group, they can introduce errors or biases when applied to all patients or be difficult to reproduce at clinical implementation time, affecting the model’s performance in clinical use. Following the fitness-for-use principle [20], we encourage readers to remain pragmatic and address only the issues relevant to their data and prediction task.

Figure 2 presents a high-level schematic overview of the recommendations, without detailing each specific challenge. For a full understanding of the context in which recommendations are made, we encourage hands-on users involved in data extraction and preparation to consult the following sections and tables below for a comprehensive understanding.

‎

Cohort Definition

Defining the cohort of interest (eg, hospital-wide population or patients with a specific condition) represents a first step in the prediction task definition (Tables 1-3). It can also be of interest during the project planning phase. An inaccurately defined cohort can lead to selection bias, resulting in performance estimates that do not accurately reflect the model’s performance in clinical practice.

Outcome Definition

Prediction models using EHR data usually focus on in-hospital or postdischarge outcomes, including mortality, length of stay, readmission, acute events (such as bacteremia, sepsis, and acute kidney injury), and chronic diseases (such as heart failure, cancer, and cardiovascular disease) [33]. We focus on outcomes that can be derived solely from structured EHR data, without linkage to external data sources or extraction of text notes, and are linked to a patient; that is, we exclude resource use and workflow optimization outcomes (Tables 4 and 5). Similar to the cohort definition, a good practice is assessing the feasibility of defining the outcome during the project planning phase.

Feature Engineering

Feature engineering is the process of selecting, transforming, and creating features (variables) from the extracted EHR data to ensure that the data are brought into a suitable format for modeling and that relevant information is processed in a meaningful way. It includes data mapping (eg, mapping medication brand names to generic drug names) and transformation (eg, grouping medical specialties in meaningful categories), converting timestamped events (eg, laboratory results, medication administration, and vital signs) into snapshot-based features, and data aggregation (summarizing multiple observations or measurements into single features). The feature engineering is typically performed during data preparation, and usually it happens concomitantly with data cleaning. Sometimes feature engineering is also performed at data extraction time, for example, for reducing the data volume for high-frequency time-series data, especially in ICU settings, or for computing scores that are calculated and displayed in the EHR software but not stored in the system. Clinical concept mapping, such as mapping medication to Anatomical Therapeutic Chemical codes, laboratory tests to LOINC (Logical Observation Identifiers, Names, and Codes) codes or procedures and clinical observations to Systematized Nomenclature of Medicine Clinical Terms can occur at different stages, including within the EHR software, during data extraction (eg, OMOP CMD Standardized Vocabularies), or as part of data preparation.

We distinguish between generic feature engineering and time-sensitive feature engineering. Generic feature engineering (Tables 6 and 7) deals with the mapping of clinical items Time-sensitive feature engineering (Tables 8 and 9), which deals with data aggregation for which correct processing of timestamps is critical and that can result in temporal leaks (ie, data available at a specific time for training the model, but not available in the system at that timestamp. Time-sensitive feature engineering is critical for dynamic prediction models, can also be useful for some static models (eg, predictions at 24 hours after admission), and has less impact on models with a prediction trigger at the end of the admission (eg, readmission prediction). We acknowledge that some minimal temporal leaks might not be very detrimental for the model performance or for its applicability, while others can have a large impact. However, following the “do it right the first time” principle, a good understanding of the extracted timestamps and correct handling of dates or times during data extraction and preparation can safeguard against future problems, big or small.

As for the previous 2 groups, certain aspects of feature engineering can be evaluated during project planning, at least for the key features relevant to the prediction task and, at a minimum, for their availability. For instance, if a key feature was only recorded for a limited period before being discontinued or if timestamps of key features are unavailable for a dynamic prediction task, the project may become jeopardized.

Data Cleaning

The data cleaning process consists of identifying and correcting data issues that could negatively impact the performance or applicability of prediction models. Errors in EHR data can arise from manual entry mistakes, improperly connected or malfunctioning devices, or bugs within the EHR software. Such errors are typically uncovered during data exploration and addressed during data preparation. Tools such as the Data Quality Dashboard [37] have been developed to support and streamline the data cleaning process. Artifacts can also be introduced during data extraction or data preparation. Unit tests for both the ETL process and the data preparation pipeline can safeguard against introducing additional errors.

Overzealous correction of errors, such as manual correction during data preparation of every error encountered, might not necessarily prove beneficial for prediction tasks when such corrections cannot be programmatically reproduced during model implementation in clinical settings. This discrepancy can lead to a situation in which training data accurately represent the true patient state, while real-time predictions rely on erroneous data, resulting in reduced predictive performance. Although measures can be taken to address common and documented errors from previous studies or identified during data exploration (Tables 10 and 11), it is impossible to anticipate and guard against all future errors. For example, new EHR software versions and updates may introduce new bugs, even as older bugs are resolved.

Discussion and Conclusion

In this work, we highlighted challenges and recommendations for the extraction and preparation of EHR data for predictive modeling. Adhering to the “garbage in, garbage out” principle, prediction models rely heavily on the quality and relevance of the input data to generate meaningful and reliable predictions. High-quality data extraction and preparation processes can support prediction models with clinical utility. While it is often argued that EHR “data are not collected for research purposes” [21,26-28,39,40], the fact that these data are sufficient for supporting patient care suggests they could be adequate for developing prediction models. Even though the list of challenges may seem extensive, it should not discourage researchers new to working with EHR data. Not all challenges will apply to every project, and as EHR systems continue to evolve, many issues might affect only historical data and not current data at implementation time. Successfully leveraging EHR data requires both an understanding of its limitations and an appreciation of its potential. Despite its complexities, EHR data offer a rich, comprehensive source of real-world clinical information that can drive impactful research and improve patient outcomes. Applied prediction models can exploit the comprehensive clinical information that is not always readily available to all health care professionals, such as nurses, doctors, hygienists, and other therapists, and have proven practical applicability [5].

Data extraction and preparation for predictive modeling using EHR data are resource-intensive processes, with time and cost varying depending on the maturity of the extraction framework, the prediction task, and the team’s experience in working with EHR data. It is estimated that this phase (which generally includes collaboration with data extraction engineers and clinical experts) takes up to 3 months for an entire team [3], but such estimations will be highly dependent on the maturity of the extraction process and of the team’s experience with EHR data. Data quality often varies significantly across EHR systems and extraction processes, as noted by Weiskopf and Weng [23]. Issues such as data gaps, temporal leaks, incorrect linking, inconsistencies in clinical concept and terminology mapping can affect the quality of extracted datasets and compromise the model’s performance and applicability. To address such challenges, we proposed a list of practical recommendations informed by our experiences with EHR data and insights from published studies. We organized the challenges and recommendations into: cohort definition, outcome definition, feature engineering, and data cleaning; the first 3 categories can also be consulted when planning a project. A clear definition of the prediction task or research question, of the intended use, and the intended users of a prediction model are the first critical steps for defining the outcome, the cohort, and the features of interest [27,31]. It is not uncommon to deem the EHR data inadequate for the prediction task before proceeding with the model-building phase [21].

For cohort definition, we recommend extracting a broader patient context beyond the immediate focus, assessing the completeness of data used for inclusion or exclusion criteria and its availability at prediction time, preventing omissions or duplications, and carefully defining episodes of interest for prediction. Outcomes can be derived in different ways, each with advantages and shortcomings. We generally advise against the use of ICD (International Classification of Diseases) codes (as these can be up- or undercoded and are generally not timestamped), unless carefully assessed as appropriate for the prediction task. A good understanding of the hospital’s processes, thorough verification of outcomes derived in code, manual inspection of labels, and agreement between data sources can also prevent incorrect outcome definition. Mapping terminology to coding systems (eg, ICD, Current Procedural Terminology, LOINC, Systematized Nomenclature of Medicine Clinical Terms, or others) can facilitate feature engineering. However, not all items in the EHR system are aligned with standardized terminologies (eg, LOINC codes for laboratory results are often not used). High-quality data extraction documentation and a good understanding of the underlying clinical concepts and health care processes are essential to support feature engineering. Data exploration can reveal unaddressed problems and further inform the construction of meaningful features. For feature engineering in prediction settings, the timestamp when a clinical item is available in the system is of greatest interest, as this is the time at which predictor values become available for prediction in clinical practice. Good documentation and correct interpretation of the extracted timestamps will prevent temporal leaks. Thorough verification of the extraction and preparation processes (using manual or automated tests), data exploration, and reproducible data cleaning can safeguard against data quality issues.

Clinical assessment of the relevance and the sequence of extracted features and outcome within a patient admission by manual verification of a random sample of admissions can further help detect problems [3]. The solutions to specific issues can be implemented at either the extraction or preparation stage. Applicable to both data extraction and preparation are a good understanding of the underlying data structure and current and historical health care processes, collaboration with relevant experts in conducting the work [3,31], and ensuring a qualitative process of extraction and preparation, supported by unit tests. We recommend maintaining consistency in data extraction and preparation between model training and clinical implementation. Exceptions might, though, exist for correcting historical data problems that are not expected to recur in future data.

We hold the opinion that, in the context of prediction models, the extraction process should not attempt to correct errors residing in the EHR database in an attempt to align the data to the clinical reality, but it should reflect the information from the EHR, presented in a simplified format. We recommend that data cleaning steps be performed during data preparation, pragmatically and programmatically, so that they can be reproduced at implementation time. Extracting and preparing the training data in a different manner than for clinical implementation (eg, temporal leaks) poses the risk of potential overoptimistic evaluation, in the light of which the model’s performance when implemented in clinical practice will be lower than expected. We acknowledge that there are divergent views on this topic and that, in the context of inferential studies, when the analysis does not need to be reproduced on future data, corrections during data extraction might be preferred. At the same time, multipurpose extractions (for both prediction and inferential studies) pose the challenge of solving this divergent view.

We do not provide recommendations for correcting (historical) biases in EHR data (eg, racial, gender, and socioeconomic). Research on preprocessing to enhance fairness has mainly focused on weighting and sampling techniques, while our work focuses on data extraction and feature engineering. Identifying and correcting bias typically requires analyzing model performance and comparison across subgroups, which happens after the data are prepared and the model is built. While the primary focus of this tutorial is methodological, several of our recommendations, such as those concerning cohort definition strategies, implicitly reflect data governance considerations, including the need for ethical approval and proper access control. For example, we encourage defining cohorts during data preparation, provided appropriate ethical approvals are in place to extract broad datasets.

While the level of detail for reporting data extraction and preparation in published prediction studies varies, we do not provide specific recommendations on this aspect. Some researchers advocate for comprehensive documentation of these processes [30,41,42], while others emphasize the importance of sharing the data preparation code to ensure transparency [43,44]. Data extraction for public datasets is generally documented in a separate publication. This can be complemented by sharing the data preparation code and describing in the main paper the key differences between the raw extracted data and the final prepared dataset [30]. Each strategy has its advantages, depending on the audience, journal requirements, and the desired trade-off between transparency and conciseness. While we advocate for transparent reporting, this paper does not specifically address reporting guidelines for data extraction and preparation.

Implementing terminology mapping and standardization during the data extraction phase (eg, through OMOP CDM) can significantly reduce the effort required during data preparation and enable research on multisite datasets extracted from different EHR systems. Furthermore, toolsets for OMOP CDM EHR data exploration and quality assessment have been developed as part of the Observational Health Data Sciences and Informatics initiative, such as Achilles [45] and Data Quality Dashboard [37], which facilitate more than 3500 data quality checks. While we recommend standardization as a solution for some of the listed challenges, we aimed to cover both standard and nonstandard extractions, as the adoption of the OMOP CDM framework may involve higher costs and effort, which can be prohibitive for some hospitals.

Our work has several limitations. First, it is based on our experiences and a selective literature review that cannot be exhaustive. We acknowledge that every project will face use-case-specific challenges. Insights from other research groups working with EHR data would likely highlight additional challenges and recommendations that we may have overlooked, offering a more complete set of recommendations. Second, our focus was on single-site structured EHR data. Extensions to multicenter datasets or unstructured data are possible. Multicenter datasets pose additional challenges with regard to aligning clinical concepts and following the same patient in different hospitals or general practitioner systems. Patient care happens across multiple systems, and single-site extractions provide only a fragmented view of the entire patient care. Third, we provide recommendations for problems that can impede the model implementation in practice, but we do not explicitly cover post implementation challenges, although some of our recommendations can inform on monitoring checks that can be implemented, which remains a subject for medical device post market surveillance. Fourth, while we emphasize the importance of high-quality data extraction and preparation as the foundation of reliable predictive models, we do not assess the impact of each challenge on the final prediction model. The impact will likely depend on the magnitude of the problem, the subgroups in which it manifests, the prediction task, or even the type of model (eg, tree-based models are generally resilient to outliers). While the impact of dataset size, missing data, and outcome definition on the model performance has been studied [46], the impact of other steps in the data preparation procedure remains unknown. Researchers identify and resolve problems during data preparation without assessing their impact on model performance, which we acknowledge having done the same. Research on measurement error [47] demonstrates that inconsistencies in predictor definitions between training and test data can affect model calibration. A similar impact may occur if correction strategies differ between model training and clinical implementation. Finally, although we focused on models with clinical applicability, by recommending an implementation-ready process to avoid discrepancies between the training data and future data due to data extraction and preparation, we did not specifically address model generalization to new hospital settings. The requirements for reproducing the data extraction and preparation can vary significantly based on the EHR software in use (whether from the same or different vendor), the data extraction platform, and the hospital workflow and data registration procedures. Sendak et al [29] estimate “approximately 75% of the effort invested in the initial data preparation for developing prediction models must be reinvested for each hospital.” We argue that the estimation would vary widely, with standardized extractions from the same EHR software potentially requiring less time. Further research could focus on extending the list of challenges and recommendations based on the experience with EHR data of other research groups, extending the scope to larger extraction contexts, or assessing the impact of erroneous or suboptimal extraction and preparation on the final model. We strongly encourage researchers to conduct such impact studies, as this remains a notable gap in the current literature. Understanding how problems propagate through the pipeline is essential for developing trustworthy and clinically meaningful prediction models. We also recognize that decisions made during data extraction and preparation have direct implications for patient safety. Model transparency, reproducibility, and clinical accountability begin early in the pipeline; poor documentation or inconsistent preprocessing can lead to silent errors that affect downstream predictions. Future work could further explore how data extraction and preparation decisions impact not just model performance but also the safe and ethical use of prediction models in clinical settings.

The extensive list of challenges and practical recommendations for EHR data extraction and preparation presented here is intended to improve the quality of research and the practical applicability of clinical predictive models. As all modeling efforts begin with the underlying data, failing to address data quality issues risks producing unreliable and nongeneralizable models. Our focus extends beyond the initial data curation stage in the artificial intelligence life cycle [48]; we also address early-stage issues that can ultimately negatively impact model deployment. Recognizing that there is no “one-size-fits-all” solution, our list of challenges and recommendations, though not exhaustive, is comprehensive enough to support many EHR-based prediction projects. Implementing the strategies applicable to each project can ultimately enhance robustness, reproducibility, and real-world impact of EHR-based prediction models.