Original Paper
Abstract
Background: Patient-centered clinical decision support (PC CDS) exists on a continuum that reflects the degree to which its knowledge base, data, delivery, and use focus on patient needs and experiences. A new focus on value-based, whole-person care has resulted in broader development of PC CDS technologies, yet there is limited information on how to measure their performance and effectiveness. To address these gaps, there is a need for more measurement guidance to assess PC CDS interventions.
Objective: This paper presents a new framework that incorporates patient-centered principles into traditional health IT and clinical decision support (CDS) evaluation frameworks to create a unified guide to PC CDS performance measurement.
Methods: We conducted a targeted literature review of 147 sources on health IT, CDS, and PC CDS measurement and evaluation to develop the framework. Sources were reviewed if they included the sociotechnical components relevant to PC CDS, covered the full IT life cycle of PC CDS, and addressed measurement considerations at different user and system levels. We then validated and refined the measurement framework through key informant interviews with 6 experts in measurement, CDS, and clinical informatics. Throughout the framework development, we gathered feedback from a 7-member expert committee on the methods, findings, and the framework’s relevance and application.
Results: The PC CDS performance measurement framework includes 6 domains: safe, timely, effective, efficient, equitable, and patient centered. The 6 domains contain 34 subdomains that can be selected to assess performance, depending on the type of PC CDS intervention or the specific research focus. In addition, there are 4 levels of aggregation at which subdomains can be measured (individual, population, organization, or IT system) that account for the multilevel impact of PC CDS. We provide examples of measures and approaches to patient centeredness for each subdomain, followed by 2 illustrative use cases demonstrating the framework application.
Conclusions: This framework can be used by researchers, health system leaders, informaticians, and patients to understand the full breadth of performance and impact of PC CDS technology. The framework is significant in that it (1) covers the entire PC CDS life cycle, (2) has a direct focus on the patient, (3) covers measurement at different levels, (4) encompasses 6 independent but related domains, and (5) requires additional research and development to fully characterize all domains and subdomains. As the field of PC CDS matures, researchers and evaluators can build upon the framework to assess which components of PC CDS technologies work; whether PC CDS technologies are being used as anticipated; and whether the intended outcomes of delivering evidence-based, patient-centered care are being achieved.
doi:10.2196/68674
Keywords
Introduction
Background
In response to a growing body of research on suboptimal health care quality in the United States [,], organizations delivering health care are amid an unprecedented transition in how they approach patient care and reimbursement. The new focus is on value-based, whole-person care that prioritizes patient values, goals, needs, and preferences to achieve positive health outcomes. This new approach to care calls for health care organizations to become “learning health systems” to generate and integrate knowledge for continuous improvement and innovation [], which includes advancement of health IT and promotion of personalized medicine, patient involvement in the clinical decision-making process, and treatment recommendations guided by evidence-based guidelines.
Clinical decision support (CDS) technology (eg, electronic health record [EHR] prompts and alerts and diagnostic and treatment guidance for clinicians based on clinical guidelines) aims to achieve these priorities but has traditionally focused on supporting clinicians at the point of care. Technology that supports patients and incorporates patient-centered knowledge and patient-generated health data, known as patient-centered CDS (PC CDS), holds promise for advancing value-based, whole-person care [,]. PC CDS encompasses a spectrum of decision-making tools that significantly incorporate patient-centered factors related to knowledge, data, delivery, and use. Knowledge refers to the use of comparative effectiveness research or patient-centered outcomes research findings. Data refer to the incorporation of patient-generated health data, patient preferences, social determinants of health (SDOH), and other patient-specific information. Use refers to directly engaging patients and caregivers across different settings. Finally, delivery focuses on facilitating bidirectional information exchange in support of patient-centered care, including shared decision-making [].
CDS developers commonly use the 5 rights [] as a benchmark for what makes a successful intervention, and several recent assessments of the CDS field have been published [-]. However, less is known about the emerging field of PC CDS []. While PC CDS has the potential to improve patient outcomes and enhance quality of care, it also has the potential to contribute to clinician burnout and undermine patient safety if it provides incorrect, irrelevant, uninformative, or nonspecific alerts and recommendations []. Measuring user acceptance performance is critical to understanding how PC CDS technology can achieve the 5 rights and prioritize outputs that are meaningful to patients and clinicians, as well as what unintended consequences may exist. PC CDS performance measurement is a complex undertaking given the intricacies of interventions, variety of data sources (eg, EHRs, remote devices, and apps), and nascency of the field. Without the understanding of what leads to a successful PC CDS intervention, PC CDS technology development, use, effectiveness, and scalability will remain limited. To ensure PC CDS technology is safely and effectively integrated into health care delivery systems, further research and measurement are needed to adapt to the goals of a patient-centered approach and facilitate understanding of the factors that lead to successful implementation.
Objectives
This paper presents a new framework that incorporates patient-centered principles into traditional health IT and CDS evaluation frameworks to create a unified guide to PC CDS performance measurement.
Methods
We used three methods in developing the new performance measurement framework: (1) literature review, (2) key informant interviews, and (3) feedback from an expert committee (refer to the Acknowledgments section for expert committee members).
Literature Review
Our literature review aimed to identify existing measurement frameworks and types of measurements used to assess health IT, CDS, and PC CDS technology performance. We began with a snowball approach, using a starting set of 10 peer-reviewed health IT and CDS measurement frameworks and measurement-related publications compiled by 1 of the authors (DFS) with expertise in CDS measurement and evaluation. We conducted backward citation tracking of the starting set of papers and then conducted a title and abstract screening to determine whether each article met the eligibility criteria (
Textbox 1).
The team reviewed the full text of each source and extracted information according to three predetermined data abstraction domains: (1) whether the source was a formal evaluation or measurement framework; (2) whether it was validated; and (3) whether the source’s findings specified levels of measurement (eg, organization and individual), measure domains, measure subdomains, and specific measures. Articles that did not meet ≥1 of these criteria were excluded from the review. This resulted in 44 peer-reviewed publications.
Eligibility criteria
- Covered at least 1 of the following 3 areas: first, the article included sociotechnical components relevant to patient-centered clinical decision support (PC CDS), such as human, organizational, environmental, and technical factors. Inclusion of these factors is important, as PC CDS needs to operate in complex health care systems, and the interaction of many different factors (including technology, people, environment, workflow, and organizational factors) has implications for PC CDS adoption and use []. Second, the article discussed the full IT life cycle of PC CDS design, development, implementation, use, and measurement. Comprehensive evaluation strategies need to account for collaboration across both technical and clinical disciplines through the full life cycle of system development and use []. Third, the article considered measurements at different levels (eg, individual, system, and health IT). PC CDS measurements need to occur at different levels to fully understand the factors that impede adoption and use at both user and system levels.
- Published as a peer-reviewed research article (ie, not abstracts, viewpoints, commentaries, editorials, or protocols).
- Full text available in English.
To begin constructing the domains and subdomains of the framework from these sources, we first used an inductive approach where 2 researchers mapped similar domains and subdomains within the literature to identify the most mentioned concepts. Two senior members of the research team (PD and DFS) then reviewed the mapping, discussed disagreements, and agreed on modifications to arrive at final domains and associated subdomains. After discovering that many of the domains and subdomains in the literature focused on aspects of quality measurement, we conducted another targeted literature review to identify existing health care quality frameworks that covered our identified domains. As a result of this review, we selected the National Academy of Medicine’s (NAM’s) 6 domains of health care quality [] as the organizing construct for our framework because it included all our identified domains, and it is well known in the CDS field.
To define the framework’s 34 subdomains, the team extracted definitions from the full text of each source. In cases where definitions or descriptions were not explicitly stated, we conducted supplemental searches specific to each subdomain in PubMed and Google Scholar to find more information. We also conducted supplemental searches to identify specific examples of measures in the subdomain and to describe the relationship of each domain and subdomain with PC CDS interventions. Given that PC CDS is a relatively new term, we used search terms encompassing key areas of CDS, such as “digital health” and “shared decision-making,” in combination with the relevant measurement concept (Textbox 2).
We screened titles and abstracts and excluded publications that were not peer-reviewed research articles, were not available in English, did not involve a PC CDS technology, or did not measure the appropriate measure domain. This yielded an additional 103 peer-reviewed articles (approximately 3 papers per each of the 34 subdomains). The final search included 147 sources (
Figure 1). To manage the screening process for all searches, we used Zotero (version 6.0; Corporation for Digital Scholarship) for reference management and Microsoft Excel for inclusion and exclusion decisions.
Keyword concepts and keywords
- Patient-centered clinical decision support: (“patient-centered clinical decision support” OR “PC CDS” OR “clinical decision support” OR “CDS”)
- Measurement: AND (“measure” OR “metrics” OR “evaluation” OR “assessment” OR “performance measure”)
- Measure domain or subdomain of interest: AND “[domain of interest]”; OR “[subdomain of interest]”
Key Informant Interviews
Using the network of the Clinical Decision Support Innovation Collaborative [], we identified a convenience sample of 8 potential informants to ensure informants represented different PC CDS stakeholder groups (patient representatives, clinicians, informaticians, measurement experts, and CDS developers). Specifically, the research team identified an initial list of known experts in the field of CDS and measurement. We then used a snowball approach to receive referrals from key informants on other experts to interview. Ultimately, we recruited 6 key informants—2 clinicians, 2 informaticians, and 2 measurement experts—to seek feedback on the framework and validate the measurement domains and subdomains.
Informants received the draft framework with the 6 NAM domains in advance of the interviews ( []). During the 60-minute interviews, we asked informants to provide feedback in three key areas: (1) perspectives on how we organized the domains and subdomains, (2) any gaps in the domains and subdomains, and (3) any refinements to the existing domains and subdomains. Transcript-style notes were created after each interview. We analyzed information from the interviews using qualitative thematic synthesis [], using an inductive approach with simultaneous phases of data collection and analysis. After each interview, 2 senior research team members (PD and DFS), who developed the draft framework and have expertise in qualitative coding, reviewed the transcript-style notes. They refined the themes identified in each interview and mapped them to the domains and subdomains identified in the literature review. Coding was conducted by 2 research analysts using Microsoft Excel. Analysts coded interviewee feedback into 4 overarching areas: subdomains to add, remove, recategorize, or refine. The research team discussed the findings within each code through biweekly collaborative sessions, where they mapped results to domains identified in the literature and resolved discrepancies through consensus. To mitigate bias, the preliminary findings were also reviewed by a multidisciplinary 7-member expert panel to further enhance the framework development.
Key informant findings helped to refine themes in the following ways: (1) validating the organization of the domains and subdomains, (2) identifying gaps in the patient centered subdomain regarding decisional quality and knowledge acquisition, and (3) providing valuable feedback on the framing of health literacy and digital health literacy concepts.
Expert Committee
The research team oversees a 7-member multidisciplinary expert committee that includes patient representatives, clinicians, informaticians, PC CDS developers, and payers. Committee members come from different academic, federal, and private institutions across the United States. Committee members were identified during the formation of the research collaborative to engage national leaders in advancing patient-centered technologies. To minimize bias that might serve the goals of 1 stakeholder group, the members were chosen to represent interdisciplinary professional backgrounds and perspectives.
The committee met on a quarterly basis through a web-based conferencing platform and provided feedback at 2 time points during the development of the framework: first, during the planning phases for the framework development, and second, during the analysis phase of findings from the literature review and interviews. Feedback from the committee was gathered via two 90-minute meetings. Meeting notes were summarized and analyzed for key themes. The expert committee validated the idea and need for a unified measurement framework for PC CDS performance, provided input on the organizing structure (NAM) and measurement domains for the framework, and reviewed the final framework.
Ethical Considerations
The NORC at the University of Chicago Institutional Review Board (IRB00000967) determined that the protocol did not meet the definition of research with human participants and was therefore exempt from further review. As such, written informed consent was waived, yet interviewers obtained verbal consent from participants at the start of each interview after informing them that participation was voluntary. No identifiable information is included in this manuscript.
Results
Overview
We identified 6 measurement and evaluation frameworks and 2 systematic reviews of such frameworks (Textbox 3). These frameworks and systematic reviews cover general health IT as well as CDS-specific evaluation.
Framework or systematic review and description
- Human, Organization, and Technology-Fit Framework []: building on previous models of health information system (HIS) evaluation, this framework proposes 8 dimensions for evaluating HIS and numerous evaluation measures within those dimensions. The framework incorporates the concept of “fit” among human, organizational, and technical factors (ie, not only does performance matter on individual dimensions and measures, but these factors must be in alignment for HIS to be successful).
- Behavior and Acceptance Framework (BEAR) []: BEAR is an integrated conceptual framework that bridges the gap between behavioral change and technology acceptance aspects of clinical decision support (CDS) implementation. BEAR presents “constructs” that are “determinants of behavioral change and acceptance of CDS” and groups them into domains.
- Evaluation in Life Cycle of Information Technology []: this is an evaluation framework for electronic health record–integrated innovations to support evaluation activities at each of 4 IT life cycle phases: planning, development, implementation, and operation. The framework also proposes 3 levels at which evaluations can be conducted—society, user, and IT—and provides measurement exemplars within each level.
- Health IT Reference-Based Evaluation Framework []: this framework builds on a prior systematic review to propose 4 evaluation concepts comprising 12 evaluation components to assess health IT.
- Measures of success of computerized CDS systems: an overview of systematic reviews []: This overview is the first to focus on evaluation metrics of CDS, which were mapped according to the updated Information Systems Success Model by DeLone and McLean [].
- Handbook of eHealth Evaluation: An Evidence-Based Approach []: this handbook provides a systematic overview of the different evaluation approaches to IT, with case examples that have been applied and reported for a wide range of health care systems and settings.
- Health information systems evaluation frameworks: a systematic review []: this paper analyzes studies on the evaluation of health information systems by applying a content, context, and process framework to address the “who,” “what,” “how,” “when,” and “why” of the evaluation processes used.
- Clinical, Human and Organizational, Educational, Administrative, Technical, Social framework []: This is an IT evaluation framework for health care that proposes six areas for assessment: (1) clinical, (2) human and organizational, (3) educational, (4) administrative, (5) technical, and (6) social. The framework recommends both qualitative and quantitative approaches to measuring the assessment areas.
Each framework offered several areas and concepts for measurement. Most began with an organizing construct for measurement and offered example measurement subcategories. However, the nature and specificity of the constructs and subcategories varied. Some frameworks were structured around broad ideas and concepts, while others began with a narrower focus. Due to the variation, we chose to categorize measurement concepts into 2 types: domains and subdomains. This enabled us to define domains as broad yet actionable measurement areas, within which more specific areas (subdomains) can be measured. We also found that the measure domains and subdomains included the structures, processes, and outcomes [] associated with the PC CDS interventions, that is, measures related to the PC CDS technology and how it is performing, measures related to the impact of the PC CDS technology on patient and clinician behavior, and measures reflecting the impact of the PC CDS technology on clinical and other outcomes.
Overview of the PC CDS Performance Measurement Framework
Overview
Our results begin with a high-level overview of the PC CDS performance measurement framework domains. We then provide further details about each domain and its subdomains, as well as illustrative measures. Given that PC CDS is a type of CDS, each domain and subdomain includes and builds on traditional CDS measures. To account for the unique features of PC CDS, we also specifically describe how to integrate patient-centered principles into each domain and subdomain to evaluate PC CDS performance.
The complete measurement framework is shown in 
Figure 2. The framework accounts for the impact of the PC CDS intervention occurring at multiple levels (eg, individual user [patient or clinician], population of users, organization, and health IT system). The various levels at which the domains and subdomains can be measured are positioned around the framework, indicating that the measurement perspective may vary depending on the goals and scope of the intervention, as well as who is conducting the measurement and interested in the outcome.
The NAM’s six health care quality domains [] are as follows:
- Safe—avoiding harm to patients or users resulting from the system itself or the recommendations it provides.
- Timely—reducing waits and sometimes harmful delays (eg, due to technology malfunction) for both those who receive and those who provide care.
- Effective—providing functional, accessible systems and services based on scientific knowledge to all who could benefit and refrain from providing systems and services to those not likely to benefit (avoiding underuse and misuse, respectively).
- Efficient—minimizing wasted or unnecessary effort by ensuring the system integrates smoothly into clinician and patient workflows, reduces user burden, and supports the intended user.
- Equitable—providing interventions, advice, and care that account for SDOH and do not vary in quality because of personal characteristics.
- Patient centered—providing interventions, advice, and care that are respectful of and responsive to individual patient preferences, needs, and values and ensuring patient values guide all clinical decisions.
Safe
PC CDS technology has the potential to improve patient safety by preventing adverse events. However, such technology can also cause adverse events (eg, when patient- or clinician-facing alerts are inappropriately overridden) if not designed, developed, or implemented correctly [-]. Concerns around patient safety and quality also prompted the US Food and Drug Administration to work with others in the federal government to develop a proposed strategy and recommendations for a risk-based framework for health information [,].
While not intended for legal compliance, the safety of PC CDS technology can be appropriately assessed by developers and implementers through measuring three subdomains: (1) error quantification, (2) system quality, and (3) completeness of data or other information (
Table 1).
| Subdomain definition | Application to PC CDSa | Example measures |
| Error quantification: the degree to which a specific intervention leads to patient risks and safety-related reportable adverse events, including any unanticipated consequences to the patient, clinician, or health care organization [,]. | Measures of errors should both (1) ensure patients are not experiencing negative clinical impacts (eg, adverse reaction to a drug due to an inappropriately overridden alert) [], as well as (2) pay particular attention to symptoms patients may be experiencing []. |
|
| System quality: the degree to which the information and functions provided by the system meet the user’s needs or expectations and give user satisfaction; the degree to which the system is free from deficiencies or defects [,]. | As the number of alerts sent directly to patients increases with PC CDS technology, measures of patient-centeredness should focus on ensuring the PC CDS can retrieve patient-contributed data accurately to drive alerts []. |
|
| Completeness: the degree to which the system provides all the information required by the user to make the intended decision or to perform the intended behavior [,,,,,,]. | Completeness should account for unique patient characteristics—patient-reported outcomes and patient preferences—that may impact care decisions. For example, when discussing treatment options for a patient on chemotherapy, the PC CDS technology needs to account for adverse patient-reported outcomes, such as peripheral neuropathy []. |
|
aPC CDS: patient-centered clinical decision support.
Timely
PC CDS interventions aim to reduce delays for care teams and patients in providing or receiving care. When properly implemented, PC CDS interventions have been shown to enhance clinicians’ ability to deliver timely care [] and help patients better manage their health. However, the CDS literature has documented numerous challenges to achieving this goal, including relevance of clinical recommendations, currency of clinical guidance, reliability of CDS tools, data availability, and more [].
The timeliness of PC CDS interventions can be assessed by measuring five subdomains: (1) computer processing time, (2) whether information is up to date, (3) availability of the PC CDS technology, (4) whether information is provided when the user needs it, and (5) whether the patient receives care at the right time (Table 2). Measurements in each subdomain should be considered and be adapted to patient lifestyles, new patient-generated data types, different methods of delivery (eg, apps and patient portals), and patient preferences to form an adequate assessment of a PC CDS intervention’s impact on care timeliness. Patient preferences and experiences are critical to include in addition to standard measurements (eg, for care timeliness, a patient-centered measurement approach should examine both the time to care delivery and patient preferences around timing of treatment).
| Subdomain definition | Application to PC CDSa | Example measures |
| Computer processing time: the degree to which the time required for the computer to complete the work required to gather data, run the CDSb algorithm, and generate an intervention is reduced []. | Developers might be interested in how long it takes the PC CDS engine to gather data and run the algorithm necessary to process PGHDc from remote devices and arrive at a recommendation. |
|
| Information provided is up to date: the degree to which the information presented by the system is based on up-to-date input [,,,]. | Use of PGHD, including PROsd, adds nuance to measurement of information timeliness (eg, a patient’s most recent blood glucose reading may not be the most relevant for long-term clinical decision-making). |
|
| Availability: the degree to which a system can be used when the user needs to use it [,,,]. | Patient-centered evaluations should assess whether the patient encounters any technical issues when accessing the system [,]. |
|
| Information is provided when the user is making the decision: the degree to which the necessary information presented by the PC CDS technology is available at the time it is needed [,,]. | Patient-centered evaluations should assess whether the PC CDS technology fires at the right time to support the patient in evidence-informed decision-making. Patient workflows or lifestyles are also key considerations for this subdomain. |
|
| Care timeliness: the degree to which the system can provide care in a timely manner after a need is recognized [,]. | Measures should incorporate patients’ preferences regarding when an intervention should be delivered (eg, preferences to try medication or lifestyle changes first). |
|
aPC CDS: patient-centered clinical decision support.
bCDS: clinical decision support.
cPGHD: patient-generated health data.
dPRO: patient-reported outcomes.
Effective
Effectiveness requires monitoring whether the PC CDS technology is benefiting those for whom it is intended. A landmark 2012 systematic review found that CDS interventions were generally effective in improving health care process measures, but evidence for effectiveness in areas such as improving clinical and economic outcomes was lacking []. Similarly, mixed findings on patient outcomes have been documented in more recent systematic CDS reviews in all health care settings []. Reviews of CDS effectiveness that focus on specific interventions or disease states (eg, medication-related CDS interventions) have also shown inconsistent results [].
Another challenge is the prioritization of CDS recommendations based on their relevance and potential benefit to patients. Due to the influx of CDS alerts and reminders in EHR systems that interrupt workflows, it is common for clinicians to override alerts [,], with some studies showing that a third of CDS interventions have had no effect []. Given this, it is important to evaluate which outputs have the most value and perhaps not share low-level alerts with patients to reduce over alerting and enable more focused and collaborative decision-making [].
Effectiveness of PC CDS interventions can be assessed by measuring six subdomains: (1) user satisfaction, (2) acceptability, (3) patient health outcomes, (4) usability, (5) clinician performance, and (6) transparency (Table 3).
| Subdomain definition | Application to PC CDSa | Example measures |
| User satisfaction: the degree to which a user is satisfied with their experience in using the system and with the system’s potential impact [,,,,]. | Evaluating different types of patients and clinicians as the end user ensures the PC CDS technology has the desired effects []. |
|
| Acceptability: the degree to which the user perceives the PC CDS technology is appropriate, adequate, and relevant and that both clinicians and patients see the CDSb as helpful []. | Measuring acceptability should involve attention to user subgroups who have different preferences and needs regarding guidelines and alerts. |
|
| Patient health outcomes: the degree to which change in health status is attributable to PC CDS interventions [,,,,,,]. | Measuring patient health outcomes involves examining changes in health status relevant to the disease or condition attributable to PC CDS []. As with many other subdomains, measures should also consider the outcomes patients desire in addition to traditional clinical indicators. |
|
| Usability: the degree to which the system enables users to carry out their tasks safely, effectively, efficiently, and enjoyably []. | Usability assessments need to account for the different methods and settings in which the PC CDS technology will be delivered to patients and clinicians [,,,,,,-]. Important concepts within usability include how the PC CDS fits with clinicians’ workflows and patients’ lifestyles, learnability of the PC CDS, and the degree to which the PC CDS is used [,,,,,]. |
|
| Clinician performance: the degree to which the clinician improves diagnosis or provides more complete preventive care, better disease management, more accurate drug dosing, and drug prescribing []. | Evaluators should assess whether technology fosters greater rates of patient-centered decisions among clinicians. |
|
| Transparency: the degree to which descriptions of the guideline source, updates to the guidelines (if applicable), and the artifact authors are clear; and the degree to which relevant metadata such as potential conflicts of interest and disclosures related to its development and use are available []. | Information needs to be detailed enough for general understanding of the subject and available in a clear and readable manner to promote patient trust in the system and empowerment to make better-informed decisions []. |
|
aPC CDS: patient-centered clinical decision support.
bCDS: clinical decision support.
cPRO: patient-reported outcome.
Efficient
PC CDS technology should support the intended goal of the intervention while enabling clinicians and patients to complete tasks without wasted or unnecessary effort. PC CDS technology holds promises to increase the efficiency of the clinician effort and other resources needed to care for patients. However, PC CDS technology can increase workload and burden when not designed or implemented properly []. PC CDS that is disease- or use case–specific can minimize efficiency and applicability across systems. Previous studies have shown that many CDS systems target a single chronic condition [], which is a critical gap given that 27.2% of US adults have multiple chronic conditions []. This results in CDS that is less scalable and less frequently used, particularly in primary care settings where clinicians do not have time to sort through multiple unrelated and often contradictory PC CDS alerts during a single patient encounter []. PC CDS that incorporates patient-specific data based on medical histories or patient-reported outcomes can often facilitate efficiency in prioritizing and integrating recommended medications, referrals, or procedures []. In addition, these interventions and user acceptance of them should be monitored to ensure they do not contribute to clinician burnout and are an efficient use of organizational resources. Similarly, PC CDS technology must integrate seamlessly into patients’ lifestyles without increasing the burden on patients or their caregivers.
Efficiency of PC CDS interventions can be assessed by measuring six subdomains: (1) relevance and appropriateness, (2) interoperability, (3) cost, (4) reuse and scalability, (5) service use, and (6) cognitive workload (Table 4).
| Subdomain definition | Application to PC CDSa | Example measures |
| Relevance and appropriateness: the degree to which recommendations are relevant for the clinical context [,,] and appropriate for patient care []. | Assessments should take a more holistic view and account for patient outcomes, preferences, and costs versus focusing solely on clinical relevance []. |
|
| Interoperability: the degree to which ≥2 systems or elements are able to exchange information and use the information that has been exchanged [,,,]. | Measurement should ensure patient data such as PGHDb can be seamlessly shared between patients and clinicians and between the device and EHRc using data exchange standards []. Interoperability metrics can also measure how quickly PC CDS technology can access FHIRd APIse within EHRs. |
|
| Cost: the amount of money required to design, develop, implement, and use the system and the amount paid recurrently to use the system [,,,]. | Evaluators should determine their own thresholds for whether interventions perform well based on their organizational context []. A patient-centered perspective would also examine benefits that are most important to patients and calculate costs to the patient, not only clinicians and health care delivery systems. |
|
| Reuse and scalability: the degree to which the deployment of CDSf capabilities is expanded with the centralized management of machine-executable knowledge resources, which are then leveraged across multiple care settings by CDS engines interfaced with different health information systems [,]. | Developing PC CDS artifacts often requires specialized clinical and informatics knowledge that not all health systems have, so it is important to measure how artifacts are being adopted by patients, clinicians, and organizations. |
|
| Use of services: the degree to which patients interact with the health care system, the types of care they receive, and the timing of that care [,]. | Measurements should account for the patient’s perspective on accessibility, continuity, and coordination of care []. |
|
| Cognitive workload: the degree to which the demands placed on a person by mental work are balanced with the person’s mental capacity [,,]. | PC CDS technology design should carefully consider the complexity of information and how information is presented to both clinicians and patients. |
|
aPC CDS: patient-centered clinical decision support.
bPGHD: patient-generated health data.
cEHR: electronic health record.
dFHIR: Fast Healthcare Interoperability Resources.
eAPI: application programming interface.
fCDS: clinical decision support.
Equitable
A plethora of research indicates the persistence of variation in US health care based on personal characteristics and SDOH, yet few studies have examined the impact of health IT on addressing gaps [-]. We consider two means by which PC CDS technology could make impacts in health care delivery by (1) unintentionally increasing the digital divide, and thereby who benefits from PC CDS technology []; and (2) incorporating SDOH data into decision support, thereby ensuring delivery systems are offering guidance to patients and caregivers that accounts for their specific needs. Given this, key informants emphasized the importance of designing and developing PC CDS interventions with addressing health care variation (and the resulting outcomes of this variation) in mind. For example, organizations should take into account differing levels of education and health literacy in designing technologies. In addition, care should be taken not to use biased clinical algorithms (eg, algorithms that correct for race but ultimately lead to undertreatment or overtreatment of minority populations) [,].
To measure the overall impact of PC CDS technology, we present 6 factors from the SDOH subdomains [] that can be used to assess relationships to performance in the framework’s other subdomains. These factors are (1) social context, (2) economic context, (3) organizational and personal health literacy, (4) digital health literacy, (5) physical infrastructure, and (6) health care context (Table 5). These areas of measurement apply both at the individual and organizational levels and across phases of the PC CDS life cycle (eg, differences in usability by social context is important to identify in the design and development phase as well as the use phase).
| Subdomain definition | Application to PC CDSa | Example measures |
| Social context: the degree to which factors that influence a patient’s social and community supports, including demographics and social cohesion []. | Social context factors can be used to conduct analyses for other subdomains. For example, patient satisfaction with PC CDS technology can be segmented by patient demographics, such as age, race, ethnicity, sex, education status, and disability status, to ensure technology is satisfactory regardless of demographic differences. |
|
| Economic context: the degree to which factors related to financial status, such as employment, income, and poverty, are included []. | To optimize patient use of PC CDS technology, measurements should consider the digital divide among users (ie, access to computers and the internet) [-]. |
|
| Health literacy (organizational): the degree to which organizations equitably enable individuals to find, understand, and use information and services to inform health-related decisions and actions for themselves and others []. Health literacy (personal): the degree to which individuals have the ability to find, understand, and use information and services to inform health-related decisions and actions for themselves and others []. | Evaluators should examine whether health literacy impacts performance in other subdomains to understand how their PC CDS and organization are performing in terms of accessibility. For example, a PC CDS tool that delivers information such as decision aids should be assessed for differences in uptake by literacy levels. |
|
| Digital health literacy: the ability to seek, find, understand, and appraise health information from electronic sources and apply the knowledge gained to address or solve a health problem []. | Similar to health literacy, evaluators should examine whether digital health literacy impacts performance in other subdomains, such as engagement and use of PC CDS technology. |
|
| Physical infrastructure: the degree to which factors related to the community in which the patient lives (eg, housing, transportation, and food availability) are included []. | Although accessing and collecting this information is challenging, PC CDS technology that can do so will be able to tailor better interventions to the specific needs and circumstances of patients. For example, frequent recommendations to schedule a clinic appointment may need to account for a patient’s access to transportation and make available other options such as telemedicine. |
|
| Health care context: the degree to which availability of, use of, and attitudes toward health care services impact participation in PC CDS interventions and behaviors after the intervention. | Evaluators should include whether the care is high quality, culturally and linguistically appropriate, and health literate; access to insurance; rurality; attitudes toward health care; and service use in their assessment of health care context []. |
|
aPC CDS: patient-centered clinical decision support.
Patient Centered
Patient centeredness involves facilitating active partnerships among all patients, families, patient representatives, and health professionals that are effective within the context of the larger health care delivery system []. Successful PC CDS technologies require a concerted effort to create accessible and targeted communication based on patient characteristics (eg, health literacy and demographics), including finding an appropriate balance between providing needed information and overwhelming people with interruptive alerts []. It also means extending decision support beyond merely providing information to identifying resources for patients and their caregivers []. One key informant noted the following: “CDS isn’t always thought of as an educational intervention, but when it comes to patients it’s worth considering if patients took away a somewhat lasting knowledge about the situation.”
As PC CDS technologies continue to expand, it is crucial that they are effective in helping patients and caregivers make informed choices about their care and engage, as needed, in shared decision-making discussions with their clinicians and care teams. Measures in the patient centered domain should be made from the patients’ perspectives as much as possible.
Patient centeredness of PC CDS technology can be measured with eight subdomains: (1) patient activation, (2) patient engagement, (3) patient satisfaction, (4) shared decision-making, (5) patient decision-making, (6) patient-relevant outcomes, (7) decisional quality, and (8) patient knowledge acquisition (Table 6).
| Subdomain definition | Application to PC CDSa | Example measures |
| Patient activation: the degree to which the patient (1) believes their role is important, (2) has the confidence and knowledge necessary to take action, (3) actually takes action to maintain and improve their own health, and (4) persists in their efforts even under stress [,]. | Patient activation is an indicator that patients are not only using or satisfied with technology but are being activated by it to manage their health. It is important to consider the influence of behavioral determinants, such as a patient’s knowledge, skills, or usual habits related to PC CDS technology and their health. |
|
| Patient engagement: the degree to which patients, families, their representatives, and health professionals work in active partnership at all levels across the health care system—direct care, organizational design and governance, and policy making—to improve health and health care []. | Patients and caregivers can be engaged in PC CDS evidence generation, design, development, implementation, use, and health care delivery []. Engagement can be measured on a continuum, reflecting both the level of activity and the extent of decision-making authority a patient has []. |
|
| Patient satisfaction: the patient’s experience in using the system and the system’s potential impact [,,,,]. | This is a subset of user satisfaction that focuses on patient preferences and expectations. This can be assessed even if the patient is not the main or only user of the system. |
|
| Shared decision-making: the degree to which (1) a clinician communicates to the patient personalized information about options, outcomes, probabilities, and uncertainties of available options and (2) a patient communicates values and the relative importance of benefits and harms []. | This can occur through conversation between the patient and clinician or through the PC CDS technology. Assessment relies on access to the latest evidence regarding treatment options, direction on how to weigh pros and cons, and a clinical culture that is supportive of patients []. |
|
| Patient decision-making: the degree to which a patient (1) wants to be involved with CDSb and (2) makes choices on test, treatment, or outcome options based on their values, experiences, and assessments of benefits and harms [-]. Patient decision-making can occur with or without clinician involvement (ie, shared decision-making). | Measures should be selected based on what matters to the patient, how much it matters, and perceptions of risks and benefits for their lives []. |
|
| Patient-relevant outcomes: the degree to which patients can choose the outcomes that match their desires, beliefs, goals, and circumstances [,]. | This is particularly salient for outcomes related to quality of life []. |
|
| Decisional quality: the degree to which patients feel satisfied during and after the decision-making process [,]. | Measurements of decisional quality should be sensitive to the preferences of patients, that is, account for the relative importance a patient attaches to various outcomes []. |
|
| Patient knowledge acquisition: the degree to which the patient gains lasting knowledge about their medical situation due to the PC CDS tool or intervention []. | This subdomain should measure the extent to which patients gain knowledge about their medical situations by taking part in the decision support activities []. |
|
aPC CDS: patient-centered clinical decision support.
bCDS: clinical decision support.
Application of the PC CDS Performance Measurement Framework
Overview
Implementers and evaluators must make a concerted effort to include patient-centered measures in their assessments, ensuring that these measures address a range of patient-focused factors, such as usability, patient digital and health literacy levels, and integration into patients’ daily lives. To contribute to the field given its nascency, we provide 2 illustrative user scenarios for applying patient-centered measures across different phases of the PC CDS life cycle and levels of measurement (Table 7).
To demonstrate the versatility of the framework, the use cases were drawn to highlight different patient populations (ie, postpartum patients and patients aged >75 years), different potential durations of symptoms (ie, acute and long term), different condition acuities, and different types of PC CDS technology (eg, patient facing and clinician facing). Rather than advocating for the universal measurement of all subdomains, the framework supports a targeted evaluation approach, with implementers and evaluators selecting measures that align with the specific goals and context of their project. The use cases provide only a subset of potential measures from each domain, illustrating how different selections may be appropriate depending on the implementation scope and needs.
| Use case scenario | Relevant domain and subdomains to measure | Measurement level |
| Design and development of an app for managing HDPa for postpartum patients |
|
|
| Use of a stroke risk assessment tool (eg, CHA2DS2-VASc []) to help clinicians and patients aged >75 years with a-fibc decide whether to initiate anticoagulation therapy |
|
|
aHDP: hypertensive disorders of pregnancy.
bEHR: electronic health record.
ca-fib: atrial fibrillation.
Use Case 1
Hypertensive disorders of pregnancy (HDP) can pose serious health complications and increase the risk of maternal mortality []. Postpartum patients with HDP will need to monitor their blood pressure and hypertensive symptoms. To assess the design and development of a patient app to manage HDP, an evaluator could assess the usability of the app through user-centered design principles, ensuring the app is easy and intuitive to use. In addition, patient activation could be assessed in the design phase to explore if the app provides patients with the confidence and skills to manage their hypertension or if additional functionalities are needed, such as an education component. Other important subdomains to measure the system quality of the app include care timeliness, cognitive workload of interpreting blood pressure data, and health literacy considerations in the design of the app.
Use Case 2
Patients aged >75 years with atrial fibrillation need to weigh the risks and benefits of taking oral anticoagulants. Anticoagulants can prevent blood clots that cause strokes but can also increase the risk of morbidity and mortality from gastrointestinal and intracerebral bleeding, particularly for older adults at greater risk of falls []. The CHA2DS2-VASc tool provides estimates of the risk that patients with atrial fibrillation will have a stroke in the absence of treatment [], and additional CDS tools provide estimates of a patient’s risk of stroke and bleeding on various treatments. These estimates can help patients, and their clinicians decide whether to initiate anticoagulation therapy. To assess the use of this tool after implementation, an evaluator could consider measurements of shared decision-making to assess the patients’ engagement with their clinician when deciding to take anticoagulants and patient health outcomes to measure the impact of the decision on the patient’s lifestyle, physical activity levels, and other outcomes important to the patient. Other important subdomains to consider would be unintended consequences in terms of adverse events, whether information is provided when the user makes the decision, the relevance and appropriateness of the recommendation provided, and the social context of the population using the tool.
Discussion
Principal Findings
Overview
In this paper, we describe a new measurement framework that can be used by researchers, operational leaders, and patients to understand the performance and impact of PC CDS technology. The framework incorporates constructs from previously developed health IT and CDS evaluation frameworks and organizes them into measure domains and subdomains. We highlight 5 key considerations for this framework and their relevance.
Covers the Entire PC CDS Life Cycle
Domains and subdomains described in this paper address the entire PC CDS life cycle—from the generation of knowledge (which includes creating evidence-based guidelines based on patient-centered outcomes research) to the design, development, and implementation of PC CDS [], and finally to its use (ie, health care delivery). Some outcome-focused subdomains are more relevant to later stages of the life cycle; for example, patient health outcomes and health care use are most relevant during the use phase, after the technology has been implemented and these outcomes can be observed. On the other hand, subdomains related to the performance of the technology (eg, error quantification and system quality) are relevant across phases but are ideally addressed to the extent possible in the design and development phases. Thus, this framework provides the foundation for a more holistic evaluation of PC CDS technology that covers structure, process, and outcome measures.
Direct Focus on the Patient
The framework’s domains and subdomains center on patient aims and specifically include patient centeredness as key components of successful PC CDS technology, representing the first framework to account for the use of PC CDS by patients and clinicians. In addition, each subdomain is a patient-centered concept (eg, patient engagement) or has potential measurements that incorporate patient-centered principles (eg, consideration of patient preferences for when decision support is delivered). By doing so, the framework can facilitate the evaluation of the impact of PC CDS technology on patient involvement in their care decisions; patient-clinician communication through shared decision-making; treatment personalization and holistic care approaches; and, ultimately, patient health outcomes []. So far, the emergence of PC CDS technology has revealed a gap in the frameworks, models, and methods commonly used for CDS evaluation, particularly in understanding the effect of including patient data in PC CDS technology and the effectiveness of PC CDS technology []. For example, patient engagement is critical to PC CDS interventions, yet few studies have focused on the factors that improve and sustain patient engagement or what types of engagement lead to improved clinical or other outcomes []. As artificial intelligence–supported decision support grows, patients’ use of these technologies will likely increase []. The framework provides a useful starting point for evaluating and measuring patient use of these tools, what is needed to support broader adoption and use, and the eventual impact of incorporating PC CDS in the patient care process.
Covers Measurement at Different Levels
Many frameworks also describe different aggregation levels at which performance metrics can, or should, be most appropriately assessed, depending on the goals of the PC CDS intervention, the nature of the domain and subdomain, and the differing perspectives of user groups (eg, clinicians vs patients). The measurement framework introduced here accounts for the impact of the PC CDS technology occurring at the level of the individual user (patient or clinician), user population, organization, and health IT system level. For example, population-level impacts may be important to evaluators trying to understand the effectiveness of PC CDS interventions on the overall health of the community, and individual-level impacts may be important to evaluators attempting to correlate individual-level use and engagement measures with specific clinical outcome measures to identify the optimal method of PC CDS intervention delivery.
Encompasses 6 Independent but Related Domains
Our performance measurement framework provides a balanced methodology to assess the extent of use and the quality of PC CDS interventions. Consequently, it provides 6 relatively independent domains in which measurements can be made and leaves the decision to the user on how to synthesize or prioritize the diverse measurements. An evaluator of an intervention does not need to assess every subdomain in the framework. Rather, they should work to assess ≥1 components of each domain most relevant to the evaluator’s particular PC CDS intervention or research focus. In addition, while we have mapped the measurements and subdomains according to their most common applications, the context and goals of an evaluation may require applications to different or multiple subdomains and domains, respectively. Subdomains and measures may also be closely related and influence each other (eg, patient activation and patient engagement). Given that PC CDS technology implementation occurs in many different contexts that may require departure from a standard framework, our intent is for the framework to serve as a foundation for users as they design their own evaluation plans.
Requires Additional Research and Development
We propose this framework as a starting point for PC CDS measurement, but further testing for usability and completeness of the framework is needed to inform refinements as appropriate. We recognize that there has been more work in some of the NAM quality domains than in others. Therefore, some domains are better understood and more fully characterized than others as they pertain to PC CDS technology, including the equitable and patient centered domains. Similarly, some phases of the PC CDS life cycle have been more thoroughly explored than others (eg, the CDS phase has more work done on measurement than the knowledge generation phase, which is a newer area for PC CDS measurement). More work is also needed to define and develop measures for our 6 domains (safe, timely, effective, efficient, equitable, and patient centered), as well as newer domains that have been added to NAM’s framework to support a learning health system []. This work will need to consider the emerging use of artificial intelligence in the generation of knowledge and patient-specific advice.
Conclusions
Taken as a whole, the new, unified PC CDS performance measurement framework we present here provides a firm foundation upon which researchers can build as the field of PC CDS matures. Clearly, more work must be done to better understand what is working, whether the specific PC CDS technology is being used as anticipated, and whether the intended outcomes are being achieved. In addition, we anticipate further development in the prioritization of measures for evaluations, as well as in the identification of relevant patient-centered measures, as PC CDS technologies become more widely available. We expect much progress to be made in PC CDS and its evaluation over the next decade, as our understanding of the needs of patients and clinicians evolves and our ability to collect and interpret more data improves.
Acknowledgments
The authors would like to acknowledge the following members of the Clinical Decision Support Innovation Collaborative’s Innovation Center Planning Committee: Angela Dobes, MPH; Gil Kuperman, MD, PhD; David Lobach, MD, PhD, MS, FACMI; J Marc Overhage, MD, PhD; Josh Mandel, MD; Ted Melnick, MD, MHS; and Jonathan Teich, MD, PhD. This work is based on research conducted by NORC at the University of Chicago under contract to the Agency for Healthcare Research and Quality, Rockville, Maryland (75Q80120D00018/75Q80121F32003).
Data Availability
The datasets generated and analyzed during this study are not publicly available due to possible reidentification but are available from the corresponding author on reasonable request.
Authors' Contributions
PD was responsible for conceptualization (lead), supervision (lead), investigation (equal), formal analysis (equal), writing—original draft (lead), and writing—review and editing (equal). CZ was responsible for investigation (equal), formal analysis (equal), and writing—original draft (support). NG was responsible for investigation (support), formal analysis (support), and writing—original draft (support). JS was responsible for writing—review and editing (equal). EL was responsible for writing—review and editing (equal). DFS was responsible for conceptualization (equal), investigation (equal), formal analysis (equal), and writing—review and editing (equal).
Conflicts of Interest
None declared.
Multimedia Appendix 1
Information provided to key informants before participating in the interviews.
DOCX File , 50 KBReferences
- McGlynn EA, Asch SM, Adams JL, Keesey J, Hicks J, DeCristofaro AH, et al. The first national report card on quality of health care in America. RAND Corporation. Apr 11, 2006. URL: https://www.rand.org/pubs/research_briefs/RB9053-2.html [accessed 2025-04-14]
- McGlynn EA. Measuring and improving quality in the US: where are we today? J Am Board Fam Med. Sep 14, 2020;33(Supplement):S28-S35. [CrossRef]
- Psek WA, Stametz RA, Bailey-Davis LD, Davis D, Darer J, Faucett WA, et al. Operationalizing the learning health care system in an integrated delivery system. EGEMS (Wash DC). Mar 10, 2015;3(1):1122. [FREE Full text] [CrossRef] [Medline]
- Sittig DF, Wright A, Osheroff JA, Middleton B, Teich JM, Ash JS, et al. Grand challenges in clinical decision support. J Biomed Inform. Apr 2008;41(2):387-392. [FREE Full text] [CrossRef] [Medline]
- Blumenfeld BH. The patient-centered clinical decision support learning network: improving the dissemination of PCOR in patient-centered clinical decision support. Agency for Healthcare Research and Quality. 2020. URL: https://digital.ahrq.gov/sites/default/files/docs/citation/u18hs024849-blumenfeld-final-report-2020.pdf [accessed 2023-11-16]
- Dullabh P, Sandberg SF, Heaney-Huls K, Hovey LS, Lobach DF, Boxwala A, et al. Challenges and opportunities for advancing patient-centered clinical decision support: findings from a horizon scan. J Am Med Inform Assoc. Jun 14, 2022;29(7):1233-1243. [FREE Full text] [CrossRef] [Medline]
- Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A, Detmer DE. A roadmap for national action on clinical decision support. J Am Med Inform Assoc. 2007;14(2):141-145. [FREE Full text] [CrossRef] [Medline]
- National Academy of Medicine. Optimizing Strategies for Clinical Decision Support: Summary of a Meeting Series. Washington, DC. The National Academies Press; 2017.
- Middleton B, Sittig DF, Wright A. Clinical decision support: a 25 year retrospective and a 25 year vision. Yearb Med Inform. Mar 06, 2018;25(S 01):S103-S116. [CrossRef]
- Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med. 2020;3:17. [FREE Full text] [CrossRef] [Medline]
- Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med. Nov 24, 2003;163(21):2625-2631. [FREE Full text] [CrossRef] [Medline]
- Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care. Oct 2010;19 Suppl 3(Suppl 3):i68-i74. [FREE Full text] [CrossRef] [Medline]
- Kawamoto K, McDonald CJ. Designing, conducting, and reporting clinical decision support studies: recommendations and call to action. Ann Intern Med. Jun 02, 2020;172(11 Suppl):S101-S109. [FREE Full text] [CrossRef] [Medline]
- Institute of Medicine, Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC. National Academies Press; 2001.
- About CDSiC. Agency for Healthcare Research and Quality. URL: https://cdsic.ahrq.gov/cdsic/cdsic-about [accessed 2024-02-28]
- Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. Aug 11, 2009;9:59. [FREE Full text] [CrossRef] [Medline]
- Yusof MM, Kuljis J, Papazafeiropoulou A, Stergioulas LK. An evaluation framework for health information systems: human, organization and technology-fit factors (HOT-fit). Int J Med Inform. Jun 2008;77(6):386-398. [CrossRef] [Medline]
- Camacho J, Zanoletti-Mannello M, Landis-Lewis Z, Kane-Gill SL, Boyce RD. A conceptual framework to study the implementation of clinical decision support systems (BEAR): literature review and concept mapping. J Med Internet Res. Aug 06, 2020;22(8):e18388. [FREE Full text] [CrossRef] [Medline]
- Kukhareva PV, Weir C, Del Fiol G, Aarons GA, Taft TY, Schlechter CR, et al. Evaluation in Life Cycle of Information Technology (ELICIT) framework: supporting the innovation life cycle from business case assessment to summative evaluation. J Biomed Inform. Mar 2022;127:104014. [FREE Full text] [CrossRef] [Medline]
- Sockolow PS, Crawford PR, Lehmann HP. Health services research evaluation principles. Broadening a general framework for evaluating health information technology. Methods Inf Med. Jan 19, 2012;51(2):122-130. [CrossRef] [Medline]
- Ji M, Yu G, Xi H, Xu T, Qin Y. Measures of success of computerized clinical decision support systems: an overview of systematic reviews. Health Policy Technol. Mar 2021;10(1):196-208. [CrossRef]
- Delone WH, McLean ER. The DeLone and McLean model of information systems success: a ten-year update. J Manag Inf Syst. Dec 23, 2014;19(4):9-30. [CrossRef]
- Lau FY, Kuziemsky C. Handbook of EHealth Evaluation: An Evidence-Based Approach. Victoria, BC. University of Victoria; 2016.
- Eslami Andargoli A, Scheepers H, Rajendran D, Sohal A. Health information systems evaluation frameworks: a systematic review. Int J Med Inform. Jan 2017;97:195-209. [CrossRef] [Medline]
- Shaw NT. 'CHEATS': a generic information communication technology (ICT) evaluation framework. Comput Biol Med. May 2002;32(3):209-220. [CrossRef] [Medline]
- Types of health care quality measures. Agency for Healthcare Research and Quality. URL: https://www.ahrq.gov/talkingquality/measures/types.html [accessed 2025-03-26]
- Wright A, Ai A, Ash J, Wiesen JF, Hickman TT, Aaron S, et al. Clinical decision support alert malfunctions: analysis and empirically derived taxonomy. J Am Med Inform Assoc. May 01, 2018;25(5):496-506. [FREE Full text] [CrossRef] [Medline]
- Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA. Mar 09, 2005;293(10):1197-1203. [CrossRef] [Medline]
- Berger RG. Computerized physician order entry: helpful or harmful? J Am Med Inform Assoc. Nov 21, 2003;11(2):100-103. [CrossRef]
- Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC, et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics. Dec 2005;116(6):1506-1512. [CrossRef] [Medline]
- Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc. 2006;13(5):547-556. [FREE Full text] [CrossRef] [Medline]
- Beeler P, Bates DW, Hug BL. Clinical decision support systems. Swiss Med Wkly. Dec 23, 2014;144:w14073. [FREE Full text] [CrossRef] [Medline]
- Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc. Jul 01, 2007;14(4):415-423. [CrossRef]
- Clinical decision support software: guidance for industry and Food and Drug Administration staff. U.S. Department of Health and Human Services Food and Drug Administration. 2022. URL: https://www.fda.gov/media/109618/download [accessed 2024-08-28]
- FDASIA health IT report: proposed strategy and recommendations for a risk-based framework. The Office of the National Coordinator for Health Information Technology. Apr 2014. URL: https://www.healthit.gov/sites/default/files/fdasia_healthitreport_final.pdf [accessed 2024-08-28]
- Yusof MM, Paul RJ, Stergioulas LK. Towards a framework for health information systems evaluation. In: Proceedings of the 39th Annual Hawaii International Conference on System Sciences. 2006. Presented at: HICSS'06; January 4-7, 2006; Kauai, HI. [CrossRef]
- Wong A, Amato MG, Seger DL, Rehr C, Wright A, Slight SP, et al. Prospective evaluation of medication-related clinical decision support over-rides in the intensive care unit. BMJ Qual Saf. Sep 09, 2018;27(9):718-724. [FREE Full text] [CrossRef] [Medline]
- Kilsdonk E, Peute LW, Jaspers MW. Factors influencing implementation success of guideline-based clinical decision support systems: a systematic review and gaps analysis. Int J Med Inform. Feb 2017;98:56-64. [CrossRef] [Medline]
- Bernhard G, Mahler C, Seidling HM, Stützle M, Ose D, Baudendistel I, et al. Developing a shared patient-centered, web-based medication platform for type 2 diabetes patients and their health care providers: qualitative study on user requirements. J Med Internet Res. Mar 27, 2018;20(3):e105. [FREE Full text] [CrossRef] [Medline]
- Garcia-Smith D, Effken JA. Development and initial evaluation of the Clinical Information Systems Success Model (CISSM). Int J Med Inform. Jun 2013;82(6):539-552. [CrossRef] [Medline]
- Hertz DL, Childs DS, Park SB, Faithfull S, Ke Y, Ali NT, et al. Patient-centric decision framework for treatment alterations in patients with Chemotherapy-Induced Peripheral Neuropathy (CIPN). Cancer Treat Rev. Sep 2021;99:102241. [CrossRef] [Medline]
- Kirby AM, Kruger B, Jain R, OʼHair DP, Granger BB. Using clinical decision support to improve referral rates in severe symptomatic aortic stenosis: a quality improvement initiative. Comput Inform Nurs. Nov 2018;36(11):525-529. [CrossRef] [Medline]
- Belard A, Buchman T, Forsberg J, Potter BK, Dente CJ, Kirk A, et al. Precision diagnosis: a view of the clinical decision support systems (CDSS) landscape through the lens of critical care. J Clin Monit Comput. Apr 22, 2017;31(2):261-271. [CrossRef] [Medline]
- Paterno MD, Goldberg HS, Simonaitis L, Dixon BE, Wright A, Rocha BH, et al. Using a service oriented architecture approach to clinical decision support: performance results from two CDS Consortium demonstrations. AMIA Annu Symp Proc. 2012;2012:690-698. [FREE Full text] [Medline]
- Sittig DF, Singh H. A new socio-technical model for studying health information technology in complex adaptive healthcare systems. In: Patel VL, Kannampallil TG, Kaufman DR, editors. Cognitive Informatics for Biomedicine. Cham, Switzerland. Springer; 2015.
- Vis C, Bührmann L, Riper H, Ossebaard HC. Health technology assessment frameworks for eHealth: a systematic review. Int J Technol Assess Health Care. Apr 16, 2020;36(3):204-216. [CrossRef]
- Van Dort BA, Zheng WY, Sundar V, Baysari MT. Optimizing clinical decision support alerts in electronic medical records: a systematic review of reported strategies adopted by hospitals. J Am Med Inform Assoc. Jan 15, 2021;28(1):177-183. [FREE Full text] [CrossRef] [Medline]
- Richesson RL, Staes CJ, Douthit BJ, Thoureen T, Hatch DJ, Kawamoto K, et al. Measuring implementation feasibility of clinical decision support alerts for clinical practice recommendations. J Am Med Inform Assoc. Apr 01, 2020;27(4):514-521. [FREE Full text] [CrossRef] [Medline]
- Muench F, Baumel A. More than a text message: dismantling digital triggers to curate behavior change in patient-centered health interventions. J Med Internet Res. May 26, 2017;19(5):e147. [FREE Full text] [CrossRef] [Medline]
- Elements of access to health care: timeliness. Agency for Healthcare Research and Quality. URL: https://www.ahrq.gov/research/findings/nhqrdr/chartbooks/access/elements3.html [accessed 2023-04-11]
- Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med. Jul 03, 2012;157(1):29-43. [FREE Full text] [CrossRef] [Medline]
- Brenner SK, Kaushal R, Grinspan Z, Joyce C, Kim I, Allard RJ, et al. Effects of health information technology on patient outcomes: a systematic review. J Am Med Inform Assoc. Sep 2016;23(5):1016-1036. [FREE Full text] [CrossRef] [Medline]
- Taheri Moghadam S, Sadoughi F, Velayati F, Ehsanzadeh SJ, Poursharif S. The effects of clinical decision support system for prescribing medication on patient outcomes and physician practice performance: a systematic review and meta-analysis. BMC Med Inform Decis Mak. Mar 10, 2021;21(1):98. [FREE Full text] [CrossRef] [Medline]
- Wong A, Rehr C, Seger DL, Amato MG, Beeler PE, Slight SP, et al. Evaluation of harm associated with high dose-range clinical decision support overrides in the intensive care unit. Drug Saf. Apr 2019;42(4):573-579. [CrossRef] [Medline]
- Olakotan OO, Mohd Yusof M. The appropriateness of clinical decision support systems alerts in supporting clinical workflows: a systematic review. Health Informatics J. Apr 15, 2021;27(2):14604582211007536. [FREE Full text] [CrossRef] [Medline]
- Weingart SN, Simchowitz B, Padolsky H, Isaac T, Seger AC, Massagli M, et al. An empirical model to estimate the potential impact of medication safety alerts on patient safety, health care utilization, and cost in ambulatory care. Arch Intern Med. Sep 14, 2009;169(16):1465-1473. [CrossRef] [Medline]
- Herasevich V, Pickering BW. Health Information Technology Evaluation Handbook: From Meaningful Use to Meaningful Outcomes. Milton Park, UK. Taylor & Francis; 2017.
- Van de Velde S, Kunnamo I, Roshanov P, Kortteisto T, Aertgeerts B, Vandvik PO, et al. The GUIDES checklist: development of a tool to improve the successful use of guideline-based computerised clinical decision support. Implement Sci. Jun 25, 2018;13(1):86. [FREE Full text] [CrossRef] [Medline]
- Jafari SM, Ali NA, Sambasivan M, Said MF. A respecification and extension of DeLone and McLean model of IS success in the citizen-centric e-governance. In: Proceedings of the IEEE International Conference on Information Reuse & Integration. 2011. Presented at: IRI 2011; August 3-5, 2011; Las Vegas, NV. [CrossRef]
- Heathfield HA, Peel V, Hudson P, Kay S, Mackay L, Marley T, et al. Evaluating large scale health information systems: from practice towards theory. Proc AMIA Annu Fall Symp. 1997:116-120. [FREE Full text] [Medline]
- Forsythe DE, Buchanan BG. Broadening our approach to evaluating medical information systems. Proc Annu Symp Comput Appl Med Care. 1991:8-12. [FREE Full text] [Medline]
- Kowatsch T, Otto L, Harperink S, Cotti A, Schlieter H. A design and evaluation framework for digital health interventions. it Inf Technol. 2019;61(5-6):253-263. [CrossRef]
- Kaplan B. Evaluating informatics applications--some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inform. Nov 2001;64(1):39-56. [CrossRef] [Medline]
- Greenes RA, Bates DW, Kawamoto K, Middleton B, Osheroff J, Shahar Y. Clinical decision support models and frameworks: seeking to address research issues underlying implementation successes and failures. J Biomed Inform. Feb 2018;78:134-143. [FREE Full text] [CrossRef] [Medline]
- Kannry J, McCullagh L, Kushniruk A, Mann D, Edonyabo D, McGinn T. A framework for usable and effective clinical decision support: experience from the iCPR randomized clinical trial. EGEMS (Wash DC). 2015;3(2):1150. [FREE Full text] [CrossRef] [Medline]
- Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. Mar 09, 2005;293(10):1223-1238. [CrossRef] [Medline]
- Richardson JE, Middleton B, Platt JE, Blumenfeld BH. Building and maintaining trust in clinical decision support: recommendations from the Patient-Centered CDS Learning Network. Learn Health Syst. Apr 11, 2020;4(2):e10208. [FREE Full text] [CrossRef] [Medline]
- Ploug T, Holm S. Right to contest AI diagnostics: defining transparency and explainability requirements from a patient’s perspective. In: Lidströmer N, Ashrafian H, editors. Artificial Intelligence in Medicine. Cham, Switzerland. Springer; 2022.
- Boersma P, Black LI, Ward BW. Prevalence of multiple chronic conditions among US adults, 2018. Prev Chronic Dis. Sep 17, 2020;17:E106. [FREE Full text] [CrossRef] [Medline]
- Sperl-Hillen JM, Rossom RC, Kharbanda EO, Gold R, Geissal ED, Elliott TE, et al. Priorities wizard: multisite web-based primary care clinical decision support improved chronic care outcomes with high use rates and high clinician satisfaction rates. EGEMS (Wash DC). Apr 03, 2019;7(1):9. [FREE Full text] [CrossRef] [Medline]
- Coulter I, Herman P, Ryan G, Hilton L, Hays RD, Members of CERC Team. The challenge of determining appropriate care in the era of patient-centered care and rising health care costs. J Health Serv Res Policy. Jul 24, 2019;24(3):201-206. [FREE Full text] [CrossRef] [Medline]
- The path to interoperability. The Office of the National Coordinator for Health Information Technology. 2013. URL: https://www.healthit.gov/sites/default/files/factsheets/onc_interoperabilityfactsheet.pdf [accessed 2023-10-05]
- Dullabh P, Heaney-Huls K, Lobach DF, Hovey LS, Sandberg SF, Desai PJ, et al. The technical landscape for patient-centered CDS: progress, gaps, and challenges. J Am Med Inform Assoc. May 11, 2022;29(6):1101-1105. [FREE Full text] [CrossRef] [Medline]
- Jacob V, Thota AB, Chattopadhyay SK, Njie GJ, Proia KK, Hopkins DP, et al. Cost and economic benefit of clinical decision support systems for cardiovascular disease prevention: a community guide systematic review. J Am Med Inform Assoc. May 01, 2017;24(3):669-676. [FREE Full text] [CrossRef] [Medline]
- Boxwala AA, Peleg M, Tu S, Ogunyemi O, Zeng QT, Wang D, et al. GLIF3: a representation format for sharable computer-interpretable clinical practice guidelines. J Biomed Inform. Jun 2004;37(3):147-161. [FREE Full text] [CrossRef] [Medline]
- Meyers EA. Health care services utilization. In: Boslaugh S, editor. Encyclopedia of Epidemiology. Thousand Oaks, CA. SAGE Publications; 2008:470-471.
- Algaze CA, Wood M, Pageler NM, Sharek PJ, Longhurst CA, Shin AY. Use of a checklist and clinical decision support tool reduces laboratory use and improves cost. Pediatrics. Jan 2016;137(1). [CrossRef] [Medline]
- Haggerty JL, Pineault R, Beaulieu MD, Brunelle Y, Gauthier J, Goulet F, et al. Practice features associated with patient-reported accessibility, continuity, and coordination of primary health care. Ann Fam Med. Mar 01, 2008;6(2):116-123. [FREE Full text] [CrossRef] [Medline]
- Singh GK, Daus GP, Allender M, Ramey CT, Martin EK, Perry C, et al. Social determinants of health in the United States: addressing major health inequality trends for the nation, 1935-2016. Int J MCH AIDS. Jan 15, 2017;6(2):139-164. [FREE Full text] [CrossRef] [Medline]
- Lagu T, Haywood C, Reimold K, DeJong C, Walker Sterling R, Iezzoni LI. 'I am not the doctor for you': physicians' attitudes about caring for people with disabilities. Health Aff (Millwood). Oct 01, 2022;41(10):1387-1395. [FREE Full text] [CrossRef] [Medline]
- Pérez-Stable EJ, Jean-Francois B, Aklin CF. Leveraging advances in technology to promote health equity. Med Care. Jun 2019;57 Suppl 6 Suppl 2:S101-S103. [CrossRef] [Medline]
- Evidence-consensus-based digital healthcare equity framework: a practical guide for implementation. Agency for Healthcare Research and Quality. URL: https://digital.ahrq.gov/sites/default/files/docs/citation/health-equity-practical-guide.pdf [accessed 2025-04-14]
- Vyas DA, Eisenstein LG, Jones DS. Hidden in plain sight — reconsidering the use of race correction in clinical algorithms. N Engl J Med. Aug 27, 2020;383(9):874-882. [CrossRef]
- Chin MH, Afsar-Manesh N, Bierman AS, Chang C, Colón-Rodríguez CJ, Dullabh P, et al. Guiding principles to address the impact of algorithm bias on racial and ethnic disparities in health and health care. JAMA Netw Open. Dec 01, 2023;6(12):e2345050. [FREE Full text] [CrossRef] [Medline]
- About SDOH in healthcare. Agency for Healthcare Research and Quality. URL: https://www.ahrq.gov/sdoh/about.html [accessed 2022-12-19]
- Hernandez-Ramos R, Aguilera A, Garcia F, Miramontes-Gomez J, Pathak LE, Figueroa CA, et al. Conducting internet-based visits for onboarding populations with limited digital literacy to an mHealth intervention: development of a patient-centered approach. JMIR Form Res. Apr 29, 2021;5(4):e25299. [FREE Full text] [CrossRef] [Medline]
- Kim E, Mayani A, Modi S, Kim Y, Soh CB. Evaluation of patient-centered electronic health record to overcome digital divide. In: Proceedings of the IEEE Engineering in Medicine and Biology 27th Annual Conference. 2006. Presented at: IEMBS 2006; January 17-18, 2006; Shanghai, China. [CrossRef]
- Gleason K, Suen JJ. Going beyond affordability for digital equity: closing the "digital divide" through outreach and training programs for older adults. J Am Geriatr Soc. Jan 22, 2022;70(1):75-77. [FREE Full text] [CrossRef] [Medline]
- Health literacy in healthy people 2030. Office of Disease Prevention and Health Promotion, Office of the Assistant Secretary for Health, Office of the Secretary, U.S. Department of Health and Human Services. URL: https://odphp.health.gov/healthypeople/priority-areas/health-literacy-healthy-people-2030 [accessed 2023-04-11]
- Norman CD, Skinner HA. eHealth literacy: essential skills for consumer health in a networked world. J Med Internet Res. Jun 16, 2006;8(2):e9. [FREE Full text] [CrossRef] [Medline]
- Grossman Liu L, Ancker JS, Masterson Creber RM. Improving patient engagement through patient decision support. Am J Prev Med. Mar 2021;60(3):438-441. [FREE Full text] [CrossRef] [Medline]
- Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res. Aug 2004;39(4 Pt 1):1005-1026. [FREE Full text] [CrossRef] [Medline]
- James J. Patient engagement. Health Affairs. Feb 14, 2013. URL: https://www.healthaffairs.org/content/briefs/patient-engagement [accessed 2025-04-22]
- Carman KL, Dardess P, Maurer M, Sofaer S, Adams K, Bechtel C, et al. Patient and family engagement: a framework for understanding the elements and developing interventions and policies. Health Aff (Millwood). Feb 2013;32(2):223-231. [FREE Full text] [CrossRef] [Medline]
- Elwyn G, Laitner S, Coulter A, Walker E, Watson P, Thomson R. Implementing shared decision making in the NHS. BMJ. Oct 14, 2010;341(oct14 2):c5146. [FREE Full text] [CrossRef] [Medline]
- Stacey D, Samant R, Bennett C. Decision making in oncology: a review of patient decision aids to support patient participation. CA Cancer J Clin. 2008;58(5):293-304. [FREE Full text] [CrossRef] [Medline]
- Zeliadt SB, Ramsey SD, Penson DF, Hall IJ, Ekwueme DU, Stroud L, et al. Why do men choose one treatment over another?: a review of patient decision making for localized prostate cancer. Cancer. May 01, 2006;106(9):1865-1874. [FREE Full text] [CrossRef] [Medline]
- Pierce PF, Hicks FD. Patient decision-making behavior: an emerging paradigm for nursing science. Nurs Res. 2001;50(5):267-274. [CrossRef] [Medline]
- Measuring patient experiences: distinguishing between patient-reported outcomes and patient preferences. National Health Council. 2019. URL: https://nationalhealthcouncil.org/webinars/coa-series-measuring-patient-experiences-distinguishing-between-patient-reported-outcomes-and-patient-preferences/ [accessed 2025-04-14]
- Kersting C, Kneer M, Barzel A. Patient-relevant outcomes: what are we talking about? A scoping review to improve conceptual clarity. BMC Health Serv Res. Jun 29, 2020;20(1):596. [FREE Full text] [CrossRef] [Medline]
- van der Elst K, Meyfroidt S, De Cock D, De Groef A, Binnard E, Moons P, et al. Unraveling patient-preferred health and treatment outcomes in early rheumatoid arthritis: a longitudinal qualitative study. Arthritis Care Res (Hoboken). Sep 29, 2016;68(9):1278-1287. [FREE Full text] [CrossRef] [Medline]
- Kaltoft M, Cunich M, Salkeld G, Dowie J. Assessing decision quality in patient-centred care requires a preference-sensitive measure. J Health Serv Res Policy. Apr 12, 2014;19(2):110-117. [FREE Full text] [CrossRef] [Medline]
- Sepucha KR, Fowler FJJ, Mulley AGJ. Policy support for patient-centered care: the need for measurable improvements in decision quality. Health Aff (Millwood). 2004;Suppl Variation:VAR54-VAR62. [CrossRef] [Medline]
- Sawka AM, Straus S, Rodin G, Tsang RW, Brierley JD, Rotstein L, et al. Exploring the relationship between patients' information preference style and knowledge acquisition process in a computerized patient decision aid randomized controlled trial. BMC Med Inform Decis Mak. Jun 19, 2015;15:48. [FREE Full text] [CrossRef] [Medline]
- Sawka AM, Straus S, Rotstein L, Brierley JD, Tsang RW, Asa S, et al. Randomized controlled trial of a computerized decision aid on adjuvant radioactive iodine treatment for patients with early-stage papillary thyroid cancer. J Clin Oncol. Aug 10, 2012;30(23):2906-2911. [CrossRef]
- CHA2DS2-VASc score for atrial fibrillation stroke risk. MDCalc. URL: https://www.mdcalc.com/calc/801/cha2ds2-vasc-score-atrial-fibrillation-stroke-risk [accessed 2023-09-19]
- Lo JO, Mission JF, Caughey AB. Hypertensive disease of pregnancy and maternal mortality. Curr Opin Obstet Gynecol. Apr 2013;25(2):124-132. [CrossRef] [Medline]
- Roth AR, Lazris A, Haskell H, James J. Anticoagulation in older adults. Am Fam Physician. 2020;101(12):748-750. [FREE Full text]
- Sirajuddin AM, Osheroff JA, Sittig DF, Chuo J, Velasco F, Collins DA. Implementation pearls from a new guidebook on improving medication use and outcomes with clinical decision support. Effective CDS is essential for addressing healthcare performance improvement imperatives. J Healthc Inf Manag. 2009;23(4):38-45. [FREE Full text] [Medline]
- Dullabh P, Leaphart D, Dhopeshwarkar R, Heaney-Huls K, Desai P. Patient-centered clinical decision support-where are we and where to next? Stud Health Technol Inform. Jan 25, 2024;310:444-448. [CrossRef] [Medline]
- Hassan M, Kushniruk A, Borycki E. Barriers to and facilitators of artificial intelligence adoption in health care: scoping review. JMIR Hum Factors. Aug 29, 2024;11:e48633. [FREE Full text] [CrossRef] [Medline]
- Learning health system shared commitments. National Academy of Medicine. 2023. URL: https://nam.edu/programs/value-science-driven-health-care/lhs-core-principles/ [accessed 2024-04-17]
Abbreviations
| CDS: clinical decision support |
| EHR: electronic health record |
| HDP: hypertensive disorders of pregnancy |
| NAM: National Academy of Medicine |
| PC CDS: patient-centered clinical decision support |
| SDOH: social determinants of health |
Edited by J Sarvestan; submitted 18.11.24; peer-reviewed by P O'Connor, K Kroeker; comments to author 14.02.25; revised version received 06.03.25; accepted 21.03.25; published 30.04.25.
Copyright©Prashila Dullabh, Courtney Zott, Nicole Gauthreaux, James Swiger, Edwin Lomotan, Dean F Sittig. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 30.04.2025.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.