TY  - JOUR
AU  - Bienefeld, Nadine
AU  - Keller, Emanuela
AU  - Grote, Gudela
PY  - 2024
DA  - 2024/7/22
TI  - Human-AI Teaming in Critical Care: A Comparative Analysis of Data Scientists’ and Clinicians’ Perspectives on AI Augmentation and Automation
JO  - J Med Internet Res
SP  - e50130
VL  - 26
KW  - AI in health care
KW  - human-AI teaming
KW  - sociotechnical systems
KW  - intensive care
KW  - ICU
KW  - AI adoption
KW  - AI implementation
KW  - augmentation
KW  - automation, health care policy and regulatory foresight
KW  - explainable AI
KW  - explainable
KW  - human-AI
KW  - human-computer
KW  - human-machine
KW  - ethical implications of AI in health care
KW  - ethical
KW  - ethic
KW  - ethics
KW  - artificial intelligence
KW  - policy
KW  - foresight
KW  - policies
KW  - recommendation
KW  - recommendations
KW  - policy maker
KW  - policy makers
KW  - Delphi
KW  - sociotechnical
AB  - Background: Artificial intelligence (AI) holds immense potential for enhancing clinical and administrative health care tasks. However, slow adoption and implementation challenges highlight the need to consider how humans can effectively collaborate with AI within broader socio-technical systems in health care. Objective: In the example of intensive care units (ICUs), we compare data scientists’ and clinicians’ assessments of the optimal utilization of human and AI capabilities by determining suitable levels of human-AI teaming for safely and meaningfully augmenting or automating 6 core tasks. The goal is to provide actionable recommendations for policy makers and health care practitioners regarding AI design and implementation. Methods: In this multimethod study, we combine a systematic task analysis across 6 ICUs with an international Delphi survey involving 19 health data scientists from the industry and academia and 61 ICU clinicians (25 physicians and 36 nurses) to define and assess optimal levels of human-AI teaming (level 1=no performance benefits; level 2=AI augments human performance; level 3=humans augment AI performance; level 4=AI performs without human input). Stakeholder groups also considered ethical and social implications. Results: Both stakeholder groups chose level 2 and 3 human-AI teaming for 4 out of 6 core tasks in the ICU. For one task (monitoring), level 4 was the preferred design choice. For the task of patient interactions, both data scientists and clinicians agreed that AI should not be used regardless of technological feasibility due to the importance of the physician-patient and nurse-patient relationship and ethical concerns. Human-AI design choices rely on interpretability, predictability, and control over AI systems. If these conditions are not met and AI performs below human-level reliability, a reduction to level 1 or shifting accountability away from human end users is advised. If AI performs at or beyond human-level reliability and these conditions are not met, shifting to level 4 automation should be considered to ensure safe and efficient human-AI teaming. Conclusions: By considering the sociotechnical system and determining appropriate levels of human-AI teaming, our study showcases the potential for improving the safety and effectiveness of AI usage in ICUs and broader health care settings. Regulatory measures should prioritize interpretability, predictability, and control if clinicians hold full accountability. Ethical and social implications must be carefully evaluated to ensure effective collaboration between humans and AI, particularly considering the most recent advancements in generative AI. 
SN  - 1438-8871
UR  - https://www.jmir.org/2024/1/e50130
UR  - https://doi.org/10.2196/50130
DO  - 10.2196/50130
ID  - info:doi/10.2196/50130
ER  -