Article
Holistic Simulation to Assess the Usefulness of AI Models For Reducing Non-Actionable Alarms in Intensive Care Units
Search Medline for
Authors
-
Julian Beimes
- idalab Gmbh, Berlin, Germany; Institute of Medical Informatics, Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
-
Anne Rike Flint
- Institute of Medical Informatics, Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
-
Sophie Klopfenstein
- Institute of Medical Informatics, Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
-
Maximilian Markus Wunderlich
- Institute of Medical Informatics, Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
-
Paul von Bünau
- idalab GmbH, Berlin, Germany
-
Felix Balzer
- Institute of Medical Informatics, Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
-
Akira-Sebastian Poncette
- Institute of Medical Informatics, Charité – Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| Published: | September 6, 2024 |
|---|
Outline
Text
Introduction: Non-actionable alarms are a well-studied problem and several AI-based models have been developed for detecting them [1]. However, the evaluation metrics for these models are insufficient to assess the real-world usefulness, as they are limited to the statistical evaluation of the model’s performance. Simulations have been shown to enable holistic evaluations of AI models in clinical workflows [2]. This study presents a novel simulation approach to assess AI-based models for reducing the number of non-actionable alarms in intensive care units (ICU).
Methods: Following IRB approval (EA1/127/18), we developed a Monte-Carlo simulation [3] to assess an AI model filtering non-actionable alarms. We model the alarm management process for an ICU with 20 patients over one day. The simulated patients generate alarms during the day that are handled by the nurses working in three shifts. If all nurses manage alarms, a new alarm is queued until a nurse becomes available. This allows a granular simulation of alarms, alarm management workload, and alarm response times. An AI model for classifying alarms and filtering non-actionable alarms can be integrated into this simulated process.
To ensure medical plausibility, the simulated alarms are randomly drawn from pseudonymized real-world data collected during the research project 'Intelligent Alarm Optimizer for Intensive Care Units' (INALO). This dataset contains more than 54,000 ICU stays from the ICUs of a German university hospital, with all alarms retrospectively annotated [4].
Results: The simulated alarms closely mirror the empirical alarm data, particularly the distribution of alarms and the correlating alarm response times throughout the day. Periods with low to regular simulated alarm activity (up to ten alarms per minute) correspond with short alarm response times, while periods with more than ten alarms per minute, so-called ‘alarm floods’ [5], also show longer simulated alarm response times.
The AI model effectively reduces the simulated workload by nearly half (49%, SD: 3.6) by filtering out non-actionable alarms. Its impact on real-world response times is substantial in alarm flood periods (61% reduction, SD: 6.3) but neglectable for periods of low alarm volume (7% reduction, SD: 1.4).
Discussion: Our simulation demonstrates that the primary benefit of the AI model lies in its ability to mitigate ‘alarm floods’. These periods can overwhelm nurses with alerts, resulting in a high workload and long response times to critical alarms. However, the AI model's impact is minimal for periods with low alarm frequency, as the medical staff can typically handle them efficiently. This implies that the application of the AI model should be restricted to periods of high alarm volume to leverage the model's benefits while minimizing the risk of falsely classified actionable alarms.
Conclusion: By simulating the alarm response and the workload, the simulation allows a holistic assessment of AI models for reducing non-actionable alarms and alarm fatigue. We show that simulations can not only support a more granular and realistic assessment of models in healthcare than model metrics, but also guide the integration in clinical workflows.
The authors declare that they have no competing interests.
The authors declare that a positive ethics committee vote has been obtained.
References
- 1.
- Poncette AS, Wunderlich MM, Spies C, Heeren P, Vorderwülbecke G, Salgado E, et al. Patient Monitoring Alarms in an Intensive Care Unit: Observational Study With Do-It-Yourself Instructions. J Med Internet Res. 2021 May 28;23(5):e26494.
- 2.
- Wornow M, Gyang Ross E, Callahan A, Shah NH. APLUS: A Python library for usefulness simulations of machine learning models in healthcare. Journal of Biomedical Informatics. 2023 Mar;139:104319.
- 3.
- Raychaudhuri S. Introduction to Monte Carlo simulation. In: 2008 Winter Simulation Conference; 2008 Dec 7-10; Miami, FL, USA. IEEE; 2008. p. 91–100. DOI: 10.1109/WSC.2008.4736059
- 4.
- Klopfenstein SAI, Flint AR, Heeren P, Prendke M, Chaoui A, Ocker T, et al. How to Annotate Patient Monitoring Alarms in Intensive Care Medicine for Machine Learning [Preprint]. Research Square. 2023. DOI: 10.21203/rs.3.rs-2514969/v1
- 5.
- Hollifield BR, Habibi E. Alarm management: seven effective methods for optimum performance. Research Triangle Park, NC: Instrumentation, Systems, and Automation Society; 2007.
