gms | German Medical Science

Introduction: The KIPeriOP project aims at improving the quality of care in the context of preoperative risk assessment by developing and implementing guideline-based clinical decision support (CDS) tools. Currently, a randomised controlled study is conducted in four German clinics to investigate if the use of CDS tools in clinical practice helps to prevent unnecessary preoperative supplementary examinations ([1], ClinicalTrials.gov Identifier: NCT05284227). It is planned to use the acquired data for training machine learning (ML) models for risk prediction which will be compared to established risk scores regarding their capability to predict perioperative complications. In this work, we implemented a retrospective study at one of the four hospitals to prepare the ML modelling pipeline. Our primary goal was to anticipate potential pitfalls and test possible solutions.

Methods: The retrospective study uses routine data from twenty-six thousand patients undergoing elective surgery from 2016 to 2022 at Charité - Universitätsmedizin Berlin. The dataset includes demographics, vital signs, clinical scores, laboratory values, comorbidities, and surgery information as well as indicators for eight common perioperative complications (e.g., pneumonia, bleeding, or death). We tested different implementations for data splitting, missing data imputation, imbalanced data handling, and ML-based and classic statistical prediction algorithms. Models were compared using performance metrics for discrimination, calibration, and fairness.

Results: The two major pitfalls we observed were missing and imbalanced data: one third of our features showed incomplete data with missing ratios between 4-88%, and prevalence of each outcome (except for delirium) was below 4%. Missing data always needed to be addressed because many standard implementations of ML algorithms only allowed complete input data. In contrast, it was possible to train prediction models without addressing imbalanced data, but ignoring this problem often led to poor model performance (e.g., sensitivity < 0.1 for 72% of models without balancing compared to 5% using balancing techniques). We quantified the influence of these two pitfalls, among other factors, in our experiments. The median balanced accuracy over all other factors was affected more by the balancing technique (under-sampling: 0.75, balancing class-weights: 0.71, no balancing: 0.52) than by the imputation method (k-nearest neighbors imputation: 0.69, mean imputation: 0.70, multiple imputation: 0.71).

Discussion: Our tests give us an overview for promising methods to apply in the planned predictive modelling on prospective data. We were able to identify which modelling steps are sensitive to the choice of methods and therefore need more careful consideration. But the findings need further validation on the prospective data. One major limitation in the retrospective data is the lack of standardized documentation. It was challenging to identify perioperative complications because ICD diagnoses had no reliable timestamps. The prevalence of many complications was lower than expected, which might indicate that false negatives occurred in our labelling process. We hope to overcome this problem with better documentation in the prospective study.

Conclusion: We used retrospective routine data to train ML models for preoperative risk assessment for eight perioperative complications. This work serves as a preparing test of methods to solve common pitfalls for predictive modelling in an ongoing prospective study.

The authors declare that they have no competing interests.

The authors declare that a positive ethics committee vote has been obtained.