gms | German Medical Science

Background: cross-sectoral routine data enable the analysis of the entire course of treatment, before, during and after hospitalization. However, there are limitations to the outcomes and risk factors that can be analyzed, so that residual confounding and the associated potentially limited validity of these quality indicators (QIs) cannot be ruled out. Clinical databases contain potential risk factors and outcome-relevant information that is not available in routine data of statutory health insurances (SHI), but which could improve quality measurement.

Objective: In the innovation fund project “Hybrid quality indicators using machine learning” (Hybrid-QI, 01VSF20013), the influence of data obtained directly from clinical information systems on risk adjustment in quality measurements was investigated.

Methods: Longitudinal SHI routine data were retrospectively linked with clinical data from 15 participating hospitals on a patient-by-patient basis (hybrid data) and harmonized in the Observational Medical Outcome Partnership (OMOP) data model. The inclusion criteria, outcomes and risk factors in the hybrid data were developed in medical expert panels. Three endpoints (30-day mortality, reinfarction after 90 days, care degree increase after 180 days) were analyzed for stroke using four different models (clustered logistic regression, elastic net, xgboost, simple neural net) on the SHI data and on the hybrid data. The models were compared using the Receiver Operating Characteristic-Area Under the Curve (ROC-AUC), Precision-Recall-Curve (PR-AUC) and the Brier Score (BS), among others. The influence on the result of the respective quality measurement was considered using the standardized event rates (e.g. SMR – standardized mortality rate).

Results: 10,902 patients were included for the period 2017–2020 (I61 and I63 ICD-10). However, clinical information was not available for all patients. The National Institutes of Stroke Scale (NIHSS) was only available for 4,501 patients. Across all three outcomes, clinical severity, as measured by the NIHSS, had a positive impact on model fit. The goodness of fit for the hybrid models was higher than that of the models based solely on SHI routine data. The SMR showed differences in the ranking of hospitals between the analyses based on the SHI routine data and the hybrid data. Particularly for the 30-day mortality endpoint, the influence of the NIHSS was greater than the influence of the most important variable “age” in the SHI model.

Implications for research and practice: The additional consideration of clinical severity improves risk adjustment in relation to the examined quality indicators of stroke. The hybrid QIs therefore appear to be superior to the purely SHI-based QIs for stroke. Due to the low completeness of the hospital data and selection effects that cannot be ruled out, the results should be replicated in more extensive data and with the participation of a more heterogeneous range of hospitals.

Funding: Innovationsfonds/Versorgungsforschung; Project name: Hybride Qualitätsindikatoren mittels Machine Learning (Hybrid-QI); Grant number: 01VSF20013