gms | German Medical Science

Introduction: Diagnostic clinical decision support systems (CDSS) integrate and present processed information to healthcare professionals at the point of care so that they can make an appropriate diagnosis [1], [2], [3], [4]. CDSS have a medical purpose; thus, they must be safe and effective [5], [6]. They should be evaluated in a diagnostic test accuracy (DTA) study using DTA indices (e.g., sensitivity and specificity) [3], [4] and they should be reported in accordance with STARD [7]; decision makers should critically appraise the study’s risk of bias with QUADAS-2 [8]. This systematic review assesses reporting quality and risk of bias in studies evaluating DTA of CDSS.

Methods: This study was registered with PROSPERO (CRD42021226570) including the study protocol. We searched PubMed/MEDLINE, Scopus, Web of Science, and the Cochrane Library for studies evaluating DTA of CDSS for use in human patients (irrespective of the target diagnosis). Studies had to be published between 2016-01-01 (i.e., after the update of STARD) and 2021-05-31 (date of database search). We excluded studies focusing on treatment, disease follow-up, severity, or economic evaluation, using a patient’s self-diagnosis or comparing pre-post CDSS implementation periods. We assessed reporting quality (using STARD 2015 with 30 items) and risk of bias (using QUADAS-2 with 14 items) of each eligible study. We rated each item as “reported transparently/low risk of bias”, “reported semi-transparently/unclear risk of bias” or “not reported/ high risk of bias” and plotted the items’ ratings stratified by year of publication. Additionally, we conducted stratified analyses according to the CDSS development phase, the subjectivity to reporting guideline policies, and the author’s expertise. This abstract is reported according to PRISMA-DTA for abstracts [9].

Results: We included 158 of 2,820 screened studies. The studies differed in terms of sample size (11 to 1.4 million individuals), data collection (2 months to 40 years), study designs, and other study characteristics. Seventy-nine percent of the studies presented a Phase-I/II CDSS. Studies differed in their reporting quality and their risk of bias with few studies at either end of the quality continuum. Applicability of findings was less concerning. The overall reporting quality and risk of bias was mostly affected by items related to participant selection, study design, flow and timing, blinding, test methods, test results, and other information (e.g., registration, study protocol, funding). Time trends were not observed. Effects of the CDSS development phase and the subjectivity to reporting guideline adherence were not apparent, while studies involving researchers from interdisciplinary scientific fields had an improved overall quality.

Discussion: Our systematic review included all studies evaluating DTA of a CDSS, of which some may have little resemblance to a ‘true’ DTA study. However, this range of studies showed that the reporting quality and risk of bias assessment is heterogeneous, particular for items related to ‘methods’, ‘results’, and ‘other information’; thus, uncertainties in research findings remain.

Conclusion: Reporting deficiencies need to be addressed by implementing incentives and measures to increase transparency and reduce risk of biases.

Funding: This work was supported by the German Federal Ministry of Health [grant number 252DAT66A].

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.