gms | German Medical Science

Background/research question: Policy makers in the health care system often have to make decisions on the one hand based on broad and high-quality evidence and on the other hand within limited time periods. Ideally, systematic reviews (SR) summarize the best available evidence and serve as a basis for an informed decision. However, SR are of varying quality. A critical appraisal of SR is necessary to identify high-quality SR, but it is also time consuming in situations with many SR for a specific research question. AMSTAR 2 [1] is a valid and moderately reliable critical appraisal instrument and has 16 items, which lead to an overall rating of methodological quality (high, moderate, low or critically low methodological quality). Decision trees offer the possibility to make decisions based on a very limited amount of information. In the present study, we aim to create a fast-and-frugal decision tree to evaluate the methodological quality of a SR with a limited set of items of the AMSTAR 2.

Methods: In previous projects, we appraised 118 SR using AMSTAR 2. The research questions were related to interventions in psychiatric disorders. The responses to the 16 items served as cues to predict a binary decision (high or low methodological quality). The decision trees were developed based on the criterion of the overall rating (SR with high and moderate quality ratings were assigned to high quality and low and critically low quality to low quality). We used the fast-and-frugal trees toolbox [2] embedded in the software R.

Results: Based on 118 SR, a fast-and-frugal tree that contained only two out of 16 items was identified. These items are related to a published review protocol (item 2) and a list of excluded studies (item 7). When applying this tree, 14 high quality reviews were correctly identified (hit) and none were missed, whereas 94 low quality reviews were correctly identified (correct rejection) and 10 were falsely identified as high quality reviews (false alarms). This is equivalent to a sensitivity of 100% and a specificity of 93%.

Conclusion: The fast-and-frugal decision tree distinguished between high and low quality SR while applying only two items. Therefore, it could save valuable time in situations with a broad evidence base (many SRs) and when decisions have to be made under time pressure. In the next step, this decision tree has to be validated in a larger set. At the conference, we also plan to present the validation results.

Competing interests: Keine Interessenskonflikte