gms | German Medical Science

Policy makers in the health care system often have to make decisions on the one hand based on a broad and high-quality evidence and on the other hand within limited time periods. Ideally, systematic reviews (SR) summarize the best available evidence and serve as a basis for an informed decision. However, SR are of varying quality and therefore results are often biased [1]. A critical appraisal of SR is necessary to identify high-quality SR, but it is also time consuming in situations with many SR for a specific research question. AMSTAR 2 [2] is a valid and moderately reliable [3] critical appraisal instrument and has 16 items, which lead to an overall rating of methodological quality (high, moderate, low or critically low methodological quality). Decision trees offer the possibility to make decisions based on a very limited amount of information. In the present study, we aim to create a fast and frugal decision tree, in order to evaluate the methodological quality of a SR with a limited set of items of the AMSTAR 2.

In our previous publications [3], [4], we appraised 118 SR in two data sets (60 and 58 SR) using AMSTAR 2. The research questions of both data sets were related to psychological interventions in psychiatric disorders. The responses to the 16 items served as cues to predict a binary decision (high or low methodological quality). The structure of the decision trees were optimized based on the criterion of the overall rating (SR with high and moderate quality ratings were assigned to high quality and low and critically low quality to low quality). We used the fast and frugal trees toolbox [5] embedded in the software R (R Core Team).

Based on 118 SR, a fast and frugal tree that contained only two out of 16 items was identified. These items related to a published review protocol (AMSTAR 2 item 2) and to a list of excluded studies (AMSTAR 2 item 7). When applying this tree, 14 high quality reviews were correctly identified (hit) and none were missed, whereas 94 low quality reviews were correctly identified (correct rejection) and 10 were falsely identified as high quality reviews (false alarms). This is equivalent to a sensitivity of 100% and a specificity of 93%.

Using the proposed fast and frugal decision tree lead to an approximation of the methodological quality by applying only two items and therefore save valuable time in situations with a broad evidence base (many SR) and decision making under time pressure. In the next step, this decision tree has to be validated in a new data set. At the conference we also want to present the validation results.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.