gms | German Medical Science

Background/research question: The methodological quality of systematic reviews (SRs) can be evaluated using a measurement tool to assess systematic reviews 2 (AMSTAR 2). Although the tool is well-designed and recommended by experts, its rating scheme for the overall confidence has been criticized.

The aim of this study was to analyze the impact of different rating schemes on the overall confidence of AMSTAR 2.

Methods: This is an analysis of an existing sample of SRs with interventions in advanced cancer patients from a methodological study (CRD42019134904). Medline, the Cochrane Database of Systematic Reviews and Web of Science were searched (January 2010 to July 2019). We included SRs with at least one statistically significant meta-analysis of at least four randomized controlled trials.

We used three schemes with varying critical items to compare the overall confidence of the 16-item AMSTAR 2 tool: 1. original 7-item scheme (Shea et al., 2017 [1]; critical items: 2, 4, 7, 9, 11, 13, 15), 2. modified 7-item scheme (Leclercq et al., 2020 [2]; critical items: 2, 4, 9, 11-14), and 3. 5-item scheme (Pieper et al., 2021 [3]; critical items: 4, 9, 11, 12, 15).

Results: A total of 6,234 hits were identified and 261 SRs were included after the screening process. The majority of SRs had a critically low methodological quality in all schemes.

1.

The original 7-item scheme resulted in the rating: high=12, moderate=8, low=11, critically low=230.

2.

The modified 7-item scheme classified 11 SRs as high, 11 as moderate, 23 as low and 216 as critically low.

3.

Using five critical items in the 5-item scheme by Pieper et al. showed the following overall confidence: high=11, moderate=21, low=80, critically low=149.

Conclusion: There was a substantial difference in overall confidence for the categories “low” and “critically low” between the two 7-item schemes and the 5-item scheme. The latter showed the best discrimination properties. While discrimination is a desirable property in general, it should not be pursued purely for methodological reasons, but valid quality requirements should be the benchmark.

The strong dependence of the overall rating from the choice of critical items and the fact that AMSTAR 2 allows for the individual choice of critical items massively affects comparability of the overall rating between SRs. When judging SRs, we suggest reporting in detail each of the 16 AMSTAR 2 items in a well-structured table or figure in the main text, and transparently justifying the critical items.

Competing interests: Siemens, W.: No competing interests. WS was employed at Roche Pharma AG from April 2020 to June 2021. Roche Pharma AG was not involved in the project and had no influence at any time on the project.

Schwarzer, G.: No competing interests. GS is an external statistical consultant of Roche Pharma AG since August 2019. Roche Pharma AG was not involved in the project and had no influence at any time on the project.