gms | German Medical Science

Background/research question: Digital interventions (DI) could support public health measures, such as physical activity (PA) promotion. Since many systematic reviews (SRs) already addressed this topic, the current study aimed to appraise such SRs using two different combinations of items on AMSTAR2, a 16-item evaluation tool [1]. AMSTAR2 assigns overall confidence ratings (high, moderate, low, critically low) to SRs according to 7/9 critical/non-critical items. In general, critically low ratings are assigned if at least 2 critical items are not fulfilled (rated as ‘no’) on AMSTAR2.

Methods: The protocol for this study was registered in OSF [2]. Of 300 reviews on DI and PA published up to March 2021 in Medline, PsycInfo and CINAHL, k=30 SRs focusing on DI for PA promotion were included in this cross-sectional study. The SRs were independently selected by two researchers and appraised with AMSTAR2 by one researcher in two steps. First, all SRs were appraised with 2 critical items (item 2: presence of SR protocol and item 7: report of a list of excluded studies). These items were selected according to a fast and frugal decision tree for the critical appraisal of SR [3]. Second, all SRs were appraised using all 16 items [1]. The outcomes of both appraisal methods were compared descriptively based on correctness and appraisal time.

Results: Based on 2 items 19/30 SRs were rated as ‘critically low’ (63.3%) due to 2 ‘no’ ratings. Further 11/30 SRs (36.7%) with 0-1 ‘no’ ratings on item 2 or 7 had to be evaluated with all AMSTAR2 items to obtain a final appraisal rating. The appraisal outcomes were 0/11 ‘high’, 1/11 ‘moderate’, 3/11 ‘low’ and 7/11 ‘critically low’. Based on 16 items, the overall confidence ratings were 0/30 ‘high’, 1/30 (3.3%) ‘moderate’, 3/30 (10%) ‘low’ and 26/30 (86.7%) ‘critically low’. Relative to appraisals derived based on 16 items, 19/26 (73.1%) SRs were correctly rated as ‘critically low’ with 2 items. The average time for individual SR appraisal was 20 min with 16 items and 5 min with 2 items. The appraisal time for all 30 SRs was 10 h with 16 items and 4 h and 20 min with 2 items.

Conclusion: Using the proposed fast and frugal decision tree it is possible to reliably assign critically low overall confidence ratings and to save time. Thus, two critical items on AMSTAR2 can help to identify SRs with the lowest confidence ratings. According to AMSTAR2 the overall confidence in the results of SRs of DI for PA promotion is disappointing and needs improvement.

Competing interests: none