gms | German Medical Science

Background/research question: In a survey among professionals conducting systematic reviews, 89% of respondents reported using automation tools, primarily Covidence, RevMan and Rayyan [1]. These tools have shown to efficiently prioritize records during abstract screening, with 95% of relevant records included within the top 50% of highest prioritized records [2], [3]). Although being predominantly designed to support systematic reviews, these tools are increasingly applied to scoping reviews. Scoping reviews differ from systematic reviews by focusing on broader research questions to map evidence rather than critically appraising it. Although they are a frequently used evidence synthesis method, studies on the efficiency of automation tools in scoping reviews are scarce. This study evaluates Rayyan’s accuracy and efficiency in identifying relevant records for scoping reviews.

Methods: This study drew on data from five scoping reviews conducted between 2022 and 2024, covering diverse topics in the context of digital health. Maintaining the imbalance of each scoping review, subsets of 2,500 records per review were created, except for one review with a smaller dataset. Rayyan was trained on a random sample of at least 50 records with at least five inclusions. Record decisions were based on the gold standard of two independent human reviewers. Ratings were computed, records ranked and inclusion/exclusion predictions noted at each 10% screening advancement. Rayyan’s performance and prioritization efficiency was calculated using Stata V.16.

Results: Although Rayyan made no prediction about inclusion, it efficiently prioritized records, identifying 95% of relevant records within the first 50% screened across all scoping reviews. After screening 20% of records, Rayyan predicted the exclusion of 65%, with additional predictions made in later stages. Initially challenged by borderline cases, Rayyan’s predictive accuracy improved throughout the screening process and the ranking of individual records became increasingly stable.

Conclusion: Our evaluation of the automation tool Rayyan for record screening in scoping reviews yielded promising results, demonstrating performance on par with that in systematic reviews. However, further research is needed to understand the factors influencing the performance of automation tools in scoping reviews, determine the ideal stopping criteria to maximize time saving, and explore the potential of large language models in the review process.

Competing interests: N/A