gms | German Medical Science

Aim: We conducted a study aimed to assess the feasibility of RobotReviewer in two real world cases of systematic reviews at the Norwegian Institute of Public Health. Feasibility is operationalized to be measured by a combination of accuracy, resource use, and acceptance. We wish to present the results regarding the participants’ acceptability for this oral presentation.

Method: This study was conducted by the machine learning team at the Norwegian Institute of Public health. We recruited two systematic review projects. The project leader in both projects chose to compare two different ways of using RobotReviewer to each other (integrated into a systematic review software and via RobotReviewer’s pilot website) rather than compare the use of RobotReviewer to usual, fully manual practices. The included studies were randomized to one of the two platforms. Acceptability was measured through individual email responses and then through a group discussion, and approached with discourse analysis.

Results: In general, newer researchers were more positive to RobotReviewer than more experienced researchers. They particularly focused on the text snippets provided that explain the basis for each automated domain asssesment, saying that these snippets guided them to where in the article’s pdf they should focus on to make their own assessment. None of the participants were willing to allow RobotReviewer to replace one of two researchers for risk of bias assessments, despite being proven to be equal in performance to humans. Instead of semi-automating the process, they suggested adding RobotReviewer as a pedagogic tool for newer researchers and to be a useful quality assurance. When asked about future use of RobotReviewer in new reviews, nearly all participants were positive to using RobotReviewer. Despite not having time estimates of RobotReviewer and manual procedures, some agreed that RobotReviewer could contribute to more effective workflows, pointing again towards the perceived time saved in identifying relevant text in a pdf.

Conclusion: Despite being presented with evidence of RobotReviewer’s equal performance to humans, an automated risk of bias assessment process was not acceptable by our institution’s excerienced systematic reviewers as a replacement for one human within this resource-demanding process. They recommended that the tool should instead be implemented as a guidance for newer researchers. However, we do not know if increasing trust over time to the tool could improve their acceptability towards increasing automation of this and other processes within systematic review.

Keywords: risk of bias assessment, semi automated, RobotReviewer