gms | German Medical Science

Introduction: RobotSearch (RS) is a machine learning system designed to select only the records judged by the machine learning model to be randomized controlled trials (RCTs). We conducted a small study to compare the results of this filter against the results of double independent human screening of the titles and abstracts.

Method: A total sample of 800 records was uploaded to RS as a RIS file and also uploaded to Covidence systematic reviewing software for assessment by two independent human reviewers. This output was compared with the RCT selection performed by RS.

Results: We found that RS resulted in a 55% record reduction that would require double independent review. Importantly, we found RS to only inaccurately exclude 1 record (0.1%).

Conclusions: RobotSearch could potentially be a useful tool during the discovery stage of a project, to help provide an estimate of how much RCT evidence may exist within a certain field.

It should be noted that the 361 records included by RS were not screened any further by the reviewers, and so it can only be inferred that these records were inaccurately included by RS. Future research could additionally screen any records included by RS and excluded by reviewers, in order to ensure we are not missing any additional records RS may accurately select that reviewers do not.

Attachment 1 [Attach. 1]

Keywords: RCT, randomized, trial, machine, learning