gms | German Medical Science

Systematic reviews aim to collate and analyze all the relevant studies for the topic in question. Potentially eligible studies, which, if included, are later scrutinized, are identified by means of electronic database searching, typically in a number of biomedical databases, of which MEDLINE is arguably the most important. While systematic reviews will now employ a range of methods to identify studies for inclusion, electronic database searching, specifically in MEDLINE, remains the most important search method. MEDLINE search strategies vary greatly among systematic reviews, in length, structure, complexity, and yield.

We aimed to address the question of how good these MEDLINE search strategies are in systematic reviews in occupational medicine. We investigated three measures that allow for an assessment of the MEDLINE search strategies: recall, i.e. the ability of the search strategy to identify the relevant MEDLINE indexed studies (the MEDLINE indexed studies that are included in the review), precision, the ability of the search strategy to identify the relevant included studies relative to the overall number of database records that the search strategy generates, and MEDLINE coverage, a measure for the ability to find relevant references through MEDLINE only (i.e. the proportion of included studies that are indexed in MEDLINE). Furthermore, we introduce the search specific coverage as a measure for the ability of the specific MEDLINE search strategy in question to identify the relevant studies (i.e. the proportion of included studies that are retrieved by the specific MEDLINE search strategy in question). The measures recall and precision are equivalent to the well-known diagnostic criteria sensitivity and specificity.

Ideally, search strategies would find all the relevant studies and not return any other studies that, on closer inspection, are revealed to not be relevant to the review. While this ideal in its absolute form is generally not practical, aiming for a reasonably high recall and precision is important to balance the amount of work and quality of systematic reviews. Further, an assessment of the quality of search strategies can be made by benchmarking to systematic reviews of a known high standard, such as Cochrane Reviews.

We therefore chose as the sample for our investigation of recall, precision, MEDLINE coverage, and search specific coverage in systematic reviews in occupational medicine the Cochrane Reviews in the sub-topic of “management of occupational disease” of the topic “health & safety at work” in the Cochrane Database of Systematic Reviews (CDSR).

A convenience sample of systematic reviews for the replication and assessment of the search strategy, but arguably with relevance for the whole field, was derived from the CDSR. For our investigation, we screened all available systematic reviews for the topic “health and safety at work” in the category of “management of occupational disease”. We included all such Cochrane Reviews which provided a replicable search strategy and which listed all included studies. Reviews at the protocol stage and reviews withdrawn from the CDSR were excluded from our present analysis. Our last search for eligible Cochrane Reviews was performed in October 2017 (https://www.cochranelibrary.com).

Data on the investigated disorders, the search strategy employed in the Cochrane Reviews and the included references with and without MEDLINE unique identifier (uid) were extracted. Reasons for absent uid were determined, if not stated in the reference list of the Cochrane Reviews, via a National Library of Medicine (NLM) Catalog search, via trial registries (ClinicalTrials.gov, ISRCTN, EU-CTR) or via a Google search. The investigated disorders were categorized as follows: ‘hearing disorders’, which comprises noise-induced hearing loss and tinnitus, ‘carpal tunnel syndrome’, ‘other musculoskeletal disorders’, including back pain and neck and limb disorders, and ‘other disorders’, which includes infectious disease, mental health disorders, poisoning, respiratory tract disorders, and dermatitis.

For our investigation the original MEDLINE search strategy was replicated (by PL) via PubMed or OVID according to the syntax given in the Cochrane Review in question. If the search strategy was provided as OVID syntax, the search was run in OvidSP. All replicated searches were verified by another author (JH) for conformity with the original searches and plausibility.

To test the quality of the MEDLINE search strategy of the included Cochrane Reviews the Inquisitio Validus Index Medicus method employing recall as search validation was applied [1]. This recall is a measure of the ability of a MEDLINE search strategy to identify the relevant studies that are indexed in MEDLINE. In this known-item search the MEDLINE-indexed included references of the original systematic review were used as ‘all relevant studies’. Recall was calculated as the number of included studies of the Cochrane Review that were retrieved in the replicated MEDLINE search divided by the number of all MEDLINE-indexed included studies in the Cochrane Review in question.

Recall =

A second measure, precision, was employed to assess the effectiveness of the search strategy. This was calculated as the number of included studies that were retrieved in the replicated MEDLINE search divided by the total number of search results generated by the replicated MEDLINE search strategy.

Precision =

Coverage is a measure that assesses to which extent relevant references can be identified by searching only in MEDLINE [2]. This coverage – in the following text called MEDLINE coverage – was calculated as the number of included studies of the Cochrane Review indexed in MEDLINE divided by the number of all included studies of the Cochrane Review.

MEDLINE coverage =

Furthermore, we here introduce the search specific coverage as a new measure describing to which extent the included references where identified through the MEDLINE search strategy. Specifically, the following formula was used:

Search specific coverage =

The search specific coverage is always smaller than or equal to the MEDLINE coverage.

Descriptive statistics (median and 10–90 percentile) were calculated for precision, recall, search specific coverage, and MEDLINE coverage for each Cochrane Review and by disorder category. Further, the overall precision, overall recall, overall search specific coverage, and overall MEDLINE coverage were also calculated. The correlation between recall and precision of the MEDLINE search strategies as well as between search specific coverage and precision was analyzed by Spearman’s rank test (Spearman R and 95% Confidence Interval (CI); SAS 9.4, SAS Institute Inc. All).

Overall, we analyzed the search strategies in 42 Cochrane Reviews, published between 2001 and 2017. Figure 1 [Fig. 1] illustrates our strategy for identifying the relevant Cochrane Reviews for our overview and reasons for excluding reviews.

The health condition that has been investigated in the respective Cochrane Review and the access to MEDLINE for the literature search as well as other characteristics of the reviews are presented in Attachment 1 [Attach. 1].

Attachment 2 [Attach. 2] shows extracted data on references (included studies) of the Cochrane Reviews and the results of our analysis of recall, precision, MEDLINE coverage, and search specific coverage.

The Cochrane Reviews included between zero and 54 studies with a uid (mean 12.3), and between zero and seven studies without a uid (mean 1.7). The included studies without a uid are detailed in Table 1 [Tab. 1]; six of these studies could be found in PubMed but not in Ovid MEDLINE (and we used either PubMed or Ovid MEDLINE to replicate the searches, in line with what had been described in the Cochrane Review in question).

The overall precision, recall, search specific coverage, and MEDLINE coverage were 0.9%, 78%, 69%, and 88%, respectively (Attachment 2 [Attach. 2]). As shown in Table 2 [Tab. 2], the median precision was 1.5%, the median recall was 83%, the median search specific coverage was 74%, and the median MEDLINE coverage was 93%.

Categorizing the Cochrane Reviews by disorder illustrates that there is some variability between the reviews.

We further investigated the relationship between recall and precision of the MEDLINE search strategies as well as between search specific coverage and precision and found no statistically significant correlations (Spearman R =0.232 (95% CI, –0.082 to 0.503) and R=0.193 (95% CI, –0.118 to 0.469)).

The median precision of 1.5%, the median recall of 83%, the median MEDLINE coverage of 93% and the median search specific coverage of 74% may serve as benchmarks for systematic reviews in occupational medicine, as we are not aware of other studies attempting such an analysis in this subspecialty. The knowledge that screening 100 database records will yield just one eligible study allows for an estimation of the work involved in conducting a systematic review in occupational medicine based on the number of hits in MEDLINE. If indeed our results are a generally applicable approximation for systematic reviews in occupational medicine, this knowledge will allow researchers to plan their work and estimate the person-power required for reviews, which may also be helpful for grant applications. This knowledge may provide a rational for performing a first comprehensive search early in the process of review writing, perhaps even at the stage of applying for a research grant. Along the same lines, it may also be something funders might in future request of research teams who propose to perform systematic reviews as part of a budget justification.

Though we are not aware of a previous analysis like this in occupational medicine, more broadly speaking, our findings can be compared with what others have reported. Sampson and McGowan pioneered the idea of calculating recall as a means of validating a MEDLINE search; they examined 6 Cochrane Reviews and calculated an overall recall of 58% and a median recall of 46% [1]. In another publication, Sampson and colleagues [3] calculated the precision for 94 reviews and found the median precision to be 2.9%, with a range of 0.7% to 35.8%. Bramer and colleagues [2] analyzed searches from 120 systematic reviews and reported an overall recall of 72.6% for MEDLINE as well as an overall precision of 2.8%; they also found that MEDLINE alone achieved 92.3% overall coverage, which is comparable to our “MEDLINE coverage” in the field of occupational medicine as a sub-specialty.

Overall, and the finding of some reviews with low recall and low precision notwithstanding, one might expect an inverse relationship between recall and precision, although the Spearman correlation was not significant.

At the one extreme, a search strategy encompassing all of MEDLINE would necessarily find all included studies that are indexed in MEDLINE, but have an extremely low precision. A review with a much focussed search strategy on the other hand would have a high precision, but typically at the expense of not detecting all relevant studies that are indexed in MEDLINE. Where along this spectrum is the optimum, in general and for a systematic review in occupational medicine?

An argument might be made for a general default of accepting low precision for the promise of high recall, but the matter may not be as simple as that, for the following reasons. The screening of large numbers of database records might overwhelm research teams, steering them away from otherwise attractive review projects. Having to screen large numbers of hits might also lead to deviations from the methodologically highest systematic review standards of independent duplication of the screening step by two reviewers to preserve time, such as by employing screening by one reviewer only or even by having several single reviewers screen non-overlapping sections of the search results where these reviewers might be inconsistent in their application of the study eligibility criteria. Moreover, having to screen a large number of hits could result in more cursory screening by reading abstracts less thoroughly or by deliberately only reading titles and not abstracts. Further, having to screen a large number of database records from one database might lead to a decision of limiting the other components of the search (other databases, non-electronic searching) for time and resource reasons, making the overall search less comprehensive. Finally, another consideration is fatigue and loss of interest and engagement with the subject matter as a result of the monotony of spending long periods of time screening abstracts on a computer screen. All these factors could lead to eligible studies being missed in the screening process. Therefore, a balance between recall and precision needs to be stuck and that balance might not be the same for different fields of study – hence there is a need for benchmarking with regard to recall and precision in different medical specialties. What the benchmark should be is of course a subject that could be discussed at length, but that would be beyond the scope of this publication; the approach of taking Cochrane Reviews as the ‘gold standard’ has appealed to the authors – there would have been other answers to the question as to what should be the benchmark [4]. Recently, Franco et al. reported that in about 70% of a random sample of Cochrane Reviews problems in the design of search strategies are present and half of these problems may limit recall and precision of the search strategies [5]. In addition to the specifics of the search strategy, the combination of bibliographic databases used in a systematic review needs to be considered. Bramer et al. [6] estimate that 60% of systematic reviews do not retrieve their chosen threshold of 95% of the relevant references. Bramer et al. argue that searching the databases MEDLINE, EMBASE, Web of Science, and Google Scholar would be a ‘minimum requirement’ for an adequate and efficient coverage of the literature. Depending on the field of study, there may be greater or lesser utility in going beyond standard biomedical database searching. For example, for the topic of health effects of environmental enhancement and conservation activities, supplementary search methods (e.g. contacting organisations and searching websites) were described as valuable [7].

As occupational medicine is a comparably small field within medicine with a relative dearth of high quality trials for a number of clinically important questions, a logical approach would be to accept a low precision for the promise of capturing all or nearly all relevant studies, the limitations outlined above notwithstanding. In other fields, where there is an abundance of high quality trials addressing the clinical questions of interest, the balance between precision and recall may be different and it may make sense to aim at higher precision with the search strategies to contain the overall effort needed for conducting the systematic review.

Further, occupational medicine is a discipline where different terms and phrases may be used in different jurisdictions to describe the same or related concepts. Contrast this with the search for trials on pharmaceutical interventions, especially when based on drug names. The precision can be expected to be higher in the latter compared to the former, for the same recall.

Some limitations of our analysis need to be considered. There was considerable variability between the Cochrane Reviews – as shown in Attachment 2 [Attach. 2] and Table 2 [Tab. 2]: it is not clear if the variability between investigated disorders is due to differences in the subject matter or difference between author teams. This would need to be explored in more extensive future studies. Further, we focussed on the management of occupational diseases, and the generalizability of our conclusions to other fields within occupational medicine will need to be confirmed.

The parameters estimated in this article – i.e. that screening 100 database records would yield about one eligible study and that about 83% of the relevant MEDLINE-indexed studies and about 74% of all relevant studies for a systematic review would be identified via a MEDLINE search – may help researchers find a balance between recall and precision of search strategies. These benchmarks for systematic reviews in occupational medicine may further be helpful for researchers in estimating the workload required for such systematic reviews and in determining what additional searching and resources they wish to employ.

• CDSR: Cochrane Database of Systematic Reviews
• uid: unique MEDLINE identifier
• NLM: National Library of Medicine
• ISRCTN: International Standard Registered Clinical/soCial sTudy Number
• EU-CTR: European Union Clinical Trials Register

The results of the replicated search strategies used for analyses of the current study are available from the corresponding author on reasonable request.

The authors declare that they have no competing interests.

TF and SS designed and supervised the study. PL and JH replicated search strategies, gathered and analysed data. SS and JH drafted the manuscript. All authors revised critically the draft and approved the final manuscript.