gms | German Medical Science

53. Jahrestagung der Deutschen Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie e. V. (GMDS)

Deutsche Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie

15. bis 18.09.2008, Stuttgart

Specially design confidence intervals for the area under the summary ROC curve

Meeting Abstract

Search Medline for

  • Pablo Verde - Universtät Düsseldorf, Duesseldorf, Deutschland
  • Christian Ohmann - Universität Düsseldorf, Düsseldorf, Deutschland

Deutsche Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie. 53. Jahrestagung der Deutschen Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie (gmds). Stuttgart, 15.-19.09.2008. Düsseldorf: German Medical Science GMS Publishing House; 2008. DocMBIO5-1

The electronic version of this article is the complete one and can be found online at:

Published: September 10, 2008

© 2008 Verde et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( You are free: to Share – to copy, distribute and transmit the work, provided the original author and source are credited.



The area under the summary ROC curve (AUC) has been proposed as a comprehensive summary statistics for meta-analysis of diagnostic test data [1]. It has the appealing interpretation to be the probability that in a pair of disease and non-disease subjects, the disease subject will classify as more likely to have the disease [2]. The AUC is calculated by numerical integration over the SROC curve in the range of the false positive rate. The available statistical methods for calculating standard errors and confidence intervals are analytically cumbersome and may deliver inaccurate results. In this work we propose specially design confidence interval (SDCI) based on cluster bootstrap methods [3] and variance stabilization techniques. Our approach is illustrated with real data from two systematic reviews [4], [5] and evaluated with an extensive simulation experiment. In this experiment we evaluate nominal coverage, length of the interval and boundary respecting of its limits.

The simulation experiment shows that for meta-analysis with small number of studies, standard statistical methods perform miserable, second order accurate bootstrap methods [6] deliver worrisome results and SDCI methods result extremely effective. Neither full analytic techniques nor full automatic bootstrap methods deliver the best results in our problem. We conclude that for complex meta-analytic inference a strategy that combine both computer power and careful analytics may be adequate approach.


Walter SD. Properties of the summary receiver operating characteristic (SROC) curve for diagnostic test data. Stat Med. 2002;21:1237–-56.
Gatsonis C, Paliwal P. Meta-analysis of diagnostic and screening test accuracy evaluations: methodologic primer. AJR. 2006; 187:271-91.
Field CA, Welsh AH. Bootstrapping clustered data. JR Statist Soc B 2007;69, Part3:369-90.
Ohmann C, Verde PE, Gilbers T, Franke C, Fürst G, Sauerland, Böhner H. Systematic Review of CT-Investigation in Suspected Acute Appendicitis. Unpublished manuscript. Coordination Centre for Clinical Trials, The Heinrich-Heine University of Duesseldorf; 2007
Liu JL, Wyatt JC, Deeks JJ, Clamp S, Keen J, Verde P, Ohmann C, Wellwood J, Dawes M, Altman DG. Systematic reviews of clinical decision tools for acute abdominal pain. Health Technol Assess. 2006;10(47):1-167, iii-iv. Review.
Davison AC, Hinkley AV. Bootstrap Methods and thier Application. Cambridge Series in Statistical and Probilistic Mathematics. Cambridge University Press; 1997.