gms | German Medical Science

63. Jahrestagung der Deutschen Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie e. V. (GMDS)

Deutsche Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie

02. - 06.09.2018, Osnabrück

Fitness landscapes for graph-based multiple comparison procedures

Meeting Abstract

Search Medline for

  • Dennis Görlich - Institut für Biometrie und Klinische Forschung, Westfälische Wilhelms-Universität Münster, Münster, Deutschland
  • Andreas Faldum - Institut für Biometrie und Klinische Forschung, Westfälische Wilhelms-Universität Münster, Münster, Deutschland

Deutsche Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie. 63. Jahrestagung der Deutschen Gesellschaft für Medizinische Informatik, Biometrie und Epidemiologie e.V. (GMDS). Osnabrück, 02.-06.09.2018. Düsseldorf: German Medical Science GMS Publishing House; 2018. DocAbstr. 239

doi: 10.3205/18gmds030, urn:nbn:de:0183-18gmds0304

Published: August 27, 2018

© 2018 Görlich et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 License. See license information at



Introduction: The simultaneous confirmatory analysis of multiple research questions usually requires the control of a global significance level, due to an increased risk of Type-I errors. A huge variety of multiple comparison procedures (MCP) have been proposed and can be applied to specific testing problems. A general approach was introduced by Bretz et al. 2009 [1], based on a graph representation of the testing problem. This approach is a framework for sequential rejective testing procedures and comprises also widely known methods, such as Bonferroni correction [2] and the Holm-Bonferroni procedure [3].

Methods: To select the best statistical design in the graph-based framework a comprehensive analysis of the procedure’s characteristics under changing parameters is necessary. To obtain fitness landscapes we performed a full search with respect to planning alternatives (expected result of a clinical trial) and target functions (optimization aim). Fitness evaluation is based on simulations of weighted Bonferroni tests. All calculations have been performed in the statistical software R [4] using the gMCP package [5], [6]. Resulting fitness landscapes can be computed for different target functions, i.e. local power values, power to reject at least 1, power to reject all null hypotheses, and the number of expected rejections. We analyzed a multitude of scenarios applying optimization of node weights and edge weights alone, as well as joint optimization. Comparisons for widely used MCPs such as Bonferroni, Holm-Bonferroni, and Fixed-Sequence designs will be presented. Effect sizes were varied in magnitude and distribution among the family of null-hypotheses.

Results: Fitness landscapes of the analyzed scenarios show specific properties, with respect to optimization aim and type. In general, fitness landscapes with respect to node weight optimization (with fixed graph) show clear gradients with increasing slopes when effect sizes increase. For very small effect sizes and also very larger effect sizes, variations in parameters have only a small impact, due to flat fitness landscapes. Our data shows that Holm-Bonferroni is only optimal in certain scenarios, e.g. equally important null-hypotheses with comparable effect sizes. In cases of unequally distributed effect estimates the optimal solutions shift towards parameter combinations favoring the null-hypothesis with the largest effect. In situations where all null-hypotheses should be rejected, procedures that exhibit two or more unconnected subgraphs (e.g. such as Bonferroni correction) are usually sub-optimal, due to the significance level that cannot be redistributed among the graph’s components. Finally, we will present the results from gradient-based optimization techniques and a differential evolution (DE) approach, and compare these with the full-search results.

Discussion: The full-search approach allows to evaluate and compare different optimization algorithms. Furthermore, our approach can also identify solutions of multi-criteria optimization. Overall, the analysis of fitness landscapes will enable statisticians to identify the best graph-based MCP. A comprehensive R-package will be developed to provide easy access to MCP optimization.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.


Bretz F, Maurer W, Brannath W, Posch M. A graphical approach to sequentially rejective multiple test procedures. Statist Med. 2009;28:586-604.
Bonferroni CE. Teoria statistica delle classi e calcolo delle probabilità. 1936. (Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze).
Holm S. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics. 1979;6(2):65–70.
R Core Team. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2015. Available from: External link
Bretz F, Posch M, Glimm E, Klinglmueller F, Maurer W, Rohmeyer K. Graphical approaches for multiple comparison procedures using weighted Bonferroni, Simes or parametric tests. Biometrical Journal. 2011;53(6):894-913.
Rohmeyer K, Klinglmueller F. gMCP: Graph Based Multiple Test Procedures. R package version 0.8-8. 2014. Available from: External link