gms | German Medical Science

Background: We are concerned with testing replicability hypotheses for many endpoints simultaneously. This constitutes a multiple test problem with composite null hypotheses. Traditional p-values, which are computed under least favourable parameter configurations (LFCs), are over-conservative in the case of composite null hypotheses. As demonstrated in prior work, this poses severe challenges in the multiple testing context, especially when one goal of the statistical analysis is to estimate the proportion pi_0 of true null hypotheses. Randomized p-values have been proposed to remedy this issue.

Methods: We discuss the application of randomized p-values in replicability analysis. In particular, we introduce a general class of statistical models for which valid, randomized p-values can be calculated easily. The proposed methodology is applied to simulated and to real data.

Results: By means of computer simulations, we demonstrate that the usage of the proposed randomized p-values typically leads to a much more accurate estimation of pi₀ when compared to the usage of LFC-based, non-randomized p-values. Applying the proposed methodology to real data from genetics leads to much more plausible estimates of pi₀, too.

Conclusion: In the case of composite null hypotheses (as typical in replicability analyses), randomized p-values have advantages over traditional LFC-based, non-randomized p-values.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.