gms | German Medical Science

As a consequence of the so-called ‘replication crisis’ [1], an increasing number of replication studies have been conducted to determine the reliability of original findings. Ideally the procedures of the replication study should be as closely matched to the original study as possible. However, selecting the same sample size in the replication study as in the original study can lead to a severely under-powered design and true effects may not be detected [2]. Larger sample sizes are usually required in replication studies and most replication projects use conditional power calculations based on the effect estimate from the original study to determine this sample size. This approach is not well suited as it does not take the uncertainty of the original result into account.

In clinical trials, Bayesian methods are often used to incorporate prior information to power calculations. We propose to adapt this methodology to the replication setting. A sensible prior in this context is a normal prior centered around the original effect estimate with variance inversely proportional to the original sample size [3]. The resulting power is called ‘predictive’ [4] and has not been used so far to design replication studies. Moreover, as many replication projects are being undertaken, optimal allocation of resources is particularly important. We describe how the methodology used in sequential clinical trials and drug development [5], [6]) can be tailored to replication studies. The predictive interim power, i.e. the predictive power of the replication study conditioned on the data already collected, is shown to be useful to decide whether to stop an experiment at interim. In addition to an informed prior, a flat prior can also be used in this setting.

At the start of the replication study as well as at interim, predictive power turns out to be a different concept than conditional power. It generally leads to smaller values than conditional power and does not always increase when increasing the replication sample size. Remarkably, for large sample sizes the predictive power cannot exceed one minus the one-sided interim p-value. Adding more subjects to the replication study can even decrease the predictive interim power if the p-value at interim is only 'suggestive', i.e. slightly below the significance level.

We illustrate these properties using data from a large-scale replication project on the replicability of social sciences [7]. This project reports replication results of 21 studies and was conducted in a sequential manner.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.