gms | German Medical Science

Introduction: The comparison of a binary endpoint between two groups is a common objective in biostatistics. In this case, the exact distribution of the number of successes in each group is known to be a binomial distribution. Since calculations with binomial probabilities are done easily and quickly, this situation suggests the use of exact tests. A number of exact tests for superiority have been proposed and shown to perform better than approximate tests, especially for small sample sizes [1]. Furthermore, exact tests for non-inferiority have been proposed for different effect measures [2]. However, usage of these methods is rare in practice. This might be due to the fact that commonly used software like R do not provide satisfying functionality to calculate p-values, power, sample size, and confidence regions for exact tests for superiority or non-inferiority of two independent binomial probabilities.

Methods: In this talk, we present a general method to construct and implement exact tests for difference, superiority, non-inferiority, or for any other null hypothesis formulated in terms of two independent binomial probabilities p_E and p_C. Starting with an arbitrary null hypothesis and an ordering criterion (i.e. test statistic), the critical value can be calculated exactly by performing a grid search over the null parameter space of (p_E, p_C). Analogously, the exact maximum p-value can be calculated. We explain Barnard’s convexity condition (BCC, [3]) and the advantage of test statistics satisfying BCC. Furthermore, we show the construction of confidence regions which are consistent with the test decision. For exact sample size calculation, we present a general iterative method based on the exact power for a specified alternative.

Results: We apply our general approach to commonly used test statistics for hypotheses formulated in terms of the risk difference, the risk ratio [4], and the odds ratio [5]. Calculations are less extensive since all test statistics satisfy BCC. We show that the maximum p-value can be calculated efficiently by finding the roots of the derivative. We compare type I error and power of the original tests and their exact alternatives. It is seen that the exact sample size calculation can be performed relatively fast. Furthermore, the exact sample sizes are compared with the approximate sample sizes.

Discussion: We see that the confidence regions which are consistent with the test decision are not always connected and we discuss the implications. Furthermore, we discuss the choice of the test statistic and its implications for the power.

Conclusion: We conclude that the exact calculation of critical value, p-value, sample size, and confidence region is straightforward. For commonly used test statistics, these calculations can be done efficiently and fast. This is especially true for small sample sizes where exact tests have a great advantage regarding type I error control as compared to approximate tests. In essence, there is no reason not to use exact tests for testing superiority or non-inferiority of two independent binomial proportions.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.