gms | German Medical Science

Introduction: Meta-analyses involving a small number of trials or rare events pose a challenge for statistical analysis. Evidence suggests that conventional two-stage statistical methods can lead to distorted results in these sparse data situations. Though better-performing one-stage methods have become available in recent years for such meta-analyses, these methods appear not sufficiently implemented and two-stage methods are still often used. The actual impact of using two-stage methods in practice, however, remains unknown. This study aims to quantify the impact by re-analysing meta-analyses from the Cochrane Database of Systematic Reviews (CDSR) in two sparse data situations: when meta-analyses included trials with zero events in one or both arms, or when meta-analyses contained only a few trials.

Methods: For each scenario, we computed one-stage statistical methods, namely the generalized linear mixed model (GLMM), the beta-binomial model (BBM) and the Bayesian binomial-normal hierarchical model using a weakly informative prior (BNHM-WIP). We then compared their impact on the results to the conventionally used two-stage methods, namely the Peto-Odds-Ratio (PETO) and DerSimonian-Laird method (DL) in case of zero event trials and DL, the Paule-Mandel (PM) and restricted maximum likelihood (REML) method in the scenario of few trials.

Results: While all methods showed similar estimates for the pooled treatment effect, the results showed large variability in the statistical precision (length of CI and statistical significance) between the methods. Specifically, two-stage methods in the zero event situation tended to estimate narrower CIs resulting in more significant meta-analyses than the one-stage methods. While differences between the two-stage and one-stage methods are less evident in the few trial situations, the one-stage methods proved less frequent statistically significant.

Conclusion: Our findings support previous studies suggesting a high number of false-positive results in real meta-analyses of sparse data. In addition, the differences in the results provide evidence that method choice has a substantial impact on the outcome of meta-analyses and encourages the careful choice of an adequate method. Specifically in the situation of zero event trials, the BBM and BNHM-WIP appear to be promising candidates while using BBM, and additionally the PM and REML model for sensitivity analyses appears reasonable in the situation of few trials. Furthermore, Bayesian methods with carefully selected priors can be an alternative in the latter situation. Overall, the results advise against relying solely on the outcome of a single meta-analysis method in the case of sparse data situations.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.

The contribution has already been published: 70th Biometric Colloquium [1], 25. Jahrestagung des Netzwerks Evidenzbasierte Medizin [2]