gms | German Medical Science

Recently, the topic to address clinical relevance on the top of statistical significance in the analysis of RCTs has been paid increasing attention. The increased interest is related to the fact that in political decisions on reimbursement a pure proof of efficacy is no longer regarded as sufficient, but that a clear clinical benefit has to be proven. For example, the German IQWiG denied recently a clinical benefit of an anti-Alzheimer drug with respect to three patient oriented outcomes, because the lower bounds of the confidence interval of the effect size was below 0.2 for all three corresponding scales in a meta-analysis.

In this talk we would like to quantify the need for and the impact of post hoc application of formal criteria to assess clinical relevance, if applied in well conducted, single RCTs. We focus on assessment of clinical relevance based on the global treatment effect, which is the method of choice if response cannot be defined at the individual level. The two criteria we consider are the comparison of the lower bound of the two-sided 95% CI for the treatment effect with a pre-specified threshold – which is equivalent to testing a shifted null hypothesis – and the comparison of the treatment effect with a pre-specified threshold.

Our results suggest that there is little need for a formal assessment, if the irrelevance limit is lower than 25% of effect assumed in the power calculation. Furthermore, in most situations a possible gain in controlling the rate of accepting a new treatment with an irrelevant effect is outperformed by the loss in power to accept treatments with relevant effects. However, these results depend on the assumed costs for accepting an “irrelevant” treatment and rejecting a “relevant” treatment. We assumed equal costs, but as long as an “irrelevant” treatment does no harm, the costs are probably lower, and any formal assessment of clinical relevance do even more harm.