gms | German Medical Science

Background: New cancer treatments are often promoted as major advances or “breakthroughs” after a significant phase III trial. Therefore, a clear and unbiased knowledge about the magnitude of the clinical benefit of approved treatments is important, so that the amount of reimbursement from public health insurance of new treatments can be assessed. To perform these evaluations of new treatments, two distinct “additional benefit assessment” methods are currently used in Europe.

The European Society for Medical Oncology (ESMO) developed the Magnitude of Clinical Benefit Scale Version 1.1 (ESMO-MCBSv1.1) classifying new treatments into 5 categories (substantial until low benefit) using a dual rule which considers the relative and the absolute benefit assessed by the lower limit of the 95% HR confidence interval or the observed absolute difference in median treatment outcomes, respectively [1], [2]. As an alternative, the German IQWiG compares the upper limit of the 95% HR confidence interval to specific relative risk scaled thresholds classifying new treatments into 6 categories (less until major added benefit) [3].

Until now, these methods have only been compared empirically [4].

Methods: We evaluate and compare the IQWiG method applying the defined relative risk scaled thresholds as well as transformed HR scaled thresholds [5] and ESMO-MCBS v1.1 by a simulation study with focus on time-to-event outcomes. The simulation includes aspects such as different censoring rates and types, incorrect HRs assumed for sample size calculation, informative censoring, and different failure time distributions. For a fair comparison between the two methods an additional placebo method as ground truth was used [4] reflecting a true (deserved) maximal score. The performance of the two methods is then assessed via ROC curves, sensitivity / specificity, and the methods' percentage of achieved maximal scores.

Results: The ESMO and IQWiG methods show similar performance in scenarios with a small effect (HR close to 1). In addition, with the implementation of the absolute benefit rule, ESMO-MCBSv1.1 achieves a downgrading of trials with a statistically significant but clinically insignificant benefit. Nevertheless, in scenarios with a quick disease progression, both methods provide a high rate of maximal scores.

In case of incorrect HR assumed for sample size calculation, both methods show liberal results in the sense that they assign higher scores than justifiable. In most scenarios with exponentially distributed failure times, IQWiGs method is more conservative than the approach from ESMO meaning that it assigns lower scores than justifiable. Moreover, in case of Weibull or Gompertz distributed failure times, both methods are liberal or IQWiG is more liberal than ESMO, respectively.

Conclusion: The results of the first step of a comprehensive comparison between the methods indicate that IQWiGs method is usually more conservative than ESMOs. Moreover, in some scenarios (e.g. quick disease progression or incorrect assumed HR) the IQWiG method is too liberal which can be reduced by using the transformed thresholds.

Nevertheless, further research is required, e.g. methods' performance under non-proportional hazards. In addition, the American Society of Clinical Oncology (ASCO) has developed another method using the HR point estimate, which remains to be compared to IQWiG and ESMO.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.