gms | German Medical Science

Background: Many publications theoretically discuss possible differences in effect estimates from randomized controlled clinical trials (RCT) and non-randomized studies (NRS) performed under routine care conditions (i.e. effectiveness-efficacy gap). The potential drivers for the difference in routine care are patient characteristics, organization of intervention (e.g. required expertise), flexibility of intervention delivery, adherence/fidelity and context/setting. In addition to these drivers for the effectiveness-efficacy gap, differences in effect estimates may arise because of methodological aspects, in particular unmeasured confounding, selection bias and analyses that rather examines the effect of starting and adhering to intervention than allocation to intervention in NRS.

Objective: Our objective was to empirically analyze differences between effect estimates from RCT and NRS performed under the usual circumstances of health care practice (uc-NRS).

Methods: We performed a systematic literature search in Pubmed (02/2019). We included studies that reported data enabling us to compare risk ratios from RCTs with risk ratios from uc-NRS for binary incidence measures (e.g. diseases specific mortality). We defined “uc-NRS” as studies in which the allocation was not investigator controlled and that were based on routinely collected health care data (e.g., registries or insurance claims). Studies were selected by two reviewers independently. Data were extracted by one reviewer and verified by a second reviewer.

We assessed the agreement between RCT and uc-NRS with a variety of measures. We counted the number of conflicting effect directions and the number of overlapping 95% confidence intervals (CIs). We performed these comparisons for unadjusted as well as adjusted uc-NRS estimates.

Further meta-epidemiological measures (e.g. expected vs observed overlap of CIs) and an exploration of possible reasons for difference (e.g. confounding bias, selection bias, analysis perspective) will be presented at the conference.

Results: We included eleven studies. The studies encompassed orthopedics (n=2), heart diseases (n=6), colorectal cancer (n=1), diabetes (n=1) and congenital diseases (n=1). Three studies considered a drug therapy, three a medical device and five a medical procedure/technique. The data base was a registry in seven studies and administrative health care data in four studies. Patient characteristics often differed between uc-NRS and RCT.

In four studies, pooled effects of uc-NRS were compared with pooled effects of RCTs.

The comparison of unadjusted effect measures included eight clinical questions. The effects were in same direction for all clinical questions (8 of 8) and CIs overlapped in most cases (6 of 8).

The comparison of adjusted effect measures included seven clinical questions. The effects were in the same direction in most cases (5 of 7). CIs overlapped in all but one case (6 of 7). As expected, the adjusted uc-NRS estimates were closer to RCT estimates than the unadjusted estimates (4 of 4 that reported unadjusted as well as adjusted measures).

Discussion: In theory, differences between RCTs and uc-NRS are well justified. We found a difference between RCT and uc-NRS, which would have resulted in a different conclusion, in only two unadjusted effect estimates and in no adjusted effect estimate. This finding is in accordance with a large amount of previous meta-epidemiological research comparing NRS and RCT effect estimates. Our finding is limited by the small sample size.

Implications for practice: Further meta-epidemiological research, in particular an in-depth analysis of drivers for differences between uc-NRS and RCT is necessary. Such research could increase the knowledge on when and how uc-NRS can be set-up that could mirror or extrapolate the effects from RCTs. A better understanding of the premises and requirements for using uc-NRS as an alternative or complement to RCT would increase the value of uc-NRS for decision-making.