gms | German Medical Science

Background/research question: Meta-epidemiological studies comparing effects from randomized controlled trials (RCT) and non-randomized studies (NRS) have shown heterogeneous results. Many publications theoretically discuss possible causes for differences in effect estimates from RCT and NRS performed during the routine delivery of health care.

Our objective was to assess disagreement between real-world-data based NRS and RCTs and to identify internal and external validity-related causes for the disagreement.

Methods: We searched PubMed in February 2019. We included studies that enabled a comparison of risk ratios from NRS and RCTs for binary incidence measures. Two reviewers selected studies independently.

We assessed the agreement with a variety of measures and assessed potential causes for disagreement using items from ROBINS-I and PRECISE-2. A second reviewer verified all tasks.

Results: We included 12 studies. The comparison of unadjusted effect measures included 8 questions. The effects had the same direction for all questions and 95% confidence intervals (95%-CIs) overlapped in 6 cases. The comparison of adjusted effect measures included 7 questions. The effects had the same direction in 5 of 7 cases. The 95%-CIs overlapped in 6 of 7 cases. Moreover, the predicted vs. observed overlap of 95%-CIs suggested no systematic difference.

In addition to confounding for the unadjusted estimates, the main potential cause for disagreement were differences in the classification of interventions, the patient population and the setting.

Conclusion: We have found a disagreement, which would probably have resulted in a different conclusion, in only two unadjusted effect estimates and in no adjusted effect estimate. In all studies, we found sources for risk of bias (RoB), for which the actual impact on the results is unpredictable in practice. Noticeable, the existence of sources for RoB that result in relevant disagreements could probably have been foreseen. We believe that this is an argument for considering NRS as a substitute for RCTs only in very carefully selected cases, when either there is very low RoB or there is the possibility that RoB can be sufficiently minimized in the analysis.

In all studies, we found potential threads to the external validity, for which the actual impact on results is difficult to predict. We believe that this is an argument for considering complementing RCTs with NRS whenever there is a possibility that effects might relevantly differ under routine care conditions.

Competing interests: Es bestehen keine Interessenkonflikte in Zusammenhang mit der eingereichten Arbeit