gms | German Medical Science

21. Jahrestagung des Deutschen Netzwerks Evidenzbasierte Medizin e. V.

Deutsches Netzwerk Evidenzbasierte Medizin e. V.

13. - 15.02.2020, Basel, Schweiz

Disagreement between non-randomized real-world evidence and randomized trials: a meta-epidemiological study on disagreement and potential causes for disagreement

Meeting Abstract

Search Medline for

  • Tim Mathes - Universität Witten/Herdecke, Evidenzbasierte Versorgungsforschung (IFOM), Deutschland
  • Tanja Rombey - Universität Witten/Herdecke, Evidenzbasierte Versorgungsforschung (IFOM), Deutschland
  • Dawid Pieper - Universität Witten/Herdecke, Evidenzbasierte Versorgungsforschung (IFOM), Deutschland

Nützliche patientenrelevante Forschung. 21. Jahrestagung des Deutschen Netzwerks Evidenzbasierte Medizin. Basel, Schweiz, 13.-15.02.2020. Düsseldorf: German Medical Science GMS Publishing House; 2020. Doc20ebmPP10-04

doi: 10.3205/20ebm053, urn:nbn:de:0183-20ebm0536

Published: February 12, 2020

© 2020 Mathes et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 License. See license information at



Background/research question: Meta-epidemiological studies comparing effects from randomized controlled trials (RCT) and non-randomized studies (NRS) have shown heterogeneous results. Many publications theoretically discuss possible causes for differences in effect estimates from RCT and NRS performed during the routine delivery of health care.

Our objective was to assess disagreement between real-world-data based NRS and RCTs and to identify internal and external validity-related causes for the disagreement.

Methods: We searched PubMed in February 2019. We included studies that enabled a comparison of risk ratios from NRS and RCTs for binary incidence measures. Two reviewers selected studies independently.

We assessed the agreement with a variety of measures and assessed potential causes for disagreement using items from ROBINS-I and PRECISE-2. A second reviewer verified all tasks.

Results: We included 12 studies. The comparison of unadjusted effect measures included 8 questions. The effects had the same direction for all questions and 95% confidence intervals (95%-CIs) overlapped in 6 cases. The comparison of adjusted effect measures included 7 questions. The effects had the same direction in 5 of 7 cases. The 95%-CIs overlapped in 6 of 7 cases. Moreover, the predicted vs. observed overlap of 95%-CIs suggested no systematic difference.

In addition to confounding for the unadjusted estimates, the main potential cause for disagreement were differences in the classification of interventions, the patient population and the setting.

Conclusion: We have found a disagreement, which would probably have resulted in a different conclusion, in only two unadjusted effect estimates and in no adjusted effect estimate. In all studies, we found sources for risk of bias (RoB), for which the actual impact on the results is unpredictable in practice. Noticeable, the existence of sources for RoB that result in relevant disagreements could probably have been foreseen. We believe that this is an argument for considering NRS as a substitute for RCTs only in very carefully selected cases, when either there is very low RoB or there is the possibility that RoB can be sufficiently minimized in the analysis.

In all studies, we found potential threads to the external validity, for which the actual impact on results is difficult to predict. We believe that this is an argument for considering complementing RCTs with NRS whenever there is a possibility that effects might relevantly differ under routine care conditions.

Competing interests: Es bestehen keine Interessenkonflikte in Zusammenhang mit der eingereichten Arbeit