gms | German Medical Science

Research in Medical Education – Chances and Challenges International Conference

20.05. - 22.05.2009, Heidelberg

Survey on the quality of randomized trials in medical education research

Meeting Abstract

Suche in Medline nach

  • corresponding author presenting/speaker Joerg Marienhagen - University Regensburg, Medical Faculty, Regensburg, Germany
  • author presenting/speaker Hans Gruber - University of Regensburg, Institute of Educational Science, Regensburg, Germany

Research in Medical Education - Chances and Challenges 2009. Heidelberg, 20.-22.05.2009. Düsseldorf: German Medical Science GMS Publishing House; 2009. Doc09rmeA2

doi: 10.3205/09rme02, urn:nbn:de:0183-09rme026

Veröffentlicht: 5. Mai 2009

© 2009 Marienhagen et al.
Dieser Artikel ist ein Open Access-Artikel und steht unter den Creative Commons Lizenzbedingungen ( Er darf vervielfältigt, verbreitet und öffentlich zugänglich gemacht werden, vorausgesetzt dass Autor und Quelle genannt werden.



Question: Although randomized trial designs are considered to be the golden standard for experimental studies, its role in medical education research is still controversial. Drawbacks in the quality of randomized trials limit the impact on a best evidence medical education. The aim of our study is to provide a survey of the quality of published randomized medical education trials. Criteria analyzed are:

reporting quality,
methodological rigour
internal validity.

Method: Six recent volumes (2002-2007) of four leading journals (Medical Education, Medical Teacher, Academic Medicine, Teaching and Learning in Medicine) were surveyed. A total of 70 randomized trial reports were assessed using 37 descriptors of reporting quality, methodological rigour and internal validity. Descriptive and exploratory methods were applied, including significance testing (Chi-square, Mann-Whitney- U, Fishers exact test, α=.05).


Less than 10% of the publications included randomized medical education trial reports; there was no significant increase in the period 2002-2007. The 70 selected papers covered many different skills and educational interventions.
87% were monocenter trials, most using two-arm parallel designs.
45 publications reported significant results, but without an association between significance and sample size (p=.75).
The assessment of both the internal validity and the methodological rigour was often hampered by missing information.
Less than 25% of the studies included a description of the randomization procedure or the concealment of allocation.
Blinding was only reported in 26 articles.
In 30 trials, no information about an intention-to treat-analysis was given.
Complete follow-up was reported only in 11 articles.
53 papers presented no description of the drop-outs; 37 did not inform about handling of missing values, 25 publications completely ignored this issue.
A priori sample size estimations were made in only 17 reports, post hoc power analysis in two studies.
Effect sizes were reported in 30 studies.
A description of the primary endpoint was found in 48 publications. Multiple endpoints were analyzed in 56 trials, but only few addressed the alpha inflation problem.

In total, an increase over time of the reporting quality could not be observed.

Conclusion: Many publications in medical education, even in top journals, still have a poor reporting quality of published randomized trials. Thus, the assessment of the internal validity was difficult. The survey identifies a number of substantial threads to validity, however, which may affect the acceptance of randomized trials in a best evidence medical education framework. Improvements of the methodological standards and of the reporting quality thus are major requirements for improving the professional quality of medical education research.