gms | German Medical Science

4th Research in Medical Education (RIME) Symposium 2015

19.03-21.03.2015, München

Empirical Structure of a Competency-Based Progress Test

Meeting Abstract

  • corresponding author presenting/speaker Andreas Möltner - Universität Heidelberg, Kompetenzzentrum für Prüfungen, Heidelberg, Germany
  • author Stefan Wagener - Universität Heidelberg, Kompetenzzentrum für Prüfungen, Heidelberg, Germany
  • author Sevgi Tımbıl - Dokuz Eylül Üniversitesi Tıp Fakültesi, Izmir, Turkey
  • author Maryna Gornostayeva - Universität Heidelberg, Kompetenzzentrum für Prüfungen, Heidelberg, Germany
  • author Jana Jünger - Universität Heidelberg, Kompetenzzentrum für Prüfungen, Heidelberg, Germany

4th Research in Medical Education (RIME) Symposium 2015. München, 19.-21.03.2015. Düsseldorf: German Medical Science GMS Publishing House; 2015. DocS1B3

doi: 10.3205/15rime14, urn:nbn:de:0183-15rime146

Published: March 12, 2015

© 2015 Möltner et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License. You are free: to Share - to copy, distribute and transmit the work, provided the original author and source are credited. See license information at



Introduction: In 2013 a formative competency-based written Progress test (144 MCQs) was developed. Items were constructed based on a two-dimensional blueprint. The first axis consisted of eight clusters of medical disciplines according to the German State Exam; the second axis consisted of five competency domains: theoretical clinical, practical clinical, communicative, scientific and professional behavior competencies. For each combination, the number of items was fixed corresponding to the blueprint [1]. The first progress test was carried out in November 2013; the second one will be run in November 2014. On this second progress test, 20 MCQs will be substituted with situational judgment items.

To provide meaningful feedback for students and faculties regarding test results on competency domains, sufficient reliability and discriminant validity of these domains are required. The aim of this study was to empirically analyze these properties of the competency domains.

Methods: Reliabilities of the competency domains were estimated by the coefficient “glb” [2]. To establish the discriminant validity of the domains, a discriminant analysis of principal components [3] of the data was conducted (“one against the rest” and multiclass discriminant analysis).

Results: All domains showed a high reliability for the 2013 data. The discriminant validity of the scientific competency domain was high. Theoretical clinical and practical clinical competencies could not be separated from each other, but differed significantly as a group from the other domains. Communication and professional domains showed a moderate independence and were significantly different from clinical and scientific competencies.

Results of the second progress test in November 2014 will also be presented.

Discussion: Items of the clinical competency domains, communication/professional competence, as well as scientific competence could be verified as three reliable and distinct clusters. With improved classification of the items, all five competency areas can be proved to be different constructs.


Wagener S, Gornostayeva M, Möltner A, Schultz JH, Brüstle P, Mohr D, Vander Beken A, Better J, Fries M, Gottschalk M, Günther J, Herrmann L, Kreisel C, Moczko T, Illg C, Jassowicz A, Müller A, Niesert M, Strübing F, Jünger J. Entwicklung eines formativen kompetenzbasierten Progresstests mit MCFragen von Studierenden - Piloteinsatz und Weiterentwicklung. Jahrestagung der Gesellschaft für Medizinische Ausbildung (GMA). Hamburg, 25.-27.09.2014. Düsseldorf: German Medical Science GMS Publishing House; 2014. DocP476. DOI: 10.3205/14gma191 External link
Jackson, P, Agunwamba, C. Lower bounds for the reliability of the total score on a test composed of non-homogeneous items: I: Algebraic lower bounds. Psychomet. 1977;42(4):567-578. DOI: 10.1007/BF02295979 External link
Jombart T, Devillard S, Balloux F. Discriminant analysis of principal components: a new method for the analysis of genetically structured populations. BMC Gen. 2010;11:94. DOI: 10.1186/1471-2156-11-94 External link