gms | German Medical Science

Gemeinsame Jahrestagung der Gesellschaft für Medizinische Ausbildung (GMA), des Arbeitskreises zur Weiterentwicklung der Lehre in der Zahnmedizin (AKWLZ) und der Chirurgischen Arbeitsgemeinschaft Lehre (CAL)

25.09. - 28.09.2019, Frankfurt am Main

Comparing learning progress in knowledge between two major subjects in medical education – a retrospective, single center, mixed model analysis of progress testing results

Meeting Abstract

Search Medline for

  • presenting/speaker Dennis Görlich - University of Münster, Institute of Biostatistics and Clinical Research, Münster, Deutschland
  • Hendrik Friederichs - University of Münster, Institute of Education and Student Affairs, Münster, Deutschland; University of Münster, Study hospital Münster, Münster, Deutschland

Gemeinsame Jahrestagung der Gesellschaft für Medizinische Ausbildung (GMA), des Arbeitskreises zur Weiterentwicklung der Lehre in der Zahnmedizin (AKWLZ) und der Chirurgischen Arbeitsgemeinschaft Lehre (CAL). Frankfurt am Main, 25.-28.09.2019. Düsseldorf: German Medical Science GMS Publishing House; 2019. DocP-03-08

doi: 10.3205/19gma265, urn:nbn:de:0183-19gma2654

Published: September 20, 2019

© 2019 Görlich et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 License. See license information at http://creativecommons.org/licenses/by/4.0/.


Outline

Text

In medical education, progress testing is a tool to allow students to identify their knowledge level in comparison to a target knowledge level. Beside the individual benefits of assessing the individual performance, the collected progress test data can be used for research purposes. Nevertheless, progress test data yields certain limitations which needs to be considered during data analysis. Due to the voluntary nature of the test, participants provide between 1 and 5 data points, at maximum one test result per study year. This leads to several problems with respect to data analysis and effect size estimation. The collected data usually exhibits a high degree of missing data on the participant level. Individual reasons (unrelated to the test itself) may lead to missed test participations (missing at random), but also study related, or even test related reasons might apply (missing not a random). Standard methods usually need complete datasets and, though, ignore participants with at least one missing assessment. With respect to the estimation of effect sizes (ES) another issue arises: To quantify the ES for knowledge gain (between two assessment times) common measures, such as Cohen’s dz, quantify paired data appropriately, but needs complete data. Using unpaired estimates in missing data situations is usually not the appropriate method. To tackle both problems we applied a linear mixed model (MM) to the available data. The MM approach allows to (i) incorporate repeated measurements with missing data, i.e. participants are included into the analysis with all available data (ii) marginal means can be estimated for each time point (iii) ES can be constructed similar to Cohen’s dz based on the model contrasts.

We applied our methodology to a cohort of 2587 students (6324 test scores) participating in the progress test between 2012 and 2018 at the University of Münster. The overall knowledge gain (1st year to 5th year) was 34.8% (percent correct answers), which corresponds to an estimated ES of dMM=1.55. Comparing the two major clinical subjects, internal medicine and surgery, we observed a comparable pattern in knowledge gain, i.e. a continuous increase until the 4th year and a maintenance of knowledge in the 5th year without a relevant increase. The ES for internal medicine for the overall gain was dMM=1.44 while the ES for surgery was dMM=1.14. The direct comparison using contrast based statistical tests within the MM showed that internal medicine and surgery knowledge gain differed significantly within each study year (all p < 0.001), except the gains between the 4th and 5th year (pre-post group difference: 0.4%-points, p=0.6092). Although, the general pattern is comparable between subjects, this direct comparison was able to detect subtle differences in the learning pattern. In summary, the mixed model approach can incorporate all participants with their collected data and provide reliable results to analyze progress test data.

Figure 1 [Fig. 1]