gms | German Medical Science

Background: The identification of curricular effectiveness using objective measures of learning outcome is widely used in the competency-orientated development of educational processes. Pre-test/post-test designs are particularly common as methods of assessing knowledge gain. Since the maximum score of a test instrument is always 100%, a strong negative correlation is observed between students’ absolute gain and their pre-test score. The aim of this study was to compare an established mathematical model for relative change with a newly developed one to calculate knowledge gain to be recorded independently of the absolute knowledge gain as well as of pre-test scores.

Material and methods: We generated a fictitious dataset to simulate pre-test scores ranging from a minimum of 10% to the maximum 100%, combined those with pre-defined absolute gains of 10, 20, 30, and 40%, and subsequently evaluated the two mathematical models for the calculation of knowledge gain: G1 defined as normalized gain (dividing the absolute gain by the maximum possible gain) and G4 defined as modified absolute gain (multiplication with a weighting factor). An empirical quasi-experimental one-group study (pre- and post-test scores from a teaching module in surgery in the fifth year of the degree course in human medicine) was used to verify the results. ANOVA was employed to compare significant differences in calculated knowledge gains as well as any interaction effects of gains with pre-test scores.

Results: For all four pre-defined absolute gains, the slope of G1 rose exponentially with increasing pre-test scores, whereas G4 correlated in a linear fashion. The gains calculated in both models G1 and G4 diverged in all combinations of absolute gains and pre-test scores. In the empirical dataset, three groups of absolute gains were defined by quartiles for absolute gain (low, medium, high). Calculation of gain using G1 and G4 led to significantly different mean scores in all of these three groups (p<0.001). In comparison, G4 attenuated the influence of the pre-test score better than G1 (see figure 1 [Fig. 1]).

Conclusion: This work adds to the area of research how to calculate knowledge gain for one-group pre-test/post-test study designs. For heterogeneous pre-test scores, G4 is considered superior to G1.Unlike the over-emphasizing effect of G1, G4 amplifies the calculated gain for high pre-test scores and reduces the calculated gain for low pre-test scores. Thus, G4 corrects for pre-test bias better than G1. This effect can be determined on both the group as well as on an individual level.