gms | German Medical Science

Question/Objective: Student performance is a reflection of the quality of teaching. However, commonly used approaches to calculate learning gain are based on Classical Test Theory (CTT) and often fail to meet the statistical requirements necessary for a robust assessment in a pre-/post-test design. Our study aimed to validate a newly developed metric called the “Weighted Gain Score” (WGS) [1], which is grounded in CTT, by comparing it with a calculation method based on Item Response Theory (IRT) known as “Rasch Learning Gain” (RLG) [2].

Methods: The WGS metric was developed as a weighting coefficient, which leads to more accurate learning gain calculations compared to conventional CTT-based approaches. The weighting coefficient includes a variable µ that allows to linearly increase or decrease the calculated learning gain. To understand its role, we created multiple versions of the metric, each with a different value of µ, ranging from 30 to 80. We used these versions to analyze an empirical dataset (n=170) and a simulated dataset (n=1000) and then compared the results with a RLG-based analysis of both datasets.

Results: Despite the variations in µ, WGS is significantly correlated (r=.93, p<001) with the IRT-based approach for both datasets. In the empirical dataset, both RLG and WGS confirmed an interaction effect that had been originally determined [1]. In the simulated dataset, both RLG and WGS showed a non-significant interaction effect. In other words, RLG and WGS have very similar inferential relationships. Regarding absolute gain values, we found that a µ value of around 30 is most similar to RLG, while WGS versions with higher µ values compute lower learning gains than RLG.

Discussion: The WGS approach is capable of calculating a learning gain that closely approximates the gain calculated by the IRT-based metric. In addition, our findings demonstrate that the gain values calculated by WGS can be adjusted to better suit a given testing situation through variations in the variable µ. However, our analysis indicates that modulating µ does not affect the inferential insights provided by the metric. Moreover, the strong correlation between RLG and WGS suggests that the newly developed metric produces inferential results that closely resemble those obtained by an IRT-based metric.

Take home messages: The WGS metric was found to be a reliable and valid method for assessing learning gain, and can be adjusted to fit different testing situations. The strong correlation between WGS and the IRT-based RLG metric indicates that WGS provides similar inferential results to a gold standard metric in the field. Mathematically speaking, the metric is straightforward to calculate and widely applicable at a routine level for individuals involved in quality assurance and curriculum evaluation.