gms | German Medical Science

Objective: TELLme, an online learning platform, has recently been implemented for the integrated, modular undergraduate medical curriculum at the Charité – Universitätsmedizin Berlin. The platform allows medical students to formatively self-test with multiple-choice questions (MCQs) linked to specific cognitive learning objectives (LOs). While the benefits of test-based learning and formative feedback for learners are well established, this study explores the potential of a formative MCQ-based platform to provide feedback to the curriculum and teachers.

Methods: The analysis is based on SQLite queries of two million initial certainty-based assessment responses collected in TELLme between 04/2021 and 04/2022, comprising 5376 MCQs on 2596 LOs. For each LO, the proportion of incorrect answers (0.3±0.14) and incorrect answers of medium certainty (0.04±0.024) and high certainty (0.035±0.024) were evaluated. The data set was then reduced to LOs for which at least 200 MCQ responses were available and whose certainty statements deviated by more than 1 SD from the mean. The selected conspicuous MCQs were examined in more detail qualitatively with regard to the alignment of LOs, MCQs and teaching materials.

Results: We identified 264 LOs (10% of all curricular LOs) with a manifest clustering in 5 modules (Blood and Immune System; Heart and Circulation; Nutrition, Digestion and Metabolism; Respiration; Kidney/Electrolytes). An analysis of module core subjects provides further insights, as illustrated for physiology and pulmonology as core subjects of the module “Respiration” 17% of all physiology LOs (39/226) and 18% of all pulmonology LOs (4/22) show presumably certain incorrect response behaviour. A qualitative probing of the associated learning objectives, teaching materials and self-testing MCQs indicates a suboptimal alignment of the identified learning objectives.

Discussion: Our study shows that student responses in a formative MCQ-based learning platform reveal curricular misalignments between LOs, MCQs and teaching materials, thereby providing valuable feedback to the curriculum and its teachers. A qualitative approach will allow more insight into the how and why, for example, through focus groups with teachers and curriculum managers.

Take Home Message: Data sets from formative self-tests are not only valuable for learners but can also be used as a source of feedback for the curriculum and their teachers.