gms | German Medical Science

GMS Zeitschrift für Medizinische Ausbildung

Gesellschaft für Medizinische Ausbildung (GMA)

ISSN 1860-3572

Using the Progress Test Medizin (PTM) for evaluation of the Medical Curriculum Munich (MeCuM)

research article medicine

  • corresponding author Ralf Schmidmaier - Klinikum der Universität München, Medizinische Klinik Innenstadt, Schwerpunkt Medizindidaktik, München, Deutschland
  • author Matthias Holzer - Klinikum der Universität München, Medizinische Klinik Innenstadt, Schwerpunkt Medizindidaktik, München, Deutschland
  • author Matthias Angstwurm - Klinikum der Universität München, Medizinische Klinik Innenstadt, Schwerpunkt Medizindidaktik, München, Deutschland
  • author Zineb Nouns - Charité - Universitätsmedizin Berlin, Dieter Scheffner Zentrum für medizinische Hochschullehre und evidenzbasierte Ausbildungsforschung, Progress Test Medizin, Berlin, Deutschland
  • author Martin Reincke - Universität München (LMU), Medizinische Fakultät, Studiendekanat, München, Deutschland
  • author Martin R. Fischer - Private Universität Witten/Herdecke, Fakultät für Gesundheitsberufe, Institut für Didaktik und Bildungsforschung im Gesundheitswesen, Witten, Deutschland

GMS Z Med Ausbild 2010;27(5):Doc70

doi: 10.3205/zma000707, urn:nbn:de:0183-zma0007072

This is the translated version of the article.
The original version can be found at: http://www.egms.de/de/journals/zma/2010-27/zma000707.shtml

Received: March 29, 2010
Revised: August 10, 2010
Accepted: August 20, 2010
Published: November 15, 2010

© 2010 Schmidmaier et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by-nc-nd/3.0/deed.en). You are free: to Share – to copy, distribute and transmit the work, provided the original author and source are credited.


Abstract

Aims: The Medical Curriculum Munich (MeCuM) has been implemented since 2004 and was completely established in 2007. In this study the clinical part of MeCuM was evaluated with respect to retention of the knowledge in internal medicine (learning objectives of the 6th/7th semester).

Methods: In summer of 2009 and winter of 2009/2010 1065 students participated in the Progress Test Medizin (PTM) from Charité Medical School Berlin. Additionally the students answered a questionnaire regarding the acceptance and rating of the progress test and basic demographic data.

Results: The knowledge of internal medicine continuously increases during the clinical part of the medical curriculum in Munich. However, significant differences between the sub-disciplines of internal medicine could be observed. The overall acceptance of the PTM was high and increased further with the study progress. Interestingly, practical experiences like clinical clerkships positively influenced the test score.

Conclusions: The PTM is a useful tool for the evaluation of knowledge retention in a specific curriculum.

Keywords: Evaluation, Progress Test, Internal Medicine, Knowledge Retention


Introduction

The Problem

The goal of medical education is to impart practical skills and knowledge enduringly [1]. In contrast to summative examinations relevant to passing examinations, the formative Medicine Progress Test (MPT) examines spontaneously retrievable, and thus enduringly stored knowledge, because the students do not prepare [2]. The main objective of progress tests is individual feedback to students [3]. But the data can also be analysed with respect to curriculum evaluation. Progress testing is a reliable tool that is used across institutions and countries [4]. Applied longitudinally (analysing knowledge growth curves), progress tests can compare various curricula or curricular changes [5], [6], [7]. Here, longitudinal growth of knowledge is a more stable and reliable parameter than measuring of individual points of reference [8]. In previous studies, a longitudinal survey was carried out and growth of knowledge measured [5], [6], [7], [8]. As part of this project at the Ludwig-Maximilians-Universität (LMU) Munich, a cross-sectional analysis was performed and the permanence of acquired knowledge in specific technical topics checked. Here we analyse an example from “conservative medicine”. In the clinical part of the Medical Curriculum Munich (MeCuM), after a share semester on the basics of clinical medicine in the 6th semester, students are divided into two streams, one on conservative and one on surgical medicine and swap in the 7th semester. Since the abolition of the State Examination before the start of the Practical Year, once students have received their certificates in internal medicine and surgery, they will not be examined in these subjects until the Part 2 of the Medical Examinations at the end of their undergraduate studies and received no feedback regarding these subjects. The Medicine Progress Test used by the Charité Berlin was introduced at the LMU Munich in the 2009 summer semester. The curriculum planners and co-managers of MeCuM until then had not known how knowledge of the individual subjects and specialist disciplines was progressing, whether knowledge was regressing, was maintained or even increased.

Goals

The aim of this study was to evaluate the permanence of knowledge gained in the “Conservative Medicine” MeCuM module.

Problem

How big is the gain of spontaneously accessible knowledge in relation to the learning targets of the “Conservative medicine” module students at the LMU acquire and how enduring is this knowledge in the subsequent course of their studies? Is MPT helpful for the evaluation of a medical curriculum?


Methods

Medicine Progress Test at LMU Munich

Figure 1 [Fig. 1] shows the rotation of student cohorts in MeCuM and the measurement time points. Students from Module 1 are distributed between Module 2 and 3, then swap in the following semester. The following semester, students are distributed among Modules 4 and 5, then swap the following semester. In the first week of the 2009 summer semester and 09/10 winter semester, the the PTM (http://ptm.charite.de/) was introduced obligatorily at three points in the clinical part of undergraduate studies: T1: start of Module 2 (“Conservative Medicine”, M2; cohorts A and B), T2: Start of Module 4 (“Nervous System and Sensorium”, M4; cohorts C + D) and T3: during the Practical Year. In the PY cohort (T3; cohorts E + F), only those students in the Practical Year or module 6 were examined who were not abroad and were registered for the State Examination Repetitorium (LMU-StaR). LMU-Star is an optional blended learning concept which accompanies the PY and is used by the majority of students in the PY. An analysis of the basic data (age, gender, school leaving grade, grade of the First Medical Examination) showed that the LMU-Star participants did not differ in a statistically significant way from the rest of PY students. Over the two measurement time points, all students in the clinical stage of studies beyond Module 1 were tested once only (N=1065; Summer Sem. 09: T1=249, T2=202, T3=156; Winter Sem. 09/10: T1=197, T2=190, T3=71). For the evaluation, the MPT questions were assigned to the MeCuM modules, in particular Module 2 “Conservative Medicine”, by the organiser of Module 2. The internal medicine sub-disciplines (endocrinology, gastroenterology, infectious diseases, cardiology, nephrology, oncology, pneumology, rheumatology) were categorised according to the learning targets of Module 2. Immediately after completing the MPT, students filled in an evaluation sheet. This included both questions on the MPT and other questions about personal data. The evaluation response rates were T1=94%, T2=92%, T3=94% (Summer Sem. 09), T1=79%, T2=82%, T3=89% (Winter Sem. 09/10 ), with a mean total of 88.3%.

Statistics

The statistical analysis and generation of the graphics was done in the Open Source statistical package R (http://www.r-project.org/), Version 2.9.0, and using Microsoft Excel. As a normal distribution could be shown for the PTM results, the calculation of correlation coefficients was done according to Pearson. For most collected variables, mean and standard deviation (SD) were also determined. The p-values listed were determined, given the normal distribution, using a t-test, otherwise using a Wilcoxon rank-sum test. To calculate the correlation of MPT performance with items of the questionnaire (leaving grades, First State Examination, self-assessment, etc.), the rank correlation coefficient Spearman’s rho was used. The box plots show the range between the 25% and 75% percentile, the line within the box indicates the median. If the notches of two groups intersect, the difference is significant.


Results

Implementation, Results and Acceptance of the MPT at the LMU

The implementation of the MPT was, in accordance with the guidelines of the Charité - University Medicine Berlin, without problem from an organisational point of view. Regarding overall performance, the LMU students with an average of 46.3 points (SD=21.7) at the beginning of the 6th semester, 61.7 points (SD=21.8) at the beginning of the 8th semester and 77.9 (SD=24.9) at the beginning of the 10th semester are comparable with the students of the control programme in Berlin (36.8, 57.7 and 71.2 points, SD=18.2, 22.6 and 27.2) and the students at the University of Witten/Herdecke (45.8 or 55.3, SD=20.7 and 17.4, no data available for the 10th semester). The acceptance of the MPT by LMU students was very high. The question on “I consider individual feedback on my performance important [Likert scale: 1=strongly agree, 6=strongly disagree]” was answered, on average, with 1.73 (N=894, SD=0.91). This desire fore feedback increases as studies progress and reaches a value of 1.20 at the beginning of the PY (T3). The MPT was globally assessed (“I rate the Medicine Progress Test with the following grade:”) with the average grade of 2.27 (N=917: SD=0.92). This leads into an overwhelming endorsement of the solid implementation of the MPT at the LMU (“The Medicine Progress Test should be introduced at the LMU. [Likert scale: 1=strongly agree, 6=strongly disagree]”). Accordingly, in particular the T3 cohort with 67% argues for conducting it once per semester (“How often do you think the Medicine Progress Test should be carried out during an undergraduate degree?”, [Each semester - once a year - less often - never]). Only 4% of students of this cohort suggested “less often” or “never” (T1: 22% T2: 12%).

Knowledge Progression in “Conservative Medicine” in the MeCuM

Using the data of the large cross-sectional analysis, an evaluation of Module 2 “Conservative Medicine” was carried out. Based on subjective judgements by professors at the LMU and in analogy to a large analysis of knowledge on internal medicine amongst practicing physicians [9] which demonstrated a significant inverse correlation between the test result and time elapsed since the certification examination, it was investigated if knowledge in “conservative medicine” decreases in the subsequent progression of studies. Moreover, the curriculum committee assumed and had the subjective experience during Module 3 “Medical Surgery”, many learning targets of Module 2 “Conservative Medicine” were also taught. Through such non-targeted redundancies, serious inhomogeneities are created in the prior knowledge of students, making it teaching in small-groups difficult (seminars, problem-oriented learning tutorials).

Figure 2 [Fig. 2] shows that the knowledge of “Conservative Medicine” (Module 2) increases in Module 2 and 3 (T2 versus T1 comparison) but contrary to initial expectations, continues to increase in the subsequent course of study until the beginning of the Practical Year (T3). The results of the 2009 summer semester and the 09/10 winter semester were pooled and the result is shown as a percentage of points achieved. The number of questions about “Conservative Medicine” and therefore the absolute number of achievable points are different in each semester. Figure 3 [Fig. 3] shows that in respect to Module 3, a significant increase in knowledge of “Conservative Medicine” occurs (A vs. B). After completion of both modules, the results return to an identical level (C, D) and continue to increase up until T3 (E/F) although this increase is not statistically significant. The learning targets in Module 2 “Conservative Medicine” are divided in the various sub-specialisations of internal medicine. It was therefore also investigated whether there are differences in the knowledge gain between these specialisations. Figure 4 [Fig. 4] shows that subjects with higher knowledge growth (for example cardiology and rheumatology) can be distinguished from subjects in which knowledge regarding specific learning targets tested in the Progress Test is virtually unchanged (for example infectious diseases and nephrology). Further analysis shows (see Figure 5 [Fig. 5]) that prior knowledge from the preclinical stage of studies and Module 1 “Clinical Basics” is low internal medicine with respect to the specific learning targets. Cardiology aside, only endocrinology, oncology and pneumology show comparably high initial levels. Regardless of this prior knowledge, there is a marked increase in knowledge in Module 2 in disciplines such as cardiology or rheumatology. These show virtually no further increase in the further course of studies. On the other hand, students make up for their lack of knowledge in nephrology and infectious diseases as their studies progress. Here the significant growth occurs only after completion of Module 2. In an interdisciplinary subject such as infectious diseases, this is can be easily understood.

Influencing Factors for MPT Results

The measured cohort was checked for the actual qualities the MPT was measuring for validity testing or more specifically, the participants’ parameters which influenced the test results.

Performance

When looking at the knowledge of Module 2 “Conservative Medicine”, a significant correlation (rho=-0.38) can be seen between the performance in the Module 2 exam and the MPT points in internal medicine. The Module 2 exam is a summative written examination to be held in two parts (at the middle and end of Module 2) and contains both multiple-choice (MCQs) and open questions. Also, the achieved MPT points in internal medicine also correlated significantly with school leaving grade (rho=-0.26) or the First Part of the Medical Examination (rho=-0.37). The gender of the participants had no influence on the result.

Motivation

Another factor that was checked was motivation to participate. As a surrogate parameter, the subjective global assessment of the MPT (“grading”, see above) and the desire for its introduction at the LMU (“implementation”, see above) was used. Here better subjective evaluation results is linked to better performance. A significant correlation (rho=-0.29) can be observed here, as well as for the (rho=-0.36) correlation between self-assessment (“How do you assess your results in the Medicine Progress Test/Internal Medicine overall? In my year, I belong to the...” [Likert scale (1-5): very good, good, average, poor, very poor participants]) and the MPT result. This correlation is reflected almost identically for the items “Conservative Medicine” but irrespective of whether the students had already taken Module 2 “Conservative Medicine”.

Experience

According to self-evaluation, the MPT displays the cross-section of the knowledge level, which is expected of a graduate on their first day at work. Many of the items contain a clinical case vignette. It should therefore be investigated whether, in addition to individual performance (reflected by the school leaving grade or the examination at the end of the pre-clinical phase), the clinical experience of students also affects the MPT test result. A significant correlation (p <0.0001) between the completion of the 4 month work placement (Famulatur) and the overall MPT result. The improved results in “Conservative Medicine” through the PY tertial in internal medicine at the end of undergraduate studies turned out not to be significant (n=60).


Discussion

The Medicine Progress Test is a formative feedback tool for students, which checks the knowledge that a doctor on their first day should possess [2]. If one brings together all students of a university at a certain point in time and only examines particular area, then the progression of expertise during the course of their studies and thus the curriculum can be evaluated using the same test. The MPT so far has only been specifically used for curriculum evaluation in a small study in 2004 in the reformed degree course at the Charité University Berlin Medical School. In a UK study, final year students were tested on their knowledge on musculoskeletal topics [10]. The most comparable study comes from the Netherlands. In this study the growth of knowledge in psychiatry and behavioural sciences was analysed in a problem-based, student-centred curriculum [11]. Here curricular changes were because no further knowledge growth could be detected in the last two years of study as would have been the expectation in a PBL curriculum. Since the students were largely self-selecting the date of when they tackled learning targets, an analysis of the permanence of acquired knowledge was not possible here. The German Medicine Progress Test, Charité University Berlin Medical School has already been used for curriculum evaluation [12] but mainly for interdepartmental comparison and not to analyse individual components of a specific curriculum.

Of course inferences on the quality of the curriculum can only be drawn very indirectly through the evaluation method described in this study. However, the permanent retention of knowledge is an important outcome parameter of a degree program. Communication and practical skills in this approach, however, are almost completely ignored. The significant impact of the 4 month work placement could either point to the fact that the largely case-based German MPT to a relevant extent tests clinically relevant practical knowledge or that an increased amount of factual knowledge is learned during the 4 month work placement. Holding the 4 month work placement early could, however, be evidence of an above-average motivation of this group of students. While the school leaving grade and the subjective self-assessment correlate with the test result, attitude towards MPT is not a major factor. Being critical towards our own work, it must be noted that for organisational reasons only those students could participate at point T3 who were taking part in the Munich State Examination Repetitorium LMU-StaR. However, in terms of basic data (age, gender, school leaving grade, grade of the First Part of the Medical Examination) this subgroup did not differ in a statistically significantly way from the other students, meaning the selection bias is probably low. Future investigations should give more weight to examining motivational aspects in this regard. Overall, a longitudinal analysis appears to be the appropriate method to for answering this question, however, there is something attractive about getting results quick. In longitudinal studies, curricular reform could onlz take place after several years and its effectiveness again would be measured after several years.

The aim of this cross-sectional analysis of all students in the clinical part of their studies at the LMU was the evaluation of Module 2 “Conservative Medicine” and clarification of whether the Medicine Progress Test can be used as an evaluation instrument for such a purpose. The MeCuM is organised into autonomous modules in which basic teaching in internal medicine and surgery is taught soon after the clinical core subjects in Module 1. There always has been, and still is, the fear amongst teachers and students that this knowledge is mostly lost by the start of the PY. Our cross-sectional analysis using MPT showed, however, that knowledge of “Conservative Medicine” continues to expand even after completion of Module 2 “Conservative Medicine”. Comparable results of knowledge increase of 16.6% in absolute terms in the permanence test after eight months were found in a Mexican study of 584 medical students in their 3rd year of training in pharmacology [13]. Here, the students had deliberately prepared for the exam and were examined again after eight months without preparation. This setting broadly corresponds to the situation under investigation at the LMU. Interestingly, these results from Mexico could only be found in a newly-designed pharmacology course; the old, traditional course did not display knowledge gain. Similar results were also found [14] in a North American study. Other analyses, mainly on basic biomedical knowledge, showed between 2.9% and 35% of knowledge loss after 15 to 21 months [15], [16], [17]. In a well planned current study, three subjects (immunology, physiology, and neuroanatomy) were compared [18], showing dramatic loss of knowledge of 17.6%, 19.4% and 52.7% after 10-11 months. The differences were not related to the results in the primary test or the acceptance of the courses by students. The acquisition of permanent knowledge is influenced by active learning and the nature of the teaching material (procedural vs. declarative, general vs. specific). Following a multi-disciplinary surgery course on mesorectal excision in Canada, there was no significant loss of knowledge even after a year [19]. Due to the different curriculum structures and sequence of the implementation of learning targets, a direct comparison between the universities investigated in Munich (LMU), Berlin and Witten/Herdecke is not possible. However, when applying the LMU learning target definition to the curricula in Berlin and Witten/Herdecke, it can be shown that there is also a continuously rising curve of knowledge in “Conservative Medicine”. Ultimately it cannot be clarified whether this is a general phenomenon in the field of internal medicine, or whether all three universities foster a particular type of knowledge retention due to their specific educational concepts. In the LMU-specific analysis, it was particularly interesting that Module 2 “Conservative Medicine” there was virtually no increase in knowledge in some sub-disciplines of internal medicine although they are not differentiated in the curriculum from other forms sub-disciplines of internal medicine as regards their teaching and examination methods. Most likely, these are down to inconsistencies between Module 2 learning targets and the examination objectives of the MPT. This feedback can lead to improvements in Module 2. Moreover, the cross-sectional analysis identify inhomogeneities in the prior knowledge of students, cause by the curriculum (especially through different module sequence, see Figure 1 [Fig. 1]) which is perceived as disturbing by students and teaching staff. This data supports the planned curriculum reform at the LMU (an organ-centred, interdisciplinary, surgical-internal medicine base year).

In summary, this analysis shows that the introduction of the MPT can contribute to the evaluation of a curriculum regarding permanence of subject knowledge. Using the example of internal medicine, it has been shown that permanent knowledge increase occurs continuously and depends on the practical experiences of students in the clinical part of their studies. Elsewhere, deficits of the curriculum could be documented. Using such cross-sectional analyses repeatedly, the effects of successful curriculum change can be evaluated.


Acknowledgements

This article was part of a project in the Master of Medical Education (MME) program of the Medical Faculty Day at the Medical Faculty of the University of Heidelberg.


Competing interests

The authors declare that they have no competing interests.


References

1.
Bundesministerium für Gesundheit. Approbationsordnung für Ärzte vom 27. Juni 2002. Bundesgesetzbl. 2002;Teil I(Nr.44).
2.
Osterberg K, Kölbel S, Brauns K. Der Progress Test Medizin: Erfahrungen an der Charité Berlin. GMS Z Med Ausbild. 2006;23(3):Doc46. Zugänglich unter/available from: http://www.egms.de/static/de/journals/zma/2006-23/zma000265.shtml. External link
3.
Blake JM, Norman GR, Keane DR, Mueller CB, Cunnington J, Didyk N. Introducing progress testing in McMaster University's problem-based medical curriculum: psychometric properties and effect on learning. Acad Med. 1996;71(9):1002-1007. DOI: 10.1097/00001888-199609000-00016 External link
4.
Freeman A, van der Vleuten C, Nouns Z, Ricketts C. Progress testing internationally. Med Teach. 2010;32(6):451-455. DOI: 10.3109/0142159X.2010.485231 External link
5.
Van der Veken J, Valcke M, De Maeseneer J, Schuwirth L, Derese A. Impact on knowledge acquisition of the transition from a conventional to an integrated contextual medical curriculum. Med Educ. 2009;43(7):704-713. DOI: 10.1111/j.1365-2923.2009.03397.x External link
6.
Peeraer G, De Winter BY, Muijtjens AM, Remmen R, Bossaert L, Scherpbier AJ. Evaluating the effectiveness of curriculum change. Is there a difference between graduating student outcomes from two different curricula? Med Teach. 2009;31(3):e64-e68. DOI: 10.1080/01421590802512920 External link
7.
Bianchi F, Stobbe K, Eva K. Comparing academic performance of medical students in distributed learning sites: the McMaster experience. Med Teach. 2008;30(1):67-71. DOI: 10.1080/01421590701754144 External link
8.
Muijtjens AM, Schuwirth LW, Cohen-Schotanus J, Thoben AJ, van der Vleuten CP. Benchmarking by cross-institutional comparison of student achievement in a progress test. Med Educ. 2008;42(1):82-88. DOI: 10.1111/j.1365-2923.2007.02896.x External link
9.
Ramsey PG, Carline JD, Inui TS, Larson EB, LoGerfo JP, Norcini JJ, Wenrich MD. Changes over time in the knowledge base of practicing internists. JAMA. 1991;266(8):1103-1107.
10.
Basu S, Roberts C, Newble DI, Snaith M. Competence in the musculoskeletal system: assessing the progression of knowledge through an undergraduate medical course. Med Educ. 2004;38(12):1253-1260. DOI: 10.1111/j.1365-2929.2004.02017.x External link
11.
Van Diest R, Van Dalen J, Bak M, Schruers K, Van der Vleuten C, Muijtjens A, Scherpbier A. Growth of knowledge in psychiatry and behavioural sciences in a problem-based learning curriculum. Med Educ. 2004;38(12):1295-1301. DOI: 10.1111/j.1365-2929.2004.02022.x External link
12.
Nouns ZM, Brauns K, HanflerS. Der Progress Test Medizin als Evaluationsinstrument. GMS Z Med Ausbild. 2005;22(4):Doc132. Zugänglich unter/available from: http://www.egms.de/static/de/journals/zma/2005-22/zma000132.shtml External link
13.
Rodriguez R, Campos-Sepulveda E, Vidrio H, Contreras E, Valenzuela F. Evaluating knowledge retention of third-year medical students taught with an innovative pharmacology program. Acad Med. 2002;77(6):574-577. DOI: 10.1097/00001888-200206000-00018 External link
14.
Saffran M, Kennedy WB, Kelley PR. Retention of knowledge of pharmacology by U.S. and Canadian medical students. Trends Pharmacol Sci. 1982;3:461-463. DOI: 10.1016/0165-6147(82)91235-4 External link
15.
Swanson DB, Case SM, Luecht RM, Dillon GF. Retention of basic science information by fourth year medical students. Acad Med. 1996;71(10 Suppl):S80-S82. DOI: 10.1097/00001888-199610000-00051 External link
16.
Watt ME. Retention of preclinical knowledge by clinical students. Med Educ. 1987;21(2):119-124. DOI: 10.1111/j.1365-2923.1987.tb00677.x External link
17.
Krebs R, Hofer R, Bloch R, Guibert JJ. Conversation et oubli des connaissances en biologie acquises pour le premier examen propedeutique de medicine. MEDUCS Bull Ass Suisse Educ Med. 1994;4:10-15.
18.
D’Eon MF. Knowledge loss of medical students on first year basic science courses at the university of Saskatchewan. BMC Med Educ. 2006;6:5. DOI: 10.1186/1472-6920-6-5 External link
19.
Cheifetz RE, Phang PT. Evaluating learning and knowledge retention after continuing medical education course on total mesorectal excision for surgeons. Am J Surg. 2006;191(5):687-690. DOI: 10.1016/j.amjsurg.2006.01.043 External link