gms | German Medical Science

GMS Journal for Medical Education

Gesellschaft für Medizinische Ausbildung (GMA)

ISSN 2366-5017

The practial use of the consensus statement on practical skills in medical school – a validation study

research article medicine

  • corresponding author Wolf E. Blaum - Charité - Universitätsmedizin Berlin, Klinik für Anästhesiologie mit Schwerpunkt operative Intensivmedizin, Berlin, Deutschland; Charité - Universitätsmedizin Berlin, Abteilung für Curriculumsorganisation, Lernzentrum, Berlin, Deutschland
  • author Katja A. Dannenberg - Charité - Universitätsmedizin Berlin, Abteilung für Curriculumsorganisation, Lernzentrum, Berlin, Deutschland
  • author Torsten Friedrich - Charité - Universitätsmedizin Berlin, Abteilung für Curriculumsorganisation, Lernzentrum, Berlin, Deutschland
  • author Anne Jarczewski - Charité - Universitätsmedizin Berlin, Abteilung für Curriculumsorganisation, Lernzentrum, Berlin, Deutschland
  • author Anne-Katrin Reinsch - Charité - Universitätsmedizin Berlin, Abteilung für Curriculumsorganisation, Lernzentrum, Berlin, Deutschland
  • author Olaf Ahlers - Charité - Universitätsmedizin Berlin, Klinik für Anästhesiologie mit Schwerpunkt operative Intensivmedizin, Berlin, Deutschland; Charité - Universitätsmedizin Berlin, Abteilung für Curriculumsorganisation, Lernzentrum, Berlin, Deutschland

GMS Z Med Ausbild 2012;29(4):Doc58

doi: 10.3205/zma000828, urn:nbn:de:0183-zma0008285

This is the English version of the article.
The German version can be found at: http://www.egms.de/de/journals/zma/2012-29/zma000828.shtml

Received: January 15, 2012
Revised: May 22, 2012
Accepted: June 20, 2012
Published: August 8, 2012

© 2012 Blaum et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by-nc-nd/3.0/deed.en). You are free: to Share – to copy, distribute and transmit the work, provided the original author and source are credited.


Abstract

Objective: The importance of the acquisition of practical medical skills during medical school is increasing. With the consensus statement “Practical Skills,” developed by the GMA as part of the National Competency-Based Learning Objective Catalogue for Medicine (NKLM), a reference frame was created for the procurement of such skills. This frame consists of 290 learning objectives divided by “organ system,” type (core or elective learning objective), current stage of medical education and level of instruction.

By comparing a large and well evaluated range of student tutorials with the consensus statement, one can analyze the practical benefit of the statement, as well as evaluate the tutorial program for completeness.

Methods: In the first stage, four evaluators in two groups independently classified all consensus statement’s learning objectives by each of the 48 tutorials currently offered. The inter-rater reliability among the evaluators of each group was calculated both collectively, and according to each organ system. In the second stage, disagreements in the classification were resolved through discussion and consensus decision-making. The coverage of the learning objectives by the tutorials, in the required level of instruction, was then analyzed separately by learning objective type and organ system. Reasons for any initial dissent were recorded and grouped thematically.

Results: The correlation between the classifications of the two evaluators was moderately significant. The strength of this correlation, and thus the precision of individual learning goals wording, varied according to organ system. After a consensus was reached, the results show that the offered tutorials covered 66% of all learning objectives, as well as 74% of the core objectives. The degree of coverage differed according to organ system and stage of medical education.

Conclusion: The consensus statement is suitable to systematically analyze and develop teaching units. The comparison with established curricula also offers possibilities for further development of the consensus statement, and therefore also of the NKLM.

Keywords: skills, practical skills, clinical skills, medical education, peer-teaching, curriculum, curricular mapping, learning objectives


Introduction

The importance of the acquisition of practical (medical) skills in medical school has increased in recent years [1], [2]. At the same time, medical curricula are increasingly relying on outcome definitions and/or (national) learning objective catalogues [3], [4]. In Germany, the Medical Education Society (GMA), in conjunction with the German Medical Faculty Association (MFT), has been working since 2009 on the development of a National Competency-Based Learning Objective Catalogue for Medicine (NKLM) [5]. As part of this project, the consensus statement “Practical Skills in Medical School” was recently published, which comprises 289 learning objectives, divided into core/elective goals and 16 organ systems. The consensus statement “should have a formative effect on the faculties, to bring the content of their practical teachings in accordance to the guidelines” [6].

The current validation study analyzes the suitability of the consensus statement as a national reference, to “map” a comprehensive curriculum – here the tutorials offered by the Learning Center of the Charité.

“Curriculum Mapping” includes the transparent, well-defined presentation of educational content and objectives, as well as their correlation and thematic association. It renders aspects such as structure, completeness, relevance, complexity, coherence, and curricular organization in a clear and understandable manner, and is also recommended by the Association for Medical Education in Europe AMEE (see [7]). In this process, a mapping of each (medical) curriculum is essential for three reasons:

1.
The medical schools are accountable, to the public and to the legislators, for the competency of their graduates to practice medicine (§41 ÄAPPO) [8].
2.
Clarity of purpose and adherence to the objectives contribute to student satisfaction and performance [9], [10], [11], [12].
3.
The integration of teaching content in the curriculum provides a reference for teachers and students, thus enhancing the quality of instruction and student examination performances [13].

The Learning Center of the Charité, which emerged from the former “Training Center for Medical Skills,” runs an extensive program of peer-teaching tutorials for the conveyance of medical skills [14]. The Learning Center currently employs 19 student tutors, who receive regular training in the performance of medical procedures, as well as in didactics. These tutors independently determine the demand for new tutorials, and are responsible for their development under the supervision of experts. Each year, about 500 tutorials are offered on 48 topics. These classes are visited by approximately 4 500 students, and the participation is voluntary and free of charge. All tutorials are systematically evaluated by the Evaluation Department of the Charité. The students are extremely satisfied with the content and scope of the tutorials, as well as the effectiveness of the training (median 1 on a seven-point Likert scale) [15]. Based on these evaluations, the tutors independently refine the individual tutorials, as well as the range of the courses offered.


Research Objectives

The aim of the current study is the mapping of the Learning Center’s peer-teaching tutorials, and a comparison with the consensus statement “Practical Skills in Medical School”. The comparison should answer three questions:

1.
Is the consensus statement suited to re-/structure a comprehensive and well evaluated range of tutorials, and therefore to “bring the content of their practical teachings in accordance to the guidelines” [6], as the statement strives to do?
2.
For the purposes of the consensus statement, is the tutorial program of the Learning Center complete, or are there gaps or unintended redundancies?
3.
Does the comparison with the consensus statement suggest that the tutorials should be visited in a particular order?

Methods

Comparison of the Tutorial Program with the Consensus Statement

The consensus statement describes 289 learning objectives, 232 of which are defined as core objectives. Each learning objective is assigned to at least one of 16 organ systems. The objective 275 is assigned to the organ system Growth/Ageing and Emergency. This learning objective was recorded twice in this study, and assigned to both “Growth/Ageing” and “Emergency Medicine” organ systems, leading to a total of 290 learning objectives.

In one of three stages of the medical education, the target level of instruction for each objective is defined as one of three levels. These stages include the beginning of the first elective clinical rotation, the beginning of the practical year, and the beginning of residency. The level of instruction is described as

1.
“was given demonstration”;
2.
“performed under supervision”; and
3.
“skilled performance” (see [6]).

In order to compare the learning objectives of the consensus statement with the tutorials, the depth at which the tutorial deals with a particular objective was classified. The classification was based on the Level of Instruction Scale of the consensus statement, which was slightly modified for this study, and is shown in Table 1 [Tab. 1].

In order to depict the level of instruction of the 290 learning objectives in the 48 tutorials, a total of 13 920 (48 x 290) classifications were needed.

Classification by the Evaluators

Four student tutors with long term experience in peer-teaching at the Learning Center were divided into two evaluation groups:

The authors AKR and AJ classified the parameters of 129 learning objectives, in all 48 available tutorials, for the following organ systems: respiratory, blood/immunology, gastrointestinal, cardiovascular, nervous, psyche, sensory organs, and Growth/Ageing. The authors KAD and TF classified 161 learning objectives, also in all 48 tutorials, for the following organ systems: musculoskeletal, endocrine, communication, emergency medicine, soft skills, urinary/sexual, dermatological, and comprehensive skills.

In the first step, the evaluators classified the objectives independently from each other, allowing for the calculation of the inter-rater reliability. In the second step, classification discrepancies were resolved through discussion and consensus decision-making, until a final classification was achieved. This classification was then used to determine the percentage of the national learning objectives that were covered by the currently available tutorials. Reasons for any initial dissent were recorded.

Data Analysis

The classifications were gathered in Libre Office 3, and the coverage of the learning objectives by the tutorials was calculated separately by organ system, type of objective (core or elective), as well as current stage of medical education.

In order to calculate the inter-rater reliability, the correlation between evaluators was determined separately for each group using Spearman’s correlation with SPSS19. Additionally, the correlation was calculated separately by organ system, in order to examine to which extent the consensus statement’s wording for the learning objectives left a margin for interpretation.

The reasons for initial dissent among the evaluators in the classification procedure were recorded and grouped thematically, so as to identify issues and aid the further development of the consensus statement.


Results

Quality of Objective Formulation in the Consensus Statement

Group 1 analyzed parameters for 129 learning objectives in 48 tutorials, for a total of 6 192 classifications, and reached a moderately significant agreement between the evaluators (r=0.66).

Group 2 analyzed parameters for 161 learning objectives in 48 tutorials, for a total of 7 728 classifications, and reached a modest, but significant agreement between the evaluators (r=0.31).

The congruence between tutors varied in relation to organ system, and the intensity of this correlation was in some cases substantial: the highest congruence rate was reached in the 192 classifications of the “Psyche” organ system (r=1.0), and the lowest in the 624 classifications of the “Urinary/Sexual” organ system (r=0.2). Table 2 [Tab. 2] shows the number of classifications and the consistency of the ratings for each organ system. As seen in Table 2 [Tab. 2], the level of congruence is independent from the number of learning objectives, and thus also the number of classifications, per organ system.

Coverage of the Learning Objectives

The current tutorial program covers 65.9% of all learning goals, and 73.7% of the core objectives listed in the consensus statement. Some of the learning goals are covered in nearly all tutorials: 42 tutorials deal with the objective “Regard to occupational health and safety in the personal medical activities and responsibilities, per example, needle stick injury, working in an ergonomic manner, recapping, wearing gloves, ...” (Soft Skills); 39 tutorials deal with the objective “Being able to explain the procedures to the patient during an examination” (Communication).

The level of instruction to which an objective must be covered, as required by the consensus statement, increases along with stage of medical education. The coverage of these objectives by the tutorial program, to the consensus statements’ required depth and at higher stages of medical education, decreases: the objectives that must be covered, to the required depth, by the time of the first elective clinical rotation amounts to 63% of all learning objectives (and 70.3% of the core objectives). The target level of instruction was achieved by the beginning of the practical year for 48.6% of the objectives (53% of the core objectives), and 39.3% of the objectives (41.8% of the core objectives) by the beginning of the continuing education.

The coverage of the learning goals varied, in some cases considerably, in relation to organ system. The tutorials offered covered 90% of the learning objectives (100% of core objectives) for the Nervous System, but only 30% of the goals (50% core objectives) for the Respiratory System.

Table 3 [Tab. 3] shows the coverage of learning objectives by the tutorials in relation to organ system and stage of education. For all organ systems, there is a decrease in the coverage of objectives (to the required depth) as the student level of education increases.

Key Issues for the Further Development of the Consensus Statement

The disagreements recorded during the classification procedure were attributed to one of three causes: ambiguous formulation of learning objectives, unspecific scope of the objective, or objective-independent causes. The most frequent problem listed was vague formulation of the consensus statement. For example, the evaluators had difficulty determining what types of infusion lines were included in objective 90 “Establishing lines” (German: “Zugänge anlegen”; usually refers to starting an IV line, but can include central, arterial, intraosseous lines, etc), or how exactly to interpret objective 115, “Indications and regulations for technical examinations.”

A second major cause for inconsistencies in the classification process was the vague scope of some objectives. While the learning objective “Active and passive examination of the upper extremities including shoulder, elbow and wrist, as well as fingers and thumb (especially inspection, locating anatomical landmarks, establishing range of motion, and functionality of the joints)” is extremely precise and well defined, others, such as the objective “Neuroradiological examinations methods,” leave a significant gap in terms of type and extent of the skills to be acquired.

The objective-independent causes refer to the consensus statement as a whole. For example, it was unclear to the evaluators whether practice on simulators, which was specifically mentioned in some learning objectives, implied its exclusion for all other objectives.


Discussion

Consensus Statement

The tutorial program in the Learning Center of the Charité was established in 1999, and has been continually evolving. Since the beginning, the tutorials have been systematically evaluated, and now the results of the seventh generation of tutors demonstrate their experience and influence. The analysis of the consensus statement against this extensive and well established program can help appraise the statement’s validity. The current tutorial program covers a significantly higher proportion of core learning objectives than the total list of objectives. This is true, without exceptions, for individual organ systems as well as for the consensus statement as a whole. The prioritization of the tutorial content supports, and is congruent with, the priorities set by the consensus statement.

Furthermore, the complexity of the learning objectives increases along with the three set stages of medical education. The coverage of these objectives by the tutorials, at the required depth, decreases both overall and according to organ system. The level of complexity required by the consensus statement matches that of the tutorials, which supports the statement’s requirements.

The stated objectives differ as to the practicality of their integration in specific courses, as suggested by the inconsistency between evaluators in relation to organ system, shown in table 2 [Tab. 2]. The following themes were listed as possible causes: “Ambiguous formulation of learning objectives,” “Unspecific scope of the objective,” and “Objective-independent causes.” A systematic review of these issues can contribute to the further development of the consensus statement. It would be also interesting to investigate whether the conveyance of practical skills, as established by the faculty in the mandatory curriculum, are complete and free of redundancy by the standards of the consensus statement.

Tutorial program

In light of the results, gaps and overlaps in the tutorials could be identified, and the data was further used to restructure the tutorial program.

By comparing the level of instruction of a particular learning objective in the tutorials with the level required at specific milestones by the consensus statement, it is possible to recommend to students the tutorial participation in a specific order. The consensus statement is therefore suitable for the restructure of existing curricula. We have started to integrate the tutorial curriculum in the existing courses and teaching platform of the Charité [13], which has so far only been used to chart mandatory courses. This encompasses the development of a student online navigation system, which uses the data collected from this study to suggest tutorials based on the student’s interest, or as a meaningful follow-up to the tutorials already visited.

Limitations

In the current study, the coverage of the learning objectives of the consensus statement “Practical Skills” [6] by the tutorials from the Learning Center of the Charité was independently classified. The inter-rater reliability was calculated using Spearman’s correlation, as the authors considered the scale shown in table 1 [Tab. 1] to be an ordinal scale. There is an extensive discussion in the literature as to whether a scale such as the one used in this study can be appraised using parametric statistics (see [16], [17]). Some authors evaluate such scales, and even nominal scales, using intraclass correlations (ICC) [18]. The use of Spearman’s correlation reduces the power of the examination (and therefore the strength of the correlation), and although the use of ICCs provide stronger correlations, it can potentially increase the α-error [16], [17]. Therefore, the correlation shown here represents solely a conservative estimate of the actual evaluator congruency.

The initial reasons for dissent among the evaluators during the classification procedure were grouped thematically. No qualitative methods were regarded during the data analysis.

Experienced student tutors, as experts on their own tutorials, were deemed most apt to estimate whether the 48 tutorials cover, and to what depth, the objectives of the consensus statement. It remains unclear how the classification by students would relate to a classification by licensed physicians, though the latter would first require extensive practical experience in the tutorial program.


Conclusion

The student-led tutorials support student independent studies. The comparison with the consensus statement “Practical Skills” is suitable to systematically analyze and develop teaching units and to “bring the content of their practical teachings in accordance to the guidelines” [6]. The comparison with established curricula offers evidence-based possibilities for further development of the consensus statement, and therefore also of the NKLM.


Note

The authors Dannenberg, Friedrich, Jarczewski and Reinsch contributed equally.


Acknowledgement

The authors thank Dr. Henrike Hölzer, Berlin, Dr. Claudia Kiessling, MPH, Munich, and Dr. Kai Schnabel, MME, Bern, as well as two anonymous reviewers for their constructive and collegial critic of the manuscript. Special thanks to Sebastian Schubert, Berlin, for his constructive involvement in the study design and the manuscript. We would also like to thank Rudi Mörgeli for the translation of the manuscript.


Competing interests

The authors declare that they have no competing interests.

WB is the head of the Learning Center of the Charité, a member of the GMA, and of the GMA commitees “Practical Skills” and “Educational Research Methods.” KAD, TF, AJ and AKR are student tutors of the Learning Center, and students of the Charité. OA is head of the Department of Curricular Organization, a member of the GMA, as well as a member of the GMA committee “Educational Research Methods.”


References

1.
Hudson JL, Tonkin AL. Clinical skills education: outcomes of relationships between junior medical students, senior peers and simulated patients. Med Educ. 2008;42(9):901-908. DOI: 10.1111/j.1365-2923.2008.03107.x External link
2.
Yudkowsky R, Otaki J, Lowenstein T, Riddle J, Nishigori H, Bordage G. A hypothesis-driven physical examination learning and assessment procedure for medical students: initial validity evidence. Med Educ. 2009;43(8):729-740. DOI: 10.1111/j.1365-2923.2009.03379.x External link
3.
Frank JR, Danoof D. The CanMEDS initiative: implementing an outcomes-based framework of physician competencies. Med Teach. 2007;29(7):642-647. DOI: 10.1080/01421590701746983 External link
4.
Bloch R, Bürgi H. The Swiss catalogue of learning objectives. Med Teach. 2002;24(2):144-150. DOI: 10.1080/01421590220120759 External link
5.
Hahn EG, Fischer MR. Nationaler Kompetenzbasierter Lernzielkatalog Medizin (NKLM) für Deutschland: Zusammenarbeit der Gesellschaft für Medizinische Ausbildung (GMA) und des Medizinischen Fakultätentages (MFT). GMS Z Med Ausbild. 2009;26(3):Doc35. DOI: 10.3205/zma000627 External link
6.
Schnabel KP, Boldt PD, Breuer G, Fichtner A, Karsten G, Kujumdshiev S, Schmidts M, Stosch C. Konsensusstatement "Praktische Fertigkeiten im Medizinstudium" – ein Positionspapier des GMA-Ausschusses für praktische Fertigkeiten. GMS Z Med Ausbild. 2011;28(4):Doc58. DOI: 10.3205/zma000770 External link
7.
Harden RM. AMEE Guide No. 21: Curriculum mapping: a tool for transparent and authentic teaching and learning. Med Teach. 2001;23(2):123-137. DOI: 10.1080/01421590120036547 External link
8.
Bundesministerium für Gesundheit. Approbationsordnung für Ärzte vom 27. Juni 2002. Bundesgesetzbl. 2002;I(44):2417.
9.
Locke EA, Latham GP. A theory of goal setting and task performance. Englewood Cliffs, NJ: Prentice Hill; 1990.
10.
Braun OL. Berufliche Zielklarheit im Studium. In: Müller GF (Hrsg). Lebenslanges Lernen. Landau: Knecht; 2001. S.261-276.
11.
Gollwitzer PM. Zielbegriffe und -theorien in der heutigen Psychologie. In: Pawlik K (Hrsg). Bericht über den 39. Kongress der Deutschen Gesellschaft für Psychologie in Hamburg 1994. Goettingen: Hogrefe; 1995. S.295-300.
12.
Kähler R. Individuelle Einflussfaktoren auf Studienzufriedenheit und persönliche Ziele von Medizinstudierenden der Charité – Universitätsmedizin Berlin. Berlin: Dissertationan der Charité; 2010. Zugänglich unter/available from: http://www.diss.fu-berlin.de/diss/servlets/MCRFileNodeServlet/FUDISS_derivate_000000007656/Dissertation_Ragna_Kaehler.pdf External link
13.
Ahlers O, Georg W, Blaum W, Stieg M, Hanfler S, Bubser F, Spies C. Der Einsatz einer interdisziplinären, webbasierten Lernzielplattform verbessert sowohl die Unterrichtsqualität als auch die Klausurergebnisse Studierender. Jahrestagung der Gesellschaft für Medizinische Ausbildung (GMA). Bochum, 23.-25.09.2010. Düsseldorf: German Medical Science GMS Publishing House, 2010, Doc10gma13.
14.
Trainingszentrum für Ärztliche Fertigkeiten. Tutorienangebot. Berlin: Charite Universitätsmedizin; 2012. Zugänglich unter/available from: http://taef.charite.de External link
15.
Kraft R, Haase C. Auswertung des studentischen Fragebogens zu den TÄF-Tutorien und des Trainingszentrums für ärztliche Fertigkeiten im Sommer 2011. Berlin: Charite Universitätsmedizin; 2011. Zugänglich unter/available from: http://taef.charite.de External link
16.
Jamieson S. Likert scales: how to (ab)use them. Med Educ. 2004; 38(12):1212-1218. DOI: 10.1111/j.1365-2929.2004.02012.x External link
17.
Norman G. Likert scales, levels of measurement and the ‘‘laws’’ of statistics. Adv in Health Sci Educ. 2010;15(5):625-632. DOI: 10.1007/s10459-010-9222-y External link
18.
Cook DA, Erwin PJ, Triola MM. Computerized virtual patients in Health Professions Education: a systematic review and meta-analysis. Acad Med. 2010;85(10):1589-1602. DOI: 10.1097/ACM.0b013e3181edfe13 External link