gms | German Medical Science

GMS Zeitschrift für Medizinische Ausbildung

Gesellschaft für Medizinische Ausbildung (GMA)

ISSN 1860-3572

Retrospective adjustment of self-assessed medical competencies – noteworthy in the evaluation of postgraduate practical training courses

research article medicine

Search Medline for

  • corresponding author Michael Nagler - Universitätsklinik für Hämatologie und Hämatologisches Zentrallabor, Inselspital, Bern, Schweiz
  • S. Feller - Universität Bern, Medizinische Fakultät, Institut für Medizinische Lehre, Abeteilung für Assessment und Evaluation, Bern, Schweiz
  • author Christine Beyeler - Universität Bern, Medizinische Fakultät, Institut für Medizinische Lehre, Abeteilung für Assessment und Evaluation, Bern, Schweiz

GMS Z Med Ausbild 2012;29(3):Doc45

doi: 10.3205/zma000815, urn:nbn:de:0183-zma0008156

This is the translated version of the article.
The original version can be found at: http://www.egms.de/de/journals/zma/2012-29/zma000815.shtml

Received: June 24, 2011
Revised: December 18, 2011
Accepted: January 17, 2012
Published: May 15, 2012

© 2012 Nagler et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by-nc-nd/3.0/deed.en). You are free: to Share – to copy, distribute and transmit the work, provided the original author and source are credited.


Abstract

Aim: The efficacy of postgraduate practical training courses is frequently evaluated by self-assessment instruments. The present study analyses the effect of a basic course in laparoscopic surgery on self-assessed medical competencies.

Methods: The 3-day course included teaching of knowledge and training of practical skills. In relation to course evaluation, a questionnaire for self-assessment was applied

1.
at the beginning of the course ('pre-course'),
2.
at the end of the course ('post-course') and
3.
at the end of the course to reassess pre-course competencies ('retrospective pre-course').

Results: 89 out of 110 participants (81%) attending 10 courses completed all the questionnaires; 83% were postgraduate trainees in surgery and 82% were inexperienced as an independent surgeon. At the beginning of the course most trainees rated themselves as 'moderately competent' or 'fully competent' with respect to the various task levels as well as to specific areas of medical competencies. At the end of the course however pronounced retrospective revisions of self-assessment to lower ratings became apparent. Statistically significant differences were seen for the task 'performing surgical procedures under supervision' and for most of the practical skills trained during the course (p <0.01). In contrast, no significant differences were observed for knowledge taught during the course as well as for 'ability to work in a team' and 'ability to concentrate', which were not foci of the course.

Conclusions: Surgeons with little experience change their self-assessment of pre-course competencies to a lower level after participation in a practical postgraduate training course. Evaluations comparing 'pre-course' and 'post-course' ratings only – without 'retrospective pre-course' ratings – may underestimate the training effects. This phenomenon needs to be taken into account when evaluations are dependent exclusively on self-assessment instruments.

Keywords: medical education, evaluation studies, diagnostic self-evaluation, clinical competence, laparoskopy/*education


Introduction

Efficacy and the achievement of learning objectives are essential considerations while conducting training programmes for medical competencies [1], [2]. In his model of evaluation, Kirkpatrick suggests evaluating the efficacy of training programmes using four levels. On the first level, the investigator asks for the perception of the participants ('Reaction'). Level two will evaluate the amount of learning that has occurred ('Learning'). Level three will evaluate the transfer that has occurred to the trainees’ everyday environment ('Behaviour') and level four measures the impact of the training to the participants’ professional life ('Outcomes'/ 'Results') [3], [4]. Kirkpatrick's model is being discussed because important factors influencing the learning process are not considered: personality traits, culture of learning and type of teaching aids [5]. Furthermore, no causal relationship between the different levels has been determined to date [5]. In addition, there is a limited amount of data available to support the concept that level four evaluations are more important than those of level 1 to 3 [5]. Nevertheless, using the four-level-model, Kirkpatrick was able to simplify and categorise the complex processes of evaluating training programmes as well as to establish measures of 'Behaviour' and 'Outcome' as important criteria [5]. However, application of these latter criteria is often not feasible due to the complexity of the object of investigation or cost constraints. Thus, training programmes are mostly evaluated on level 1 and 2 using self-assessment instruments.

The ability to assess one's own competence is recognised as a key factor for medical competencies [6]. It is essential for medical professionals to be able to identify their personal strengths and weaknesses, to have confidence in their expertise and to be able to restrict their medical practice activities as appropriate [7]. With regard to the learning process, self-assessment skills are required for the determination of aims and to verify existing skills. However, it has been illustrated in a recent review article that physicians are predominantly unable to assess their abilities accurately. The authors were particularly worried about those professionals with the fewest skills but who overestimated their abilities the most [8]. These observations have been confirmed by other investigations, which revealed that the least competent individuals were unable to assess the competence level of either their own skills or those of other individuals accurately [9], [10], [11]. Self-assessment at the beginning of a training programme may contain an over-estimation of skills due to inexperience [12]. Appropriate training programmes may though improve not only the competence of the trainees, but furthermore self-assessment skills [12], [13]. Therefore, the data from previous investigations suggest that evaluation of training programmes using self-assessment measures may contain systematic errors due the effect mentioned above.

Medical education is traditionally organised as an apprenticeship, but recently more and more training courses are held [14]. Training courses are needed particularly in surgical education because advanced technical skills are required for highly specialised procedures [15], [16]. Several studies have demonstrated the efficacy of practical training courses, especially in laparoscopic surgery [17], [18], [19].


Aim

Using the example of a basic course in laparoscopic surgery, we aimed to investigate whether and how self-assessment scores of medical competencies changed as the participants progressed through the postgraduate practical training courses. Furthermore, we intended to discuss the possible effects on course evaluation measures. We considered the following questions:

1.
How do participants rate
1.1.
their competencies at the beginning of the course (‘pre-course’),
1.2.
their competencies at the end of the course (‘post-course’) and
1.3.
their pre-course competencies at the end of the course (‘retrospective pre-course’)?
2.
Is there an alteration in self-assessment over the course of the training (‘post-course’ versus ‘pre-course’ and ‘retrospective pre-course’ versus ‘pre-course’, respectively) and which medical competencies are affected?
3.
Are there differences in the changes reported between areas of competencies, which are trained and those that are not?

Methods

The present investigation was conducted for 10 basic courses in laparoscopic surgery at the ‘European Surgical Institute’ of the Ethicon Endo-Surgery Company in Norderstedt, Germany.

Practical postgraduate training course

Three-day courses were performed to impart knowledge and train practical skills in a complex operative setting. In the first phase, basic knowledge was taught and essential skills trained. Lectures of 3¼ hours were held on basic anatomical, physiological and clinical principles as well as surgical procedures. In a video demonstration of 1½ hours possible risks and sources of error were discussed. The handling of laparoscopic instruments was shown in a practical demonstration (1 hour). Exercises in practical skills comprised of suturing and knotting techniques in a microsurgical setting (2 hours) and in a computer simulation (2 hours). In the second phase, the skills learnt were transferred into the laparoscopic surgery setting; suturing exercises were performed on a simulated abdomen with an opened or closed abdominal wall (‘Pelvitrainer’). In the last phase, an animal organ was added (pig’s liver) and a laparoscopic cholecystectomy was performed by the trainees (6 hours). The training was provided within a teamwork setting and feedback was given continuously. Overall, the aim of the courses was to enhance medical competencies with regard to knowledge and practical skills in a laparoscopic setting.

Questionnaire and data acquisition

A questionnaire was developed for self-assessment of different aspects of laparoscopic medical competencies (see Table 1 [Tab. 1]). A complete item is illustrated in Table 2 [Tab. 2]. The confidence of the trainees was assessed, for example, by the question ‘Do you feel confident to perform laparoscopic procedures independently?’ and they were asked to rate themselves according to four task levels using a 5-point scale ranging from 0 (‘not applicable’) to ‘4 (‘fully applicable’). In addition, confidence was assessed regarding different areas of competency: ‘How do you rate your competencies with regard to…?’. A 5-point scale ranging from 0 (‘low’) to 4 (‘very high’) was used by the participants to indicate their responses. The questions were then reformulated to assess knowledge, practical skills, judgment and professional behaviour (see Table 1 [Tab. 1]). The questionnaire has not yet been evaluated with regard to psychometric properties.

Participants of 10 basic courses in laparoscopic surgery in 2000 and 2001 were asked to complete the questionnaire. They rated their current laparoscopic medical competencies at the beginning of the course (‘pre-course’), at the end of the 3 day course (‘post-course’) and re-assessed their pre-course competencies at the end of the course (‘retrospective pre-course’).

Data analysis

The data were pseudonymised and data analysis was performed using SPSS® 15.

For presentation purposes with regard to question 1, the ratings 0 and 1 were merged into the category ‘minimally competent’, rating 2 to ‘moderately competent’ and ratings 3 and 4 to ‘fully competent’ and displayed in table form. The Wilcoxon test was used for analysis regarding questions 3 and 4 (comparison of self-assessment ‘post-course’ vs. ‘pre-course’ and ‘retrospective pre-course’ vs. ‘pre-course’ respectively) because significant differences from normal distribution were found. Bonferroni-Holm correction for multiple measurements was used as multiple items in the questionnaire were compared. In this case, the maximum statistical value of the single comparisons was evaluated on a significance level of α`=α / m (i.e. 0.05 / number of single comparisons). If this statistical value was significant, the next value was evaluated on a level of α / (m-1) and so on, until no significance was found [20].


Results

The basic characteristics of the study cohort as well as the experiences of the surgeons gained before commencing the course are displayed in Table 3 [Tab. 3].

Self-assessment of medical competencies

‘Pre-course’ more than half of the participants rated their competencies as low (‘minimally competent’) if asked about independently performed surgical procedures, whereas most of the trainees rated themselves as ‘moderately competent’ or ‘fully competent’ for surgical procedures performed under supervision, as a surgical first assistant or as a camera assistant (see Table 4 [Tab. 4]). With regard to specific areas of medical competencies more than 90% of the trainees assessed themselves ‘pre-course’ as ‘moderately competent’ or ‘fully competent’ regarding ‘knowledge about laparoscopic procedures’, ‘manual skills’, ‘mastering basic surgical techniques’, ‘careful manipulation of the tissue’ and ‘orientation and navigation in the surgical field’. Similarly, most trainees rated their skills ‘pre-course’ in the following aspects of social behaviour ‘problem management and handling of errors’, ‘ability to work in a team’ and ‘ability to concentrate and handle stressful situations’ as ‘moderately competent’ or ‘fully competent’. More than half of the trainees regarded their ‘knowledge of laparoscopic instruments and technical equipment’ as ‘moderate’ (see Table 5 [Tab. 5]).

‘Post-course’ a significant increase of self-assessments was found with regard to different task levels (except for ‘work as a camera assistant’; see Table 6 [Tab. 6]). Furthermore, a significant increase was seen ‘post-course’ for the following areas of medical competencies: ‘knowledge on laparoscopic procedures’, ‘knowledge of laparoscopic instruments and technical equipment’, ‘orientation and navigation in the surgical field’. Self-assessment was comparable to the ‘pre-course’ ratings in the remaining areas of medical competencies (see Table 7 [Tab. 7]).

The comparison of ‘pre-course’ vs. ‘retrospective pre-course’ ratings indicated a significant adjustment of self-assessment with regard to the task level ‘performing laparoscopic procedures under supervision’: ‘pre-course’ 74% of the trainees rated themselves as ‘fully competent’, which decreased to 57% on the ‘retrospective pre-course’ questionnaire (see Table 4 [Tab. 4]; Table 5 [Tab. 5]). Such significant adjustment was in fact seen in most of the areas of medical competencies trained during the course (‘manual skills’, ‘mastering basic surgical techniques’, ‘careful manipulation of the tissue’ but not in ‘orientation and navigation in the surgical field’) as well as ‘problem management and handling of errors’ and ‘clinical reasoning’. The proportion of participants, who rated their competencies ‘pre-course’ as ‘minimally competent’ increased in the ‘retrospective pre-course’ assessment. In contrast, no significant difference was seen with regard to the knowledge taught (‘knowledge of laparoscopic procedures‘, ‘knowledge of laparoscopic instruments and technical equipment’) as well as with regard to 'ability to work in a team', which was not the focus of the course (see Table 6 [Tab. 6], Table 7 [Tab. 7]).


Discussion

The present investigation provides evidence that surgeons with limited experience overestimate their competencies. First, participants rated themselves as ‘moderately competent’ or ‘fully competent’ before attending the course with regard to different task levels (with the exception of ‘performing laparoscopic procedures independently) as well as specific areas of medical competencies. This is astonishing because the participants had applied for training of laparoscopic competencies and only a minority already had finished postgraduate surgical training. Second, our data revealed substantial retrospective revisions to lower ratings over the course of the training. More than 85 percent of the trainees rated their ‘pre-course’ competencies as ‘moderately competent’ or ‘fully competent’ in 9 out of 10 areas of medical competencies. Relevant adjustments were seen in the ‘retrospective pre-course’ assessment: Only about a quarter of the trainees (or a third, respectively) rated themselves as “moderately” or “fully” competent in 5 (or 3, respectively) competency areas. These adjustments were statistically significant in those tasks most important to the focus of the training course ('performing surgical procedures under supervision') and in 5 out of 10 specific areas of medical competencies. No statistically significant difference was seen regarding the knowledge taught (‘knowledge of laparoscopic procedures’, ‘knowledge of laparoscopic instruments and technical equipment’). Likewise, no significant difference was seen in the competencies ‘ability to work in a team’ and ‘ability to concentrate and handle stressful situations’. The latter aspects of social behaviour were not the focus of the training course and no feedback was applied to the participants.

Our results confirm previous data on the tendency of minimally competent surgeons to overestimate their abilities [10], [13]. Furthermore, our data support previous publications which suggest that trainings lead to increased ‘metacognitive abilities’ and more realistic self-assessment abilities [12], [13] via improved competencies [17], [18], [19] and the feedback obtained. It cannot, however, be overlooked that retrospective assessments may also have socio-psycological influences (eg. absence of threats to self esteem) and distorting phenomena (eg Hindsight bias) [21].

In the present study we investigated the adjustment of self-assessment ratings exclusively in a typical postgraduate practical training course. An additional objective assessment of competence improvement would have added to the validity of the results. However, instruments for objective assessment of surgical competencies are very limited and require extensive resources [22]. Therefore, an objective assessment of surgical competencies was not made.

Our data suggest that the initial over-estimation of trainees’ abilities will influence the results of self-assessment instruments to evaluate the effectiveness of training. In particular a comparison restricted to ‘pre-course’ vs. ‘post-course’ assessments may lead to an under-estimation of the training effect that has occurred. Instead, a ‘retrospective pre-course’ assessment should also be considered if self-assessment instruments are in use. ‘Retrospective pre-course’ assessments are not a substitute for an objective measure or ‘gold standard’. However, it may reveal valuable information regarding the level of over-estimation of personal capabilities at the outset. If such over-estimation could be recognized and incorporated into the curriculum of training programs it may play an important role in the learning process of trainees.


Acknowledgement

The authors thank Ethicon Endo-Surgery, Norderstedt (Germany) and Mr. Thomas Bürger for the opportunity to conduct this investigation over the 10 basic courses in laparoscopic surgery run at their institution. Furthermore, we thank Dr. Anja Rogausch for the revision of the statistical analyses.


Competing interests

The authors declare that they have no competing interests.


References

1.
Musick DW. A conceptual model for program evaluation in graduate medical education. Acad Med. 2006;81(8):759-765. DOI: 10.1097/00001888-200608000-00015 External link
2.
Fabry G, Lammerding-Köppel M, Hofer M, Ochsendorf F, Schirlo C, Breckwoldt J. Hochschuldidaktische Qualifizierung in der Medizin IV: Messung von Wirksamkeit und Erfolg medizindidaktischer Qualifizierungsangebote: Ein Positionspapier des GMA-Ausschusses Personal- und Organisationsentwicklung für die medizinische Lehre der Gesellschaft für Medizinische Ausbildung sowie des Kompetenzzentrums für Hochschuldidaktik in Medizin Baden-Württemberg. GMS Z Med Ausbild. 2010;27(4):Doc62. DOI: 10.3205/zma000699 External link
3.
Kirkpatrick DL. Evaluating Training Programs: The Four Levels. San Francisco/CA: Berrett-Koehler; 1994.
4.
Kirkpatrick DL. Evaluation of training. In: Craig RL (Hrsg). Training and development handbook: A guide to human resource development. New York: McGraw Hill; 1976.
5.
Bates R. A critical analysis to evaluation practice: the kirkpatrick modell and the principle of benevolence. Eval Prog Plan. 2004;27:341-347. DOI: 10.1016/j.evalprogplan.2004.04.011 External link
6.
Gordon MJ. A review of the validity and accuracy of self-assessments in health professions training. Acad Med. 1991;66(12):762-769. DOI: 10.1097/00001888-199112000-00012 External link
7.
Eva KW, Regehr G. Self-Assessment in the Health Professions: A Reformulation and Research Agenda. Acad Med. 2005;80(10 Suppl):S46-54. DOI: 10.1097/00001888-200510001-00015 External link
8.
Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006;296(9):1094-1102. DOI: 10.1001/jama.296.9.1094 External link
9.
Chi MT, Glaser R, Rees E. Expertise in problem solving. In: Sternberg R (Hrsg). Advances in the psychology of human intelligence. Hillsdale/NJ: Erlbaum; 1982. S.7-75.
10.
McPherson SL, Thomas JR. Relation of knowledge and performance in boy's tennis: Age and expertise. J Exper Child Psychol. 1989;48:190-211. DOI: 10.1016/0022-0965(89)90002-7 External link
11.
Hofer M, Galonska L, Sievers K, Önenköprülü B, Heussen N. Evaluation eines Trainingskonzepts "Plenardidaktik" für Dozenten in Vorlesungen. GMS Z Med Ausbild. 2010;27(3):Doc47. DOI: 10.3205/zma000684 External link
12.
Kruger J, Dunning D. Unskilled and unware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. J Pers Soc Psychol. 1999;77(6):1121-1134. DOI: 10.1037/0022-3514.77.6.1121 External link
13.
Hodges B, Regehr G, Martin D. Difficulties in recognizing one`s own incompetence: novice physicians who are unskilled and unaware of it. Acad Med. 2001;76(10 Suppl):S87-89. DOI: 10.1097/00001888-200110001-00029 External link
14.
Ericsson KA. Deliberate practice and the aquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;79(10 Suppl):70-81. DOI: 10.1097/00001888-200410001-00022 External link
15.
DesCôteaux JG, Leclere H. Learning surgical technical skills. Can J Surg. 1995;38(1):33-38.
16.
Hamdorf JM, Hall JC. Acquiring surgical skills. Br J Surg. 2000;87(1):28-37. DOI: 10.1046/j.1365-2168.2000.01327.x External link
17.
Rosser JC, Rosser LE, Savalgi RS. Objective evaluation of a laparoscopic surgical skill program for residents and senior surgeons. Arch Surg. 1998;133(6):657-661. DOI: 10.1001/archsurg.133.6.657 External link
18.
Ritz JP, Grone J, Hopt U, Saeger HD, Siewert JR, Vollmar B, Lauscher JC, Lehmann KS, Buhr HJ. Practical course for visceral surgery in Warnemunde" 10 years on. Significance and benefits of a surgical training course. Chirurg. 2009;80(9):864-871. DOI: 10.1007/s00104-009-1782-0 External link
19.
Grone J, Ritz JP, Buhr HJ, Lauscher JC. Sustainability of skill courses for general and visceral surgery--evaluation of the long-term effect. Langenbecks Arch Surg. 2010;395(3):277-283. DOI: 10.1007/s00423-009-0568-7 External link
20.
Bortz J. Statistik für Human- und Sozialwissenschaftler. Heidelberg: Springer-Verlag; 2005.
21.
Fischhoff B. Hindsight not equal to foresight: the effect of outcome knowledge on judgment under uncertainty. Qual Saf Health Care. 2003;12(4):304-311; discussion 11-12. DOI: 10.1136/qhc.12.4.304 External link
22.
van Hove PD, Tuijthof GJ, Verdaasdonk EG, Stassen LP, Dankelman J. Objective assessment of technical surgical skills. Br J Surg. 2010;97(7):972-987. DOI: 10.1002/bjs.7115 External link