gms | German Medical Science

GMS Journal for Medical Education

Gesellschaft für Medizinische Ausbildung (GMA)

ISSN 2366-5017

The use of a portfolio in postgraduate medical education – reflect, assess and account, one for each or all in one?

article

Search Medline for

  • corresponding author Sylvia Heeneman - Maastricht University/MUMC, Department of Pathology, HX Maastricht, The Netherlands; Maastricht University, School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht, The Netherlands
  • Erik W. Driessen - Maastricht University/MUMC, Department of Educational Development and Research, HX Maastricht, The Netherlands; Maastricht University, School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht, The Netherlands

GMS J Med Educ 2017;34(5):Doc57

doi: 10.3205/zma001134, urn:nbn:de:0183-zma0011345

This is the English version of the article.
The German version can be found at: http://www.egms.de/de/journals/zma/2017-34/zma001134.shtml

Received: November 8, 2016
Revised: February 7, 2017
Accepted: March 20, 2017
Published: November 15, 2017

© 2017 Heeneman et al.
This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 License. See license information at http://creativecommons.org/licenses/by/4.0/.


Abstract

Competency-based education has become central to the training and assessment of post-graduate medical trainees or residents [1]. In competency-based education, there is a strong focus on outcomes and professional performance. Typically, holistic tasks are used to train, practice and assess the defined outcomes or competencies. In residency training, these tasks are part of the day-to-day clinical practice. The performance of residents in the workplace needs to be captured and stored. A portfolio has been used as an instrument for storage and collection of workplace-based assessment and feedback in various countries, like the Netherlands and the United States. The collection of information in a portfolio can serve or be used for a variety of purposes. These are:

1.
The collection of work samples, assessment, feedback and evaluations in a portfolio enables the learner to look back, analyze and reflect.
2.
The content is used for assessment or making decisions about progress. And
3.
the portfolio is used as an instrument for quality assurance processes.

In post-graduate medical education, these purposes can be combined but this is not always reported transparently. In this paper, we will discuss the different perspectives, how a portfolio can serve these three purposes and what are opportunities and challenges of combining multiple purposes.

Keywords: postgraduate education, feedback, portfolio, reflection, quality insurance


Portfolios in post-graduate medical education

Portfolios are instruments to collect and assess evidence of the learners’ progression in tasks or competencies. The content can be prescribed or is to the choice of the learner. The performance and feedback information is used by the learner to show progress, look back and reflect on feedback, and formulate plans/ learning objectives for improving performance or competence [2], [3]. In the post-graduate setting, Smith and Tillema [4] made a differentiation in types of portfolios based on

1.
the purpose of the portfolio, being either selection or developmentally orientated, and
2.
the setting of use, being either mandated or self-directed.

This led to 4 types:

1.
The dossier portfolio, containing mandated records of achievement, with specified levels of performance, used for selection or promotion.
2.
The training portfolio, containing a mandated collection of acquired skills and competencies, in a fixed format, with some reflective comments on selected evidence.
3.
The reflective portfolio containing a purposeful collection of evidence for personal growth and development, used for promotion or selection. And
4.
the personal development portfolio, containing reflective and personal evaluations of progress in time, enabling discussion and valuing of these activities.

In practice, there may be more of a continuum. A portfolio in post-graduate education is often mandatory, containing prescribed information such as 360 degree feedback rounds, evaluations of clinical performance etc., and it is used for assessment. Therefore, it would be best typified as a training portfolio. However, portfolios in postgraduate education can also have a stronger developmental aspect, with more reflective accounts of personal and professional growth, used for a dialogue with a supervisor or mentor. This diversity in purposes is further discussed in the paragraphs below.


The portfolio to reflect and learn

There are many studies on the effect of reflection in a portfolio on learning in the undergraduate setting. Through reflection, the learner can scrutinize the personal performance, analyse and come to terms with what went well and what went wrong and set out learning objectives for strategies to improve. This supports the learner taking responsibility for his or her learning [5]. Reflection was shown to have a small, positive impact on performance (case-solving capacity) [6]. In addition, a study by Sobral [7] showed that reflection led to progression in learning and was (weakly) correlated to improved academic performance.

In postgraduate education, the learning is done at the workplace. There is some evidence that reflection on experience in the clinical workplace, as in the undergraduate setting, led to a deeper approach to learning, allowing integration of what was learned with existing knowledge and skills [8]. In addition, reflection improved the diagnostic reasoning on complex and unusual patients or cases [9].

The implementation of a portfolio can be used to facilitate or encourage reflection just by the fact that information is collected and reviewed by the learner. However, the perception of residents whether a portfolio should be a tool for reflection varied, ranging from a positive 72% of surgical residents [10] to 42% of General Practitioner trainees [11]. The lack of willingness to reflect in a portfolio could be caused by an uncertainty or lack of confidence on how this information is used [12], especially if the reflective activities are also assessed (see next paragraph). Another finding was that learners considered themselves already as reflective practitioners and the obligation to reflect in the portfolio actually hindered their own approach to professional learning [13]. In addition, time is an important factor. In workplace-based education, workload is high and the collection of workplace-based feedback and evaluation for the portfolio and having to reflect can be a burden [14], [15].

Lack of acceptance is an important issue. Portfolios need a supportive structure and cannot be implemented as an independent device. Research has shown that portfolios work if certain conditions have been fulfilled: mentoring, a structure that allows some flexibility for the learner (agency), an environment that supports learning and providing a meaningful learning experience for the learner [16]. For reflective activities in the portfolio, these conditions are very similar. Very often portfolios in post-graduate training have an electronic format, and this can be used to support or facilitate reflection, e.g. by the use of electronic devices [17], [18], [19]. Keim et al. [20] studied whether the introduction of a portfolio as such increased reflective activities and indeed showed more self-reflective entries in trainees with a portfolio, compared to a control group with no portfolio. The introduction of a smartphone App as support for reflection also increased reflective activities [19]. So, when conditions are met, this seems to result in better acceptance. It will however, remain a challenge to ensure sufficient learner centeredness. For the later, a mentoring system is a prerequisite. Research is unequivocal that mentoring supports and enhances reflective learning [21], [22], [23]. Regardless if reflection is part of the portfolio, a mentor is also considered to be essential in portfolio learning and the portfolio process [11], [16], [21], [24]. Given the previously mentioned lack of confidence or fear that the portfolio content is used ‘against’ the learner, it is important that the mentor is independent, and as noted by Dekker et al., not the daily supervisor or the head of the department [24].

In conclusion, the use of a portfolio, in combination with a mentor, helps and stimulates to look back and reflect at the collected information and this seems helpful for learning and further professional development. Pitfalls are time-constraints and an obligatory nature.


The portfolio as an instrument for decision-making

Competency-based education needs continuous, elaborate and comprehensive feedback systems and assessment [25]. In post-graduate training, the use of a portfolio can facilitate the continuity and comprehensiveness of assessment by collecting information about the trainees progression towards outcomes, milestones, competencies and Entrustable Professional Activities (EPAs) [26].

For portfolio assessment as such, sufficient reliability and validity has been shown, given a number of qualitative criteria such as feedback cycles incorporated in the assessment process, and procedural safeguards in the assessment procedure [16], [27] , [28], [29]. Potential weaknesses are the time-consuming nature of assessment, need for an active feedback culture and acceptance by the users [30]. In addition, as expressed above, we need to be aware of the effects of the assessment of reflection. As expressed by Hodges et al., there is a tension between assessment and accountability on the one hand, and the requirement of trainees to reflect on their actions on the other hand [31], [32]. As indicated in the previous paragraph, reflection in medical education has its merits, it is essential for learning from clinical practice and experiences, and therefore it should be common practice of every medical professional [33]. However, a careful balance should be sought, this could be a portfolio containing reflection in a dialogue with a mentor [33], with an assessment that is fit for purpose, meaningful and not judgmental. This format of assessment of reflection is certainly not easy or common within the field of medical education.

Nevertheless, a portfolio can be a useful and meaningful instrument for the assessment of competencies if certain principles of continuity, comprehensiveness and opportunities for a meaningful dialogue on feedback are applied in the design of the assessment process. Such a design of continuous assessment was advocated by van der Vleuten and Schuwirth [34], [35]. In their model of programmatic assessment, assessment activities are rich in feedback and informative, they are purposefully chosen, aggregated and arranged in time. This will provide a longitudinal flow of information about the trainee, which the trainee can use to learn from feedback and assessment and to plan future learning opportunities. In this model, there are no single (high-stake) decisions, rather the aggregation of all information available is used to come to pass-fail or high-stake decisions, or a decision on promotion. This model resonates with the view of Eva et al., that a continuous model of assessment has the potential to mitigate potentially unwanted features of competency-based assessment. A program of assessment could help to overcome the psychometric challenges and context-specificity of competency assessment, and combine the formative and summative assessment, by broad and purposeful sampling [36]. In addition, in competency-based education, progression through residency is no longer simply time-dependent. Trainees can progress and complete residency at the time that final outcomes or qualifications are met. Therefore, decisions on the level of expected proficiency need to be made. Ten Cate introduced the Entrustable Professional Activity (EPA) to facilitate and aid in the decision on proficiency level [37]. An EPA is a professional unit or task that is entrusted to the trainee to perform without direct supervision once the trainee has shown or provided evidence that the expected level of competency has been reached (37). The Entrustment decision itself resembles the assessment of a portfolio in that multiple sources of information are used, by different supervisors, in various context, to support the validity of the decision [38]. The models and theoretical foundation of entrustment decisions are complex and currently under deliberation, e.g. by the International Competency-Based Medical Education Collaborators (38). It should be kept in mind that an EPA represent a complex set of behaviors that have to be recognized and observed by clinical supervisors in the workplace. Oerlemans et al. suggested that observing clinical performance during a series of encounters by the same supervisor revealed consistent behavior and added valuable information on the trainees’ performance [39]. This aligns with the broad and purposeful sampling strategy used in programmatic assessment. In addition, clinical supervisors need to be supported and trained, Calaman et al. described the development of standard-setting videos in the context of EPAs and entrustment decisions to support direct observation of trainee performance by clinical supervisors [40]. An electronic portfolio format could support the entrustment decision process by integration of information through automatically generated overviews of forms, narrative feedback and levels of proficiency. In the e-portfolio, a timeline per EPA can be presented or generated in which the decisions and levels are shown and the supervisor can easily navigate back to the underlying forms and information.

In conclusion, the portfolio as an instrument for continuous, programmatic assessment is usable in post-graduate medical training, as assessment instruments (such as mini-CEX, observational instruments, technical skills assessments), and feedback instruments (multi-source feedback) are usually operational and collected in the resident portfolio. EPAs are increasingly used in post-graduate education. The theoretical foundation of entrustment decisions is complex. An electronic portfolio format could be helpful in supporting the process and procedures of entrustment decisions.


The portfolio as a quality assurance tool

A portfolio can also be used as a quality assurance instrument, e.g. how many procedures have been performed, success rate of procedures, registration of professional development or courses, patient evaluations etc. The purpose of the portfolio will then be more tailored to a dossier function, in which information is being collected and if needed aggregated and reported. The portfolio as a dossier for management issues can be applicable to both individual trainees and (senior) physicians, as for training programs. An electronic format has greatly enhanced the potential of portfolios as a quality assurance tool as data can now easily be aggregated and compared. In undergraduate programs, the portfolio has also been used to get insights in the hidden curriculum, e.g. to monitor for the practices and values that are taught or transmitted through role modeling [41] or to explore the content of e-portfolio on gender issues entries, a topic often taught in the hidden curriculum [42]. In post-graduate education, the portfolio could also fulfil this role on the quality of clinical teaching, e.g.by monitoring the richness and quality of the narrative feedback.

In addition, many countries have reformed the official requirements for revalidation processes for physicians [http://www.gmc-uk.org/guidance/good_medical_practice.asp], [43], [44], [45], in which evidence of (reflective) activities and practice are often mandatory elements. For these revalidation procedures, portfolios are used [46]. Given the current commonality of portfolio use in post-graduate medical training, this will most likely be a natural continuation of portfolio use once residency training has ended. For senior doctors, this may be the first time that they have to complete a portfolio. As already addressed, the bureaucracy and often complexity of portfolio or in this case revalidation procedures have the danger of transgressing into a requirement, and the process is perceived as less meaningful for stakeholders.


Three purposes: one portfolio for each or all in one portfolio?

We have discussed three purposes of portfolios, i.e. to enable reflection, to assess competencies and as an instrument for quality assurance processes. As was already evident, there is a natural overlap in these purposes, one cannot reflect if no information is being collected, and information on performance or proficiency level is used for assessment and entrustment decisions. A potential pitfall of having everything in one portfolio is that it becomes too complex, bureaucratic and time-consuming. The challenge is to keep portfolio processes meaningful for all stakeholders. We argue that a portfolio can serve all three purposes, but much is dependent on the implementation and how it is used. The different goals of the portfolio should be clear for all stakeholders, residents, clinical teachers and supervisors and mentors. Instruction and communication should be transparent and clearly designed. The mentor is an important source of information for the learner and should serve as an intermediary between the learner/ resident and the program. We believe that a mentor also has an important role in the above mentioned problems with acceptance or reluctance to work and use the portfolio for learning. Best-practice examples for implementation of portfolio in undergraduate and post-graduate medical programs can be found in Driessen et al. [47], Dannefer et al. [48], and Fung et al. [49].

In conclusion, the portfolio needs to be a learner chart [33], in which information is collected (dossier), used for learning guided by a mentor or coach (reflect) and serves as the basis for decisions on proficiency or outcome requirements (assess). Then it can be a learner chart that comprehensively documents the trainees’ progress and which is discussed between a trainee and a mentor to support the trainees’ development. When the portfolio is integrated in a program of assessment, and decision-making is removed from individual assessments and decisions are only based on longitudinally gathered assessment and learning information: then the portfolio becomes the key instrument that fully services learning, assessment and quality assurance.


Competing interests

The authors declare that they have no competing interests.


References

1.
Hawkins R, Welcher C, Holmboe E, Kirk L, Norcini J, Simons K, Skochelak SE. Implementation of competency-based medical education: are we addressing the concerns and challenges? Med Educ. 2015;49(11):1086-1102. DOI: 10.1111/medu.12831 External link
2.
Tillema H. Portfolios as developmental assessment tools. International J Training Develop. 2001;5(2):126-135. DOI: 10.1111/1468-2419.00127 External link
3.
van Tartwijk J, Driessen E. Portfolios for assessment and learning: AMEE Guide no. 45. Med Teach. 2009;31(9):790-801. DOI: 10.1080/01421590903139201 External link
4.
Smith K, Tillema H. Clarifying different types of portfolio use. Ass Eval Higher Educ. 2003;28(6):625-648. DOI: 10.1080/0260293032000130252 External link
5.
Fernsten L, Fernsten J. Portfolio assessment and reflection: enhancing learning through effective practice. Reflect Pract. 2005;6(2):303-309. DOI: 10.1080/14623940500106542 External link
6.
Koole S, Dornan T, Aper L, Scherpbier A, Valcke M, Cohen-Schotanus J, Derese A. Does reflection have an effect upon case-solving abilities of undergraduate medical students? BMC Med Educ. 2012;12(1):75. DOI: 10.1186/1472-6920-12-75 External link
7.
Sobral DT. Medical students' reflection in learning in relation to approaches to study and academic achievement. Med Teach. 2001;23(5):508-513. DOI: 10.1080/01421590126488 External link
8.
Mann K, Gordon J, MacLeod A. Reflection and reflective practice in health professions education: a systematic review. Adv Health Sci Educ. 2009;14(4):595-621. DOI: 10.1007/s10459-007-9090-2 External link
9.
Mamede S, Schmidt HG, Penaforte JC. Effects of reflective practice on the accuracy of medical diagnoses. Med Educ. 2008;42(5):468-475. DOI: 10.1111/j.1365-2923.2008.03030.x External link
10.
Webb T, Aprahamian C, Weigelt J, Brasel K. The Surgical Learning and Instructional Portfolio (SLIP) as a self-assessment educational tool demonstrating practice-based learning. Curr Surgery. 2006;63(6):444-447. DOI: 10.1016/j.cursur.2006.04.001 External link
11.
Pearson D, Heywood P. Portfolio use in general practice vocational training: a survey of GP registrars. Med Educ. 2004;38(1):87-95. DOI: 10.1111/j.1365-2923.2004.01737.x External link
12.
Swallow V, Clarke C, Iles S, Harden J. Work based, lifelong learning through professional portfolios: Challenge or reward? Pharm Educ. 2006;6(2):77-89. DOI: 10.1080/15602210600682228 External link
13.
Austin Z, Marini A, DesRoches B. Use of a learning portfolio for continuous professional development: A study of pharmacists in Ontario (Canada). Pharm Educ. 2005;5. DOI: 10.1080/15602210500282434 External link
14.
Kjaer N, Maagaard R, Wied S. Using an online portfolio in postgraduate training. Med Teach. 2006;28(8):708-712. DOI: 10.1080/01421590601047672 External link
15.
Hrisos S, Illing J, Burford B. Portfolio learning for foundation doctors: early feedback on its use in the clinical workplace. Med Educ. 2008;42(2):214-223. DOI: 10.1111/j.1365-2923.2007.02960.x External link
16.
Driessen E, Van Tartwijk J, Overeem K, Vermunt J, Van Der Vleuten C. Conditions for successful reflective use of portfolios in undergraduate medical education. Med Educ. 2005;39(12):1230-1235. DOI: 10.1111/j.1365-2929.2005.02337.x External link
17.
Driessen E, Muijtjens A, Van Tartwijk J, van der Vleuten C. Web-or paper-based portfolios: is there a difference? Med Educ. 2007;41(11):1067-1073. DOI: 10.1111/j.1365-2923.2007.02859.x External link
18.
Gordon J, Campbell C. The role of ePortfolios in supporting continuing professional development in practice. Med Teach. 2013;35(4):287-294. DOI: 10.3109/0142159X.2013.773395 External link
19.
Könings K, van Berlo J, Koopmans R, Hoogland H, Spanjers I, ten Haaf J, van den Vleuten CP, van Merrienboer JJ. Using a Smartphone App and Coaching Group Sessions to Promote Residents' Reflection in the Workplace. Acad Med. 2016;91(3):365-370. DOI: 10.1097/ACM.0000000000000989 External link
20.
Keim K, Gates G, Johnson C. Dietetics professionals have a positive perception of professional development. J Am Diet Ass. 2001;101(7):820-824. DOI: 10.1016/S0002-8223(01)00202-4 External link
21.
Driessen E, van Tartwijk J, van der Vleuten C, Wass V. Portfolios in medical education: why do they meet with mixed success? A systematic review. Med Educ. 2007;41(12):1224-1233. DOI: 10.1111/j.1365-2923.2007.02944.x External link
22.
Kalén S, Ponzer S, Seeberger A, Kiessling A, Silén C. Continuous mentoring of medical students provides space for reflection and awareness of their own development. Intern J Med Educ. 2012;3:236-244. DOI: 10.5116/ijme.50ad.328c External link
23.
Sargeant J, Lockyer J, Mann K, Holmboe E, Silver I, Armson H, Driessen E, MacLeod T, Yen W, Ross K, Power M. Facilitated reflective performance feedback: developing an evidence-and theory-based model that builds relationship, explores reactions and content, and coaches for performance change (R2C2). Acad Med. 2015;90(12):1698-1706. DOI: 10.1097/ACM.0000000000000809 External link
24.
Dekker H, Driessen E, ter Braak E, Scheele F, Slaets J, van der Molen T, Cohen-Schontanus J. Mentoring portfolio use in undergraduate and postgraduate medical education. Med Teach. 2009;31(10):903-909. DOI: 10.3109/01421590903173697 External link
25.
Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676-682. DOI: 10.3109/0142159X.2010.500704 External link
26.
ten Cate O, Scheele F. Viewpoint: Competency-Based Postgraduate Training: Can We Bridge the Gap between Theory and Clinical Practice? Acad Med. 2007;82(6):542-547. DOI: 10.1097/ACM.0b013e31805559c7 External link
27.
O'Sullivan P, Reckase M, McClain T, Savidge M, Clardy J. Demonstration of portfolios to assess competency of residents. Adv Health Sci Educ. 2004;9(4):309-323. DOI: 10.1007/s10459-004-0885-0 External link
28.
Michels N, Driessen E, Muijtjens A, Van Gaal L, Bossaert L, De Winter B. Portfolio assessment during medical internships: How to obtain a reliable and feasible assessment procedure? Educ Health. 2009;22(3):313.
29.
Gadbury-Amyot C, Kim J, Palm R, Mills G, Noble E, Overman P. Validity and reliability of portfolio assessment of competency in a baccalaureate dental hygiene program. J Dent Educ. 2003;67(9):991-1002.
30.
Van Tartwijk J, Driessen E, van der Vleuten C, Stokking K. Factors influencing the successful introduction of portfolios. Qual High Educ. 2007;13(1):69-79. DOI: 10.1080/13538320701272813 External link
31.
Hodges B. Sea monsters & whirlpools: Navigating between examination and reflection in medical education. Med Teach. 2015;37(3):261-266. DOI: 10.3109/0142159X.2014.993601 External link
32.
Ng S, Kinsella E, Friesen F, Hodges B. Reclaiming a theoretical orientation to reflection in medical education research: a critical narrative review. Med Educ. 2015;49(5):461-475. DOI: 10.1111/medu.12680 External link
33.
Driessen E. Do portfolios have a future? Adv Health Sci Educ. 2016:1-8.
34.
van der Vleuten C, Schuwirth L, Driessen E, Dijkstra J, Tigelaar D, Baartman LK, van Tartwijk J. A model for programmatic assessment fit for purpose. Med Teach. 2012;34(3):205-214. DOI: 10.3109/0142159X.2012.652239 External link
35.
Schuwirth L, van der Vleuten C. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach. 2011;33(6):478-485. DOI: 10.3109/0142159X.2011.565828 External link
36.
Eva K, Bordage G, Campbell C, Galbraith R, Ginsburg S, Holmboe E, Regehr G. Towards a program of assessment for health professionals: from training into practice. Adv Health Sci Educ Theory Pract. 2016;21(4):897-913. DOI: 10.1007/s10459-015-9653-6 External link
37.
Ten Cate O. Entrustability of professional activities and competency-bases training. Med Educ. 2005;39:1176-1177. DOI: 10.1111/j.1365-2929.2005.02341.x External link
38.
Ten Cate O, Hart D, Ankel F, Busari J, Englander R, Glasgow N, Holmboe E, Iobst W, Lovell E, Snell LS, Touchie C, Van Melie E, Wycliffe-Johnes K; International Competency-Based Medical Education Collaborators. Entrustment decision making in clinical training. Acad Med. 2016;91(2):191-198. DOI: 10.1097/ACM.0000000000001044 External link
39.
Oerlemans M, Dielissen P, Timmerman A, Ram P, Maiburg B, Muris J, van der Vleuten C. Should we assess clinical performance in single patient encounters or consistent behaviors of clinical performance over a series of encounters? A qualitative exploration of narrative trainee profiles. Med Teach. 2017:1-8. DOI: 10.1080/0142159X.2017.1270427 External link
40.
Calaman S, Hepps J, Bismilla Z, Carraccio C, Englander R, Feraco A, Landrigan CP, Lopreiato JO, Sectish TC, Starmer AJ, Yu CE, Spector ND, Wwest DC; I-Pass Study Education Executive Committee. The creation of standard-setting videos to support faculty observations of learner performance and entrustment decisions. Acad Med. 2016;91(2):204-209. DOI: 10.1097/ACM.0000000000000853 External link
41.
Hall P, Byszewski A, Sutherland S, Stodel E. Developing a sustainable electronic portfolio (ePortfolio) program that fosters reflective practice and incorporates CanMEDS competencies into the undergraduate medical curriculum. Acad Med. 2012;87(6):744-751. DOI: 10.1097/ACM.0b013e318253dacd External link
42.
Phillips C. Student portfolios and the hidden curriculum on gender: mapping exclusion. Med Educ. 2009;43(9):847-853. DOI: 10.1111/j.1365-2923.2009.03403.x External link
43.
Catto G. Building on the GMC's achievements. BMJ. 2005;330(7501):1205- 1207. DOI: 10.1136/bmj.330.7501.1205 External link
44.
Cain F, Benjamin R, Thompson J. Obstacles to maintaining licensure in the United States. BMJ. 2005;330(7505):1443-1445. DOI: 10.1136/bmj.330.7505.1443 External link
45.
Smeenk F. Medisch-specialistische Vervolg-opleidingen in een Veranderende Maatschappelijke Context. In: Smeenk F, Rutten H, van de Laar E (Hrsg). Toegewijde dokters: Waarom de niet-medische competenties geen bijzaak zijn. 6. Antwerpen-Apeldoorn: Garant; 2016. S.15-43.
46.
Furmedge D, Griffin A, O'Keeffe C, Verma A, Smith L, Gill D. Paper trials: a qualitative study exploring the place of portfolios in making revalidation recommendations for Responsible Officers. BMC Med Educ. 2016;16. DOI: 10.1186/s12909-016-0592-6 External link
47.
Driessen E, van Tartwijk J, Govaerts M, Teunissen P, van der Vleuten C. The use of programmatic assessment in the clinical workplace: A Maastricht case report. Med Teach. 2012;34(3):226-231. DOI: 10.3109/0142159X.2012.652242 External link
48.
Dannefer EF, Henson L. The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine. Acad Med. 2007;82(5):493-502. DOI: 10.1097/ACM.0b013e31803ead30 External link
49.
Fung M, Walker M, Fung K, Temple L, Lajoie F, Bellemare G, Bryson SC. An Internet-based learning portfolio in resident education: the KOALA™ multicentre programme. Med Educ. 2000;34(6):474-479. DOI: 10.1046/j.1365-2923.2000.00571.x External link