Assessment in the legal and medical domain: two sides of a coin Cees P.M. van der Vleuten1 Maastricht University INTRODUCTION It has been such an honour to read the assessment papers in legal education that were written with an earlier paper of mine (C. P. Van der Vleuten & Schuwirth, 2005) as a frame of reference. The papers provide an excellent insight in a number of assessment practices in different law schools. Very striking were the similarities of the issues that are discussed from the legal domain to my own domain, the field of medicine. The papers are addressing notions of reflections, reflective practice, the importance of learning (and assessing) in context (either simulated or real) developing professional competences, definitions of professional competence, the relevance of general skills (professionalism, ethics, values, altruism, empathy, client- centeredness, managing themselves and others in work), and new approaches to assessment (journals, portfolios, extracted examples of work, observation, think- aloud in practice and holistic approaches to assessment). All these notions completely resonate with developments in the medical domain. For this contribution I thought of summarizing some recent developments in the medical domain having 1 Cees is Professor in the Faculty of Health, Medicine and Life Sciences, School of Educational Development and Research ( Special Issue Problematising Assessment in Clinical Legal Education ) ( 156 ) relevance to all these topics: competency frameworks, assessment of performance in context, reflection, and programmatic assessment. This is meant merely as an informative mirror on what happens in this other domain. COMPETENCY FRAMEWORKS The issue of competency and competency definitions has been articulated strongly in the medical domain in recent years. A whole number of countries around the world have engaged in consensus procedures leading to a set of competency frameworks that are nationally agreed upon. Among the most prominent ones are the ones from the US, Canada and the UK such as described in table 1 below. United states (ACGME)2 Canada (Canmeds)3 United Kingdom (Good medical practice)4 · Medical knowledge · Patient care · Practice-based learning & improvement · Interpersonal and communication skills · Professionalism · Systems-based practice · Medical expert · Communicator · Collaborator · Manager · Health advocate · Scholar · Professional · Good clinical care · Relationships with patients and families · Working with colleagues · Managing the workplace · Social responsibility and accountability · Professionalism Many other countries have similar agreed competency frameworks and they vary to some degree. Each of these competencies within the framework is further defined 2 Accreditation Council of Graduate Medical Education (2009). Common Program Requirements: General Competencies. From http://www.acgme.org/outcome/comp/GeneralCompetenciesStandards21307.pdf 3 Frank, J. R., & Danoff, D. (2007). The CanMEDS initiative: implementing an outcomes-based framework of physician competencies. Medical teacher, 29(7), 642-647. See also: http://www.royalcollege.ca/portal/page/portal/rc/canmeds/canmeds2015 4 General Medical Council (2013). Good medical practice: working with doctors for patients. From: http://www.gmc-uk.org/guidance/good_medical_practice.asp into behaviours. The frameworks have been developed in extensive consensus procedures with abundant stakeholder input. What is really interesting is that most competencies emphasize skills outside the “knowledge domain”. They also bear similarity to the skills summarized above that were found in the legal papers. These skills are apparently less bound to the domain in which they were developed. The descriptions of the competencies show remarkable similarity across frameworks. So apparently when different organizations consult stakeholder groups for reaching consensus on what professionals should be able to do, they reach rather similar outcomes. The frameworks have had and still have vast consequences in medical education. They have become the standard by which medical training programs are increasingly being structured both at the undergraduate level as well as at the postgraduate level. To give you an example, The Netherlands has adopted the Canmeds system and has given it legal status. Training programs have to be built around the competency framework, assessment strategies have to be developed to assess these competencies and accreditation procedures inspect the attainment of the competencies. What typically happens is that longitudinal curricular lines are created in which teaching, learning and assessment activities take place in a more coordinated fashion. This is not easy to achieve change, because most training programs are very modularly structured with little transfer of information from one module to the other. Good implementation of competency-based education is therefore challenging and requires good governance of the curriculum as a whole. Many universities and their programs are not used to such kind of governance. Nevertheless, the rising importance of the competency frameworks requires universities and postgraduate training institutions to change. An interesting more recent development has been an alternative way of defining what is competence. It is very difficult for clinical teachers to understand exactly what collaboration means or professionalism or communication and how to define if you master enough of it. When a critical professional activity is taken, say handling a normal delivery of a child, it is clear for any clinician with whom to collaborate, how to act professionally and with whom to communicate. Subsequently a decision can be taken on the level of entrustment of the learner in relation to performing the critical professional activity independently. Often this is done on an entrustment scale with varying degrees of supervision: observing the activity, acting with direct supervision present, acting unsupervised, providing supervision to juniors. Standards are now defined language that clinicians understand feel acquainted with it. They continuously make judgments about patient safety and that is what this entrustment related to (Kogan, Conforti, Iobst, & Holmboe, 2014). The critical professional activities have been termed Entrusted Professional Activities (EPAs) (ten Cate, 2013). EPAs are currently conquering the medical education world and various disciplines have identified their EPAs. By mapping these EPAs on competencies and by formulating at which level of education “milestones” of competencies in the form of rubrics need to be achieved a comprehensive framework can be developed. EPAs have helped shaping what we wish to train and assess in the words of the professionals in the domain thereby creating a natural buy-in and a formal language on what to train and assess.5 ASSESSMENT OF PERFORMANCE IN CONTEXT Many competencies in table 1 are behavioural in nature. Experiential learning is imperative for learning these skills either in the form of simulation or in a real world work setting. Behavioural skills can only be assessed by direct observation. Therefore many observation instruments have been developed and validated in medical education. For simulated performance simulated performance testing is widely used. They are called Objective Structured Clinical Examinations and virtually every medical school in the world uses it (Harden, Lilley, & Patricio, 2015). However, since a number of years assessment methods are developed that used in the unstandardized real clinical environment (Norcini & Burch, 2007). One very popular method is called the Mini-Clinical Evaluation Exercise (Mini-CEX). An assessor directly observes a learner while doing a clinical activity, fills in an assessment form (usually structured according to the Canmeds competencies), and then gives feedback often in the form reflective questioning. Finally some actions are formulated. Rubrics are often used to describe the performance quality. Narrative written feedback is strongly encouraged. The Mini-CEX is repeated a number of times while the learner is in a same clinical setting. 5 An illustration of such a mapping exercise can be found here: https://www.acgme.org/acgmeweb/Portals/0/PDFs/Milestones/PediatricsMilestones.pdf Another popular instrument is the multi-source feedback (MSF). A questionnaire is (electronically) sent to a range of assessors who are relevant to the learner (clinical supervisors, peers, nurses, patients, secretary at the desk, etcetera). The learner also has to complete one as a self-assessment exercise. The questionnaire is also structured according to a competency framework. Each assessor completes the questionnaire (mostly anonymously) and data are aggregated across assessors. A feedback report is generated, for example in a spider chart format showing the self- assessment score, the average assessor score and the cohort score. Narrative information is also here very much encouraged and is part of a feedback report. Often the feedback is moderated in a discussion between supervisor (or mentor) with the learner after the MSF has been completed. MSF procedures are becoming also increasingly popular to assess clinicians in their daily working role as part of their continuous professional development (Overeem et al., 2010). Finally portfolios have become very popular. In a portfolio the evidence burden is reversed. Not the teacher but learner has to prove competence. Therefore the portfolio contains evidence and reflections from the learner. Portfolios have been well researched in medical education (E. Driessen, Van Tartwijk, Van Der Vleuten, & Wass, 2007), (Buckley et al., 2009). Many portfolios now are online and provide all kinds of assessment services (e.g. conducting an MSF assessment) and aggregation tools. There are many more instrument to assess the performance of learners in a clinical context. The more enriching the feedback is, the more serious assessor and learner take these assessments, the more engaging these assessments can be. As is often mentioned in the legal papers on assessment, time is a concern and finding ways to embed these assessment activities in the routine of daily practice is a challenge. REFLECTION AND MENTORING Experiential learning and reflection are closely related. A number of the legal papers discussed the use of reflections for example in the use of diaries. Reflective learning is emphasized in educational theories such the well known model from Schön (Schön, 1983) and Korthagen (Korthagen, Kessels, Koster, Lagerwerf, & Wubbels, 2001). Reflection is the link between the feedback and the performance improvement (Sargeant, Mann, van der Vleuten, & Metsemakers, 2009). Most of the feedback is ignored by learners (Kluger & DeNisi, 1996) and making learners reflect may facilitate the use of feedback. Just like in legal education reflection is not always considered to be enjoyable by learners. Reflection should have a clear education value or otherwise learners disengage with it. In medical education this is often done through mentoring either in peer groups of with faculty coaches or both. Mentoring has been broadly studied and has shown considerable effectiveness on increased use of feedback, improved professional development, career preparation and success and prevention of production loss such as for example through burnout (E. W. Driessen & Overeem, 2013). It has also been a key issue to the success of the use of portfolios and self directed learning (E. Driessen et al., 2007). Reflection has therefore had considerable attention in medical education in recent years and is part of many modern assessment approaches where learners are connected to mentors or coaches and their longitudinal performance on competency development is being monitored and discussed. Learning complex skills, experiential learning, assessment providing feedback, longitudinal monitoring and coaching are all important ingredients that mutually influence each other in a positive way. The ingredients provide the bricks of a highly powerful learning environment. PROGRAMMATIC ASSESSMENT In recent years a more synthetic approach to assessment has been proposed that integrates many of the insights discussed above and is called programmatic assessment (C. P. Van der Vleuten et al., 2012). In this approach a whole assessment program is purposefully designed very similar to a full curriculum design. Methods are carefully chosen for their educational function in that moment in time and in relation to other methods being used in the program. Methods purposefully require a variation in activities: verbalizing, writing, arguing, defending, synthesizing, all following the educational purpose of the learning program. Each moment of assessment is considered to be one data point. Decision-making on pass/failing is disconnected from individual data points. Individual data points only provide feedback to the learner. Decisions are based on many data points by aggregating the information across data points being gathered. The higher the stake of the decision the more data points are needed. Learners are coached in using the assessment information for planning their learning or for remediation. An overarching structure such as a competency framework is used to aggregate the assessment information (and other learning information such a learning or work products) in a meaningful way. Independent committees take progression decisions based on all the information. Currently a number of education practices are using programmatic assessment in their curriculum (Dannefer & Henson, 2007), (Bok et al., 2013), (Heeneman, Oudkerk Pool, Schuwirth, Vleuten, & Driessen, 2015), (Chan & Sherbino, 2015) and many more are working towards it. Although educationally appealing, changing towards programmatic assessment presents a great challenge requiring substantial staff buy- in, good leadership and strong central governance over the curriculum. Many universities lack such organizational virtues. Nevertheless, parts of programmatic assessment, i.e. the feedback orientation or the mentoring, are very valuable approaches to modernize our assessment more evolutionary. Often one hears that assessment drives learning. In programmatic assessment learning drives assessment. Perhaps many more ways of assessment are viable in our educational practices inspired on this mantra. CONCLUSION Medical education has embraced the move towards competency-based education in which consensus is sought on what to train. Assessment methodology is following this movement resulting in considerable more performance assessment in the reality of the professional context. This move has been strongly promoted to the problems in health care and patient safety (Frenk et al., 2010). Without responding to the needs of society education will fail on its mission to prepare our learners for the labour market. It is difficult to compare the needs in legal and medical education, but from the papers it is clear that many parallels do seem to exist. As has been mentioned a number of times in assessment papers in legal education, cost is an issue. All the assessment approaches above are not cheap. In reality staff- student ratios are probably worse in legal education as compared to medical education. Despite of the cost and the realization that we will not get more funding, we need to think of ways how to implement some of these ideas. We will not be able to resolve this resource constraint without more fundamental scrutiny of our funding allocation in education. In my view we spend too much resources on information transmission to learners (C. Van der Vleuten & Driessen, 2014). Learning is about information processing and not about information consumption. In my view it is a waste of resources that the same but different professor gives the same lecture across rather similarly across the world. Expensive teacher time should not be wasted to information delivery but to the scaffolding of the information processing of learners, preferably in small face-to-face settings. Meaningful assessment information providing the necessary feedback to the learning is part of this scaffold. Two of the most powerful effects on learning are then united: the teacher and feedback. What more could you wish for? References Bok, H. G., Teunissen, P. W., Favier, R. P., Rietbroek, N. J., Theyse, L. F., Brommer, H., Jaarsma, D. A. (2013). Programmatic assessment of competency-based workplace learning: when theory meets practice. BMC Medical Education, 13(1), 123. Buckley, S., Coleman, J., Davison, I., Khan, K. S., Zamora, J., Malick, S., Popovic, C. (2009). The educational effects of portfolios on undergraduate student learning: a Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 11. Medical teacher, 31(4), 282-298. Chan, T., & Sherbino, J. (2015). The McMaster Modular Assessment Program (McMAP): A Theoretically Grounded Work-Based Assessment System for an Emergency Medicine Residency Program. Academic medicine: journal of the Association of American Medical Colleges. Dannefer, E. F., & Henson, L. C. (2007). The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine. Academic Medicine, 82(5), 493-502. Driessen, E., Van Tartwijk, J., Van Der Vleuten, C., & Wass, V. (2007). Portfolios in medical education: why do they meet with mixed success? A systematic review. Medical education, 41(12), 1224-1233. Driessen, E. W., & Overeem, K. (2013). Mentoring. In K. Walsh (Ed.), Oxford Textbook of Medical Education (pp. 265-284). Oxford: Oxford University Press. Frenk, J., Chen, L., Bhutta, Z. A., Cohen, J., Crisp, N., Evans, T., . . . Zurayk, H. (2010). Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet, 376(9756), 1923-1958. doi: 10.1016/S0140-6736(10)61854-5 Harden, R. M., Lilley, P., & Patricio, M. (2015). The Definitive Guide to the OSCE: Elsevier Health Sciences. Heeneman, S., Oudkerk Pool, A., Schuwirth, L. W., Vleuten, C. P., & Driessen, E. W. (2015). The impact of programmatic assessment on student learning: theory versus practice. Medical education, 49(5), 487-498. Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. . Psychological Bulletin, 119, 254-284. Kogan, J. R., Conforti, L. N., Iobst, W. F., & Holmboe, E. S. (2014). Reconceptualizing variable rater assessments as both an educational and clinical care problem. Academic Medicine, 89(5), 721-727. Korthagen, F. A. J., Kessels, J., Koster, B., Lagerwerf, B., & Wubbels, T. (2001). Linking theory and practice: The pedagogy of realistic teacher education. Mahwah, NY: Lawrence Erlbaum Associates. Norcini, J., & Burch, V. (2007). Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach, 29(9), 855-871. doi: 10.1080/01421590701775453 Overeem, K., Lombarts, M., Arah, O. A., Klazinga, N. S., Grol, R. P., & Wollersheim, H. C. (2010). Three methods of multi-source feedback compared: a plea for narrative comments and coworkers' perspectives. Medical teacher, 32(2), 141- 147. Sargeant, J. M., Mann, K. V., van der Vleuten, C. P., & Metsemakers, J. F. (2009). Reflection: a link between receiving and using assessment feedback. Advances in Health Sciences Education: Theory and Practice, 14(3), 399-410. doi: 10.1007/s10459-008-9124-4 Schön, D. A. (1983). The reflective practitioner: How professionals think in action (Vol. 5126): Basic books. ten Cate, O. (2013). Nuts and bolts of entrustable professional activities. Journal of graduate medical education, 5(1), 157-158. Van der Vleuten, C., & Driessen, E. (2014). What would happen to education if we take education evidence seriously? Perspectives on medical education, 3(3), 222- 232. Van der Vleuten, C. P., & Schuwirth, L. W. (2005). Assessing professional competence: from methods to programmes. Medical education, 39(3), 309-317. doi: 10.1111/j.1365-2929.2005.02094.x Van der Vleuten, C. P., Schuwirth, L. W., Driessen, E. W., Dijkstra, J., Tigelaar, D., Baartman, L. K., & van Tartwijk, J. (2012). A model for programmatic assessment fit for purpose. Med Teach, 34(3), 205-214. doi: 10.3109/0142159X.2012.652239