Reflection and assessment in clinical legal education: Do you see what I see? Georgina Ledvinka1 Introduction This paper discusses issues surrounding reflection, and assessment of reflection, in clinical legal education. The first section of the paper examines what reflection is and why it plays an important role in learning. It considers the educational theory underlying reflection and how this can help to inform the way in which reflection is employed in individual programmes. Suggestions are noted for how to encourage students to reflect. There is discussion of issues concerning assessment of reflection including whether it is acceptable to assess reflection at all, and if so, how fair and consistent assessment might be achieved given its inherently subjective nature. The second section of the paper discusses a case study on assessment of reflective work within the clinical law programme at Northumbria University. The study considers the implications of current assessment methods and whether they achieve acceptable levels of consistency between markers. SECTION1: Reflection and Assessment What is reflection and why is it important? In clinical legal education students learn by engaging in some form of hands-on legal experience such as simulated case work, work-based placement or live client environment.2 The experience gives students an opportunity to apply and enhance the legal knowledge they gain in the lecture theatre and seminar room, and it gives context to such learning. In addition to the hands-on legal experience there is a second main element of clinical legal education, and that is reflection. Reflection is a vital part of the process; it is the magic ingredient Reflection and assessment in clinical legal education: Do you see what I see? 29 1 Georgina Ledvinka is a senior lecturer at Northumbria University in the United Kingdom. She works as a solicitor/supervisor within the law school's clinical legal education programme. 2 According to Grimes these are the current main forms of clinical legal work. See "The Theory and Practice of Clinical Legal Education", Grimes, R, in Effective Learning and Teaching in Law, Burridge, R, et al (eds) Kogan (1996) at p. 140. which converts legal experience into education. Imagine two students, A and B, who are participating in a live client programme. Both students are asked to interview a client for the first time. Student A's interview goes well; she seems to have a natural flair for eliciting information from the client and explaining how the case is likely to progress. Later, however, in discussion with her tutor and peers, when Student A describes the interview she is only able to do so in the most basic, descriptive terms. She has little perception of what made the interview proceed well and cannot express her feelings about the experience. Student B also has a successful first interview with her client. Afterwards she meets with her tutor and peers and discusses the experience in detail, including B's own perception of the strengths and weaknesses of her performance. B relates the interview to previous experiences and explains to the group what she feels she would do differently next time. There are few who would dispute that B has undergone a higher quality learning experience than A. The difference is due to reflection. A has learned little, if anything, from having had the interview whereas B has actively rationalised the experience and related it to her existing mental framework, resulting in true learning. But what, exactly, is reflection, and why is it so important to learning? Reflection is something which human beings do naturally as part of everyday life. Whilst we tend not to mull over routine, day to day experiences, if something interesting or out of the ordinary occurs it is quite common to think about it afterwards, to replay the experience in our mind's eye and think what we might have done differently. We often indulge in this kind of reflection even if there is no possibility of changing what happened. Sometimes we think about our experiences in order to evaluate our performance, and identify what we could have done better. On other occasions we may be confronted with some complicated or difficult scenario for which there is no immediate solution. By reflecting on the problem, even at times when we might be engaged in another activity, we can sometimes find a solution or way forward. This is good news for clinical law teachers because it means that students come to us with an innate ability to reflect. However, reflection in a clinical law context is generally not the same as the informal mulling over described above. Programmes have (or ought to have) a clear vision of the way in which students must reflect in order to meet the requirements of the course, and this often involves something more formal than students may be used to. One of the challenges for clinical law teachers is to get students to recognise they already have reflective skills, which they can harness and develop in order to maximise the learning opportunity offered by clinic. Dewey was one of the earliest theoreticians to appreciate the importance of reflection in learning. In 1933 he described reflection as: '... active, persistent and careful consideration of any belief or supposed form of knowledge in the light of the grounds that support it, and further conclusions to which it leads...it includes a conscious and voluntary effort to establish belief upon a firm basis of evidence and rationality.'3 30 Journal of Clinical Legal Education August 2006 3 How We Think, Dewey, J, D C Heath and Co (1933) Boud, Keogh and Walker have said that: 'Reflection is an important human activity in which people recapture their experience, think about it, mull it over and evaluate it. It is this working with experience that is important in learning.'4 Moon has defined reflection in the following terms: 'Reflection is a form of mental processing - like a form of thinking - that we use to fulfil a purpose or to achieve some anticipated outcome. It is applied to relatively complicated or unstructured ideas for which there is not an obvious solution and is largely based on the further processing of knowledge and understanding and possibly emotions that we already possess.'5 Race says that: 'The act of reflecting is one which causes us to make sense of what we've learned, why we've learned it, and how that particular increment of learning took place. Moreover, reflection is about linking one increment of learning to the wider perspective of learning - heading towards seeing the bigger picture. Most of all, however, it is increasingly recognised that reflection is an important transferable skill, and is much valued by all around us, in employment, as well as life in general.'6 What emerges from these definitions, and from the educational theory discussed below, is that reflection is a method of learning and teaching. Moon makes the point that if we can encourage students to be reflective we are helping them to develop a habit of processing cognitive material which can lead students to ideas beyond the curriculum, beyond learning outcomes, and beyond their teachers.7 In other words, we are helping them to develop tools for life long learning.8 Reflection is an important aspect of many, if not most, current theories of education and learning. While it may not be necessary for the clinical law teacher to have an intricate appreciation of all of these theories, it can be helpful to have at least a working knowledge of some of the more prominent ones, as they can inform us about how to incorporate and employ reflection as part of Reflection and assessment in clinical legal education: Do you see what I see? 31 4 Reflection: Turning Experience into Learning, Boud, D, Keogh, R, and Walker, D (eds) Kogan Page (1985) at p. 19. 5 See Reflection in Higher Education Learning, by Moon, J, PDP Working Paper 4, LTSN (2001). See also Reflection and Employability, Moon, J, LTSN (2004) where Moon refers to the work of Goleman (1995) on emotional intelligence and its role in learning. 6 Evidencing Reflection: Putting the "w" into reflection, Race, P, ESCALATE Learning Exchange (2002) 7 Reflection in Higher Education Learning, Moon, J, op cit. 8 Boon argues that reflective skills are particularly pertinent to students of law as they, especially, need to develop a perspective which enables them to ask why they should act in a particular way. He says this must involve a scholarly inquiry into action, motivation and ethics, laying the foundation of an ability to reflect not only on performance, but on the underlying rationale for action. See "Skills in the initial stage of legal education: theory and practice for transformation" by Boon, A, in Teaching Lawyers' Skills Webb, J, and Maughan, C, (eds) Butterworths (1996). 9 There is a considerable repertoire of published material in the area of educational theory which can help to guide clinical law teachers in how to incorporate reflection as part of their own programmes. This paper does not seek to reprise this body of work but rather to discuss a selection of educational theories which may be of particular relevance and application. For a comprehensive literature review of reflection see Reflection in Teacher Education: Towards Definition and Implementation by Hatton, N, and Smith, D, of the University of The University of Sydney, Australia (2006) which can be located at http://alex.edfac.usyd.edu.au/LocalResource/Study 1/hattonart.html (accessed 18.09.06). An earlier (1995) version of the paper appears in Teaching and Teacher Education, 11, (1) 33 - 49. our clinical programmes.9 This paper briefly discusses the seminal theories of Kolb and Schön and also relates clinical legal learning to constructivist learning theory. It is also worth noting that if clinical law teachers have some knowledge of the theoretical background of reflection, we can share it with our students. As discussed below, some students are not natural reflectors.10 Not everyone finds it easy or comfortable to articulate their innermost thoughts and feelings about an experience. Such students can be quite resistant to giving reflection, and they can find it extremely difficult when they try. If we can share some of the educational theory which underlies reflection, it can aid such students to have a broader understanding of why they are being asked to reflect and why it matters. For a student showing borderline reflecting effort/abilities, this could make all the difference.11 Kolb's learning cycle Kolb's experiential learning cycle can be depicted as follows:12 Figure 1: Kolb's experiential learning cycle13 This cycle, or spiral, represents the process by which Kolb suggests students engage in learning. It is a cycle of experience, reflection, thinking and acting.14 The cycle can be entered at any point, but the learner always follows the same sequence, and indeed the learner may 'go round' once or several times as part of a learning process. 32 Journal of Clinical Legal Education August 2006 10 Moon notes that just as some students find reflection a difficult thing to do, some staff find it hard to understand also. See supra note 7 at p. 10. 11 See "Taking reflection seriously: How was it for us?" by Maughan, C and Webb, J in Teaching Lawyers' Skills by Webb, J, and Maughan, C (eds) Butterworths (1996), in which the authors discuss using materials on reflection and educational theory as part of their legal process course, to encourage students to have a greater appreciation of the theoretical basis for reflection. 12 Experiential learning - experience as the source of learning and development Kolb D A, Prentice-hall (1984). 13 This diagram is reproduced from Gibb's publication Learning by doing: a guide to teaching and learning methods, Further Education Unit (1988) which includes examples of the application of Kolb's learning cycle in various teaching scenarios. 14 Or as Moon describes the cycle, there is having an experience, reflecting on the experience, learning from the experience, and then trying out what you have learned. See supra note 7. Applying Kolb in the context of clinical legal education, we can take the example of a student who is learning how to interview clients. In this example the student will engage in a simulated interview with an actress playing the role of the client and the entire interview is to be video recorded. The cycle might be entered at the conceptualisation stage, with the student having been asked to read a list of materials on effective client interviewing. Having digested the materials the student begins to form generalisations about what makes a successful interview. Next he experiments with these concepts and generalisations as he prepares his plan for the interview. The interview takes place as an active experience, with the student actively testing and exploring his ideas and assumptions.15 Following the interview the student engages in reflection. As teachers we can employ any number of methods to assist our students to engage in effective reflection, thus maximising the potential for learning.16 For example, in this scenario the actress could be asked to give feedback to the student saying how effective the interview was from her point of view. In a group session the student could be asked to report on the interview to his peers and give an assessment of his own performance, and this could be followed by the group watching the video recording and giving peer assessment. Such feedback given in a timely manner can assist the student to elucidate and articulate his thoughts and feelings about the interview, thus facilitating effective reflection, and it can guide him towards developing his concepts about good interview techniques, which can be put into practice next time the student prepares for and carries out another interview, thus going around the cycle again. Schön's theory of the reflective practitioner Another theory of relevance is Schön's theory of the reflective practitioner.17 Schön observed that professional education tended to distinguish between knowledge and action by assuming that professional practice is merely the application of a body of knowledge to a practical situation. However, Schön notes that rarely if ever are things that simple in practice; that real life tends to involve messy, indeterminate situations which professionals try to sort out by a combination of knowledge, intuition and action. Schön argues that reflection is an integral part of this problem solving process. Whilst we have a body of tacit knowledge which helps us to respond spontaneously and unconsciously to get through every day tasks, to deal with novel situations we need to expand our repertoire of responses. We use reflection to bring our tacit knowledge to the surface so that we can consciously confront and assess its application to the novel problem at hand.18 Reflection and assessment in clinical legal education: Do you see what I see? 33 15 Gibb, G, supra note 13 at section 2. 16 These methods are discussed in further detail below. See also Gibb, G, supra note 13. As Gibb says, this kind of experiential learning is not the same as 'discovery' learning where it is hoped that learners will discover things for themselves in a haphazard way through sudden bursts of inspiration. The activity should be carefully designed by the teacher and the experience carefully analysed afterwards for learning to take place. A crucial feature of this kind of experiential learning is the structure devised by the teacher within which learning takes place. 17 See The Reflective Practitioner: How Professionals Think in Action by Schön, D, Arena Publishing (1983) and also Educating the Reflective Practitioner by Schön, D, Jossey-Bass (1987). 18 See Maughan and Webb, supra note 11. The potential for applying Schön to clinical legal education is immediately apparent. In conventional class room teaching legal problems given to students tend to be nicely confined, solvable (without too much difficulty) and limited to the subject area in question. On the other hand, as practitioners and clinicians well know, cases in the 'real world' are nothing like this. Almost any client seeking legal advice will have a problem which encompasses different areas of law, and the facts are never presented neatly on a plate. This means that students dealing with such situations cannot simply fall back on conventional class room teaching. They need to develop new strategies and approaches to problem solving, through the process of reflection. Constructivism Constructivist learning theory holds that learning is not something that happens passively, but rather that students participate actively in learning and construct their own knowledge.19 As Ormrod has put it, "learning involves constructing one's own knowledge from one's own experience."20 This is done through two processes, known as accommodation and assimilation. When a learner has an experience which aligns with their internal understanding of the world they assimilate that experience into the existing framework. If an experience does not fit the existing framework then the learner re-frames his or her internal understanding of the world to accommodate what has happened. In this way the learner constructs new knowledge. Constructivist-based pedagogies tend to be founded on a belief that learning is best accomplished by a hands-on approach. The idea is that learning is a personal endeavour where students engage in experimentation and draw their own discoveries and conclusions. Under this model the teacher acts as a facilitator who encourages students to discover principles for themselves and to construct knowledge by working to solve realistic (or indeed, real) problems, often in collaboration with others. DeVries suggests that the teacher ought to engage with students whilst they are undertaking activities, wondering aloud and posing questions to promote students' reasoning.21 This notion can be applied in a clinical law context. For example, if students are engaged in a negotiation role play exercise, the teacher could be present and ask probing questions of students such as 'What does that disclosure make you think about the strengths and weaknesses of the other side's case?' or 'Why do you think it is appropriate to make the opponent an offer at this stage?' Such contributions by the teacher can encourage the student to reflect more deeply on his approach to the activity, whilst he is doing it, rather than being swept along by momentum and acting without conscious thought.22 When the exercise is finished the teacher could go over the questions posed during the exercise and ask the students to discuss them again, with the benefit of hindsight. Thus the questions could aid the student to reflect upon the activity and reach a deeper level of learning. Constructivism can help to explain the distinction between 'surface' learning and 'deep' learning.23 With 'surface' learning the student is likely to have little interest in what he is learning. His prime motivation is assessment. He will commit topics to memory for regurgitation upon assessment, often to be forgotten as soon as the exam has passed or the essay has been handed in. He does not 34 Journal of Clinical Legal Education August 2006 19 The theory of constructivism is generally attributed to Piaget. See The Psychology of Intelligence by Piaget, J, Routledge (1950). 20 'Educational Psychology: Developing Learners', Ormrod, J E, (4th edn) (2003). 21 Developing constructivist early childhood curriculum: practical principles and activities, DeVries et al., Teachers College Press (2002). 22 This tallies with Schön's concept of reflection in action. See supra note 17. 23 Moon also uses the terms 'meaningful' and 'non- meaningful' learning. See Reflection and Employability, Moon, J, LTSN (2004). aim to achieve any real understanding of the subject matter. In constructivist terms, this person is learning in a passive sense; he makes little or no effort either to accommodate or assimilate new learning to past experience and thus he never moves beyond a surface appreciation of the topic. By contrast, the 'deep' learner often has a genuine interest in the topic. He enjoys learning; he wants to understand what he is learning about. The 'deep' learner relates what he is taught to past experience and takes the time to think about feedback in order to improve future performance. In this manner the 'deep' learner constructs new knowledge.24 It can be seen that reflection plays an important role in distinguishing between the two learning styles. Whereas a 'surface' approach is marked by unrelatedness, memorisation and unreflectiveness,25 the 'deep' learner reflects on experience: 'Reflection is a way of getting students to realise that learning is about drawing on life experiences, not just something that takes place in a classroom. It enables students to think about what and how they learn and to understand that this impacts on how well they do.'26 As wonderful as reflection is, it is not a cure-all which is guaranteed to turn out sensitive, ethical lawyers, or those who have particularly good negotiation/advocacy/interviewing skills, or whatever it is we particularly want our students to achieve. We must remember that reflection is a method of learning which students can employ and teachers can facilitate, but the eventual outcome is in the hands of the students and the teachers. If, for example, we wish to promote ethical awareness amongst students, we must give them concrete experience which gives rise to ethical issues, and then as teachers we must facilitate their reflection in a meaningful way. If we wish to promote technical drafting skills, we need to give students experience of drafting and then we should encourage their reflective attention towards the technical aspects of drafting.27 One of the most useful aspects of reflection is its chameleon-like versatility. Reflection can be applied in any number of learning contexts, from professional skills to broader issues of social awareness and justice and even to living a fuller intellectual, emotional and professional life. Indeed, as Macfarlane says, 'A reflective model encourages the development of both cognitive and affective theories of moral and ethical behaviour, challenging students to integrate these into their personal belief systems as a result of their experiences instead of (at best) passively absorbing the 'rules' of professional conduct.'28 Reflection and assessment in clinical legal education: Do you see what I see? 35 24 See also supra note 7 at p. 5. 25 "Diversifying assessment and developing judgment in legal education" Hinett, K, and Bone, A, in R Burridge et al (eds) Effective Learning and Teaching in Law, Kogan Page (1996) at p. 54. 26 Developing Reflective Practice in Legal Education, Hinett, K, UKCLE, LTSN, at p. 6. 27 The decision about what kind of lawyers we are, or should be, trying to produce from law clinics is probably down to individual teachers or institutions and is, in any event, outside the scope of this paper. 28 "Pedagogic Principles, Certification Needs and the Assessment of "Reflective Practitioners"", Macfarlane, J, International Journal of the Legal Profession 5(1) 1-23 (1998). And there is an added benefit of encouraging students to reflect. By engaging in reflection students come to have a better understanding of their own cognitive functioning, making them more aware of how they learn. This increased self- awareness of learning, or metacognition, is correlated with better learning.29 In other words, by encouraging students to reflect we are helping them build for themselves a self-awareness which will promote more successful learning in the future.30 How to promote student reflection Now that we know what reflection is, and what its potential benefits are, it is useful to consider some practical ways in which clinical law teachers can encourage students to reflect. a) Course design - at a fundamental level, the ways in which courses are designed can have a significant impact on whether effective reflection is likely to occur. For example, in their legal process course Maughan and Webb took the view that in order to learn, students had to know how they were learning. Accordingly Maughan and Webb devoted a number of workshops to learning theory as part of which students were required to examine ideas such as Schön's concept of reflection, behaviourist and cognitive learning theories and discrepant reasoning.31 By having this theoretical background it was believed that students would have greater understanding of reflection and why it formed part of their course. b) Teacher knowledge - it is beneficial if the teacher has knowledge of the educational theory regarding reflection. c) Arrangement of class rooms - the way in which a class room is arranged can have a significant impact on whether reflection is forthcoming. Compare the conventional set-up with students seated behind rows of desks and the chalk-wielding teacher standing at the front, with a small group setting where students and teacher are ranged equally around a table, or better still, in a circle with no furniture to divide the group. Immediately the latter does away with traditional power props; the teacher is present at the same level as students, and the circle arrangement encourages discourse amongst the group. d) Teacher to act as facilitator - the role adopted by the teacher should be that of facilitator, rather than playing the master who can give the answers on every issue.32 36 Journal of Clinical Legal Education August 2006 29 Moon refers to two studies, one by Ertmer and Newby in 1996 (evidence that good learners have better metacognitive processes than poor learners) and one by Main in 1985 (study skills programmes that support learners' awareness of their learning processes seem to be more successful than those which focus on techniques). See supra note 7 at p. 7. 30 In addition to these student centred benefits of reflection, there are other wider advantages to be had from encouraging reflection amongst students. Hinett, supra note 26, and Race, supra note 6, both refer to current UK government agendas for widening participating in higher education, which mean that higher education will rapidly have to cater for a wider and more varied population with diverse cultures and differing learning needs and capacities. Hinett argues that in this context reflective practice offers a flexible framework in which students can make sense of their own development, and it can encourage them to become lifelong learners. Race comments that with increased attention to student retention in higher education, reflection can be one of the most powerful vehicles for alerting teaching staff to 'at risk' students, so that appropriate compensations and adjustment may be made to reduce the risk of withdrawal from higher education. 31 See supra note 11. 32 Indeed to be an effective facilitator the teacher should resist the temptation to give answers, and try to guide the students towards finding them for themselves. See Webb's comments on his view of the facilitator's role at p. 268 of Maughan and Webb, supra note 11. e) Learning environment - reflection can be further encouraged by the creation of a supportive and non-judgmental learning environment. If students know there are no right and wrong answers in reflection, and that they can be free to say things which might otherwise appear stupid or 'un-cool' in another setting, it can be enormously liberating and conducive to quality discussion.33 f) Small groups - having students work on problems or cases in small groups can further facilitate reflection. As students become more comfortable working closely with peers there is greater opportunity for peer and self assessment. Further to these 'background' considerations there are the actual methods which can be employed with students to promote reflection.34 These include: g) Self and peer assessment, which Boud describes as the involvement of students in identifying standards and/or criteria which apply to their work, and in making judgments about the extent to which they have met these criteria and standards.35 With self assessment the student can contemplate not only whether the work he has produced meets the relevant standards and criteria, but also the process of learning involved in producing the work, thereby promoting metacognition as discussed above. h) Learning journals, logs and diaries, which may be structured or unstructured. Students can be encouraged to use these items to reflect regularly over a period of time with the aim of improving or supporting learning.36 i) Oral presentations including some reflective element. j) Reflective exercises, to encouraging effective reflection. k) Reflection on work experience, work-based learning, placement learning etc. l) Portfolios, which generally include a reflective element. m) Personal development planning, which the Quality Assurance Agency has defined as a structured and supported process undertaken by an individual to reflect upon his or her own learning, performance and/or achievement and to plan for their personal, educational and career development.37 Within these structures it can be helpful to give students more detailed guidance about how to reflect. Moon suggests a two stage guidance process may be helpful for students: an initial presentation stage to introduce ideas about reflection, and then a second stage to focus on deepening the process of reflection.38 The first stage might involve consideration of points such as what reflection is and how it differs from more familiar forms of learning, why reflection is being Reflection and assessment in clinical legal education: Do you see what I see? 37 33 Although, as Maughan and Webb note, supra note 11, it should be a supportive group, not a support group. The function is to facilitate learning, not just make people feel better. 34 See Hinett, K, supra note 26, and Moon, J, supra note 7. 35 Enhancing Learning Through Self Assessment, Boud, J, HERDSA (1995). 36 As Moon observes, learning journals have been used successfully in most disciplines including the sciences and mathematics. See Learning Journals: A Handbook for Academics, Moon, J, Kogan Page (1999). If students are required to submit reflective written material later in the course, for example as part of a portfolio, learning journals, logs and diaries can form useful raw material on which students can draw. This would involve second- order reflection, as discussed below. 37 Guidelines for HE Progress Files, Quality Assurance Agency (2001) 38 Moon, J, supra note 7 starting at p. 10. used to facilitate the relevant area of learning, why it is acceptable to write reflective work in the first person, and examples could be given to the group of good and poor reflective writing, in order to generate discussion.39 The second stage is based on a developing awareness of knowledge and how it is constructed, including the way in which events can be conceived of differently according to emotions and frames of reference. For example, students could be asked to reflect upon a legal dispute from the point of view of both claimant and defendant. Moon also refers to second-order reflection, where a student is asked to look through previous reflective work and write a reflective overview.40 Race comments that it is probably unwise to attempt to 'teach' reflection. He suggests that the process can be illustrated but in the final analysis reflection remains an individual act in most circumstances.41 Race argues that the most efficient way to help people reflect, and to evidence their reflection, is by providing them with questions as devices to help them to focus their thinking, and to direct their thinking towards those areas of work where reflection can pay highest dividends. He suggests that deep reflection can be generated by clusters of questions. These might include past, present and future-tense questions, such as: 1. What worked really well for you? (past tense) 2. Why do you now think that this worked well for you? (present tense) 3. What are you going to do as a result of this having worked well for you? (future tense) Alternately clusters of questions can comprise a scene-setting starter, and the sub-questions which follow should be probing or clarifying questions, intentionally leading towards deeper or more focussed reflection. Race notes that often such clusters begin with interrogatives such as 'who', 'what', 'when', 'where', why' and 'how'. Some examples given by Race include: • What was the most boring or tedious part of doing this assignment for me? Can I see the point of doing these things? If not, how could the assignment have been re-designed to be more stimulating and interesting for me? • What have I got out of doing this assignment? How have I developed my knowledge and skills? How do I see the payoff from doing this assignment helping me in the longer term? • What are the three most important things that I think I need to do with this topic at this moment in time? Which of these things do I think is the most urgent for me to do? When will I aim to start doing this, and what is a sensible deadline for me to have completed it by?42 38 Journal of Clinical Legal Education August 2006 39 Following the first stage it could be helpful to ask each student in the group to produce a short document setting out his or her views on how to approach reflection and what makes good reflection, or any of the other topics covered in stage one. Rather than the teacher providing a hand-out, this will encourage students to take ownership of the material as well as increasing their understanding of reflection. 40 Moon suggests this can be done by way of a double entry journal. Students write only on one half of a vertically divided page. They leave the other space blank until another time, when they go through the initial material writing further comments that emerge from their more coherent overview of the initial work. See Moon, supra note 7 at p. 14. 41 This accords with the view that reflection is a method of teaching and learning rather than a substantive topic with right and wrong answers. To the extent that it is possible to teach a method of learning, reflection is capable of being taught, however, Race's approach to teaching reflection is in itself facilitative by encouraging students to construct the method for themselves through use of clusters of questions. See supra note 6. 42 Race, P, supra note 6. In addition to occasions when students are required to engage in reflection as part of their course, we can encourage them to reflect whenever they have a particularly acute learning experience. Say, for example, a student prepared diligently prior to representing a client at an Employment Tribunal hearing but, despite her best efforts, the hearing was a disaster. As soon as possible after the event the student should be encouraged to reflect on the experience. The reflection could take place in a small group setting with the student's peers and tutor, where the student could discuss her perceptions of what went wrong and how she felt about it. Other students could be encouraged to offer constructive comments. In addition, the student could make a written record of her reflection. This method would encompass several of the reflection mechanisms discussed above, with the added benefit that capturing all of this contemporaneously is likely to focus the student's mind and maximise the potential for learning. Should we assess reflection? On a traditional undergraduate law degree, assessment of substantive areas of law such as tort or property focuses on what the student knows. The student's knowledge is adjudged against a set of learning outcomes. Assessment on a clinical legal education programme is subtly, but importantly, different. Not only are we assessing the student's substantive knowledge and skills, but also the learning journey he or she has taken from the beginning to the end of the course. In order to assess the learning journey we must have some evidence that it took place and what it encompassed. Reflection, especially written reflection, provides this evidence. As mentioned earlier in this paper, reflection is a normal human activity but many students find it challenging to engage in the more formalised type of reflection which is often required as part of a clinical law programme. Although reflection need not be structured or formal compared with traditional academic work, there is generally an obligation for students to reflect in a disciplined manner, often at set times or occasions, and there is also a requirement for students to evidence their reflection usually in writing. Some students find it extremely difficult and feel very self- conscious engaging in this kind of writing, and it can be a challenge for clinical law teachers to coax good quality reflection out of such students. On the clinical legal education programme at Northumbria University43 we try to identify any problems with reflection early in the academic year by requiring students to submit a sample piece of written reflection for formative feedback. The idea is to let students know, early, if they are reflecting in an appropriate manner or if they need to adapt their approach. Clinic staff report that from these early sample pieces of reflection a substantial proportion of students, even good students, perhaps as many as one fifth overall, fundamentally fail to grasp what reflection is and how it can be evidenced. The most common error seems to involve submitting a piece which is purely descriptive of the experience on which the student has chosen to 'reflect', and which contains little or no qualitative analysis. Even where students seem to grasp what is required, an Reflection and assessment in clinical legal education: Do you see what I see? 39 43 Northumbria University offers a 'combined' law degree which is unique in the UK. It enables students to complete, within four years, their undergraduate law degree as well as their postgraduate professional qualification as solicitor or barrister. At the present time clinic is introduced during the third year of the course, during which students engage in simulated legal cases. During the fourth (final) year of the course all students, currently around 114 in number, participate in a full clinical model with students offering legal advice and representation to members of the public in a wide range of areas including employment, family, criminal appeals, personal injury, consumer, general civil disputes, welfare benefits, education, and construction law. even higher percentage (perhaps as many as half of them) initially seem not to appreciate why they are being asked to reflect and what benefit reflection is ultimately likely to have in terms of their learning.44 Regrettably, for many students learning is driven largely by assessment. If reflection is not to be assessed there must be a risk that some students will view it as less important than assessable work, and therefore potentially expendable.45 The risk is likely to be more acute amongst students who struggle with reflection, either because they have trouble doing it or cannot see the point. Moon suggests that if we see value in students' reflective work and they will not engage in unassessed work, then reflection will need to be assessed in some way.46 Accordingly there seem to be good arguments for making reflection assessable. Not everyone agrees with this view. Bolton refers to a seminar given by Boud at Sheffield University in February 2001, during which he argued that assessment is inappropriate because it will stultify or even destroy 'raw reflection', including students' confidence in expressing themselves freely and exploratively, and that it may lead to unethical levels of disclosure and confession.47 The UK Centre for Legal Education (UKCLE) notes that assessing reflection is a sensitive issue because it is highly personal and developmental and because it can raise difficulties around parity and validity of assessment.48 Whilst recognising the validity of these concerns, it is possible to assess reflection in a manner which does not significantly risk destroying the openness and freedom of students' raw reflection. This can be done by directing assessment towards reflective work which draws upon, but does not necessary include (unless it is a student's wish to do so), raw reflection. This way the student benefits from recording his or her raw reflection initially, revisiting it at a later date to mull over the experience again, and then preparing the reflective piece for assessment.49 Winter et al comment that the various difficulties described above are capable of being resolved, and are not in any event so very different from the problems of academic assessment in general. They say that if assessment is based on professional criteria and if examiners spend time sharing and discussing their responses to groups of texts, judgments can be agreed as to whether work fulfils the given criteria and with what degree of success.50 Moon agrees. She says that technically the issues surrounding assessment of reflection are no more difficult that those involved in the assessment of anything. Although staff may have differing views about reflective practice the potential for unfair diversity of assessment can be minimised by having staff sessions in which understandings, proposed methods, and assessment techniques are explored and approaches 40 Journal of Clinical Legal Education August 2006 44 This early lack of understanding about why students are being asked to reflect seems to lend weight to the approach of Maughan and Webb, who have tackled the situation by having sessions in their legal process course devoted to the educational theory surrounding reflection. See supra note 11. 45 See further Hinett, supra note 26 at p. 40, and Hinett and Bone, supra note 25 at p. 57. 46 See Moon, supra note 7. 47 Reflective Practice: Writing and Professional Development, Bolton, G, Paul Chapman (2001) at p. 83. 48 http://www.ukcle.ac.uk/resources/trns/clinic/nine.html (accessed on 17.09.2006) 49 This is the approach taken by Bolton, supra note 47 at p.83. Similarly at Northumbria University there is no requirement for students to submit raw reflection for assessment. Moon comments that greater learning is likely to result if a student is required to 'secondarily' reflect on their initial reflection. See Reflection and Employability Moon, J, Learning & Employability Series, LTSN (2004) and The Module and Programme Development Handbook Moon, J, Kogan Page (2002a). 50 See Bolton, supra note 47 at p. 84 and Professional Experience and the Investigative Imagination: The Art of Reflective Writing, Winter, R, Buck, A, and Sobiechowska, P, Routledge (1999) at p. 148. agreed.51 The Quality Assurance Agency for Higher Education in the UK has published a code of practice for the assurance of academic quality and standards in higher education.52 Clearly, if reflection is to be assessed, it must be done so in a way which meets the requirements of the code especially as to clarity and consistency. Amongst other things there must be clearly expressed learning outcomes, which say what is required in terms of reflection, and the criteria for assessment of the reflection should relate to these learning outcomes.53 A tension emerges between the need to prescribe clear criteria for assessment, and the inherently subjective nature of reflection which is personal, unique and unboundaried. Boud, who is opposed to assessing reflection in the first place, argues that the unboundaried nature of effective reflective practice renders it inappropriate for an assessed formal learning context, where clear boundaries are necessary.54 Others acknowledge the tension but do not view it as a bar to assessing reflection. Maughan and Webb say they have identified guidelines and criteria for assessing their students' reflective work, but not ones which are based on detailed written standards or competencies. They say their approach of not being too prescriptive has led to some extremely innovative work from their students, including a video reconstruction of a case, students presenting their experiences in a quiz show format, and a video diary recording the progress of students' work on a case. Maughan and Webb comment, however, that this enabling approach is difficult to incorporate into assessment criteria which are flexible enough to reflect what are often very diverse presentations from across the same year group.55 Burridge raises an interesting point, namely, which part should the student reflect upon when faced with a complex case? He notes the view expressed by Blasi that in guiding students as to what they should be reflective about, the tutor should point out the most critical aspects of a situation or problem. He also notes the alternate view, that the choice should be left to the student to discover from experience.56 When discussing cases with students on a live client programme, it is inevitable that attention will focus on the most challenging or pressing aspects of the case at the given time. Generally the student perceives importance and urgency in the context of needing to deal with an issue on a practical, case management level. The appropriateness for reflection tends to be perceived afterwards, once the crisis has passed. Thus the most salient aspects of cases tend to present themselves for reflective attention. If written reflection is to be assessed, what criteria should be applied in terms of structure, grammar and punctuation? Should we expect students to submit a polished piece of work or should we accept something less refined? Although Boud and Walker suggest that reflective writing should be judged "in terms of criteria for the recognition of reflective writing" rather than by standard academic writing conventions,57 experience on the live client programme at Northumbria Reflection and assessment in clinical legal education: Do you see what I see? 41 51 See Moon, supra note 23. 52 http://www.qaa.ac.uk/academicinfrastructure/codeOfPr actice/section6/default.asp (accessed on 18.09.2006) 53 See Moon, supra note 23. 54 Bolton, supra note 47 at p. 83. 55 Maughan and Webb, supra note 11 at p. 287. 56 "Learning Law and Legal Expertise by Experience" by Burridge, R, in Effective Learning and Teaching in Law, R Burridge et al (eds) Kogan Page (2002) at p. 44. Hinett, supra note 26, at p. 42, refers to Boud's observation that it is naïve to expect students to restrict their reflection to matters outlined by the tutor. 57 'Promoting Reflection in Professional Courses: the Challenge of Context', Boud, D, and Walker, D, Studies in Higher Education 23(2) 191 - 206 at p. 194. University suggests there is some degree of parity between reflective work which is clearly expressed and meets normal academic writing standards, and that which shows depth of thought and perception. An abundance of spelling errors, and poor structure and layout, are usually indicative of a student who has rushed his reflection and failed to see the point, and as a result submits a fairly poor piece of work.58 It may of course turn on what type of reflection is being assessed. If one is assessing 'raw' reflection such as learning diaries or journals, it would make sense to make some allowance for errors of expression and grammar. Finally, there is the 'old chestnut' of how to assess the student who performs brilliantly with her live client work but turns in a relatively shallow piece of reflection, and conversely, the student who is clueless when it comes to dealing with cases but submits an excellent piece of reflection analysing why it all went wrong. In the former situation (good performance, poor reflection) it seems appropriate that the poor reflection should warrant a substantial reduction in the student's overall grade. As discussed above, if a student is unable to perceive the reasons for her good performance and cannot extrapolate any lessons for the future, the quality of her learning experience is thereby downgraded. This should be reflected in her assessment. With the converse situation (poor performance, good reflection) it is sometimes more difficult to know how grade a student's work. Obviously, if a student has performed badly he deserves a relatively low grade, but ought not we give some kind of upgrade to take account of excellent reflection? If not, what is the point of assessing the reflective work? Then there are further questions: to what degree should excellent reflection be able to 'remedy' poor performance? Is it fair for this kind of student to score better overall than another whose live client work was much better but whose quality of reflection more average? Inevitably such issues will persist wherever reflection is assessed alongside performance. What are we assessing, when we assess reflection? If we make the decision to assess students' reflective work, it is pertinent to consider exactly what we are intending to assess. This depends on the purpose of the reflective work and what it is intended to achieve. As argued above, reflection is a method of teaching and learning which can be employed to great effect in a wide range of educational scenarios. By being clear about the purpose for which students are being asked to reflect in any particular circumstance, we can begin to formulate appropriate assessment criteria. Moon comments that a crucial decision in the development of assessment criteria for reflective tasks is whether we are assessing the content of the reflective learning or the reflective process itself.59 Say, for example, clinical law students are being assessed on drafting skills and as part of the course requirements they are required to submit a reflective journal which records the development of those skills over the course. The main focus of this assessment will be students' drafting skills, with the reflective journal forming part of the evidence. On the other hand, learning outcomes may state that students will become proficient in reflective practice, in which case the assessment should focus on the reflective process as well as the content. This raises the question of 42 Journal of Clinical Legal Education August 2006 58 This can of course be clarified from the outset by making explicit to students the requirements as to use of language, structure and presentation. See Moon, supra note 7 at p. 15. 59 Supra note 23 at p. 13. how the reflective process ought to be assessed. It is sometimes said of reflection that 'you know a good one when you see it' but obviously we need to be more rigorous than this if we are to identify fair and consistent standards for assessing reflection. To a certain extent this can be circumvented if a decision is taken to assess reflection on a 'competent' or 'not yet competent' basis. Then all that needs to be established are base-line criteria for a pass, avoiding difficult judgments about degrees of success and awarding of grades. If, however, we decide to grade reflection then it is necessary to identify clear guidelines for awarding grades. Hatton and Smith, working in the context of teacher education, have developed a system of criteria for the recognition of evidence for different types of reflective writing. They identify four categories as follows, which are applicable to reflective work produced in a clinical law context:60 1. Descriptive writing - this is not reflective writing. It may take the form of a description of events that occurred or a report of literature. There is no attempt to provide reasons or consider justifications for events. 2. Descriptive reflection - this has a reflective element. There is a description of events and some attempt to provide reasons and/or justifications but in a reportive or descriptive way. There may be one perspective or rationale identified (for example, 'I chose this problem solving activity because I believe that students should be active rather than passive learners') or there may be some recognition of alternative factors and perspectives (for example, 'Tyler (1949), because of the assumptions on which his approach rests suggests that the curriculum process should begin with objectives. Yinger (1979), on the other hand argues that the 'task' is the starting point'). 3. Dialogic reflection - this demonstrates a 'stepping back' from the events or actions being discussed to reveal a different level of mulling over, and the author may engage in discourse with him/herself. Such reflection is analytical and/or integrative of factors and perspectives and may recognise inconsistencies in attempting to provide rationales and critique. For example: 'While I had planned to use mainly written text materials I became aware very quickly that a number of students did not respond to these. Thinking about this now there may have been several reasons for this. A number of the students, while reasonably proficient in English, even though they had been NESB learners, may still have lacked some confidence in handling the level of language in the text. Alternatively a number of students may have been visual and tactile learners. In any case I found that I had to employ more concrete activities in my teaching.' 4. Critical reflection - this demonstrates an awareness that actions and events are not only located in, and explicable by, reference to multiple perspectives but also that they are located in, and influenced by, multiple historical and socio-political contexts. For example: 'What must be recognised, however, is that the issues of student management experienced with this class can only be understood within the wider structural locations of power relationships established between teachers and students in schools as social institutions based upon the principle of control'. Reflection and assessment in clinical legal education: Do you see what I see? 43 60 See Hatton and Smith, supra note 9. Moon's work on assessment of reflective writing in a clinical law context builds upon the work of Hatton and Smith. Moon's is a three stage analysis which identifies features that can be indicative of different levels of reflection:61 A. A descriptive account which contains little reflection: • it describes what happened, sometimes mentioning past experiences, sometimes anticipating the future, but all in the context of an account of the event • there are some references to the author's emotional reactions, but little or no exploration of how these relate to her behaviour • ideas are taken on without questioning them or considering them in depth • the account is written only from the author's point of view • external information is mentioned but its impact on behaviour is not subject to consideration • generally one point is made at a time and ideas are not linked B. An account which shows evidence of some reflection: • there is description of an event, but where there are external ideas or information, the material is subjected to consideration and deliberation • the account shows some analysis • there is recognition of the worth of exploring motives for behaviour • there is a willingness to be critical of action • relevant and helpful detail is explored where it has value • there is recognition of the overall effect of the event on self - in other words, there is some 'standing back' from the event • however, there is no recognition that views can change with time and more reflection, i.e. that that frames of reference affect the manner in which we reflect at a given time. C. An account which shows quite deep reflection: • there is evidence of self-questioning, possibly including internal dialogue. There is deliberation between different views of the author's own behaviour • the author takes account of the views and motives of others and considers them against her own • the author recognises how prior experience and thoughts interact with the production of her own behaviour • there is clear evidence of standing back from an event • the author may indicate a clear divergence between the reflective process and the points she wishes to learn • there is recognition that the personal frame of reference can change according to the emotional state in which it is written, the acquisition of new information, the review of ideas and the effect of time passing. 44 Journal of Clinical Legal Education August 2006 61 Supra note 7, appendix 1. It can be seen that effective reflection in a clinical law setting involves students engaging in an assessment of themselves in their complex new role as lawyers. Inevitably, and quite constructively, much of the focus will be on students' skills and personal performance seeking to identify areas for improvement. However, the best quality reflection will move beyond this with students considering themselves and their actions in a wider holistic sense encompassing their role in the legal profession and society at large. An important thing to note is that there is no universally accepted set of criteria for assessment of reflective work. Reflective tasks are set in order to achieve different purposes, and therefore assessment criteria need to be tailored accordingly.62 If the purpose of reflection is to become adept at the reflective process itself, then the valuable analysis of Hatton and Smith, and Moon, discussed above can lend assistance to clinical law teachers in formulating criteria which are applicable to their own particular circumstances. Can assessment of reflection ever be fair? So far we have discussed the personal and subjective nature of reflective work. It has been argued that despite its nature, reflective work is capable of assessment provided that staff are clear about the purpose for which the reflective task has been set, and they develop assessment criteria accordingly. If the reflective process itself is to be assessed, the work of Hatton and Smith, and Moon, above, provides a useful starting point for developing assessment criteria. Whilst this aims to introduce objectivity into the process of assessing reflection, the question nonetheless arises whether the process can ever be truly objective and fair because the teacher marking the reflection often has a close working relationship with the student which could, consciously or unconsciously, affect the grading of their work. The assessment of clinical work therefore contrasts significantly with the assessment of most university work, where there is a general trend towards anonymous marking. Reflection in a clinical legal programme usually takes place within a framework where teachers and students develop a close working relationship over a period of time. Students' work is often assessed by those teachers according to the first hand knowledge they have of the students' performance over the programme. Grimes argues that the intensive nature of clinical work gives the supervisor and student a rare opportunity to demonstrate to each other their roles in, and understanding of, the assessment process.63 However, it could also be argued that the close working relationship between teacher and students gives rise to a possibility of bias and unfairness. Say, for example, a student has annoying personal habits, might these subconsciously influence the teacher towards giving him a lower mark than is warranted? Or if a teacher happens to get on particularly well with another student, might he receive a higher mark than is fair? At the very least the close supervision implicit in clinical programmes makes objective assessment difficult.64 One way to try to surmount these subjective difficulties on assessment is to have a system of double marking reflective work, so that if the close relationship between teacher and student has Reflection and assessment in clinical legal education: Do you see what I see? 45 62 Supra note 7 at p. 15. Moon comments that it is entirely reasonable to engage students in the process of developing or fine-tuning assessment criteria, if not for their own work, for the work of next year's students. 63 Supra note 2 at p. 159. 64 Supra note 2 at p. 152. in some way unfairly influenced the assessment of the student's work, this can be neutralised upon appraisal by an unconnected second marker. As discussed below, a double marking system can also have the added benefit of ensuring consistency in assessment of reflective work between different teachers on the programme.65 SECTION II: Assessment Case Study The first part of this paper has considered the nature of reflection, concluding that reflection is a valuable method of teaching and learning which can be employed in numerous educational contexts. The first part has also considered issues relevant to assessment of reflective work. The remainder of this paper focuses on a case study within the context of the clinical law programme at Northumbria University. The study considers issues surrounding double marking and asks whether current methods for assessment of the programme's reflective essay achieve acceptable levels of marking consistency. Reflection within the live client programme at Northumbria University All students undertaking the final year of the 'combined' law degree at Northumbria University participate in the live client programme known as the Student Law Office (SLO), where students provide legal advice and representation to members of the public in a wide range of legal cases.66 Students are divided into 'firms' of four to six individuals and each firm is supervised by a practising solicitor/barrister or welfare benefits officer. Currently there are around 114 students and 16 supervisors involved in the programme. Reflection, both formal and informal, occurs at many points during the SLO programme. Each firm meets weekly with its supervisor, to discuss and share progress of the firm's cases and to talk about any problems or challenges the students have encountered. So as well as providing a forum for deciding on how to proceed with cases, firm meetings are designed to give students an opportunity to reflect with their peers and supervisor on anything they wish to discuss. The format of firm meetings is deliberately unstructured in order to promote reflection. Because the students are not pressured to raise or disclose anything they are not comfortable with, they tend to open up quite naturally to share and reflect upon their experiences.67 None of this informal reflection is assessed. Students are required to submit a portfolio of their work for assessment at the end of the year. The portfolio must contain evidence of certain key areas of practical work over the course of the year, together with reflection on those areas. For example, one key area of practical work is written communications. Students must include copies of all substantive written communications they have produced, and a reflective commentary which refers to three specific items (such as a letter of 46 Journal of Clinical Legal Education August 2006 65 Hinett and Bone, supra note 25 at p. 66, discuss consistency in assessment and they say that with planning, it ought to be possible to achieve intra- reliability of assessment within a single faculty. They refer to a need for law teachers to discuss assessment and what is valued in their teaching. The UKCLE has funded research into consistency of marking across law schools, further details of which can be located at http://www.ukcle.ac.uk/research/projects/mitchell.h tml (accessed 18.09.06). 66 See supra note 43. 67 In addition to weekly firm meetings, students often meet with each other and/or with their supervisor during the week to discuss cases and this provides a further opportunity for informal reflection. advice, a witness statement and a letter to an opposing party setting out a case) and which: (1) compares the different approaches the student took when preparing each of the three items, and (2) discusses the development of the student's drafting and writing skills throughout the year and his or her strengths and weaknesses in this area. Seventy percent of students' overall grade for the SLO comes from assessment of their practical work, as evidenced by their portfolios. The remaining thirty percent comes from a 3,500 word reflective essay based upon students' experience of live client work. Assessment criteria for the essay say: 'There is no set title for the essay element of the SLO assessment. This is to allow you scope to select a subject area that has affected your own particular experience within the SLO. The essay should lead you to consider the work you have done in the SLO in its wider context. It is not a summary of your work through the year. You should pick a subject area and relate it to your SLO experiences. For example you may wish to look at the practical effect of an area of academic law on the conduct or outcome of a case or you may wish to look at the role of the SLO in the wider context of provision of legal services. Your discussions in firm meetings may have raised a number of appropriate issues. Like any other essay it should be structured and informed.' Although there is no set structure for the essay, by far the most common approach taken by students is to talk about a case they were involved in, describing the way the case progressed and the outcome, and how this affected them. Most students discuss their feelings about the case and some consider wider social and political issues such as potential areas for law reform. What is not wanted is a dry piece of academic writing. While there may be some discussion of substantive law, the idea is for students to use their personal experiences in the SLO and the reflective skills they have developed over the course of the year to facilitate discussion of wider issues concerning the legal system. Criteria for assessing the essay The essay is assessed by the student's own supervisor and is given a grade out of 100. To promote a consistent approach between supervisors, grading criteria have been discussed and it has been agreed that essays are often characterised as follows: • First class (70 or more) - an outstanding piece of work which stimulates the marker's interest in the topic. Draws on the student's experience in the SLO but goes considerably beyond. Presentation, structure, spelling and grammar immaculate or very good. Exciting, new ideas raised and analysed in a cohesive and persuasive manner. Student's emotions often not referred to, or only in passing. Mature appreciation of time frames, points of reference and wider socio-political issues. Workable law reform proposals often identified and discussed. • High pass (60 to 69) - an interesting and stimulating essay which draws upon the student's SLO experiences and transcends them. Well structured and well argued, usually with a central theme linking discussion of cases. Student stands back from the events described to see the bigger picture/wider ramifications. Emotions may be referred to but are not the focus. Descriptive element is present, but minor, with analysis being the focus of the essay. Links are Reflection and assessment in clinical legal education: Do you see what I see? 47 made between the student's SLO experiences and the world at large. Appreciation of different frames of reference and standpoints. • Low pass (50 to 59) - the essay focuses entirely on the student's experiences in the SLO. Content is largely descriptive with only a small amount of analysis and reflection. Some identifiable structure but not very good. The student's emotions are discussed, quite often as the focus of the piece, however there is only minimal understanding of effect of emotions upon the student's behaviour. Makes links between different cases or ideas but in a rudimentary way. Analysis is simplistic and uninformed. Little or no consideration of wider ramifications. No awareness of frames of reference and the impact of time upon qualify of reflection. • Fail (50 or less) - exceeds or falls significantly short of the word limit. Little or no structure. May be unconnected with the student's experience in the SLO. Content is merely descriptive, e.g. a report of all the cases the student worked on. May be some reference to the student's emotions but no discussion of how they influenced his or her behaviour. No attempt to make links. No attempt to consider or analyse events or to draw conclusions. No awareness of alternative viewpoints or wider issues. Often characterised by sloppy presentation with little or no attention to spelling, grammar, use of language. No central theme or argument. It can be seen that these criteria for assessment of the SLO essay are broadly reflective of the analysis of Hatton and Smith, and Moon, as discussed above.68 Method of assessing the essay With large numbers of students and staff involved in the SLO programme it is important to aim for consistency in assessment, so that a student can be confident her essay will receive a fair grade regardless of the identity of her supervisor. Accordingly once students' essays have been graded by their own supervisors, some are selected for double marking by a different supervisor. These include all first class essays, all fails, any bare passes, any essays that supervisors have asked to have double marked, and one or two essays from each supervisor.69 When an essay is marked for the first time by the student's supervisor, the supervisor completes a brief report which includes the names of the student and supervisor, the grade given for the essay and the supervisor's written comments. Some supervisors also annotate written comments on the essay itself while they are marking. 48 Journal of Clinical Legal Education August 2006 68 There is one interesting observation involving the emotional content of the essay. Moon, supra note 7 at pp. 13 - 14 and supra note 23 at p. 5 suggests that it is an important aspect of reflection to encourage students to refer to their emotional reactions and to perceive how this can influence their behaviour. Looking at the SLO essay overall there tends to be a great deal of reference to students' emotional reactions to cases, but often students do not analyse this or seek to draw any conclusions such as whether emotion had any effect on their conduct of the case. For this reason many of the essays which spend a significant amount of time discussing the student's emotions tend to be awarded relatively low grades. If the student refers to emotion with some degree of perception about how it has influenced his or her behaviour, that tends to be indicative of a better quality essay, but the first class papers rarely seem to refer to emotion much at all focussing instead of wider issues. 69 These tend to include essays which have been marked at the border of a grading band (i.e. given a grade of around 60 or around 70), or any right in the middle of a band (i.e. given a grade of around 55 or 65). If an essay is selected for double marking, the double marker is given the essay together with the first marker's report. Based on their own assessment of the essay, and the first marker's report, the double marker can retain the grade given by the first marker or substitute their own grade. Whilst this process aims to promote consistency and fairness in assessment of the essay, there may be some potential shortcomings. Earlier in this paper it was discussed how the intensive nature of clinical work can make objective assessment difficult. This could be true of the SLO essay, which is marked in the first instance by the student's supervisor. Although having some essays double marked is designed to further objectivity in the assessment process, might the reverse in fact be true? The double marker knows not only the student's identity, but also the grade given by the first marker and his or her written comments on the work. This could make it difficult for the double marker to carry out his function objectively. For example, if the double marker is a new member of staff, he might lack the confidence to change a grade given by an experienced, senior colleague. It was decided to design and carry out a case study based on assessment of the SLO reflective essay in order to examine this issue in greater detail. The case study The case study was initially designed to consider one main question: whether double markers are influenced in their assessment of SLO essays by information available to them at the time of double marking. At discussed above, the double marker knows the name of the student and the identity of the first marker, and also has an assessment report showing the grade given by the first marker and his or her comments on the essay. If this information was not available on double marking an essay, would the grade given by the double marker be more or less consistent with the grade given by the first marker? It was decided to look at this issue by taking a group of essays which had been double marked under usual SLO procedures, and then re-marking them under blind marking conditions, and then comparing the two sets of results. At the end of the 2003/04 academic year, 101 reflective essays were submitted for assessment in the SLO, of which 35 were selected for double marking based on the usual SLO criteria discussed above. Under the case study conditions, all 35 of the essays were subjected to a further round of blind marking, which took place towards the end of 2004. The method for getting the essays ready for blind marking was as follows. Each of the essays was typewritten, so there was no potential problem of students being identified by their handwriting. The essays were photocopied, then 'white out' was applied to the copies where required to obliterate any handwritten comments or other markings which had been made by the first markers. Care was taken, where possible, to delete the markings in a manner which concealed that they had been made at all. In addition, 'white out' was used to conceal students' names, first markers' names, and the grades given by first markers where these had been written on the essay. After this, the essays were photocopied again to hide the use of 'white out'. At this point the essays were considered to be ready for blind marking. Of the five supervisors who had been involved in the double marking, the author excluded herself, which left four markers. It was ensured that none of the four was given an essay they had previously seen or discussed, and then the 35 papers were randomly assigned between them. Markers were Reflection and assessment in clinical legal education: Do you see what I see? 49 given instructions which asked them to assess and grade the essays in exactly the same way as they would ordinarily double mark a SLO essay. They were also asked to provide brief written comments giving their views about the current marking procedure. Results In this results section, comparison is made between the grades given in respect of the essay papers when they were double marked at the end of the 2003/04 academic year, and the grades which were given when the same papers were later blind marked under the case study conditions. First, results were analysed to see whether grades tended to go up or down as a result of blind marking. It was found that of the 35 papers, grades went up in 18 cases (51%) and down in 14 cases (40%), and there was no change of grade in the remaining 3 cases (9%). The next question was, to what degree did the grades given upon blind marking vary from the grades given upon double marking. The following results were found: Tribe states that 5% is the percentage variation which is generally accepted by commentators as being the norm between staff markers.70 Adopting this figure, the results show there was an acceptable degree of variation in the case of 22 essays (63%), and a higher than acceptable degree of variation for 13 essays (37%). This means that for more than one third of essays, the degree of variation was higher than the acceptable norm of 5%. Looking at the papers where there was more than 5% variation, it was asked whether there was a correlation between the degree of variation and the marker's subject knowledge of the essay topic. Put another way, would the grade given by the blind marker be more likely to match the grade given by the double marker, where the blind marker had knowledge of the subject area of the essay (i.e. where there was a 'subject match')? It was found that where there was a subject match, only two out of 13 papers had a degree of variation above 5%. However, where there was no subject match, 11 out of 13 papers had more than 5% variation. This suggests (unsurprisingly) that the subject knowledge of the marker is important when marking an essay: that it is much better for an essay to be double marked by a supervisor who has subject knowledge of the essay topic. The next question which was asked, was whether the variation in marks would have made a difference to the student's grade for the essay (assuming the blind marker's grade was substituted for the grade given on double marking). It was found that the variation would not have made a 50 Journal of Clinical Legal Education August 2006 70 "DIY Learning: Self and Peer Assessment", Tribe, D, in Teaching Lawyers' Skills, Webb, J, and Maughan, C, Butterworths (1996) at p. 365. 0 to 5% variation 22 essays 6 to 10% variation 8 essays 11 to 15% variation 5 essays difference in 20 cases (57%) but it would have made a difference in 15 cases (43%). In those 15 cases (43%), would it have made a difference to the student's overall grade for his or her SLO work (given the 30% weighting for the essay)? It was calculated that in 4 cases (11%) there would have been such a difference.71 Looking at the general comments made by the markers engaged in the case study, the following remarks were observed: • The factors most likely to influence double markers are the grade given by the first marker and his or her comments on the work, but having this information to hand makes the process of double marking less time consuming. • The student's identity is the factor least likely to influence double markers. • Where the first marker is a long-standing member of staff with a high level of experience in assessing students' work, a double marker who is less experienced may be reluctant to alter the first marker's grades. • Supervisors would prefer to double mark essays in an area of law where they have at least a comfortable degree of subject knowledge. Potential limitations of the case study design Certain potential limitations of the case study design were perceived. In numerous cases the removal of the first marker's name from the essay was ineffective because it was easy to infer his or her identity in other ways. The SLO handles some high profile cases and staff generally know which of their colleagues, and often which students, are involved in them. Accordingly if an essay refers to one of these cases it is easy to infer the identity of the first marker and sometimes also the student. In addition, although there are currently around 16 supervisors in the SLO, and some of them share areas of expertise such as family or employment law, other supervisors have their own unique areas of expertise. For example, there is only one supervisor who specialises in construction litigation and another supervisor with expertise in welfare benefits. If a student essay refers to a case in this area, the first marker can therefore be identified. In reality this means that many of the papers were not completely 'anonymised'. Although the first marker's name was deleted from the essay, in many cases the blind marker would have known who the first marker was and could therefore have been influenced by this (see above). Another issue relates to a potential skewing of one of the results of the case study. It was reported, above, that in 13 out of 35 cases the degree of variation between the grade given by the blind marker and the double marker exceeded 5%. Of these 13 cases, 11 occurred where the blind marker had little or no subject knowledge of the essay topic while only two occurred when the blind marker had the relevant subject knowledge. It is possible that there is some skewing of this result. One of the supervisors who was involved in the case study has expertise in one particular area of law, and limited knowledge of other subject areas. If this staff member is excluded from the calculation, however, the result is still significant. The figures would then be six cases of more Reflection and assessment in clinical legal education: Do you see what I see? 51 71 For three students their grade would have increased from a 2:2 to a 2:1, and for one student the grade would have increased from a 2:1 to a first. than 5% variation where the blind marker has little or no subject knowledge of the essay topic, compared with two cases where the blind marker as such knowledge. From this it can still be concluded that the grade given by a blind marker is far more likely to resemble the first marker's grade, where the blind marker has subject knowledge of the essay topic. A further issue is raised by the nature of the essay itself. The essay is not a dry piece of academic writing which is intended to describe the current state of substantive areas of law. As discussed above previously, it is a piece of reflective work drawing upon students' practical experience of their live client work. To the extent that an essay may say, "I worked on the Jones case, I did x and y, x went really well but y didn't and this is what I would change..." the student's own supervisor is in a position to assess the veracity of the factual content, whereas another marker who was not involved in the student's supervision would not be able to do so. To that extent, at least, the second marker must necessarily rely on the first marker's comments about the factual content. When the first marker's comments were removed under the blind marking conditions, the blind marker had no way to judge the veracity of the factual statements made in each essay; the blind marker was obliged to take them at face value. This means, potentially, that a well written, well structured paper that was nevertheless full of factual misrepresentations could have received a high grade upon blind marking. Further, it is occasionally the case that a student will insert into their essay material which they have recycled from earlier SLO work, such as a research report on a particular case. The student's supervisor will be able to spot this whereas the blind marker would not know whether an essay contained recycled material or not. In the case of a blind marker who does not have subject knowledge of the essay topic, he would not be in a position to judge any statements of law contained in the essay. He would be obliged to take them at face value, and could potentially give an essay a high grade even when it contained errors of law (which would ordinarily result in a significant reduction in the essay grade). A final observation relates to the relatively small numbers involved in the case study. Only 35 papers were involved, and perhaps even more significantly, only four supervisors were involved in the blind marking exercise. It would be interesting to see whether similar results were obtained by repeating the study in subsequent years. Implications of the case study In an excellent review paper, Brooks observes that the practice of double marking has flourished in higher education where a growing number of university assessment policies require students' coursework to be double marked. Interestingly, however, Brooks notes that this expansion in use at degree level is set against limited interest in double marking as a research topic, and that it has in fact received very little attention in literature published towards the end of the 20th century.72 Bone on the other hand states that double marking has had bad press as there is evidence that there is little to be gained from doing it. She states that second markers tend to come up with similar marks regardless of whether they have seen the first marker's marks or not.73 52 Journal of Clinical Legal Education August 2006 72 'Double Marking Revisited', Brooks, V, British Journal of Educational Studies 52(1) 29 - 42 (2004) at pp. 30-31. 73 Ensuring Successful Assessment, Bone, A, National Centre for Legal Education (1999) at p. 46. However, Bone's comments may refer to double marking of regular academic work, rather than reflective written work. Brooks refers to an interesting episode in 1949 when a researcher called Wiseman pioneered a radical new approach which embraced inconsistency between markers: multiple marking. Under this system each script was marked independently by teams of four markers so that the final mark for each script was the sum of four independent assessments. Markers were selected for inclusion in the teams by having a high degree of self-consistency, also know as mark re-mark consistency. Wiseman said that provided markers are experienced teachers, lack of high inter-correlation is desirable since it points to a diversity of viewpoints in the judgment of complex material.74 Wiseman's comments about diversity of assessment are extremely interesting. One wonders whether multiple marking is a technique which could be helpful in the assessment of reflective work. Regrettably, it seems unlikely in the current academic climate, where constraints of time and money cause staff availability already to be considerably stretched, that one could readily find teams of four markers to assess each student essay. If multiple marking is not feasible, what alternatives might there be? Baume describes a method for assessing portfolios where, before each round of assessment, each assessor first reads and assesses the same portfolio. They then work together for a day to share their judgments, and try to reach agreement. Two assessors then independently assess each portfolio, and if there are any disagreements a third assessor, a course team member, resolves them.75 This method appears to be a thorough way of approaching assessment of portfolios, although from the description it seems that many staff may need to be involved and would need to invest a substantial amount of time in the process. Tribe describes a method where students play a role in the assessment process. Students prepare the objectives to be achieved in a piece of course work and they give a mark, either for themselves or their peers, indicating the extent to which the objectives have been met. The same work is also assessed by staff. Where the student mark and the staff mark are within 5% of each other, the student mark will be retained. If there is greater discrepancy, Tribe says that discussion will take place in which justifications for the mark variance are explained and agreed.76 One attraction of this method is the way in which students are engaged, participating in both the formulation of learning objectives and the assessment process. Each of these assessment methods provides interesting food for thought. With more than one third of essays in the case study showing greater than 5% variation on blind marking, one's first reaction may well be to say that a different assessment method is required. This is because, as academics, we have been conditioned to regard variation in assessment as a bad thing, to be avoided at all costs. If the case study had been looking at assessment of a standard academic essay then we might validly have thrown our hands up in horror at the results. However, it is most important to bear in mind the nature of the work which is the subject of the case study. Far from being a standard piece of academic writing, the SLO essay has a strong reflective element and we have seen above the way in which such work is inevitably highly personal and subjective. Although we can attempt to introduce objectivity by devising appropriate assessment criteria for reflective work, it Reflection and assessment in clinical legal education: Do you see what I see? 53 74 Brooks, supra note 72 at p. 34. 75 A Briefing on Assessment of Portfolios, Baume, D, LTSN Generic Centre Assessment Series No. 6, at pp. 10-11, which can be accessed at http://www.heacademy.ac.uk/resources.asp?process =full_record§ion=generic&id=6 (accessed 19.09.06) 76 Tribe, supra note 70 at p. 365. is probably true to note that a subjective vein runs through the entire process, from the student who writes the essay based on his or her unique personal experiences in the clinic, to the marker who has close knowledge of the student's capabilities and exertions. Given the unique and personal aspect of this process, is it so very surprising that a blind marker, who has no knowledge of the student or his or her experiences, may perceive the contents of a reflective essay quite differently? It may therefore be inevitable that assessment of reflective work involves a degree of subjectivity and therefore variation. Nonetheless the case study has been a worthwhile exercise as it highlights certain alternations which could be made to improve the way in which the SLO essay is assessed. First, the student's name could be left off the paper when an essay is selected for double marking. Although supervisors report that the student's identity does not influence their approach to assessing the student's essay, certain 'difficult' students attract a degree of notoriety each year, and also the brightest students tend to become known. It would therefore make sense to neutralise this factor by deleting the student's name. For reasons discussed above it is very often not possible to conceal the identity of the first marker. While the first marker's comments on the work could be withheld, it has been seen that the second marker would then be unable to tell whether the student has made untrue factual claims in his or her essay or recycled previously used material. For these reasons it may be better to keep the first markers' names and make their comments available to the second marker. Is it helpful for the second marker to know what grade the first marker attributed to the essay? This is a difficult question. Supervisors report that having this information makes it quicker for them to double mark an essay but it influences their own assessment of the work. Provided that double markers are willing to make an additional investment of time, it may be better for them not to know the grade given by the first marker so they cannot be influenced by it. The study suggests that the single greatest improvement which could be made to the current assessment method is for double markers to assess essays which are within their subject specialism or where they have at least a comfortable level of knowledge. Supervisors report that they feel more at ease marking papers where they know the relevant area of law, and this was borne out by the results of the case study which found that second markers were far more likely to grade a paper within a 5% degree of variation if they had the relevant subject knowledge. Practically, however, this has important ramifications. Currently the clinic offers a diverse range of practice areas, including employment law, family law, crime, public law, personal injury, consumer, construction litigation, welfare benefits and so on. The diversity is regarded as beneficial both for clients and students. Currently around 16 members of academic staff work on the clinic and each brings to bear his or her own personal experience of legal practice. The clinic is able to offer such diversity because of the scale of the operation and the number of staff involved. It also means that in some instances there is only one supervisor in a particular area of law, such as construction litigation or welfare benefits. Within areas of law such as personal injury and family law, where there are several supervisors with subject expertise, it may be possible to arrange matters so that essays are double marked by other supervisors with subject knowledge. Currently this would not be possible where there is only one supervisor in an area of law. One way it could happen would be to drop areas of practice so that the clinic would be confined to one or two areas of law with many supervisors in each. However, 54 Journal of Clinical Legal Education August 2006 this would be undesirable from the viewpoint of all concerned: students, clients and staff. Another way would be to greatly increase the numbers of staff involved in the clinic to ensure that each practice area was staffed by at least two or three supervisors. The SLO has strong ambitions for growth over coming years, to greatly increase both staff and student numbers, and if the growth is carefully managed it may be possible to achieve this. It is certainly something to aim for. It may be possible to ask a non-clinical member of staff with the relevant subject knowledge to double mark the essay, perhaps together with a clinical tutor. Alternately, if there was a web-based database of clinic supervisors and their areas of expertise, it may be possible to find a clinical tutor with subject expertise from another institution who could assist with double marking.77 Conclusion It is widely known that practical experience and reflection form the two main elements of clinical legal education. While clinical staff are usually comfortable with the practical side of clinic, many feel some degree of unease about reflection including what it is, why we ask students to do it, and how to assess it. If we feel such uncertainties as clinicians it is hardly surprising that many of our students struggle with reflection. The role that reflection plays in clinical legal education mirrors the role that reflection plays in learning, full stop. All human beings have a capacity for reflection; it is something we naturally employ, usually quite subconsciously, on a day to day level when we mull over events in our minds. Numerous educational theories recognise reflection as an integral part of learning including models developed by Kolb and Schön and constructivist learning theory. Clinical staff should have an awareness of these theories so they know what reflection is and why it is important. This can inform staff in the development of programmes which are conducive to the occurrence of good quality reflection. To encourage reflection in clinic, staff should move away from the traditional 'teacher' role of authority and giving answers, towards a facilitator role which encourages students to open up and share their thoughts about their experiences. We need to think about wider issues of programme design but equally important are considerations such as class size, the way students work together and the arrangement of teaching spaces as all of these can have a significant impact on whether reflection is likely to occur in practice. While it may not be possible to teach reflection directly, we can provide students with helpful guidance to simulate their interest and their willingness to reflect. One way of doing this is to give students clusters of questions to get them going. For example, questions can challenge students into considering the general and the specific, and to think about how the perspective of time can affect the quality of their reflection. If we can get students to produce quality reflection there are obvious benefits for them. Not only will they perform better in their clinical course but reflection will make them more aware of how they learn. There is evidence that increased self-awareness of learning is correlated with better learning, so by engaging in reflection students are teaching themselves how to learn better in the future. Reflection and assessment in clinical legal education: Do you see what I see? 55 77 While this could be a helpful resource for many reasons, not only relating to assessment, it would raise issues of consistency of marking between law schools which would need to be addressed. See also supra note 65. There are conflicting views about whether reflection ought to be assessed. Some say that reflection is unique and raw, and we risk destroying this if we seek to impose the strictures of an assessment regime. Others say that if reflection matters to us, and if students will not engage in unassessed work, we have little alternative than to assess it. If we decide to assess reflection, this opens a further can of worms given its inherently subjective nature and the close working relationships which often exist between students and the staff who are to assess their work. Can assessment of reflection in such a context ever be fair and objective? Arguably yes, provided that assessment criteria are carefully thought though and made explicit, and that staff meet to discuss and agree assessment methods and approaches. In particular it is vital to be clear whether the reflection is being assessed as part of the assessment of a wider project or skill (for example, drafting) or whether the focus is the learning journey itself. If the latter, and especially if reflection is to be fully graded rather than merely adjudged competent/not yet competent, carefully considered assessment criteria need to be developed and circulated amongst staff. Valuable work has been done by others in identifying criteria for assessing reflection in clinical teacher education and legal education. These provide a helpful starting point for assessing reflection in our own programmes but we should be wary of a 'one size fits all' approach. Just as we need to give careful thought as to how to employ reflection as part of our clinical programmes, we need to individualise assessment criteria to ensure they are relevant and fair for our own purposes. If we follow these guidelines it should be possible to arrive at a fair assessment of the reflective work produced by law students in clinic. But is this enough: should we also double mark students' reflective work to check that assessment criteria are being applied consistently by different markers? Experience of doubling marking reflective work in the law clinic at Northumbria University suggests that double marking has a role to play in ensuring consistency between markers. However, care needs to be taken in the design of a double marking scheme. Second markers can be swayed by the grades awarded and comments made by first markers, so consider whether your second markers need this information or not. It is vital to ensure that second markers have the right subject knowledge to mark the reflective work they are given; experience at Northumbria University shows that second markers cannot meaningfully contribute to the process unless they have knowledge of the area of law on which the reflection is based. 56 Journal of Clinical Legal Education August 2006