Can We Assess What We Purport to Teach In Clinical Law Courses? Roy Stuckey* "Assessment - evaluation: A judgment about something based on an understanding of the situation."1 "Assess: to judge . . . the . . . quality . . . of something."2 Many claims are made about the educational value of clinical education in law schools. Unfortunately, the first generation of clinical law teachers did not clearly articulate our educational goals nor did we fully explore how to assess the effectiveness of our instruction. Subsequent generations of clinical teachers adopted the practices of their predecessors and mentors. Consequently, many issues related to assessments of clinical students remain unexplored, and current practices tend to be neither valid nor reliable. While clinical teachers in the United Kingdom have made more progress than those in the United States,3 all clinical teachers need to work together to improve our understanding of assessments and to develop improved methods for finding out whether our students are learning what we purport to teach. This article explains the importance and nature of assessments, illustrates some of the issues presented by current practices, and proposes some new directions to consider. It concludes that much work remains to be done to clarify the goals of clinical legal education and to develop valid and reliable assessment tools.4 Can We Assess What We Purport to Teach In Clinical Law Courses? 9 * Webster Professor of Clinical Legal Education, University of South Carolina School of Law, USA. 1 Encarta Dictionary, http://encarta.msn.com/encnet/dictionary (last visited May 22, 2006). 2 Cambridge Advanced Learners Dictionary, http://dictionary.cambridge.org (last visited May 22, 2006). 3 Evidence of this includes multiple assessment- related projects in the U.K., some of which are accessible via the webpage titled "Resources on Assessment in Legal Education" maintained by the UK Centre for Legal Education, http://www.ukcle.ac.uk/resources/assessment/index. html (last visited August 15, 2006). There is no equivalent resource in the United States. 4 Portions of this paper were adapted from Roy Stuckey, Best Practices for Legal Education: A Project of the Clinical Legal Education Association (March 31, 2006, draft). The most current draft is posted on-line at http://professionalism.law.sc.edu (look in the "news" section on the main page). The drafting phase of the project should be completed in September 2006. The Purpose and Importance of Assessments The main purpose of assessments in educational institutions is to discover if students have achieved the learning outcomes of the course studied.5 In other words, we use assessments to find out whether our students are learning what we want them to learn. In law schools, as in medical schools, one purpose of assessment is to determine which students should receive degrees, but other purposes of assessment are more important. Aside from the need to protect the public by denying graduation to those few trainees who are not expected to overcome their deficiencies, the outcomes of assessment should be to foster learning, inspire confidence in the learner, enhance the learner's ability to self-monitor, and drive institutional self-assessment and curricular change.6 The goals and methods we select for assessment are important. "Assessment methods and requirements probably have a greater influence on how and what students learn than any other single factor. This influence may well be of greater importance than the impact of teaching materials."7 [C]hanging the assessment procedure is one of the most effective ways of changing how and what students learn. Surface approaches are induced by excessive workloads, a narrow band of assessment techniques and undue emphasis upon knowledge reproduction. Deep approaches are influenced by choice, a variety of assessment methods, project work and an emphasis upon tasks that demand demonstration of understanding.8 Thus, legal educators, including clinical teachers, should consider carefully what we are trying to assess and how we are doing it. Types of Assessments in Clinical Courses There are at least three types of assessments in clinical courses: evaluating overall competency, helping students understand what they learn from individual, unique experiences, and determining whether students are learning what we are trying to teach. Consider the differences in the following questions that a clinical teacher might ask: "How competent is the student?," "What did the student learn?," and "Did the student learn what I intended?" The first question focuses on how well a student performs as a lawyer. This is a natural and important question to ask, because clinical courses, especially those involving actual client representation, present opportunities for students to test for the first time on a personal level a number of abilities that are essential for lawyers. Perhaps the most important of these is whether a student is able to engage in appropriate behaviors and integrity in a range of situations. 10 Journal of Clinical Legal Education August 2006 5 Alison Bone, Ensuring Successful Assessment 3 National Centre for Legal Education Guidance Note (Burridge & Varnava: series eds., 1999), available at www.ukcle.ac.uk/resources/assessment/bone.pdf 6 Ronald M. Epstein, MD, & Edward M. Hundert, MD, "Defining and Assessing Professional Competence", 287 JAMA 226, 226 (Jan. 9, 2002). 7 Bone, supra note 5 at 2 (quoting D.Boud, Enhancing Learning Through Self-Assessment (1995)). 8 Id. at 4 (citing G. Brown, Assessment of Learning: Its Implications for Quality, Paper Delivered at Open University Conference on Changing Patterns of Student Assessment and Examination (Jan. 1994)). That is, students in client representation courses are beginning to learn the extent to which they are able to conduct themselves professionally and provide competent legal services. The assessment of students' professional strengths and weaknesses is an important function of clinical teachers, and it occurs to some degree or another in every type of clinical course. These "holistic" assessments begin with the very first task that a student undertakes in a clinical course. Some of the knowledge, skills, and values that are reflected in each performance were acquired before students enrolled in the clinical course. Thus, when we ask "How competent is the student?," we are potentially assessing everything that a student has learned during law school, or for that matter, during their lives, not just what a student is learning in the clinical course. This is an important reality for clinical teachers to keep in mind, especially in the United States where it is common practice for clinical teachers to mark or grade everything a student does from the very first day in the course. "Holistic" assessments are not addressed in this article. Ross Hyams discusses issues related to overall competence assessments of students in the article that he submitted to the Journal.9 Now consider the second question, "What did the student learn?" In clinical courses students might learn what we intend for them to learn, but they also frequently learn lessons that are unexpected, unplanned by the instructor, and unique to the particular student. These lessons may be quite valuable to the student, and clinical teachers can play a role in maximizing the educational value of these experiences. One would not, however, necessarily want to assess whether or how well a student learns such lessons or use such assessments for assigning grades or advancement decisions, because they are not educational objectives of the course or lessons that the student needs to learn to earn credit for the course. On the other hand, we might want to assess the student's self-reflection skills and ability to learn from practice, if developing these skills are educational goals of the course. The third question, "Did the student learn what I intended?," is the focus of this article - how do we tell whether our students are learning what we intend for them to learn? I begin with the assumption that a clinical course has clear educational objectives, the instructors and students are aware of these goals, and they are interested in employing assessment tools to find out whether those objectives are being accomplished. Basic Principles of Assessment The purpose of this section is to establish a shared vocabulary about assessments. An assessment may take the form of a final exam, a test administered after a unit of instruction is covered, a paper, observation of performance, a discussion between student and teacher, portfolio (profile) reviews, or some other method of determining what a student has learned. Before selecting an assessment tool, we should be clear about the goals of the assessment and the purposes for which it will be used. Can We Assess What We Purport to Teach In Clinical Law Courses? 11 9 Ross Hyams, Student Assessment in the Clinical Movement - What Can We Learn From the U.S. Experience?, Paper Presented at the Fourth International Journal of Clinical Legal Education Conference in London (July 12-13, 2006) (copy on file with Roy Stuckey). The goals of a particular assessment may be to evaluate a student's knowledge, behavior (what a student does before and after a learning experience), performance (ability to perform a task), attitudes/values, or a combination of these.10 The purpose of an assessment can be formative, summative, or both. Formative assessments are used to provide feedback to students and faculty. Their purpose is purely educational, and while they may be scored, they are not used to assign grades or rank students. A summative assessment is one that is used for assigning a grade or otherwise indicating a student's level of achievement. "Summative assessment occurs at the end of a course of study and is primarily used for the purpose of making a final judgement of the student alongside his or her peers - final in the sense that (unless there are mitigating circumstances) it is how a student performs in this assessment that will be used to decide whether a student can proceed, e.g., to the next level of the course or be admitted to a vocational course."11 An assessment tool should be valid. An assessment tool is valid if it allows the teacher to draw inferences about a student's acquisition of the skills, values, and knowledge that the tool purports to assess.12 Congruence is a necessary aspect of validity, that is, the goals of the assessment must agree with the goals of the instruction.13 For example, a professor who seeks to build students' ability to apply and distinguish cases might administer an essay question that raises issues that test the outer limits of a set of precedents. On its face, the assessment appears to be a valid test of the skill. If, however, students must take the test in a closed-book setting or without sufficient time to review the relevant authorities while taking the test, students who have developed the ability to apply and distinguish cases but possess poor memorization skills would likely perform poorly. Thus, the assessment tool would not be valid. An assessment tool should also be reliable; that is, it should accurately rate those who have learned as having learned and those who have not learned as having not learned.14 It should not matter whether a student is being assessed first or last or whether one teacher or another is conducting the assessment. Assessments can be norm-referenced or criteria-referenced. Assessments in the United States tend to be norm-referenced; assessments in the United Kingdom are typically criteria-referenced. Norm-referenced assessments are based on how students perform in relation to other students in a course rather than how well they achieve the educational objectives of the course. Normative assessment is often done to ensure that certain grade curves can be achieved. This approach allows law schools to sort students for legal employers. Norm-referenced evaluations inform students how their performance relates to other students, but they do not help students understand the degree to which they achieved the educational objectives of the course. This can have a negative effect on student motivation and learning. 12 Journal of Clinical Legal Education August 2006 10 See Gregory S. Munro, Outcomes Assessment for Law Schools 111-17 (2000). 11 Bone, supra note 5, at 4. 12 Gerald Hess and Steve Friedland, Techniques for Teaching Law 289 (1999). See also Patricia L. Smith and Tillman J. Ragan, Instructional Design 95 (2d ed. 1999). 13 Smith and Ragan, supra note 12, at 95. 14 Id. at 97. [S]tudents . . . perceive that something different is going on in the current circumstance, and wonder whether the "sorting" process reflects an artificial or arbitrary allocation of rewards. In the absence of a clearly stated explanation of the actual standards to be achieved, it is easy to become frustrated, then angry, wasting energy that might otherwise be invested in meaningful efforts to learn. Students also powerfully articulate their hunger to link assessment and learning. They want to learn to take exams, and they want feedback so they can improve.15 Norm-referenced assessment allows grades to be distributed along a bell curve, but this should be neither a goal nor an expectation of assessments. What matters is whether students adequately achieve the learning outcomes of the course. A bell curve outcome actually reflects a failure of instruction. Our goal should be for every student to achieve the learning outcomes we establish for each course, whether those are to learn certain information, understand key concepts, or develop skills to a specified level of proficiency. Some students may get there faster or easier, but if our teaching is effective and successful, all students should learn what we want them to learn and earn high marks on assessments. If a student is incapable of learning what we are trying to teach, the student should not be allowed to become a lawyer. Criteria-referenced assessments rely on detailed, explicit criteria that identify the abilities students should be demonstrating (e.g., applying and distinguishing cases) and the bases on which the instructor will distinguish among excellent, good, competent, or incompetent performances.16 The use of criteria minimizes the risk of unreliability in assigning grades.17 Criteria-referenced assessment enables teachers to "judge whether certain criteria have been satisfied and normally operates on a pass/fail basis: an example would be the driving test. It is not important to establish whether more or less drivers pass this test in any one year (or at any one centre) but only to ensure that the national pass standard is maintained."18 The use of clear criteria helps students understand what is expected of them as well as why they receive the grades/marks they receive and, even more importantly, it increases the reliability of the teacher's assessment by tethering the assessment to explicit criteria rather than the instructor's gestalt sense of the correct answer or performance.19 The criteria should be explained to students long before the students undergo an assessment. This enhances learning and encourages students to become reflective, empowered, self-regulated learners.20 In creating assessment criteria, teachers should recognize that the development of professional expertise takes time and there are stages with discernable differences, for example, novice, advanced beginner, competent, proficient, and expert.21 Therefore, our assessments should communicate to students where their development of professional expertise stands. Defining the level of proficiency that we want law students to achieve at each stage of their professional development is a task that warrants the attention of clinical teachers. Can We Assess What We Purport to Teach In Clinical Law Courses? 13 15 Judith Wegner, Thinking Like a Lawyer About Law School Assessment (Draft 2003) (unpublished manuscript on file with Roy Stuckey) (hereinafter Wegner, Assessment). 16 Sophie Sparrow, "Describing the Ball: Improve Teaching by Using Rubrics-Explicit Grading Criteria", 2004 Mich. St. L. Rev. 1, 6-15. 17 See N. R. Madhava Menon, "Designing a Simulation-Based Clinical Course: Trial Advocacy" in A Handbook on Clinical Legal Education 177, 181 (N. R. Madhava Menon ed., Eastern Book Company, India 1998) ("Students and evaluators need a clear understanding of the criteria on which performances will be graded."). 18 Bone, supra note 5, at 4. 19 Sparrow, supra note 16, at 28-29. 20 Id. at 22-25. 21 Wegner, Assessments, supra note 15, at 11. Clinical Courses Clinical courses are courses that use experiential education as the primary method of instruction. Experiential education can occur in many contexts in U.S. law school courses and in U.K. law school courses, vocational courses, and law firm training. There are three primary types of clinical courses: simulation-based courses, in-house clinics, and externships.22 These courses differ from each other in the following ways: -in simulation-based courses, students assume the roles of lawyers and perform law- related tasks in hypothetical situations; -in in-house clinics, students represent clients or perform other professional roles23 under the supervision of members of the faculty; and -in externships, students represent clients or perform other professional roles under the supervision of practicing lawyers or they observe or assist practicing lawyers or judges at their work. Experiential education integrates theory and practice by combining academic inquiry with actual experience. Learning is not education, and experiential learning differs from experiential education.24 Learning happens with or without teachers and institutions.25 For example, eavesdroppers learn about the things they hear,26 yet they are not educated simply by the fact of eavesdropping because the activity is not accompanied by a teacher's or institution's participation in the learning process. Education, in contrast to a learning opportunity, consists of a designed, managed, and guided experience.27 28 Thus, while part-time work experiences of law students in legal settings can be valuable learning experiences, they are not considered experiential education because the learning in such environments is not necessarily accompanied by academic inquiry. Optimal learning from experience involves a continuous, circular, four stage sequence of experience, reflection, theory, and application. Experience is the immersing of one's self in a task or similar event - the doing. Reflection involves stepping back and reflecting on both the cognitive and affective aspects of what happened or was done. Theory entails interpreting the task or event, making generalizations, or seeing the experience in a larger context. Application enables one to plan for or make predictions about encountering the event or task a second time.29 14 Journal of Clinical Legal Education August 2006 22 Some people define experiential education as involving "real life," not simulated, experience. See, e.g., Hess and Friedland, supra note 12, at 105. 23 Other professional roles include, for example, serving as mediators or teaching street law. 24 See Lewis Jackson and Doug MacIsaac, "Introduction to a New Approach to Experiential Learning" in 62 New Directions for Adult & Continuing Educ. 17, 22-23 (1994). 25 Ronald Barnett, "What Effects? What Outcomes?", in Learning to Effect 3, 4 (Ronald Barnett ed., 1992). 26 Id. 27 Robert Stevens, Law School: Legal Education in America from the 1850s to the 1980s 24 (1983). 28 James E. Moliterno, "Legal Education, Experiential Education, and Professional Responsibility", 38 WM. & Mary L. Rev. 71, 78 (1996). 29 Steven Hartwell, "Six Easy Pieces: Teaching Experientially", 41 San Diego L. Rev. 1011, 1013 (2004). There are three domains of learning, and students who are being educated experientially are involved in all three: -the cognitive domain (increasingly complex sorts of understandings and analytical processes), -the psychomotor or performance domain (complex patterns of physical or motor activity such as lawyering activities), and -the affective or feeling domain (values, attitudes, and beliefs).30 Jay Feinman further describes the cognitive, performative, and affective skills that law students need to develop: Cognitive skills range from simple recall of facts, through the ability to apply prior knowledge to solve new problems, up to the ability to evaluate the use and implications of one's knowledge. In law school, these skills involve the understanding of substantive law, legal process, and related matters such as professional responsibility. Performative skills in law are increasingly defined by the MacCrate Report's catalog of skills beyond legal analysis and reasoning, including legal research, factual investigation, counseling and the management of legal work. Affective skills include personal and professional issues: how students feel about their competency as lawyers, how they relate to the client, how they respond to problems of professional responsibility, and how their values inform their role.31 Experiential education is especially beneficial to students because it gives them opportunities to be actively involved in their own education, and it has positive effects on their motivation, attitudes toward the course, willingness to participate in class, ability to ask insightful questions, and acquisition of knowledge and skills. Any subject can be taught using experiential methodology. The challenge is to determine what lessons can be taught more effectively and efficiently using experiential education than through other methods of instruction and to focus our time and energy on accomplishing those learning objectives. Once we select our desired outcomes and decide how we will try to achieve them, we need to find out if our students are learning what we purport to be teaching. This requires valid, reliable assessment tools. It would not be fair to grant credit for course work or to base decisions about marks or grades on anything other than solid evidence showing which students are learning and which are not. Unfortunately, as we will see in the following section, this can be a difficult, sometimes impossible, task. Can We Assess What We Purport to Teach In Clinical Law Courses? 15 30 Kenneth R. Kreiling, "Clinical Education and Lawyer Competency: The Process of Learning To Learn From Experience Through Properly Structured Clinical Supervision", 40 Md. L. Rev. 284, 287, n.10 (1981). 31 Jay M. Feinman, "Simulations: An Introduction", 45 J. Legal Educ. 469, 472 (1995). Specific Examples The remainder of the article will examine four outcomes that might be among the educational objectives of a clinical course to illustrate some of the issues related to the assessment of clinical students: 1. Understanding litigation and alternative dispute resolution. 2. Autonomy and ability to learn. 3. Ability to establish rapport in an initial client interview. 4. A commitment to seeking justice. I do not mean to imply that these goals would necessarily exist in a given clinical course. They are presented here to provide a context for discussing assessment issues. 1. Understanding litigation and alternative dispute resolution. It is easier to set and control educational goals in simulation courses than in client representation courses. Some of the educational goals of client representation courses are predetermined and unavoidable. We have to teach students about office procedures, including the central importance of avoiding conflicts of interests and maintaining confidences. We have to teach students about their relationships with us and the restrictions we are placing on their freedom to engage in client representation. We often have to teach the rules of evidence and professional conduct and basic lessons about lawyering skills and how to act as lawyers. We also have to teach students about the law, procedures, systems, and protocols of the various practice settings they will encounter in our courses. A common objective of clinical courses is to teach students about litigation and alternative dispute resolution. These are topics that both the MacCrate Report32 and the Law Society of England and Wales indicate that all lawyers should understand before beginning practice. The MacCrate Report includes a description of the "fundamental lawyering skills essential for competent representation" for all lawyers graduating from law school in the United States, no matter what practice settings they are entering. Among the items on the MacCrate list is that a lawyer should understand the potential functions and consequences of litigation and alternative dispute resolution processes and should have a working knowledge of the fundamentals of litigation at the trial-court level, including, inter alia: (a) An understanding of the litigative process, including: (I) The functions and general organization of the trial courts; (ii) Basic concepts of jurisdiction; (iii) The availability of alternative forums and the importance of choice of forum; (iv) The basic procedural rules and principles governing jurisdiction in a trial court of general jurisdiction; (v) The basic rules and principles of evidence; 16 Journal of Clinical Legal Education August 2006 32 American Bar Association, Section of Legal Education and Admissions to the Bar, Legal Education and Professional Development- An Educational Continuum, Report of the Task Force on Law Schools and the Profession: Narrowing the Gap (1992) (the MacCrate Report). (vi) Knowledge of the means by which additional pertinent rules of procedure and evidence may be efficiently ascertained (including legal research).33 The Law Society is more ambitious than the MacCrate Report in describing the knowledge and skills related to litigation and advocacy that new lawyers should have on day one in practice. The Law Society expects that, "[o]n completion of this compulsory area [litigation and advocacy] the student should have an appreciation of the nature of civil and criminal litigation, be able to identify the critical steps in the process of litigation and have gained experience through practice of some or all of the stages of litigation."34 The Law Society also includes a list of the specific abilities related to dispute resolution that students should be able to demonstrate, including various skills and an appreciation of the range of methods available to resolve disputes. It is common for client representation clinical courses, and even simulation courses, to teach many of the topics on the MacCrate and Law Society lists by involving students in specific litigation contexts, such as criminal prosecution and defense, domestic violence, divorce, landlord-tenant, and consumer disputes. The educational objectives of such courses almost necessarily include helping students acquire knowledge of the fundamentals of litigation at the trial level in those specialized contexts. When I taught an in-house divorce clinic, I expected my students to learn how divorce cases are processed in South Carolina, the relevant rules of the Family Court, and the requirements for drafting and serving various types of court documents. This was not knowledge they had before enrolling in my course. It would be fairly easy to conduct a summative assessment to determine whether we are achieving the educational objectives described above as well as other items on the MacCrate and Law Society lists. After all, they refer to "legal knowledge," "understanding," and "ability to identify" which can be demonstrated on traditional written tests. Although I could have conducted summative assessments of knowledge and understanding that I was purporting to teach my clinical students, I did not. I am unaware of any clinical teachers who conduct summative assessments of such subjects. Why do clinical teachers not make greater use of written instruments for summative assessments? The main reason is probably that the use of written tests to find out what clinical students are learning was not, and still is not, part of the tradition of clinical teachers in the United States - or anywhere else that I am aware of. Another reason may be that clinical teachers feel they are adequately assessing their students' knowledge and understanding of these topics by working with them on cases. This is certainly true to a degree, but it is not clear how much of this assessment should be considered formative. Evaluating knowledge and understanding by observing students' case work is also somewhat subjective and not uniform from student to student. This reason still does not explain why clinical teachers do not also administer end of the term summative assessments to find out if these educational goals are being achieved. Can We Assess What We Purport to Teach In Clinical Law Courses? 17 33 Id. at 191. 34 Education and Training Unit of the Law Society of England and Wales, Legal Practice Course: Written Standards 11 (Version 10, September 2004), http://www.lawsociety.org.uk/documents/download s/becominglpcstandards.pdf. The Legal Practice Course is the year long vocational training course that aspirant solicitors are required to take following their undergraduate law degrees and before beginning their training contacts (articles). One also might speculate that the absence of formal assessments in clinical courses reflects our insecurity. The fact that we do not administer final exams is popular with students. Would students' interest in clinical courses drop if we gave final exams? 2. Autonomy and ability to learn. In 1982, Tony Amsterdam said "the most significant contribution of the clinical method to legal education" is giving students an opportunity to learn how to learn from experience.35 The importance of helping students become independent learners with the ability to engage in effective self-reflection has long been recognized by legal educators in the United Kingdom. The current benchmark standards as well as the draft statement of benchmark standards for Law in England, Wales, N. Ireland36 and in Scotland37 include "autonomy" and "ability to learn" among the abilities that all undergraduate law students must demonstrate before graduation. 5. Autonomy and ability to learn: A student should demonstrate a basic ability, with limited guidance • to act independently in planning and undertaking tasks in areas of law which he or she has already studied; • to be able to undertake independent research in areas of law which she or he has not previously studied starting from standard legal information sources; • to reflect on his or her own learning, and to seek and make use of feedback. 18. Autonomy and ability to learn: This is perhaps the key feature of graduateness. The ability to learn and make use of learning in an independent fashion is generally taken to distinguish the final year student from the first year student. The learning activities required by a Law School should be such that students should be required to demonstrate what they can do independently, rather than just demonstrating that they have learnt what they have been told. This can be demonstrated by the structure of a particular module. For example, all students may be required to study a module without lectures and which requires them to prepare material for individual seminars, not all of which is directed by the teacher. This could provide a basis of evidence on whether individual students are able to learn on their own with minimal guidance. (emphasis added) Minimal Guidance: Obviously, an independent learner will need some support and some broad structure within which to operate. The extent of guidance required will depend on a student's stage of development in the field and the complexity of the material. However, by the honours stage the 18 Journal of Clinical Legal Education August 2006 35 Anthony Amsterdam, Professor, New York University School of Law, address at the Deans' Workshop conducted by the ABA Section of Legal Education and Admissions to the Bar (January 23, 1982) (unpublished). 36 Quality Assurance Agency for Higher Education, Draft Statement Benchmark Standards for Law (England, Wales, N. Ireland), http://www.qaa.ac.uk/academicinfrastructure/bench mark/evaluation/law.asp (last visited July 12, 2006). 37 Quality Assurance Agency for Higher Education, Draft Statement Benchmark Standards for Law (Scotland), http://www.qaa.ac.uk/academicinfrastructure/bench mark/evaluation/law.asp (last visited July 12, 2006). teacher input should indeed be small. The independent undergraduate should be able to take the initiative to seek support and feedback. Ability to reflect critically: A student should be able not only to learn something, but to reflect critically on the extent of her or his learning. At a minimum, a student should have some sense of whether s/he knows something well enough or whether s/he needs to learn more in order to understand a particular aspect of the law.38 The draft statements also define the level of performance expected at the vocational level as follows: Can act independently in planning and managing complex tasks with limited guidance within a defined framework; able to identify own resources; Can reflect on own learning; can seek and make use of feedback.39 While autonomy and ability to learn are described as "perhaps the key feature of graduateness" for undergraduate law students, it is also important for lawyers to continue developing these attributes at all levels of legal education and throughout their careers. Autonomy and ability to learn are skills that can be developed in any type of law school course, but experiential education courses are particularly well-suited to this purpose. In his 1982 remarks, Tony Amsterdam said: The students who spend three years in law school will next spend 30 or 50 years in practice. These 30 or 50 years will be a learning experience whether we like it or not. It can be, as conventional wisdom has it, merely a hit-or-miss learning experience in the school of hard knocks. Or it can be a mediated and systematic learning experience if the law schools undertake as part of their curricula to teach students techniques of learning from experience. Clinical courses can do this - and should focus on doing it - because their very method is to make the student's experience the subject of critical review and reflection.40 As Amsterdam points out, students are unlikely to develop fully their ability to learn from experience during law school, particularly in systems of legal education such as the United States' where legal education only lasts three years before a person can obtain an unrestricted license to practice law. No matter how long it takes to become a lawyer, however, lawyers continue to develop problem-solving expertise throughout their careers. The ability to learn from experience is, therefore, an important life-long skill for lawyers to acquire. At the outset, therefore, it seems that helping students further develop their autonomy and ability to learn should be an articulated goal of all clinical courses, simulated and real life. One can only speculate about the percentage of clinical courses that articulate this as a goal and consciously pursue it. For our purposes, we will assume it is a goal, and turn our attention to how one might assess our success in achieving it. Can We Assess What We Purport to Teach In Clinical Law Courses? 19 38 Benchmark Standards, England, supra note 36, at Guidance Note for Law Schools on the Benchmark Standards for Law Degrees in England, Wales and Northern Ireland, items 5 & 18. 39 Id. at app. A (Illustration of Possible Modal Statements, Autonomy). 40 Amsterdam, supra note 35. As mentioned above, the benchmark standards suggest that a basis of evidence on whether individual undergraduate students are able to learn on their own with minimal guidance could be provided by requiring students "to study a module without lectures and to prepare material for individual seminars, not all of which is directed by the teacher." Similar opportunities can be afforded students in clinical courses. Students could be assigned to teach portions of classroom meetings in simulation, in-house, and externship courses. While this might enable students to demonstrate some skills, it does not enable them to demonstrate an ability to learn from experience, certainly not from the experience of law practice. Two preliminary issues are presented. The first is that students at every level of professional training will have a range of skill levels. This means that some students will probably exceed the level of proficiency required at a given level of professional training and others will not. Some students will, therefore, require more opportunities and assistance to reach an acceptable level. The second issue is how to define the desired level of proficiency at each level of professional training. These are issues beyond the scope of this paper, but critical for assessment. Clinical courses, especially courses in which students represent actual clients, are well-suited for giving students opportunities to demonstrate and develop the ability to work autonomously. Students who are ready or nearly ready for law practice will take charge of their cases, and show up in their supervising lawyers' offices after accomplishing what they can do on their own and with clear ideas about what help they need before going forward. Their files will be organized, and tasks will be completed before they are due. Other students will need more help figuring out what needs to be done, what they should do on their own, and when and how to seek assistance. Instructors must monitor their work carefully and spend time helping students understand the level of autonomy they should exercise in professional settings. How might one conduct a summative assessment of autonomy at the end of a course of study? It may not be possible, even if we had the resources to assign similar legal problems to all students and observe them at work. One might consider giving students a set of facts from an initial interview, then asking the students to describe the initial steps they would take toward resolving the problem (or even to perform the tasks they would undertake on their own), and they could be asked to describe what more they would need before going forward and how they would try to obtain it. Turning now to assessment issues related to the ability to learn from experience, it is once again difficult to conceptualize how to describe the level of skill expected at each stage of professional training, and it is clear that students' skill levels will vary. If we assume that an instructor is consciously trying to assess a students' ability to learn from experience, one might begin by applying Ken Kreiling's methodology. Kreiling was among the first to help us understand that the most effective way to learn from experience is to use "theories of practice" to develop and articulate "espoused theories of action."41 "Theories of practice" provide a basis upon which students can evaluate behaviors they observe and their own performances. These theories may involve information about how lawyers should conduct themselves, how certain aspects of the judicial system should work, or whatever else is relevant to the educational objectives of the course. "Theories of action" explain how a student hopes to perform in a 20 Journal of Clinical Legal Education August 2006 41 Kreiling, supra note 30, at 286. lawyering situation, for example, to build a close and trusting relationship in an initial client interview, to use only leading questions during a cross examination, to be flexible about means and rigid about goals in negotiation. Following a performance, the espoused theory of action can be compared to the behavior actually exhibited, the "theory in use."42 If the comparison discloses that the student was ineffective in applying the espoused theory of action, the student and the teacher can analyze what caused the ineffectiveness - the quality of the espoused theory; the student's skills, values, or knowledge; or some other factor. Thus, we should first help the student learn theories of practice, that is, how a particular task should be done. Then, make sure the student has articulated a plan for how the student intends to perform a particular skill. After the performance is finished, discuss with the student how the actual performance related to the planned performance. Finally, analyze why any differences existed and what the student would try to do differently the next time. These discussions would not be limited to the technical aspects of performances. They could also include ethical, moral, and affective issues. The conversations with students might occur in one-on-one or group meetings. Once students understand the method of analysis expected, the instructor might simply ask, "What did you learn from your [trial, interview, phone conversation, meeting with opposing counsel, and so forth]?" Instructors could also ask students to write reflective journals in which they organize their thoughts about their experiences and describe what they learned from them. The quality of the information produced by such approaches may be affected by a student's reluctance to discuss certain lessons learned, even if the student in fact is aware of them and benefitted from them. To reduce this risk, it is important to ensure that students understand that a goal of the course is to help them learn from experience, the instructor will be evaluating their skill level, and the kinds of information that will demonstrate whether the student is or is not demonstrating a sufficient level of skill. Even then, such assessments may not be reliable or valid. Perhaps the only valid and reliable method to evaluate their abilities to learn from experience would be to follow our students into practice for a period of time. Perhaps even that would not work. It may not be possible to develop valid and reliable summative assessments of some of our desired outcomes, and autonomy and ability to learn may be among these. We can, however, determine whether our students understand how to apply theories of practice to concrete situations. There are a variety of ways we could do this. A simple test would be to describe a task to be performed and ask the student to articulate several "espoused theories" of how the task should be performed. Another way would be to show a videotape of a lawyer's performance and ask the student to evaluate it in light of theories of practice that were studied during the course. Or both of these techniques could be combined. While success on these tests would not conclusively establish a student's ability to learn from experience, failing these tests would indicate that the student has not yet developed these skills. Assessing a student's autonomy and ability to learn is problematic in clinical courses that assign letter grades, which is common practice in the United States. Unless a valid summative assessment can be devised and administered, one is forced to decide whether to give the highest grades to Can We Assess What We Purport to Teach In Clinical Law Courses? 21 42 Id. at 291-97. students who come into the course with highly developed skills and perform better throughout the course or to give the highest grades to the students who demonstrate the greatest improvement during the course. In criteria-referenced courses, on the other hand, the question is whether the student passes or fails, or perhaps deserves honours. Theoretically, improvement should be irrelevant, although as a practical matter it is difficult to ignore. 3. The ability to establish rapport in an initial client interview. One of the most common tasks that lawyers perform is the initial meeting with a potential client. This is an important meeting because it is the basis upon which the lawyer and client decide whether to form an attorney-client relationship and it sets the stage for the lawyer's initial work on the case. Assessment criteria for evaluating lawyer-client communication skills, beginning with client interviewing, are being developed in an on-going project by faculty at Georgia State University School of Law, the Glasgow Graduate School of Law, and the Dundee Medical School.43 The project is taking the components of effective client interviewing skills and breaking them down into discrete segments with descriptions of various levels of proficiency. Hopefully, more collaborations like the Glasgow/Georgia State project will lead to the development of additional rubrics and a growing consensus about what we should be teaching students and how we can measure our success. Establishing rapport with a potential client is an important goal of a lawyer conducting an initial client interview. If a lawyer cannot establish rapport with a potential client, the client may decide to hire another lawyer, the client may not feel comfortable being open and honest with the lawyer, and fee collection might become difficult. "'[R]apport' means mutual trust. Clients must trust you in order to open up and be candid."44 "Rapport" also "connotes a certain personal regard between you and the client, though this regard is not necessarily the same as friendship. Rather, the rapport one seeks is one of genuineness and respect, the client knows you care about and respect the client as a person and the client returns those feelings."45 Establishing rapport seems to come more naturally for some people than for others. How might clinical teachers assess the ability of a student to establish rapport with a client? The most common method is to observe a student conducting a real or simulated interview. We form our own opinions of the student's success, and we can seek the student's and client's opinions. The use of "standardized clients" is a key part of the methodology being used in the Georgia State/Glasgow project.46 The clients are trained to provide consistent information in interviews, and they play the role of the client for multiple students. Following the interview, the simulated clients are asked, among other things, to evaluate the degree to which the students achieved the goal of establishing rapport. 22 Journal of Clinical Legal Education August 2006 43 Karen Barton et al., Do We Value What Clients Think About Their Lawyers? If So, Why Don't We Measure It?, Paper Presented at the UCLA Law/University of London Sixth International Clinical Conference (Oct. 28, 2005), available at http://www.law.ucla.edu/docs/barton__cunningham __jones__maharg-_what_clients_think.pdf. 44 Robert M. Bastress and Joseph D. Harbaugh, Interviewing, Counseling, and Negotiating 67 (1990). 45 Id. at 66. 46 The use of standardized clients is modeled after the medical profession's use of the Objective Structured Clinical Examination (OSCE), a timed multistation examination using standardized patients to simulate clinical scenarios. Epstein & Hundert, supra note 6, at 230. Deciding how to describe the level of rapport-building skill that should be achieved at each level of professional training is a difficult problem. In the Georgia State/Glasgow project, points for rapport building were awarded on the following scale: 1-2 points: Lawyer was bored, uninterested, rude, unpleasant, cold, or obviously insincere. 3-4 points: Lawyer was mechanical, distracted, nervous, insincere, or used inappropriate remarks. 5 points: Lawyer was courteous to you and encouraged you to confide in him or her. 6-7 points: Lawyer was generally attentive to and interested in you. 8-9 points: Lawyer showed a genuine and sincere interest in you. There was a sense of connection between you and the lawyer. These points were added to points awarded for other aspects of the interview to determine if a student passed or failed. Instead of basing the pass/fail decision on the total point score, one could require a minimum level of achievement on each component that is separately scored. For example, one could reasonably argue that a student who scores only 1-2 points for rapport building should not be allowed to pass, no matter how many other points were earned for the overall interview. People could also have differing opinions about the degree of proficiency that lawyers should be required to demonstrate before being fully licensed, that is, whether the pass mark should be set at the 5, 6-7, or 8-9 point level. And what level of proficiency should we require of a law student who is conducting an initial interview for the first or second time? Perhaps we should concede at the beginning that we cannot validly or reliably assess a student's ability to establish rapport with a client. Even when we use a hypothetical problem with a well- trained standardized client, it is still a make believe situation. The feelings of trust, etc., between the client and the student may or may not be real feelings, but they are not formed in a true attorney client relationship where the existence or nonexistence of rapport would have real consequences. Thus, the assessment would not be completely valid. In a real life client clinic, we could try to measure the degree of rapport between students and real life clients, but we could not measure the same client's feelings of trust toward each student, because each student would have a different client. Thus, the results would not be reliable. Even if we cannot accurately determine the degree of rapport that a student can establish with a client, we can measure whether and how well a student employs techniques that, in theory, will build rapport, and how well a student avoids words and actions that would hinder it. These techniques47 might include such things as: -be friendly. -stand up to greet the client warmly and personally. Can We Assess What We Purport to Teach In Clinical Law Courses? 23 47 This list was compiled from the following sources, Robert F. Cochran, Jr., John M.A. DiPippa, and martha M. Peters, The Counselor-At-Law: A Collaborative Approach to Client Interviewing and Counseling 29-106 (2d ed. 2006); Stefan H. Krieger and Richard J. Neumann, Jr., Essential Lawyering Skills (2d ed. 2003); Robert M. Bastress and Joseph D. Harbaugh, Interviewing, Counseling, and Negotiating 59-232 (1990); David A. Binder and Susan C. Price, Legal Interviewing and Counseling: A Client-Centered Approach 6-123 (1977). -engage in appropriate ice-breaking talk before the interview. -have a comfortable and inviting office. -give the client your full attention; avoid interruptions during the meeting. -give the client an immediate opportunity to explain why the client is there and how the lawyer can help. -attend to any immediate questions or other needs of the client. -explain the purpose and structure of the meeting, including any costs. -listen to the client without being judgmental. -use verbal and nonverbal communication facilitators, such as arranging the office so that there is no physical barrier between the lawyer and the client, employing good body language, and using active listening techniques.48 -avoid or adjust for communication inhibitors such as ego threat, case threat, etiquette barriers, trauma, perceived irrelevancy, and greater need. -employ a structure for organizing the discussion. -appear confident and competent. -appear empathetic and concerned about the client's problem. -ensure before the meeting ends that the client knows when to expect the next contact with the lawyer and what the client and the lawyer are to do before then. Of course, this raises the question of whether some techniques are more important or effective than others, that is, should a student's success employing certain techniques be weighted more heavily in our assessment of the student's rapport building skills? This is a topic worthy of empirical research by clinical teachers. The most effective assessment method, of course, would be to observe new lawyers in actual practice over a period of time and with a variety of clients who would be questioned by the assessor throughout their relationships with the lawyers. In an academic setting, the most effective method is probably to observe and record students' meetings with clients, and discuss with them the extent to which they believed rapport was established and how well the student employed rapport building techniques, that is, to compare theories of practice with actual performance. Additional discussions would occur after faculty, students, and clients review the recording of the interview, and perhaps after the student drafts a reflective self-evaluation memo. Due to resource constraints in the real world, clinical teachers use variations on the performance, discussion, reflection method, for example, by using students to play the clients, using people other than the instructor to provide feedback, only having one meeting, not recording the interview, and so forth. Most of these feedback sessions are formative assessments, though they are often formative and summative in United States schools because some clinical teachers assign grades to every student performance. Although I once did the same thing, I have come to doubt the fairness of this practice and the validity of any grade assigned to a single performance by a single observer. Feedback should be formative until the student has had an opportunity to study and practice the required task. Some students will demonstrate good practice skills in their first performance, but 24 Journal of Clinical Legal Education August 2006 48 Additional facilitators are discussed in Binder & Price, supra note 47, at 14-18. those who do not should not suffer a grade penalty because other students came into the course with more highly developed skills or knowledge. Instead, students who have not demonstrated an adequate level of proficiency should be required to continue practicing that task until the desired level of proficiency is achieved. Helping all students achieve an appropriate level of rapport building proficiency for that stage of their professional development should be our goal, not simply measuring which students are better at certain tasks than others. Ideally, we should conduct summative assessments at the end of the unit of study. Very few schools could afford to use actual client interviews for summative assessments. We should therefore consider whether we can create an end-of-the-course summative assessment that does not involve an actual performance of an initial interview. One idea would be to show a videotape of an interview to a student and ask the student to evaluate the skill with which the lawyer on the tape employed effective or ineffective techniques. This would not indicate the ability of the student being evaluated to establish rapport, but it would evaluate the degree to which the student can observe a performance and analyze it in light of theories of practice about establishing rapport. Asking a student to analyze a transcript of an interview could produce some indication of the student's knowledge about theories of practice as well as his or her ability to analyze the performance of the task. Even multiple choice questions could produce valid and reliable data about a student's knowledge of theories of practice related to rapport building. Of course, even if a student demonstrates knowledge of why rapport building is important and what might impede or enhance rapport building, this does not necessarily predict how the student will actually perform. We assume, however, that students who have this knowledge are more likely to perform adequately than students who do not. Why else are we producing books and videos about rapport building? 4. A commitment to seeking justice. We may have some educational objectives that cannot be measured or that we do not want to measure. Many clinical teachers believe that an important objective of clinical courses is to teach our students about the values of the legal profession and to instill in them a commitment to conform their law practice to those values.49 All professional values deserve attention by law schools, but teaching students to strive to seek justice may be the most important goal of all. Andrew Boan concluded that "[t]he integration of skills and knowledge should assist practitioners in achieving the good of legal professions; achieving justice. The development of virtues consistent with this social good must be a central goal of legal education."50 Richard Burke reached similar conclusions: Can We Assess What We Purport to Teach In Clinical Law Courses? 25 49 The Global Alliance for Justice Education (GAJE) was created to promote justice education by law schools and NGOs. Membership in GAJE is free. For more information about GAJE including how to join, visit the GAJE website, http://www.gaje.org (last visited August 4, 2006). 50 Andrew Boon, "History is Past Politics: A Critique of the Legal Skills Movement in England and Wales", in Transformative Visions of Legal Education 151, 154-155 (Anthony Bradney & Fiona Cownie eds., 1998), published simultaneously in 25 J. Law & Soc. 151 (1998) (citing Ronald Dearing, The National Committee of Inquiry into Higher Education, Report of the National Committee (1997)). Truth, justice, and fairness, both in means and ends, are paramount on the scale of legal values, and when those are at stake, the other values must yield.51 . . . . . First, we should say that truth and justice are our goals; that, though we may never find totally objective truth or achieve perfect justice, we will seek and strive for them to the best of our professional ability. Second, we should make clear that this quest for truth and justice is a professional responsibility upon which rests the reliability and integrity of the entire legal system. Hence, an individual client's desires and objectives must be subordinate to that quest. Third, our rules of conduct should specifically prohibit lawyer or lawyer participation in lying, falsification, misrepresentation, or deception in every aspect of practice from courtroom advocacy to office consultation and practice.52 Calvin Woodward believed that teaching students to seek justice should be the central focus of legal education. Woodward considered the impact of the centuries-long process of secularization and concluded that this process had undermined the influence of religion and discredited legality as a social sanction, especially in western democratic societies. He also determined, however, that "the course of secularization has been led, almost without exception, by men seeking substantial justice. And therein lies the clue - a straw in the wind - for modern law schools. In a world populated by ultra-rational men, Law must find its strength in Justice, not Legality."53 Woodward called on law schools to train students to regard themselves as agents of justice as well as officers of the court. Law schools must rid themselves of the vestiges of mysticism that, in days past, held laymen in awe of law and legality; and students must be trained to regard themselves as agents of Justice as well as officers of the court. More important, they must be shown precisely what this responsibility entails. And establishing a course of instruction that will serve this purpose should be the great issue with legal education today.54 Woodward proposed two governing maxims for law schools. "First, within the House of the Law there are many mansions - in which practitioners of all kinds, counsellors, judges, public servants, scholars and philosophers work in their several ways to further the course of, and to implement, Justice. Second, legal education, as an adjunct of Justice, must start with the proposition that the greater includes the lesser, the higher the lower, and not vice versa. That is, law schools must assume, as their basic premise, that the man who first understands his obligations to Justice will be better able to fulfill his legal 'function,' whatever it might be. Justice, in a word, must take precedence over law."55 In light of the importance of instilling a commitment to justice in all law students, it is easy to conclude that every clinical course should make this an explicit educational objective and make every effort to design our courses to accomplish this goal. Having said this, however, we should 26 Journal of Clinical Legal Education August 2006 51 Richard K. Burke, ""Truth in Lawyering": An Essay on Lying and Deceit in the Practice of Law", 38 Ark. L. Rev. 1, 22 (1984). 52 Id. at 3-4. 53 Calvin Woodward, "The Limits of Legal Realism: An Historical Perspective", in Herbert L. Packer and Thomas Ehrlich, New Directions in Legal Education 329, 380 (1972). 54 Id. 55 Id. at 381. recognize that it is difficult to define justice much less to know with any certainty what is and is not justice in real world situations. Assuming we can resolve these issues satisfactorily, is it possible to tell if our efforts are having the desired impact? Can we assess our students' commitment to justice without following them into practice and tracking what they do with their careers? Lawyers have different degrees of commitment to justice. Is there a way to describe the minimal level of commitment that all lawyers should have before being fully admitted to practice? Can we describe levels of commitment to justice that students should have at each stage of their professional development? This is probably unlikely. What assessment tools might produce any information about a student's commitment to justice? I concede that I do not have very good answers. In clinical courses, especially real life clinics, we can observe students working on cases and listen to what they say about justice-related issues. This may be the best indicator of a student's feelings and beliefs about justice. We could assess our students' knowledge and understanding of justice-related topics. This could produce valuable data. One could reasonably conclude that a student who has never considered what "justice" is, how injustices can destroy people and societies, or why it is important for lawyers to have a commitment to justice, are less likely to have or develop a commitment to justice than students who have these understandings. We could acquire valid and reliable information about our students' "justice knowledge" through written tests, including essays by students demonstrating their understanding of the importance of seeking justice and role of each individual lawyer in providing access to justice. We could also provide students with scenarios based on our clinics' or other cases and ask them to identify injustices in those scenarios and to discuss how lawyers contributed to the injustices or might contribute to resolving them. It might provide useful information about the effectiveness of our instruction to conduct such assessments at the beginning and end of clinical courses that seek to instill a commitment to justice in their students. In the final analysis, it is difficult to imagine that we would ever refuse to pass a student for not developing a commitment to justice or an "adequate" level of commitment. Would we assign grades/marks to students on the basis of their commitment to justice or is this a desirable educational outcome that should have a formative, but not a summative, assessment? We may not even want to share our conclusions about a student's commitment to justice with the student, even if we could acquire valid and reliable data. Perhaps, we should only use it to evaluate the effectiveness of our instruction. My only point in this section is that if clinical teachers continue to claim that our educational objectives include instilling a commitment to justice or otherwise developing our students' professional values, we need to think very carefully about what we intend to teach about these matters and how or whether we will try to measure our success. Can We Assess What We Purport to Teach In Clinical Law Courses? 27 Conclusion This article raises more questions than it answers. Some things are clear, however: - assessments are important to students and institutions. - assessments should be as reliable, valid, and fair as possible. - we need to be very precise about our educational objectives. - we need to articulate our assessment criteria and communicate them to our students. - we should not profess that students will learn something in our courses if we cannot assess whether such learning occurs. There are probably some outcomes that legal education would like to accomplish that cannot be assessed adequately. We may need to distinguish between desired outcomes and measurable outcomes. The conclusions on this issue in the report of the project to map best practice in clinical legal education in the United Kingdom are very insightful. For anyone reading this report thinking of setting up a program and despairing at the difficulty of identifying the objectives that really matter to them personally - changing people, planting a lifelong interest in justice, etc. - our experience (personal and through this research) is that you will never succeed but do not need to try. It is perhaps legitimate to settle for more mundane objectives which are recognizable within the normal academic structures or programmes. Then you will not fail. This does not prevent you achieving the higher order objectives, but if you defined them and claimed to assess your achievement of them, you would face potential disappointment and a range of challenges in terms of actual assessment of students' work.56 The ultimate question we would like to answer is whether a student will practice law effectively and responsibly. Unfortunately, the question is enormously complex - what is "law practice" and what do "effectively" and "responsibly" mean? A student's performance in practice will depend on future circumstances that we cannot predict or control. Perhaps the best that legal educators and licensing authorities can hope to achieve is to identify certain aspects of the legal knowledge, skills, and values that we believe are associated with competent law practice, and evaluate as many of these as we can evaluate with valid and reliable tools. Clinical educators should be leading the way in developing innovative methods for assessing legal competence, but so far the assessment topic has not received much attention by clinical teachers in our scholarship or conferences. Hopefully the next generation of clinical teachers will develop the theories and tools to make progress on this issue that my generation has largely chosen to ignore. 28 Journal of Clinical Legal Education August 2006 56 Richard Grimes & Hugh Brayne, Mapping Best Practice in Clinical Legal Education 14 (2004), http://www.ukcle.ac.uk/research/projects/clinic.htm l.