How Should We Assess Interviewing and Counseling Skills?1 Lawrence M. Grosberg Introduction. The need to teach interviewing and counseling skills has long been established among clinical legal educators.2 Even among our non-clinical colleagues, these skills are recognized as integral to competent lawyering.3 While there remains considerable difference of opinion within the United States as to whether teaching such skills should be in a required course or simply be available as an elective, there is no doubt that a twenty-first century American law school must include the teaching of these skills in its curricular array.4 How Should We Assess Interviewing and Counseling Skills? 57 Lawrence M. Grosberg, Professor of Law, New York Law School 1 This essay is derived from a talk given at the Journal's conference at Edinburgh in July, 2004. While it is written from the perspective of an American clinical teacher, the underlying skills concepts increasingly are reflected in clinical education programs around the globe. 2 See Robert M. Bastress & Joseph D. Harbaugh, Interviewing, Counseling, and Negotiating: Skills for Effective Representation (1990); Gary Bellow & Bea Moulton, The Lawyering Process: Materials for Clinical Instruction in Advocacy (1978); David A. Binder, Paul Bergman, Susan C. Price & Paul R. Tremblay, lawyers as counselors: A Client-Centered Approach (2d ed. 2004); Stefan H. Krieger & Richard K. Neumann, Jr., Essential Lawyering Skills (2nd ed. 2003). Philip G. Schrag & Michael Meltsner, Public Interest Advocacy: Materials for Clinical Legal Education (1974). 3 See Jay M. Feinman, "Simulations: An Introduction", 45 J. Legal Educ. 469 (1995); Cf. American Bar Association Section of Legal Education and Admissions to the Bar, Task Force on Law Schools and the Profession: Narrowing the Gap, Legal Education and Professional development- An Educational Continuum (1992) [hereinafter MacCrate Report] (report on the essential components of competent lawyering). 4 The ABA accreditation standards do not now require that all students take clinics or interpersonal skills courses. They do require, however, that every school have a clinical and skills program. See, e.g., Richard A. Matasar, "Skills and Values Education: Debate About the Continuum Continues", 46 N.Y.L. Sch. L. Rev. 395 (2002); Robert MacCrate, "Yesterday, Today and Tomorrow: Building the Continuum of Legal education & Professional Development", 10 Clinical L. Rev. 805 (2003). Recent amendments to ABA accreditation standard 302 may add to the pressures on law schools to require such interpersonal skills learning. But it is far from certain whether the changes in the standards will result in any significant increase in skills teaching, or indeed, in any meaningful curricular reform. See John Sebert (June 8, 2004). http://www.abanet.org/legaled/standards/standardsd ocuments/standardsreview302305commentmemo.d oc. Accompanying the teaching of these skills has been the challenge of evaluating the student performance of such skills. Clinicians knew from the outset that assessment would be qualitatively different from the evaluation methods used for doctrinal teaching.5 We also knew there would be two fundamentally different functions of clinical assessments: the traditional grading or sorting purpose and the feedback/constructive learning function. It is the latter which is and always has been at the core of clinical teaching.6 In the competitive world of law students, however, as with anything that is taught, the formal assessment or grading label remains crucial.7 If it is not tested and graded, it will be devalued by the students.8 The question of how best to do that remains the challenge with which we are grappling.9 A central aspect of assessment is the learning goal. What are we trying to teach? What is the ideal outcome we would like a student to achieve? It is critical that the method of assessment be directly correlated with those learning objectives. An evaluation is valid if it accurately measures the student in terms of the instructional objectives.10 To state the obvious, if we want the student to learn how to draft advocacy briefs, we do not assess by conducting an oral exam with the student. And then, there is the level of proficiency. Are we talking about an introductory or very elementary level with respect to the skill involved? Or, is a much more advanced level of learning the desired goal? Thus, in discussing alternative methods of assessing interviewing and counseling skills, we need to be clear about what our objectives are. Different methods of evaluation may be more or less appropriate depending on the desired outcome. Thus, clinical evaluations of interviewing or counseling skills could include assessing the examinee's knowledge of interviewing models or theories, the method of preparation, the actual performance of the skill, a self-critique of her or his performance, the resolution of ethical dilemmas or all of the above.11 This paper first briefly describes the structure of legal education in the United States (insofar as clinical and skills teaching is concerned) and the almost total absence of any bar admission training or apprenticeship requirements. If the law schools are not required to fully train all future lawyers 58 Journal of Clinical Legal Education August 2006 5 Student performance and critique assessment were central to skills teaching. E.g.: "Intellectual mastery is not sufficient .... the goals must include both the student's ability to understand and ability to perform effectively."... "It is essential that each student have at least one opportunity to be critiqued in conducting a full interview". David A. Binder & Susan C. Price, Instructor's Manual for Legal Interviewing and Counseling, 4 and 22 (1979). 6 See, e.g.,Robert Dinerstein, "Report of the Committee on the Future of the In-house Clinic", 42 J. Legal Educ. 508 (1992); Peter Toll Hoffman, "Clinical Course Design and The Supervisory Process", 1982 Ariz. St. L.J. 277, 292 (1982); David F. Chavkin, "Matchmaker, Matchmaker: Student Collaboration in Clinical Programs", 1 Clinical L.Rev. 199, 202 (1994); Kenneth R. Kreiling, "Clinical Education and Lawyer Competency: The Process of Learning to Learn From Experience Through Properly Structured Clinical Supervision", 40 Md. L. Rev. 284, 330-334 (1981). 7 Stacy L. Brustin & David Chavkin, "Testing the Grades: Evaluating Grading Models in Clinical Legal Education", 3 Clinical L. Rev. 299 (1997) ( a report on a survey of students as to whether they prefer a pass/fail grade or the usual number or letter grades, together with a recommendation for grades with an option for pass/fail). 8 See Steven Friedland, "A Critical Inquiry into the Traditional Uses of Law School Evaluation", 23 Pace L.Rev. 147, 171 (2002); Brustin & Chavkin, supra note 7, at 306; Lawrence M. Grosberg, "Should We Test for Interpersonal Lawyering Skills?", 2 Clinical L. Rev. 349, 351 (1996 ). 9 The broader subject of law school testing generally, is of course, worthy of re-examination, and as has been observed, "has been widely overlooked", Friedland, supra note 8, at 149. 10 Michael Josephson, Learning and Evaluation in Law School, Volume 1, Principles of Testing and Grading, Learning Theory and Instructional Objectives at 8, Submitted to Association of American Law Schools' Annual Meeting, January, 1984. 11 Roy Stuckey, Assessing Law Student Learning in Clinical Courses: Issues for Discussion, Slide 6 (Power Point presentation, 4th International Clinical Education Journal Conference, July 14, 2006) and the bar admission authorities likewise disavow responsibility for doing so, should clinical law professors assume the burden?12 I then go on to discuss the primary clinical evaluation technique of directly observing the student's performance, sometimes referred to as the gold standard method of assessment. Against the backdrop of the assertion that it is beneficial to use multiple methods of assessment, I then describe the several methods I have used to address the question of how best to assess interviewing and counseling skills. As an aside, it becomes clear that much more empirical analysis is in order.13 1. U.S. Legal Education and Bar Admission Standards. Whether you reside inside or outside the United States, it is important to recognize some key differences between American legal education and licensure and the comparable legal institutions abroad. Much of the rest of the world has a law school and licensing structure different from that in the United States. In the U.S.14 almost all lawyers first obtain a four year undergraduate college degree (typically a BA or BS) and then go on to complete either a full-time three year law school program or a four year part-time program, both culminating with a JD degree.15 Following the acquisition of the JD, an applicant for admission to the bar must successfully pass a bar examination in the state in which the future lawyer wishes to practice.16 Once admitted to the bar, the new attorney is licensed to practice in any and all areas of law and inside or outside of the courtroom. While nearly all states have some form of post-graduate mandatory continuing education requirements17, there are almost no apprenticeship requirements18, as there frequently are elsewhere in the world. In the U.S., the American Bar Association Section on Legal Education and Admission to the Bar is the entity that is authorized by the U.S. Department of Education to accredit law schools. It is the ABA, therefore, that determines what a law school is required to do or not to do in order to be accredited. With the exception of a very small number of states, one cannot sit for a bar examination unless one is a graduate of an accredited law school. The ABA has a fairly detailed set of regulations covering everything from the size of the faculty and the physical How Should We Assess Interviewing and Counseling Skills? 59 12 More specifically, should clinicians devise and recommend ways to teach and evaluate skills learning for that large group of law school graduates who elect not to take clinics of skills courses? 13 I am in the process of addressing this dearth of data by conducting various studies of the efficacy of various assessment methods. 14 There are exceptions to all of the following generalizations in the text, however, this abbreviated summary of the differences, I believe, will be useful in considering the main theme of this essay regarding methods of evaluating interviewing and counseling skills. 15 Robert B. Stevens, Law School: Legal Education in America from the 1850s to the 1980s at 209 (1983). See also ABA LSAC Official Guide to ABA- Approved Law Schools 2007 (Wendy Margolis ed., 2007 ed. April 2006). 16 There is one state, Wisconsin, where taking a bar exam is not required if the bar applicant graduated from an accredited law school in Wisconsin. This is referred to as the "diploma privilege." New Hampshire has just begun the experimental Webster Scholar Program pursuant to which a limited number of students are selected to participate after completing the first year. The students then will participate in clinical and skills activities during their last two years of law school. If they satisfactorily complete those tasks, they may be admitted to practice without taking a bar examination. See the discussion of this program in Clark Cunningham, "Rethinking the Licensing of New Attorneys - - An Exploration of Alternatives to the Bar Exam: Introduction", 20 Ga. St. U. L. Rev. vii at xiii - xv (2004). 17 Much, if not most, of American Continuing Legal Education requirements do not involve graded evaluations of any written or oral work. Rather, using New York as an example, the only requirement is that the attendee sign an attendance sheet at the beginning and at the end of a CLE session. 18 Vermont and Delaware require forms of an apprenticeship. See V.R.A.B. ยง 6(i) and Del. Sup. Ct. R. 52(a)(8). plant to guidelines on the curriculum.19 With respect to experiential learning, either in the form of students working on real cases under the supervision of law professors or practicing lawyers (cf. apprenticing), or in the form of simulation exercises, the ABA does not require that every law graduate complete such training. The accrediting standards do encourage law schools to develop clinical education and they do mandate that every law school actually maintain some kind of a clinical or skills program.20 Likewise, the state bar admission authorities, all of which are completely independent of the ABA and the law schools, do not require any type of apprenticeship or experiential testing as part of the requirements for admission to the bar.21 There are some bar exam pen and pencil tests that call on bar applicants to demonstrate some familiarity with lawyering skills such as investigation or interviewing or counseling or trial advocacy skills. But, these questions, called "performance tests"22, do not in any way require the applicant to perform an oral skill. What all of this means is that it is very possible for a law student to graduate law school in the U.S., pass a state bar exam and be admitted to practice law, without ever having interviewed or counseled a client (real or simulated) or tried a case or negotiated a transaction or even without having observed such lawyering activities. Thus, in terms of assessment of the actual performance of interviewing or counseling skills, there is neither a formative nor a summative evaluation requirement in law school or after, prior to being licensed to practice law. I am not aware of any data that specifies precisely how many American law graduates fall into this category of persons bereft of any clinical or skills training. But, extrapolating from my own law school experience and my familiarity with many others, it is clear that there is a significant number in this category, perhaps even half or more of American law graduates. Moreover, the remaining students may have taken only a single clinical or skills course, hardly a basis upon which one might conclude that such a student is well versed or even minimally competent in those skills. It is in this context that law schools must address their institutional responsibilities to produce competent lawyers. If the bar admission officials are not taking steps to ensure the public that this is the case, then law schools, and particularly its clinical faculty, it seems to me, must confront this reality. Even if the ABA does not require that law schools address this need, the law schools certainly are not prohibited by the ABA standards from trying to meet this deficiency. Assuming then that law schools want to address and, indeed, are addressing this need, we come full circle, back to the task of determining how best to teach and evaluate these skills, beginning with interviewing and counseling. 2. Professorial Observation of Student Performance. How does a teacher evaluate a specific student's skills, either in a typical one-on-one live client clinic or in a larger simulation course such as Trial Advocacy or Negotiating, Counseling & Interviewing? A professor's first response to this question must take account of the teacher's objectives. One clinician might focus on ethics and community empowerment23, while another 60 Journal of Clinical Legal Education August 2006 19 See MacCrate Report, supra, note 3 at 260-268. 20 But, see f.n. 4 supra. 21 But, see f.n. 18 supra, regarding apprenticeship requirements in Delaware and Vermont. Ibid at 274- 284. 22 I do not mean to diminish the value of this form of written testing. I believe performance tests are an excellent format. I use them extensively, with and without an additional video component. See Grosberg, supra n. 8 at 2 Clinical L. Rev. 366-379. 23 Daniel S. Shah," Lawyering for Empowerment: Community Development and Social Change", 6 Clinical L. Rev. 217 (1999). targets interpersonal skills such as the effective use of empathy. What are the skills or knowledge to be assessed, therefore, is the threshold issue for any teacher. A second issue is whether the assessment is in the nature of a constructive critique24 or a "graded" evaluation that directly affects a law student's standing and progression in law school or entry into the profession. There has been a considerable amount of clinical scholarship that focuses on giving constructive feedback, but much less on graded evaluations. Among other things, the critique- feedback literature has produced the detailed criteria that clinicians invariably use in one-on-one supervision sessions.25 The critique literature also appropriately centers on self reflection and the life-long learning process that is so central to clinical teaching.26 A critical dimension of any graded evaluation of a skills performance is that it be fair. This raises the same kinds of basic testing issues - - the reliability and the validity of the testing measures - - that ought to be confronted by anyone who is administering a formative or summative assessment tool of any kind.27 Does the assessment tool accurately and consistently measure what it purports to measure? This holds true for an elementary school teacher giving an English exam to a fifth grade student as well as for a law professor grading a torts essay exam or a bar examiner testing an applicant for admission to the bar.28 Despite the many years that I have been evaluating law students' performance of interpersonal lawyering skills - - in live client clinics, upper class simulation courses and first year introductory lawyering courses - - I still have recurring bouts of concern as to whether I am being fair in applying commonly accepted assessment criteria. For example, in the case of counseling skills, can I apply the criterion that a client receive a "clear summary of options" in a manner that would justify giving one student a B+ and a second, a B? These doubts were fueled by a group of studies done by medical educators who examined the results of analogous clinical evaluations conducted by medical professors.29 What they found was that the medical professors were not consistent in their evaluations, as a result of which, the analysts concluded, those assessments were not fair. The same performance might be assessed differently by different medical professors. Or, the nature of the student-patient interactions observed were so different as to preclude consistent comparative grading of the students. Indeed, similar results were reached with respect to the inconsistency of law professor essay grading.30 One consequence of the medical findings was that medical educators began to resort to other techniques - - one might say unorthodox or previously untried How Should We Assess Interviewing and Counseling Skills? 61 24 See sources cited, supra note 6. 25 See MacCrate Report, supra note 3, at 138-148. 26 See Richard K. Neumann, Jr., "A Preliminary Inquiry into the Art of Critique", 40 Hastings L.J. 725 (1989); Amy L. Ziegler, "Developing a System of Evaluation in Clinical Legal Teaching", 42 J. Legal Educ. 575 (1992); Kenneth R. Krieling, "Clinical Education and Lawyer Competency: The Process of Learning to Learn from Experience Through Properly Structured Clinical Supervision", 40 Md. L. Rev. 284 (1981). 27 See Rachael Slaughter et al., "Bar Examinations: Performance or Multiple Choice?", The Bar Examiner, August 1994, 7. 28 This is not the place to examine generally the fairness or unfairness of all of our law school assessment tools. My focus here is only on the individual and systemic aspects of evaluating interviewing and counseling skills. See Friedland, supra note 8. 29 M.H. Swartz et al., "Global Rating of Videotaped Performance Versus Global Ratings of Actions Recorded on Checklist: A criterion for Performance Assessment with Standardized Patients", 74 Acad. Med. 1028 (1999). 30 See, e.g. Friedland, supra note 8, at 184 n.154. See also Greg Sergienko, "New modes of Assessment", 38 San Diego L. Rev. 463, 471-72 & n.37-38 (2001). (Discussing first a study that showed that when a law professor re-read an exam paper, there was only a seventy-five percent chance that the second reading would produce the same pass/fail result. A second study showed a bar exam question had only a sixty-seven percent chance of being graded consistently as to pass/fail by a second reader). methods - - of evaluating medical students' clinical performance.31 If the medical professors could not assess fairly the clinical interactions of their students, why do we think we are more able to do the same with our students? Traditionally, clinicians have graded clinic students' interviewing and counseling skills by observing them performing these tasks. This continues to be the most generally accepted method of assessment. The clinical professor presumably is the most qualified person to assess whether a student's overall interviewing and counseling skills performance meets the standards of competent lawyering and to ensure that their evaluations are measuring what they purport to measure.32 This is the so-called "gold standard" evaluation.33 Its validity is also reinforced when the students receive the applicable criteria with adequate advance notice.34 Also, typically, the same criteria would be the learning vehicle for constructive non-graded feedback and student self-reflection as well as the bases upon which an ultimate grade would be determined.35 As stated above, however, while teacher assessments are quite valid, they may not be reliable in terms of consistency. The professorial observations more often than not would be faculty reviews of videotaped simulations. Usually, in a typical clinic that I have taught, however, we would be lucky to have completed one videotaped simulated preparatory session prior to the student meeting with a real client or a witness. Thus, the notion of repetition as the most valuable aspect of preparation, practice and more practice, is difficult to achieve.36 Observing a student interviewing or counseling a real client might also take place, but the scheduling conflicts and unavoidable distractions and supervisory tensions often would interfere with the accuracy of these assessments or even prohibit these observations.37 3. Multiple Methods of Evaluation. The question raised here is: Is the teacher's direct observation of a student's performance of interpersonal skills the most effective way or, indeed, the only way to grade such skills? Most importantly, is it fair? Or, to use testing terminology, is it reliable, accurate and consistent from student to student? Would another clinician give the same grade? And is it the most valid way to assess the overall levels of interviewing and counseling competency, or for that matter, specific components of the skills such as question form or the use of empathy? Why shouldn't we consider other potentially supplementary or complementary ways to assess a student's interviewing and counseling skills? 62 Journal of Clinical Legal Education August 2006 31 One of these is the "standardized patient" (using actors first to portray patients and then second, to give evaluations of performances). See infra. 32 See Kristin Booth Glen, "Thinking out of the Bar Exam Box: A Proposal to "MacCrate" Entry to the Profession", 23 Pace L.Rev. 343, 428 n. 357 (2004). (To be valid, any assessment must be certain to accurately measure what it purports to measure.) 33 See Swartz, supra note 29. 34 Cf. Sophie M. Sparrow, "Describing the Ball: Improve Teaching by Using Rubrics-Explicit Grading Criteria", 2004 Mich. St. L.Rev. 1 (2004). 35 In my case, the interviewing and counseling grades would be part of the computation of a final grade. See Gerald F. Hess, "Heads and Hearts: The Teaching and Learning Environment in Law School", 52 J. Legal Educ. 75, at 107 (2002) (reporting on the importance of detailed criteria). 36 See David A. Binder & Paul Bergman, "Taking Lawyering Skills Training Seriously", 10 Clinical L. Rev. 191 (2003). 37 See Joshua D. Rosenberg, "Interpersonal Dynamics: Helping Lawyers Learn the Skills, and the Importance of Human Relationships in the Practice of Law", 58 U. Miami L. Rev. 1225, 1234 (2004) (noting that the absence of repetition and the stepped development of these skills necessarily affects the quality of both feedback and grading. There are at least two rationales that favor using more than one method of evaluating a student's performance of interviewing and counseling. Assume, for example, that the objective is achieving an overall novice level in initial client interviewing. First, using one method - - for example, observing a single live client-student interaction - - could have, depending on the circumstances, a distorting effect for any one of several reasons and, therefore, an unfair impact on any grade. It could be an interaction with a extremely difficult client. It could involve a much more complicated legal situation. There could be unreasonable time constraints or difficult physical conditions under which the meeting took place. The student might be feeling bad that day, etc. For any of these reasons a grade for interviewing based on this single interaction would be an inaccurate assessment of the student's overall competence level.38 Why not err on the side of caution by not relying too heavily on only one method of evaluation? How can it hurt to use multiple assessment methods? One can also give different weights to different methods. Still another reason for using more than one kind of testing device (e.g. multiple choice as well as essay questions) is that different students learn differently and, therefore, do better or worse depending on the type of assessment tools used.39 Using a variety of methods enables the students to demonstrate their talents in at least some of the tests. This also improves the "validity" of the overall grade, because it accounts for different ways in which to register the multiple competencies that are necessary for lawyering.40 In addition, there is an element of repetition that favors giving students an opportunity to improve their abilities to both perform the same skill and reflect their full understanding of the skill and to improve their mastery of the testing device. The latter, I would characterize - - not in the pejorative phrases "teaching to the test" or teaching "test-taking skills" - - but rather as the opportunity to become familiar and comfortable with a particular kind of exercise. Just as the typical law student learns through repetition how to handle the usual law school essay exam question, they are less prepared for some of these skills testing methods that they may encounter only once. For example, in the case of the standardized client exercises discussed below, the students conducting the second, third or fourth standardized client exercise know and understand the mechanics of administering the exercises better than they did for the first exercise. Their added comfort level often facilitates a higher level of performance. It likewise seems indisputable that a student's experience with repeated tests of an interactive skill such as interviewing or counseling will benefit greatly from such opportunities. The value of repetition was noted earlier.41 Another way to describe the repeated use of testing tools is that it is a way of giving stepped feedback to a student thereby reflecting that student's progress which is recognized as an "essential ingredient for How Should We Assess Interviewing and Counseling Skills? 63 38 Cf. Friedland, supra note 8, at 185-86 (referring to this similar distorting impact when the grade is based solely on a single end of term exam and even worse, if the single question exam deals with the 1 of 6 course topics the student didn't prepare for.) See also Glen, supra note 32, at 405 n.264; id. at 446. Likewise, with respect to evaluating the overall competency of an interviewing performance, one student might be superb in establishing rapport with a client but quite ineffective in the use of properly formed questions, whereas a second student would be the reverse, but in a way that totally undermines the efficacy of the interaction. Limiting the evaluation to a single event or to only one aspect of the skill may result in a misleading overall assessment. 39 See Ian Weinstein, "Testing Multiple Intelligences: Comparing Evaluation by Simulation and Written Exam", 8 Clinical L. Rev. 247 (2001). See also, John. M. Bauman, "Oral Examination as a Method of Evaluating Students", 51 J. Legal Educ 130, 136 (2001)(students performed well on oral exams while others did better on written exams.) 40 Friedland, supra note 8, at 196. 41 See Binder & Bergman, supra, note 36. This would seem to be true whether it is repetition of overall assessments of complete client interviews or repetition (in the sense of drills) of more narrow skills (e.g. active listening or use of T-funnel questioning). advancement".42 Professors Binder and Bergman have certainly persuasively made this "practice and more practice" point regarding interviewing, deposition and counseling skills.43 Finally, before reviewing specific new assessment tools, it is worth noting clinicians' dislike of the grading process generally. Brustin and Chavkin recognized this reality in constructing their study as to whether a clinic should be graded in the usual fashion (A, B, C, etc.) or with a Pass/Fail.44 While I very much believe in detailed feedback and constructive advice as to how students might improve their skills, I frequently find myself delaying the final calculation of grades until the last possible minute. In my experience, the notion of giving a clinic student a "test", therefore, has an even more unpleasant sound to most clinicians. Yet, most of us ultimately do in fact generally give a formal grade of some sort. If giving clinic students something that might be characterized as a test would make our grading better, fairer and more accurate, why not use these other devices. It is against this backdrop of more traditional methods of clinical assessment that I continue to experiment with different ways to evaluate students' interviewing and counseling skills. The following assessment tools are intended to complement or supplement, NOT replace, the traditional clinician's direct observation.45 [A] Videotaped Performance Test.46 This is a written exam in which a student views a videotape depiction of a lawyering activity and is asked to analyze all or part of what is on the tape. The ability of a student to articulate (either or both verbally or in writing) why a lawyer effectively counseled a client would suggest a cognitive understanding of skills theory and its requisite components. It not only would reflect a student's knowledge of skills models and theories, but an understanding of how someone succeeded or failed in applying those theories. Assuming one's teaching objectives included an understanding of skills theories, this is a valid way to assess that understanding. Being able to parse the deficiencies of a golf swing will not make you a top golfer.47 But, it certainly cannot hurt in the development of the reviewer's swing. In simulation classes, we have asked students to view a videotape of someone else conducting a client interview or a counseling session and to write a critical analysis using those same criteria that I use in evaluating them.48 We have used this method in large skills classes both with an in-class exam (open book and closed book) as well as in a take-home exam. I have also given students a narrative of a factual situation or a transcript of a lawyer-client interaction, not unlike a traditional law school exam, and asked them to write an essay as to how they might handle counseling a client in that situation, or 64 Journal of Clinical Legal Education August 2006 42 Id. at 204. 43 Id at 201; See also Rosenberg, supra note 37. 44 See Brustin & Chavkin, supra note 7, at 300-01. 45 I would be extremely reluctant to eliminate direct professorial observation of student performances even in the face of questionable reliability and accuracy data. Rather, in addition to implementing the theme of this paper (namely, using multiple methods of assessment) I would take steps to improve the observation method. 46 See Grosberg, supra, note 8 at 366-378. 47 Contra Rosenberg, supra, note 37 at 1234. 48 To make this assessment a more meaningful one, I give the students a case file that provides the context for the session. For example, for a counseling session, the file would include an initial client interview memo, as well as file memos and documents reflecting an investigation. It utilizes the same technique used on many bar exams (the "performance test") but adds the video component as part of the exam question. I have referred to these as "videotaped performance tests". Grosberg, supra, note 8. See Stephen P. Klein, "An Analysis of the Relationship Between Trial Practice Skills and Bar Examination Results," January 10, 1983 at 272 of Learning and Evaluation in Law Schools, supra n. 10. (In an analysis comparing the scores of a videotaped performance type exam and a traditional bar exam, the conclusion was that the former scores "were just as reliable" as the traditional essay questions Id at 293.) in the case of a transcript, analyze the efficacy of the lawyer-client encounter. The former would call for an outline of a counseling session; the latter a critique. These kinds of questions would more explicitly call for an effective application of the readings and videotaped lessons that I typically assign in any course involving interviewing and counseling.49 The desired teaching outcome is for the student to clearly reflect an understanding of the applicable theories and why the lawyer was effective or ineffective. In many ways such a written critique of another's performance is comparable to a student's self appraisal, especially if it is facilitated by a videotape of the performance.50 In the ideal Binder and Bergman world, the student would have several opportunities to repeat performances of a skill. In that context, it seems to me, the ability to do an effective critique of another's performance would be transferable and identifiable in the student's next performance of the skill. For that reason, the critical analysis skill can play an important role in the student's development of the performance skill. This is in addition to using this assessment method for the independent purpose of testing the student's understanding of and the ability to apply the skills theories. The term "performance test" (PT) has come to mean in the world of bar exams and legal education, an opportunity to draft a written document that a lawyer might produce; e.g. a deposition outline, a jury summation, an opinion letter, etc. Thus far, bar examiners have not yet extended it to include any oral lawyering performance or, indeed, to include a video component.51 The PT typically gives the test-taker a file with original fact documents and some law (cases, statutes, etc.) and asks the student to produce the requested document in a stated period of time based on that file. The PT is in stark contrast to the typical traditional law school exam which gives a one paragraph hypothetical and asks the student to write a judicial opinion resolving the dispute. In the "videotaped PT", the student is given similar fact and law files and then asked to provide an analysis of both doctrine and the interpersonal skill. For example, the student would then observe a lawyer counseling a client in the case reflected in the case files. In addition to analyzing the case and possibly producing a practice document of some sort (e.g. a negotiation outline), the student would also be asked to provide a critical analysis of the lawyer's counseling skills performance. In evaluating a student's response to a videotaped PT, the clinician can assess the students' legal reasoning and analysis skills as well as their understanding of what worked or didn't work in the interviewing or counseling performance and why. This depiction of a lawyer applying the law in How Should We Assess Interviewing and Counseling Skills? 65 49 See books cited supra note 2. These works offer a theoretical foundation for a full discussion in an examinee's response to any test device that asks for an analysis of some aspect of a lawyer-client interaction. 50 See Steven A. Lieberman, "Introduction of Patient Video Clips into Computer-Based Testing: Effects on Item Statistics & Reliability Estimates", 78 Academic Med. S48 (2003). Here again our medical colleagues have examined the use of such video exams and found them to be sufficiently reliable compared to text based questions. 51 The genesis of the performance test (PT) was an experiment in California in 1981 when the California Board of Bar Examiners established an experimental "Assessment Center" in which applicants for bar admission completed both oral simulation exercises, videotaped performance tests as well as traditional bar exam questions. The results of the experiment were positive, however, both oral simulation exercises and video taped components were rejected for administrative and cost reasons. To my knowledge, the recommendation or even a modest suggestion to implement these more comprehensive testing techniques was never to be heard again, at least not publicly. Those 1981 experiments did, however, lead to California's implementation of the three- hour PT and later to the development and implementation by the National Conference of Bar Examiners a ninety minute PT. See Stephen P. Klein & Roger B. Bolus, "An Analysis of the Relationship Between Clinical Skills and Bar Exam Results", July 1, 1982 at 164, Josephson, Learning and Evaluation in Law Schools, Volume 1, supra note 10. context offers the students the opportunity to analyze the lawyer's attempt to synthesize the law and the facts and then explain the law and the options available to the client. To make this assessment tool even more realistic, a professor might dispense with the typical in-class short time limits. For example, instead of giving students two or three hours to complete a videotaped PT, they might be given 2-3 days.52 As far as the ultimate grade in a live-client clinic, the video PT could play a role there as well.53 This, of course, assumes that learning and understanding skills models and theories is an objective of the clinician. The notion of a written final exam in a clinic, and often even in a simulation course, is contrary to most clinicians' pedagogical instincts. Based on my experience, the use of any kind of a written exam in a clinic remains rare. Once again, however, the starting point or premise for this survey of alternative methods of evaluating interviewing and counseling skills, is that what we're now doing might not be as fair - - or accurate for that matter - - as we might like to think it is. If that is true, we should be considering ways to move it closer to that ideal.54 The question is whether we remain open to improving our method(s) of evaluation. [B] Multiple Choice Questions. The use of multiple choice questions on law school exams or the bar exam has often elicited controversy.55 For some law teachers, this testing method constitutes a poor substitute for an exam that calls on the student to write an answer demonstrating the ability to use and apply the law to a factual situation. The essay question continues to be the dominant form of law school exams.56 I was among those who reacted skeptically (even negatively) to the suggested use of multiple choice questions both on the bar exam and for traditional doctrinal courses. I never even contemplated using them for assessing interpersonal skills proficiency. Yet some professors rely exclusively on multiple choice questions (evidence is one example57) and others are even now proposing their use for skills evaluations.58 Overcoming a certain amount of resistance, we first began to use this assessment method in our first-year Lawyering course. Initially, we used multiple choice questions as ungraded in-class quizzes and then later as part of a final exam. One set of questions was based on a transcript of a videotaped interview that the students had viewed. 66 Journal of Clinical Legal Education August 2006 52 See Glen, supra, note 32 at 368 (criticism of bar exam questions because of the disparate racial effect of exams with time limits). 53 See Larry Grosberg, "Evaluation of Oral Lawyering Skills Through a Video Exam", in Gerald F. Hess & Steven Friedland, Techniques for Teaching Law 308 (1999). 54 Sparrow, supra, note 34 at 28-9, noting a colleague's use of her suggested rubrics as a way to make grading more consistent. 55 Committee on Legal Education and Admission to the Bar, Association of the Bar of the City of New York, "Report on Admission to the Bar in New York in the Twenty-First Century - - A Blueprint for Reform", 47 Record 464, 482 (1992). (Criticism of bar exam because of too much use of multiple choice questions.) 56 See Greg Sergienko, supra n.30. 57 Interview of my colleague, Professor Eugene Cerruti, with respect to evidence. October, 2004. 58 My colleague, Professor Stephen Ellmann, encouraged me to use multiple choice questions for some limited teaching objectives regarding the assessment of interpersonal lawyering skills. Professor Sergienko discusses the use of multiple choice questions for "skills testing" in a recent article, but his focus is on "higher level cognitive skills" (Sergienko, supra n. 30 at 493-501) and not the kind of interpersonal verbal skills that are the focus of this paper. The students were asked to choose the best of four assessments of an exchange.59 Or, they might have to select the applicable concept or rule of law.60 Still another multiple choice question we used was to select the best assessment(s) of a lawyer-client interaction.61 There is much to be said for using such quizzes simply because of the speed with which the student receives the test results. The quick turn-around to students can be extremely useful in terms of their improvement of their performance.62 As with other methods to assess interviewing and counseling skills the threshold issue is does it assist us in achieving our teaching objectives, and, therefore, is it adding anything useful to our How Should We Assess Interviewing and Counseling Skills? 67 59 Q#1. At the beginning of the excerpt from the session between attorney Mary Hogart (AT) and Maureen Redhail (MR), this exchange takes place: MR: So, can you tell me what to do? I'm torn between my choices. AT: Well, I understand why you might be torn in this situation, you're facing a really tough decision. But, really it's your decision. Ummm, and as tough as it is, you're in a much better position than anybody, including me, to make the decision. Because you know what's important to you and also you're, you know, you're going to have to live with the consequence of your decision. So I'm- MR: Right. AT: I'm very happy to help in any way that I can. Is there anything that I can tell you that might help you sort this out? In light of the options that Maureen Redhail had (as explained in the prior dialogue), which of the following assessments of this exchange is the soundest? A. Since no one could reasonably have chosen to go to trial in this case, the lawyer should simply have said right that right away. B. In responding this way, the lawyer is showing no empathy for the client's understandable uncertainty and desire for help. C. The lawyer bluntly and properly refuses to answer this question since lawyers should never state their opinions about what to do in case. D. The lawyer rightly urges the client to reach her own decision, since it is her case and her life, but does not absolutely refuse to answer the question. [correct answer to Q#1: D] Unpublished quiz, Lawyering, Spring 2004 (on file with author) 60 Friedland, supra note 8, at 165. This would be relevant to the legal reasoning and analysis skills as opposed to oral lawyering skills. 61 Q#3. Based on prior dialogue between the attorney Mary Hogart (AT) and her client Maureen Redhail (MR) regarding the client's concern for the welfare of her two children, they go on:. AT: So, one of the things that might help you with this decision is trying to sort out, ummm, what is the most important priority. Knowing that the kids are really important to you- MR: Right. AT: Is the chance of getting a better financial settlement at the end of trial, ummm, and being in a slightly better position financially more important to you than the risk of having them deposed and having this continue? MR: Yeah. I see. Well, the settlement, what they're offering, is already better than what I've had this past year. AT: It is better than what you've had. It's absolutely better... Please consider the following assessments of this excerpt: 1. Because the lawyer's job is to help the client to clarify her own thinking, it makes good sense for the lawyer to ask the client to compare two different ways that she might help her children and to judge which is more important to her. 2. The lawyer potentially skews the comparison she is asking the client to make by recognizing the extent of the financial improvement that the trial might produce. 3. The lawyer's statement that the settlement is "absolutely better" than what Maureen has had for the past year is accurate but may be so strongly phrased as to encourage Maureen to lean towards settlement. 4. By putting the comparison in terms of the client's own values, the lawyer avoids taking on the role of telling the client what values she should hold. Which of these statements are well-founded A. 1 & 3 B. 2 & 4 C. 1, 2, 3, & 4 D. 1, 3, & 4 [correct answer to Q#3: C] Quiz, supra note 49. 62 See Sergienko, supra, note 30, at 493. repertoire of evaluation tools. I think the answer is a modest yes. While it does not evaluate a student's ability to perform, it does enable us to evaluate a student's understanding of skills theories and lawyering models. One prerequisite, of course, is that the questions are valid, reliable and efficiently administered.63 Do the questions measure what they purport to measure? In our case, did the questions measure a student's cognitive understanding of basic interviewing and counseling concepts? I believe the answer is yes. While intellectual understanding of applicable lawyering theories does not mean someone will be a good interviewer, it does mean something, and it could later aid in someone's actual ability to perform. More importantly, if a lawyer understands why certain interpersonal concepts are valid, that lawyer is more apt to ground her behavior in those principles than to just act instinctively or spontaneously in a very ad hoc manner. Finally, the use of multiple choice questions simply adds an alternative evaluation device. Its use means that a student's entire assessment is not unduly placed on one evaluation tool.64 It enables us to minimize the deficiencies of an one particular method of assessment. To the extent such questions are used in mid-semester quizzes, and then repeated in a course summative final exam, it also is an effective and efficient method both of providing ongoing or interim feedback (always useful from a learning perspective) and giving them notice of the testing method to be used on the final. From an instructor's perspective, it's also much easier and speedier to grade such questions than essay or other textual test responses, an especially important factor in large classes. [C] Self Reflection or Self Evaluation. This has certainly been one of the traditional ways in which clinicians have engaged students in the process of evaluation. Indeed, teaching the skill of self- critique is itself often a clinical teaching objective.65 To the extent that student development of life-long learning habits is a goal, self evaluation can be a valuable tool. A student's self- evaluation/critique often is done in the form of a written document, typically without any (or much) time pressure. But it also could be done more informally and only verbally. For me, in clinics, this is often part of a semester-end one-on-one meeting with students. The care with which a student analyzes her or his own performance will necessarily reflect the student's understanding of and application of interviewing and counseling theories to the specific situation that is the object of the critique.66 And as discussed above, the same would be true of an analysis of someone else's performance. Both provide additional components of what might be a student's final grade on her interviewing and counseling skills. [D] Standardized Clients. Still another method of evaluating student skills performance is the "standardized client". This is a method that is based on the medical education model - - the "standardized patient"- - in which a lay person is trained to portray a patient and then to give written feedback to the student interviewing and examining them. I have been experimenting with my adaptation of this tool to law school in several contexts.67 Essentially, we train actors first, to 68 Journal of Clinical Legal Education August 2006 63 Friedland, supra, note 7 at 157-163. 64 Id. at 182 (measuring skills has "perplexed" professors, especially those that typically administer essay exams). 65 See Nina W. Tarr, "The Skill of Evaluation as an Explicit Goal of Clinical Teaching", 21 Pac. L. J. 967 (1990) and Ziegler, supra, note 26. 66 "Helping students identify when they had difficulty acting consistently with their intentions enhances skills learning." Don Peters, "Mapping Modeling, and Critiquing: Facilitating Learning Negotiation, Mediation, Interviewing, and Counseling", 48 Fla. L.Rev. 875, 924(1996). 67 See, e.g. See e.g., Lawrence M. Grosberg, "Medical Education Again Provides a Model for Law Schools: The Standardized Patient Becomes the Standardized Client", 51 J. Legal Educ. 212 (2001); and Lawrence M. Grosberg, "Standardized Clients: A Possible Improvement for the Bar Exam", 20 Ga. St. U. L. Rev. 841 (2004). role-play clients or witnesses and second, to complete an assessment of the student's performance on a checklist68 that the professors prepare. While dispositive results are not yet in, the preliminary findings69 suggest that this is a valuable assessment tool which gives students multiple70 opportunities to perform a skill and to receive valid and reliable feedback as to how they are doing. In our first-year Lawyering course, these exercises facilitate achieving the objective of introducing students to the basics of interviewing and counseling and fact analysis. We do not purport to do more. At most, a student might reach a novice level as an interviewer. On a more mundane level, the SC exercises also offer students some feedback on how a stranger might perceive them. It also affords them the opportunity to achieve a higher comfort level in trying to engage a client or a witness in a conversation. For many students, it is the first time they have had such a human interaction with a stranger. As I suggested at the outset, a major concern is whether our overall assessments are fair, sufficiently objective, and, ultimately, accurate measures of a student's interviewing and counseling skills. One of the reasons that this device began to be used in medical education was that the professors concluded that their direct observations of their students' clinical skills were too variable and unpredictable.71 Observing a student performing a tonsillectomy is not the same as observing that student operating on a broken back. Similarly, a medical professor might be called away from an observation of a student for an emergency. The use of standardized patients was one way to ensure that all students got the full, as well as the same, array of cases to deal with.72 Very similar obstacles (and others as well) get in the way of law school clinicians in observing student performances on real cases. Thus, it could be a useful supplement even in our live-client clinics and simulation courses. The standardized client can also be valuable as a vehicle for introducing students to skills learning in situations where it is practically or financially impossible to provide professorial feedback on skills performances. That in fact is the context in which I have been conducting most of my experiments with this technique. Our first-year required course in Lawyering is a large class (we have four sections of 110 students each) which has as its purpose, the introduction to fact analysis and interviewing and counseling.73 One-on-one teacher feedback is not financially feasible. Providing each of them with three standardized client opportunities, How Should We Assess Interviewing and Counseling Skills? 69 68 For an example of such a checklist, see www.nyls.edu/grosberg. 69 The institutional research specialist at New York Law School, Dr. Joanne Ingham, compared the scores given by forty-two actors and four law professors on a random sample of 99 professor- actors pairings from three different SC exercises conducted during the actor training sessions. She found that the average agreement levels for SC exercises I, II and III were 64%, 70% and 53% respectively. An assessment of the data indicated that future modifications to the training program will produce an increase in the current agreement levels. In her opinion, these data are sufficient to support the reliability and the validity of low stakes grading of the SC exercises. See also Karen Barton, Clark D. Cunningham, Gregory Todd Jones & Paul Maharg, "Valuing What Clients Think: Standardized Clients and the Assessment of Communicative Competence", Clinical L. Rev. (2006) (forthcoming). (The preliminary results of their full-scale study of the use of standardized clients in Scotland as part of the Scottish requirements for admission to the Bar suggest strong correlations with assessments by the Scottish tutors, attorney-teachers already admitted to the Bar.) 70 Binder & Bergman, supra, note 36 at 201 (noting the significance for value of opportunities for repeat performance). 71 See also Sergienko, supra, note 30 at 471-472. 72 See Mark H. Swartz & Jerry Colliver, "Using Standardized Patients for Assessing Clinical Performance-An Overview", 63 Mount Sinai J. Med. 241 (1996); Grosberg, "Medical Model", supra note 67 at 215. 73 For a fuller description of the Lawyering course, see Grosberg, "Bar Exam Improvement", supra note 67. however, is a viable option.74 As a result, each of our students has the chance to conduct an initial interview of a potential client, an interview of a witness in another case and a counseling session with a client in still another matter.75 We also use a videotaped PT as part of the final exam in this course. In the simulation skills courses (e.g interviewing, counseling and negotiating) as well as the live- client clinics76, the use of standardized clients may be the only way to accomplish the kind of repetition of skills exercises which some feel is critical to meaningful skills learning.77 It is worth noting that repetition is integral to law students' learning of legal reasoning and analysis skills; they are developing those skills in all of their first year classes, as well as most of the upper class courses. The same should be true as to interpersonal skills. Again, the goal in the use of standardized client exercises is not to displace other evaluation methods, especially direct professorial feedback, but to supplement those clinical assessment tools with additional and repeated measures of competence. [E] Computerized Assessment Tools. The videotaped performance test and the use of multiple choice questions are assessment devices that may now be used via the computer.78 The technology (either on line of via CD's of DVD's) is available for use either as a formative or summative assessment tool or as a self-learning device for students to take home and use as desired. In the latter case, it raises basic questions about how to lay the groundwork for effective student use of such a self-learning vehicle. In its more extreme form, the "distance learning" concept is similarly raising these questions.79 To what extent can students develop the necessary lawyering skills (legal reasoning and analysis and writing, as well as skills such as interviewing and counseling) simply by sitting next to a computer? Has the law school Socratic discussion become an anachronism? This is not the place to address these fundamental issues about how people learn and the extent to which it is reasonable to rely on self-learning techniques. But, it is clear to me, that just as the video PT assessments and multiple choice test questions can have some benefit in students' learning, the computerized versions of those methods might do the same.80 They cannot be a replacement, however, of interpersonal live interactions. Professor Maharg has taken things a step further by 70 Journal of Clinical Legal Education August 2006 74 The cost of standardized clients is not insignificant. To conduct nearly 1500 standardized client exercise entails approximately $45,000 out-of-pocket costs. But consider how much it would cost if professors gave feedback on that many lawyer-client interactions. Also, we hope to establish a center that could be used by more than one law school, thereby reducing the costs via the economy of scales. This is what medical schools often do as well. 75 Last year we received faculty approval to giving grades on the standardized client exercises; 10 % of the students' grades in this two credit course is based on their SC evaluations. This would be considered a "low stakes" evaluation, as contrasted with the "standardized patient" exercises that are now, as of June, 2004, an integral part of the medical licensing exam - - a "high stakes" evaluation. The results from this past semester suggest that a small but vociferous minority had strong objections to being assessed by a non-lawyer. Similar complaints, however, are made to medical educators' use of standardized patients, even after thirty years of use and full acceptance by the medical schools and the medical licensing authorities. 76 Some clinics may not include skills training as a teaching goal. 77 See, e.g., Binder & Bergman, supra, note 36 at 201. 78 Professor Paul Maharg at the Glasgow Graduate School of Law has been doing innovative work with regard to interactive computer programs. I have also produced an interactive CD for use in a first- year property class. The CD contains alternative video depictions of a lawyer-client counseling session, the contents of the case file, the relevant law and a series of multiple choice questions calling for analysis of the various aspects of the case. It is available by contacting the author at lgrosberg@nyls.edu. . 79 See e.g. Phil Agre, "The Distances of Education", 85 Academe 37 ((1999)(discussion of distance learning in education). 80 The Grosberg CD is essentially a videotaped performance test via a computer. Again, I believe these computer modes are not a substitute for traditional skills feedback, but simply supplements or complements. developing a virtual world in which students can practice law and have their virtual world law work evaluated.81 Practicing law in the virtual world gives Maharg students the opportunity to be creative and do things not possible in a clinic, or even in a traditional simulation course and to do so without the concern that they will make a mistake with a real client. For example, they can exercise initiatives to bring a lawsuit or alternatively, to propose an ADR device to resolve a dispute without litigation, just to name two. The opportunities in the virtual world provide a potentially richer base upon which to assess the students' strategic as well as interpersonal lawyering skills. The computer records provide the basis upon which the assessments of student work are based. While he has not yet developed avatars who could be interviewed and counseled by the students in an interactive and interpersonal sense, he has taken simulation several steps further than anything I have seen in the U.S. There is certainly room for some American pedagogical initiative here. Conclusion Evaluating interviewing and counseling skills calls on us to apply different measurement tools from those used to assess students' grasp of doctrinal law. We should approach this responsibility as clinicians in the same open-minded manner we would like to think we approach all educational issues. Numerous ways to evaluate skills have been developed and we should use or at least consider the potential use of all of them, having in mind the ultimate refinement of the most appropriate mix in each situation. While clinicians have made much headway in developing sophisticated critique and feedback tools, there has been less effort expended on the more formal grading evaluation. To the extent we make progress in developing law school measuring tools, we should try to carry over developments to the bar exam. How Should We Assess Interviewing and Counseling Skills? 71 81 Paul Maharg, "Legal Sims: From Everquest to Ardcalloch (and back again)" 2004; and Patricia McKellar and Paul Maharg, "Virtual Learning Environments in Action" , 2002 and Patricia McKellar and Paul Maharg, "Presence, Emergence and Knowledge Objects: User Interaction in A Virtual Learning Environment", all available at http//www.ggls.strath.ac.uk/Itdu/research/default.ht m (description of Professor Maharg's virtual world of the city of Ardcalloch in Scotland in which his law students conduct law practices). 72 Journal of Clinical Legal Education August 2006