AVOIDING A BOGEY; GRADING CASE DISCUSSIONS SCIENTIFICALLY Developments In Business Simulations & Experiential Learning, Volume 24, 1997 AVOIDING A BOGEY; GRADING CASE DISCUSSIONS SCIENTIFICALLY Ralph Roberts, The University of West Florida Brian Peach, The University of West Florida ABSTRACT Cases have been established as important pedagogical tools. The level and quality of students’ case discussion is a major objective of the case method, and discussion also provides, frequently, a major measure of student learning. However, little research has been done on appropriate measures of student participation despite the fact that some instructors are uneasy in their evaluation of student-case discussion. This paper proposes a classroom proven approach to methodically capturing data about on-going student participation, and then using the data to enhance further student discussion. INTRODUCTION The case method was adopted by the Harvard School of Business Administration in 1910; since that time it has been adopted in a wide variety of disciplines. From its Harvard beginnings, the case method is a primary pedagogical tool in strategic management/policy courses usually presented as capstone courses for business schools (Alexander, O’Neill, Snyder and Townsend, 1986). In a recent survey of policy courses, Jennings (1996) concluded that case study is a central element in the overall course pedagogy. He found that the three most important instructional goals in policy courses, in order of importance, were the following: (1) developing strategic analysis & strategic thinking, (2) illustrating real world situations, and (3) integrating the functional areas of business. The effectiveness of a case study was assessed in two ways: (1) the instructor’s evaluation of success in achieving student participation, and (2) the instructor’s ratings for achievement of objectives. Student participation is clearly considered a critical measure of case method success, so it is interesting to note that there was no discussion of methodology to assess student participation. In the survey instructors were asked to rate the level of student participation on a five-point scale - from excellent to poor. Seventy-five percent of the respondents rated student participation as excellent or very good. One can only wonder at the basis for such a positive assessment and wonder further if students’ perceptions are as positive. The authors’ experience is a little more problematic, and may better accord with other case instructors’ experiences. AVOIDING THE BOGEY A bogey may be an evil spirit, or it may be something that haunts and annoys. And who of us evaluating case discussions, as part of students’ grades has not been at least a little haunted and annoyed at our task? We are haunted that we may not be giving accurate evaluations, and we are annoyed when good students ask us to explain to them why they received a score less than they expected or than believe they deserve. We are haunted because we average perhaps one hundred students per semester and use perhaps ten cases in each of our classes. As each student may potentially speak several times on any case, this means we have over one thousand evaluation opportunities to some how, fairly we hope, capture student participation and record it “in the grade book”. We use a number of approaches to attempt the capture of this data. A seating chart may be imposed to enable an effort to make some sort of check marks to indicate who talked and how many times, or an even more 289 Developments In Business Simulations & Experiential Learning, Volume 24, 1997 elaborate code may be developed to indicate not only who commented but also the quality of the comments. But that scheme can be disruptive to our train of thought as we lead the case discussion. Another approach is to memorize the students’ names, and periodically use our class rolls or photos of the students to stimulate our memory and record some subjective assessment of: relative student contribution But if we get busy or distracted, or as standard procedure we may find ourselves at the end of the semester, even during the final exam, trying to identify names and faces and then challenge our memory to determine how much each student “contributed” over the last four months. Or use the student photos so we can challenge our memory in the privacy of our offices. Having worked so hard at being accurately subjective, we are annoyed when students ask us to explain and, yes, defend our grading methodology after we assign course grades. Both good and poor students will likely tell us they talked a lot more than the credit we have assigned. We are left claiming we did it correctly, but both student and professor know there is little real evidence we can offer. The well-known fact that 70% of us think we are above average applies here. Students usually believe, barring intervening feedback to the contrary, that they are doing better than they really are, and they are shocked when they receive average or below average scores for which’ we offer little specific information. They assert that they would have tried harder had they known, and we have to agree that they did not know, because we did not know, until that fateful moment in our office when we attempted to power our memory to a prodigious feat: evaluation of over one thousand student/case discussions. We feel another real annoyance, because we know the students are somewhat correct, when they say, “I had no chance to talk -- the big mouths always monopolized the discussions.” Others say, “I am not comfortable volunteering to talk in class - this grading scheme is unfair to me on its face.” And our only leverage for the non-talkers is to attempt to call on those who are reticent. But if we call on students, then those waving their hands eager to speak are annoyed because they want to maximize their grades; since they do not know where they stand, they think they are being denied an “A” grade, at least for case discussion points. We are also annoyed as we view the range of points we have assigned for case discussions. Since we have no accurate information, and since withholding all participation points causes the course grade to drop one grade or more, we find ourselves greatly compressing this part of our grading criteria. Thus the lowest person, whom we do not remember speaking, may get half credit while the top speakers get full credit. So our grading scheme does not range from zero to one hundred percent for case discussion, it ranges from fifty percent as a base gimmee up to another fifty percent for discussion; is that fair to the students who really do speak; is it just for those who never participate? A CLASS PARTICIPATION METHODOLOGY So how are evaluations of over one thousand student/case discussion interactions made more scientific, systematic and accurate? An evaluation system that gives timely as well as accurate feedback to students - in other words, timely for them to change their level of participation? That allows a full range from zero to one hundred percent, to accurately assign correct participation credit? That provides real documentation to explain grades to students -- documentation that they all accept as fair? Also that allows calling on students who are truly non-speakers? And to control, in a way they agree is fair, the dominant over-talkers? The evaluation method is out there to accomplish all this -- and it is easy. The writers have developed, and used for years, a system, which uses comment cards. It is very simple -- and too good to be true. Each student is given a copious number of 3x5 290 Developments In Business Simulations & Experiential Learning, Volume 24, 1997 cards. Students are requested to jot down, briefly, comments they make about a particular case, during each class. The comment cards are collected at the end of each class. Occasionally, if a case is continued to the next meeting, the students will be advised to hold the card for the next meeting. Working immediately after class, or at least the same day, the instructor reviews the comment card, and with name associated to face, can easily recollect the quality and quantity of each student’s participation for that case, that day. The instructor then assigns a point score based on the breadth and depth of the student’s comments. Some discussion must be made in class about what is/are full credit or partial credit comments, but that also gives the students accurate, before the end-of-semester, evaluation information about a course expectation and grading criterion. An additional supportive technique is employed with the comment cards which the authors find very helpful, but which may not be essential, to the scientific evaluation of case discussion. Pictures are taken of student’s in-groups of about four immediately at the beginning of the semester (this can be student teams if teams are used). The next class period names are appended to the pictures. These pictures are helpful (essential in the authors’ view) in learning names and they serve a very important function in relationship to the comment cards. The pictures allow the association of name to face from the very first case. As the instructor learns the students’ names, reference to the pictures becomes unnecessary. Points are written on each 3x5 card and entered into the grade book under that particular case, and returned to students promptly the next class meeting. These writers assign a score of one to five for each comment card/student discussion. In a class of 30 students there may be 20 cards turned in for each case and the grading time is ten to fifteen minutes. It is essential that the number of comment card points necessary to earn the fill course points for case discussion be established the first class day, if not in the syllabus itself This likely will require some experimentation for each instructor -- to determine the comment card raw points necessary for earning the full course points. This will vary by number of cases in the course, grading of cards by the instructor, typical number of students in class, and amount of discussion which the instructor allows on each case. Cases that are only reviewed briefly and kept tightly to specific and limited teaching points will require different grading approaches than cases that are broadly discussed for a full class period to generate any and all teaching points. The writers have assigned fifteen percent of the course grade for case discussion in our Business Policy courses. We use approximately twelve cases in the course and we allow a full ranging discussion. With comment cards providing raw points, we find that a ratio of two raw points to one case discussion/grading point works about right. In other words a student who generates thirty comment card points earns the full fifteen- percent of the course grade. The comment card points are totaled after about one-fourth of the cases, and then frequently thereafter; and after each of the last three or so cases. Students’ totals are written on the cards along with the score for that particular card/discussion. Students also keep a running total; most are aware enough to do so without being told! After the one-fourth mark, with the first total, the instructors go to class with a short list, say five students, of those who have zero or very low (compared to peers) comment points. These students are called on during the case discussion, so they can earn points as well as the dominant students who are restrained by this technique. Calling on some students provides a priming effect; 291 Developments In Business Simulations & Experiential Learning, Volume 24, 1997 that is they realize that they can talk in class and not die so they begin to do so without having to be called upon for each discussion point. Students who achieve the full thirty points can be encouraged to reduce, if not cease, their class discussions, if their continued class preparation and attendance can be ensured through other methods such as pop quizzes and handing in of case preparation questions. Students have been fully accepting of these class procedures which relate cumulative discussion points to speaking opportunities. Those who normally speak little, or who admit to never having spoken in any class, are very appreciative of being called on and encouraged to speak. They are also very appreciative that the instructor has a mechanism to shut-up the loud mouths, as they put it. The dominant participants, on the other hand, recognize that they have, or are approaching, the maximum credit for case discussion and are thus more willing to minimize their remarks. Bright, extroverted students will always want to participate, but the instructor can save them for times when there are no other students willing to talk or able to move the discussion forward. Our view is that it is so difficult for them to completely be silent that the better course is to control their comments, rather than to turn off the freely flowing spigot completely. A dribble seems to satisfy everyone -- talkers and non-talkers alike. In the writers’ experience up to seventy percent of the class members may achieve the maximum discussion points (fifteen percent of course grade), while the less voluble students earn about half the discussion points. This is a loss of approximately seven- percent of the course grade, which seems reasonable to us. True non-participants can receive lower scores, but it comes as no surprise to them. A number of clear benefits derive from the consistent use of comment cards. The instructor has a fully defensible, up front, clear, understandable process of arriving at potentially more than one thousand case/discussion evaluations. There is frequent, almost continuous, feedback to class members about how they are doing in this element of the course. The instructor can adjust the class discussion to encourage and enhance less voluble students and control highly voluble ones. Both groups are either completely accepting and/or highly pleased with the process. Students have instant and complete feedback about the here-to-fore nebulous and arcane assignment of case discussion points, so they can either pick up the pace of discussion or slack off depending on their circumstances. There are no end-of-semester confrontations with students about their discussion grades which they knew all along, and which they knew with finality, with the return of the last comment card. The comment card process requires very little time of the instructor, perhaps three to four hours per course, per semester, and it provides documented and timely information for the grading process. Probably less time than trying to guess (yes guess) back for four months and for over one thousand interactions, in the attempt to accurately and thus fairly evaluate our sacred trusts -- the students. CONCLUSION Comment cards are a new, easy, exact, and fully acceptable approach to the difficult task of evaluating case discussion (or any class discussion) in our courses. There are no negatives, no drawbacks, no disincentives. How often can we say that about an element or process in our crucial but challenging task of educating and evaluating our students? REFERENCES Alexander, A.L., O’Neill, H.M., Snyder, N.H., and Townsend, J.B. (1986) How Academy Members Teach The Business Policy/ Strategic Management Case Course. Journal of Management Case Studies, 2(3), 333-344. 292 Developments In Business Simulations & Experiential Learning, Volume 24, 1997 Jennings, D. (1996) Strategic Management and the Case Method: Survey and Evaluation. In A.L. Patz & J.K. Butler, (Eds.) Developments In Business Simulation and Experiential Exercises, Association for Business Simulation and Experiential Learning, 105-110. 293 Table of Contents Volume 24, 1997 Incorporating Computer Telephony into the MIS Course A Learner Oriented Infrastructure for videoconferencing Based Distance Education Courses The Identification of Temporally Related Structural Elements of the Experiential Component of Electronic Spreadsheet Tasks Does Involvement Influence Learning from Simulation Participation? Some Relationships with Helpfulness and Performance Outcomes Musings on Business Garne Performance Evaluation Performance on a TE Simulation: What does it represent? The Business Policy Game: An International Simulation - An Assessment Tool Students in Free Enterprise as Experiential Learning The Role of Computer Models in Wargames: A Practitioner's view Using Focus Groups as a Tool to Research Learning: A Demonstration and a Discussion Using Medical Simulations to Teach Multi-Cultural Diversity Using the World Game to Internationalize International Exchange Game Community Services Needs Assessment: An Innovative Approach Introducing Students to Potential Total Quality Management Ethical Dilemmas Privacy in the Workplace: A Situational Analysis Antecedents of Learning in Simulations Demonstrating the Learning Effectiveness of Simulation: Where we are and Where we need to go The Use of Computer Simulations as a Pedagogical Aid in Teaching Management Information Systems Evaluating Simulation Learning in a Distant Learning Instructional Model Financial Engineering of Global Investments The Labor History Game: Playing with the Past Perks Participation Assessing Negotiator's Proficiency with a Negotiation role-play The Art of Negotiating Enhancing Learning and Employee Development through the Assessment of Learning Pedagogy Preferences across selected Dimensions of Culture: A Preliminary Investigation Cooperative Learning: What are we Learning? Effective Use of Mastery Based Experiential learning in a Project Course to improve skills in system analysis and Design Simulations and Learning: Can we prove a relationship? (Seminar) Predicting and Reviewing NYSE Stock Prices by use of Basic Statistical Analysis and Logic Toy Car Depreciation Exercise ABSEL as Home Community: An Interactive Exploration Service Learning: Linking Academic Study to Community Development and Business Enhancement Ability of Efficient Evaluation of Knowledge-Based Management Strategies Corporate Ethics Training Programs Modeling Attributes in Demand Functions of Computerized Business Simulations: An Extension of Teach's Gravity Flow Algorithm Expert Systems Combined with Neutral Networks: Tools to Benefit the Marketing Researcher Computer Game Design: New Directions for Intercultural Simulation Game Designers Consistency in Simulation Performance over Time and Across Simulation Games The Impact of an Artificial Market Leader on Simulation Competitor's Strategies Business Plans, Case Studies, and Total Enterprise Simulations: A Natural Co-existence An Exploration into the Non-Use of Business Simulations The Market Game: Interactive Learning Through Market Simulation Plotting Brand Trajectories with the COMPLETE PPM package: A Market Segmentation Analysis and Positioning Tool Me and Mine Inc: An Exercise in Management Theory Learning to Differentiate Leadership from Managerial Position Business Ethics Survey: A Perspective from the Retail Industry Experiential Learning in Demand Analysis for an Agricultural Commodity Marketing on the Internet: A Pedagogical Exercise College Students Need Simple Computer Simulations, Especially for International Business Courses An Analysis of Student Attitudes, Performance, and Strategies in a Simulation Competition Based on a Controlled Product-Market-Entry Game Structure The Crystal Enterprise: Application of a Non-Computerized Simulation Model in a Process of Organizational Change Threshold: A Windows-Based Behaviorally Oriented Total Enterprise Simulation Coaching Business Game Teams Using a Decision Variable Optimizer Rock and Roll is here to Stay: Enhancing Experiential Pedagogy with Musical Exercises An Example of Business Process Analysis Simulation for Customer Software Support The Use of Business Gaming in Hong Kong Academic Institutions Using the Integrative International Simulation INTOPIA Mark 2000 in a Concentrated MBA Curriculum A Group Experience as an Integrated Part of the Core Management Course Empowered Learning in the Classroom The Inter-Group Interaction: An Innovative Approach to Cooperative Learning Courses that Utilize Student Team: An Approach to enhancing their Effectiveness Thoughts about the Measurement of Learning: The Case for Guided Learning and Associated Measurement Issues Measuring Student Learning Using Business Simulations: A Theory Based Perspective On the Use of PC Fingame in an Undergraduate Finance Course How Managers get Things Done: A Virtual Soundbite Internet Experiential learning in the Principals of Marketing Classroom: A Pedagogical Approach Current Student Perceptions Relative to Business Simulations Beyond Capitalism: Designing Business Policy and Social Justice The Use of Boards of Directors to Evaluate Reports and Presentations in an Undergraduate Business Policy Course Beacon Lumber: An Experiential Introduction to Financial Accounting The Application of Organizational Motivation Principles: The Experiential Business Simulation Motus Manufacturing Avoiding a Bogey: Grading Case Discussions Scientifically Communicating Consumer Behavior II: A Modified Exercise Using Personal Consumption Journals in Condensed Courses Designing Instruments for Assessing the Effectiveness of Simulations, (Seminar) 360º Performance Feedback: Appraisal vs. Assessment The Cafeteria Approach to Managing an Academic Career: Remaining Non-Perishable while Doing Your Own Thing An Experiential Exercise Related to Person-Organization Fit and it's Consequences For Today's Dynamic and Changing World, Business Simulations need to be expanded to encompass Much Greater Complexity: A Demonstration The Incident Process: A Case in Reverse Contextually-Anchored Business Simulations The Energy Factor: Building Motivation in the Simulation Gaming Environment The Design of an Internet Game