FROM CASE PRESENTATION TO CASE FACILITATION: HOW ASSESSMENT CHANGED THE CAPSTONE COURSE Developments in Business Simulation and Experiential Learning, Volume 34, 2007 85 FROM CASE PRESENTATION TO CASE FACILITATION: HOW ASSESSMENT CHANGED THE CAPSTONE COURSE Peter M. Markulis SUNY Geneseo Markulis@geneseo.edu ABSTRACT Assessment has become an integral ingredient for business schools and most accredited business schools now have an assessment program in place. Since an emphasis on assessment is a relatively new phenomenon, many schools lack the knowledge and experience to use assessment to change and enhance student learning. This paper describes how an assessment program put into place two years earlier produced curriculum changes which the program sponsors believed enhance some of the most important learning goals for a business administration program. The paper outlines the process and substantive changes that took place due to the assessment program. Inherent in the paper are suggestions for schools and faculty interested in using assessment protocols to enrich student learning. INTRODUCTION Business schools are now quite familiar with the AACSB mantra: Assessment! The AACSB believes, as do many others in the education, that assessment leads to--or at least promotes--“better” learning on the part of students (Martell & Calderon, 2005, 2). Assessment is defined as “an ongoing process aimed at understanding and improving student learning,” (Angelo, 1995, 10). Since 2002, the AACSB has sponsored numerous assessment seminars, as do other accrediting organizations, so that schools can learn how to integrate meaningful programs into their schools. At this point, most business schools have completed the initial steps and some type of assessment program is in place (Martell & Calderon, 2005, 21). What schools of business are now wrestling with is how to alter or augment their present curricula in order to take advantage of assessment results, referred to in the assessment literature as “closing the loop” (Martell & Calderon, 2005, 8; Redle & Calderon, 2005, 227). This paper describes how a set of assessment protocols developed for a capstone course (the strategic management course, in this case) and how these protocols were used to change and –it is believed--enhance the learning objectives for the course. Business schools which are developing or fine-tuning assessment protocols may benefit from the experience described in this paper, both in terms of making substantive curricula changes as well as learning how to use the process of assessment to enhance student learning. COURSE BACKGROUND AND DESCRIPTION Like most schools, the Business School at SUNY Geneseo had developed its mission, and vision statements, its educational philosophy and set of general educational learning goals in preparation for initial accreditation (see Appendix A for TIMELINE & PICTORIAL SCHEMA). Subsequently, it became the province of the assessment committee to devise a set of assessment measures for each of the school’s learning goals and present these to the entire business school faculty for review and approval. Most learning goals had multi-measures. Some focused on knowledge areas, while others dealt with skill-building or behavioral issues. Some were value-based; others were performance-based (Muraki, Hombo, & Lee, 2000; Martell & Caledron, 2005, 8), and some were formative while others were summative—many were course-embedded (Gerretson & Golson, 2005). An obvious choice for much of the assessment work was the capstone course, where a variety of assessment protocols could be used, especially summative measures. A summative assessment protocol is one which is used to assess a student’s skill or knowledge at the end of their career or at the completion of a prescribed curriculum. Most business schools offer a capstone course to students; often it is the strategic management (SM) course that serves this function. The typical SM course contains several instructional modalities: • Case study discussions, • One or more computerized simulation experiences, • Written case analysis, • Oral case presentations, • CEO interviews, • Various types of tests. The SM course frequently employs case studies and simulations so that students get a holistic and comprehensive picture of real-world business situations. One instructional purpose is to have students serve in the capacity of surrogate decision makers using the knowledge and skills they gained from pervious courses (as well as the capstone course) to resolve complex organizational situations. As Wood and Anderson note, “Case teaching can help students learn to grow and become proactive in a dynamic environment,” (2001, 1). Often, the culmination Developments in Business Simulation and Experiential Learning, Volume 34, 2007 86 for the course is an oral case presentation by a student, or more often, a student team. Student teams usually make an oral case presentation near the end of the semester displaying the team’s abilities and skills at case analysis. This paper describes the learning goals for the course, how course specific learning objectives were developed for those learning goals and assessed, and how changes were made based on the assessment protocols. COURSE LEARNING GOALS. The learning goals for the SM (capstone) course at the Business School are: In the Strategic Management course, students will learn about the process of strategy formulation, strategy implementation and strategy evaluation. Upon completion of the course, students should know that: • strategic management is an emerging and rapidly developing discipline, and students should be knowledgeable about recent developments in the field; • students should be able to evaluate a firm from the perspective of top management; • students should be able to identify and evaluate an organization’s goals and objectives as well as weigh the pros and cons of alternative organizational strategies; • students should be able to identify the elements of the strategic management process in actual organization’s and evaluate their effectiveness; • students should be able to understand the elements of a business plan and be able to critique actual business plans; • in making judgments about an organization's long term survival and success, students will incorporate an ethical dimension as part of their analysis; These course learning goals are a synthesis of goals formulated by the different instructors of the course and are based on the Business School’s core learning goals (see Appendix B). It is likely that similar learning goals would be found in the SM courses of many business schools across the country. Each individual instructor, however, had the responsibility for determining how these learning goals were to be taught in his/her respective course or courses. That is, each instructor had to devise a set of course specific learning objectives (CSLOs) for his/her own course which served to operationalize the course learning goals. Readers may want to refer to Appendix A which chronicles and illustrates the various relationships and flows among the levels of learning goals. Because there was one principal instructor for the SM course at the Business School who taught most of the sections, this paper will focus on the CSLOs and assessment protocols for that section. While the Business School did not monitor each instructor’s individual learning objectives, each instructor of the course was required to assess at least one learning goal per semester and submit the results (as well as any remedial actions) to the assessment coordinator. The Dean appointed an assessment coordinator and he attended some assessment workshops sponsored by the AACSB and the AIHE. COURSE SPECIFIC LEARNING OBJECTIVES. The course learning goals as described above are fairly general and need further elaboration and specification if they are to be measured in a meaningful way. As noted, instructors were required to subscribe to the general learning goals for the SM course but were free to develop their own ways to both assess and evaluate student progress in meeting the learning goals. This was done through the use of course specific learning objectives (CSLO’s). Four CSLO’s for the SM course are listed below. 1. students should be able to demonstrate the use of critical reasoning through a case analysis; 2. students should be able to apply comprehensive business knowledge and techniques to a complex business situation; 3. students should be able to intelligently convey points 1 & 2 above to an informed audience (an “informed” audience refers to business professionals, business faculty and/or senior-level business students); 4. students should be able to engage in constructive discourse regarding points 1 & 2 above with an informed audience. Ostensibly, these CSLOs should flow naturally from the more general course learning goals. These four CSLO’s were believed to meet both the course learning goals, as well as address Bloom’s higher order cognitive learning dimensions, such as application, analysis, and synthesis (Bloom, et al., 1956). ASSESSMENT. As the CSLOs were developed, thought was given as to how these CSLOs were to be assessed (and eventually, graded). Several informal meetings took place among the three instructors, as well as with selected members of the schools business advisory council (BAC). BAC members are business professionals who participate in various business school activities. As noted by Walvoord and Anderson, the CSLOs need some kind of specification or rubric to measure achievement for the CSLOs (1998). It was decided that one of the principal means for assessing the four CSLOs was through the use of an oral case presentation. For the 2004-05 academic year a rubric was developed and used by all participating instructors (See Appendix C). Instructors invited one to two BAC members to attend the sessions and use the rubric to help assess the students’ progress. These BAC members were ones who were particularly interested in teaching-related activities. Students in the class were also asked to use the rubric to assess oral case presentations. This type of assessment was summative rather than formative. Since one of the purposes was assessment, the principal instructor tallied certain items on the rubric for assessment purposes. Also, some interviews were conducted with BAC members and selected students after the first year’s experience in order to get their views as to how the process was working. As will be described later, this is a crucial step and should not be left out of the assessment program. Developments in Business Simulation and Experiential Learning, Volume 34, 2007 87 As many business instructors probably can attest, oral presentations often mark the “high-water point” for the course (program). The oral presentation provides a venue for student teams to perform—to be on stage--“to show their stuff”--as it were. While most instructors might agree that the oral presentation is quite important to the student team and the instructor (and/or other reviewers), the remaining students in the audience seem to feel that it is their time to passively observe. For our students we felt that this passivity seemed to result in a semi-comatose state. This observation was one of the byproducts of the assessment program RESULTS. It was decided to “test” the assessment system during the 2004-05 academic year. For both the Fall 2004 and Spring 2005 semesters, the new assessment protocols were implemented in the SM course. Of particular interest for purposes of this paper were the oral presentation assessment results. An important result was that several assessors rated the teams as “unsatisfactory” on a number of items for the oral case presentations (See Appendix D1). These items indicate that the students were not meeting two important CSLOs, particularly #3 & #4. In order to help ascertain why this was so, the instructor met with two student focus groups and 4 BAC members to discuss these issues. While the BAC members said that the student teams—while making very professional, carefully planned presentations—did not interact much with the audience. One member pointed out that the presentation structure seemed “stilted, and confining.” Another said, “Very polished, but unrelated and distant.” An especially noteworthy point was that several students as well as BAC members, pointed out that the presenters did not “engage” the rest of the class while the students complained that they did not really know how to do this. This led one BAC member to say, “Well, if the team really knew the case well, they could engage the students.” (CSLO #4—constructive discourse). Back and forth discussions lead one student to raise this issue (and I will paraphrase): The instructor discussed 4 or 5 cases himself before any student team did their presentation. The instructor seemed to get the class involved without making a formal presentation, but by asking a lot of interesting questions about the case or by directing students toward certain issues in the case. Yet, when we do our presentation, they seem to be completely different from the way he did his. Since the author was the instructor referred to, he simply pointed out his goal in using cases was to “facilitate” the case—not to “present” the case. The activity of learning for a case facilitation is multi-way or “symbiotic” as opposed to one-way for the traditional case presentation, (see Figure1). This means that both facilitators and presenters have fully researched, analyzed and have developed viable recommendations for the case, but the facilitators have an important additional role. Facilitators must lead the participants through a discovery process in which the students help determine the appropriate modes of analyses, bring forth a set of reasonable alternatives, derive a set of sound recommendations and most importantly, discover important lessons and insights—lesson and insight which come only through a case discussion, not through a case presentation. Facilitation means that there are learning goals for each case and that the facilitating team needs to discover what those goals are, as well as what types of analyses should be undertaken and what additional research might be necessary to fully understand the organization and industry. The FIGURE 1: ONE-WAY VERSUS SYMBIOTIC LEANRING Developments in Business Simulation and Experiential Learning, Volume 34, 2007 88 facilitating team must determine the case issues, an appropriate approach--or approaches—to analyzing the case, and delineate a set of feasible alternatives as well as realistic recommendations for the case. Not all of the discussions went smoothly, but after discussing my approach and why I used it, several of us began to hit upon the idea of having the teams try their hand at facilitating, as opposed to presenting a case. It was thought that this might be a better way of meeting some of the CSLOs. A few students felt that if nothing else, the facilitation approach had the chance of enlivening what many of them felt were very boring and tedious presentations. “You really don’t understand it unless you can teach it.” Someone once said, "Perhaps case facilitation is one means for applying this nostrum". Basically, students conducting a case facilitation do everything they would do for a case presentation except for two important additional tasks. First, the facilitating team must assume the role of surrogate instructor. They must determine what the learning goals for the case are. This role was not without controversy. It is not an easy task for a student team to derive learning goals from a case study. Some students and BAC members felt uneasy about having the students perform this task. Nevertheless, most of us agreed that we were providing the students with an opportunity to try their hand at a “worthy intellectual task” (Wiggins 1990). Ultimately, case facilitation was seen as a constructive challenge for the student team, as they would really have to think—not only about how to analyze and recommend solutions for the case, but to consider why this was a case (i.e., why was this case written), and what should one expect to take away from the classroom discussion and analysis of the case. Secondly, they must figure out how best to get the class involved in the case facilitation. They could use a variety of skits, role-plays, short video clips, debates, small group discussions, etc. to help engage the class. But ultimately, the team would have to figure out what to do. Because case facilitation has two additional tasks, the assessment rubric for the oral cases was modified (see Appendix D) for the 2005-06 academic year. The modifications are easily viewed by comparing the assessment rubric for oral case presentations (Appendix C) with the revised rubric (Appendix D). The facilitation approach was substituted for the case presentation approach for the academic year 2005-06. Student teams performed better and performed better for the same measures than they had done a year earlier. These results can be viewed by comparing the selected results shown in Appendix C with those in Appendix D. CLOSING THE LOOP. It should be noted that case facilitation—at least from one year’s experience—requires more involvement of the instructor with the student teams than the case presentation format. All teams had to meet with the instructor at least once before facilitating their case and several teams met more than once. All of these meetings were outside of class and an attempt was made to limit the meeting time to ½ hour. For my part, I required that the team read the case carefully and come to the case prep meeting with their learning goals for the case, their mode of presentation (e.g., class involvement techniques), their analysis of the organization and industry (e.g., research) and their basic recommendations in hand. That way, one could tell before the facilitation if the team had a good grasp of the case and how to facilitate it. In the future, we intend to develop a student-faculty meeting assessment rubric. Some faculty have suggested that such meetings be graded, which is an intriguing idea. CONCLUSION Assessment is now a sine qua non for business schools. Business schools must have a complete assessment program in place by 2007 (AACSB, 2004). This means learning goals and outcomes must be well established, assessment rubrics used in selected courses, assessment results gathered and changes—if any are required—implemented. This paper reports on how a capstone course (viz., SM) was changed due to assessment protocols set into the course two years earlier. Perhaps what is most important here is not so much what happened (the change from case presentation format to the case facilitation format—time will tell if this is really better), but how it happened. Instructors are always trying to figure out how to use assessment to “close the loop.” This paper may be useful to those instructors who are in that situation and are looking for innovative ways on how assessment can be used to improve or enhance the learning process. REFERENCES AACSB. (2004) Overview of assessment. Tampa, Florida: AACSB Retrieved July 20, 2006 from http://www.aacsb.edu/resource_centers/assessment/over view-why.asp. Angelo, T. A. (1995). Reassessing Assessment. AAHE Bulletin, 47 (8), 7-13. Bloom, B. S. (1956). Taxonomy of educational objectives: The classification of educational goals, and book 1: The Cognitive Domain. McKay Press, New York. Redle, David A. & Calderon, Thomas G. (2005). “Closing The Loop in Finance,” in K. Martell and T. Calderon (eds.), Assessment of Student Learning in Business Schools: Best Practices Each Step of the Way. The Association for Institutional Research and AACSB International Volume 1: Number 1, pp. 227-238. Gerretson, Helen & Golson, Emily. (2005) Synopsis of the Use of Course-Embedded Assessment in a Medium Sized Public University's General Education Program, Journal of General Education vol. 54, no. 2, p.139-149. Martell, Kathryn and Calderon, Thomas G. (2006) “Assessment in Business School: What It Is, Where We Are, and Where We Need to go Now,” in K. Martell and T. Calderon (eds.), Assessment of Student Learning http://www.aacsb.edu/resource_centers/assessment/overview-why.asp http://www.aacsb.edu/resource_centers/assessment/overview-why.asp Developments in Business Simulation and Experiential Learning, Volume 34, 2007 89 in Business Schools: Best Practices Each Step of the Way. The Association for Institutional Research and AACSB International Volume 1: Number 2, pp. 1-26. Muraki, E., Hombo, C. M., & Lee.Y. (2000). “Equating and Linking of Performance Assessments.” Applied Psychological Measurement, 24 (4), pp. 325-337. Walvoord, Barbara E & Anderson, Virginia J. (1998) Effective Grading: A Tool for Learning and Assessment. San Francisco: Jossey-Bass. Wiggins, Grant (1990) The Case for Authentic Assessment. ERIC Digest. American Institutes for Research, Washington, DC.; ERIC Clearinghouse on Tests, Measurement, and Evaluation, Washington, DC. Report: EDO-TM-90-10. 4p. Dec 1990. Wood, Alexander, T. & Anderson Carol H. (2001) “The Case Study Method: Critical Thinking Enhanced by Effective Teacher Questioning Skills,” Paper presented at International Conference of World Association of Case Method Research & Application, Lund, Sweden June 17-20th, 12 pages. Developments in Business Simulation and Experiential Learning, Volume 34, 2007 90 APPENDIX A TIMELINE & CHRONOLOGICAL SCHEMA University-wide Mission/Vision and Educational Philosophy 2000-01 Development of B-School’s Mission, Vision, & Educational Philosophy 2002 ---Assessment Committee Formed ---Assessment Coordinator appointed 2003------Assessment Workshops for Faculty ------- Assessment Criteria set up For BA Program -------Program learning goals established Some program learning goals were assessed Some program learning goals were assessed Using course-embedded learning goals using more general means (e.g., ETS national (e.g., the capstone course). business test for seniors). 2004—CSLOs and assessment rubrics established for capstone course 2005—assessment protocols implemented 2006—“Closing the Loop” (i.e., Case facilitation substituted for Case Presentation to meet CSLOs for capstone course Developments in Business Simulation and Experiential Learning, Volume 34, 2007 91 APPENDIX B EDUCATIONAL PHILOSOPHY & CORE LEARNING GOALS FOR THE BUSINESS SCHOOL AT SUNY GENESEO The Business School student body is expected to remain highly qualified academically and to consist predominantly of traditional college-aged students from XXX STATE. To prepare our students to meet the challenges of business, we must provide them with high quality degree programs that offer an integrated combination of knowledge, skills and professional orientation. We seek to do this by creating a student-centered active learning community in our classes and co-curricular activities. Our program is designed to ensure that our students leave The Business School with: • A thorough grounding in the principles, tools and best practices of a business discipline. • An appreciation of the many issues that affect the business world, including diversity, globalization and technology. • Competence in the analytical, interpersonal and communication skills necessary to succeed in the business world of today and tomorrow. • An exposure to and interaction with the world of business. We work to make explicit the connection between the liberal arts and business disciplines. We believe that the liberal arts core of SUNY Geneseo's education combined with our curriculum provides the broad foundation that will lead our students to develop sound business judgment and effective problem solving skills. Developments in Business Simulation and Experiential Learning, Volume 34, 2007 92 APPENDIX C SELECTED RESULTS FOR ORAL CASE PRESENTATIONS CASE______________________________________ Section_______________ Team Members:_______________________________________________________________ SUBSTANCE/CONTENT of presentation Excellent Satisfactory Unsatisfactory Major issues recognized Major issues discussed Critical Analysis of All viable alternatives Implementation Recommendations are realistic and “doable” Choice of Actions(s) make “sense” Recommendations Appropriate use of theories/concepts/readings Lessons/insights Case update Application of Critical Reasoning (defined below) to case analysis & presentation 4 5 3 PRESENTATION FORMAT Excellent Satisfactory Unsatisfactory Well organized/coherent Creative Team was enthusiastic Engaged class in Constructive Discourse (defined below) 2 2 8 Critical Reasoning (defined): …an investigation whose purpose is to explore a situation, phenomenon, question, or problem to arrive at a hypothesis or conclusion about it that integrates all available information and that can therefore be convincingly justified, [(from Kurfiss, J.G (1988) Critical Thinking: Theory, research, practice and possibilities. ASHE-ERIC Higher Education Report #2, Association for the Study of Higher education.] Constructive Discourse (defined): aggressively seeking different and/or opposing viewpoints. Positively discussing and evaluating alternatives views provided by others. COMMENTS (include on back side of page): NOTE: The numbers represent the number of teams in each category. Developments in Business Simulation and Experiential Learning, Volume 34, 2007 93 APPENDIX D SELECTED RESULTS FOR CASE FACILITATION TEAM___________________________________________________________ Facilitating Team got students involved in: Excellent Satisfactory Unsatisfactory Delineation of case issues Determining case learning goals Developing org. alternatives Critical assessment of alternatives Eliciting sound recommendations Thoroughly discussed implementation issues Derived lessons/insights from case Application of Critical Reasoning to case analysis 5 4 3 Facilitating Team: Excellent Satisfactory Unsatisfactory Supplemental research used to augment facilitation Creative use of facilitation aides Challenged students Team was enthusiastic Facilitation flow & structure Data/facts utilized Case update In general, Engaged class in Constructive Discourse 4 6 2 NOTE: In order to maintain some consistency (for assessment purposes) the assessors were asked to rate the teams on their use of critical reasoning and constructive discourse as separate items so as to compare these factors with previous semesters. COMMENTS (include on back side of page): NOTE: The numbers represent the number of teams in each category. Table of Contents Volume 34, 2007 Experiential Teaching May Lead To Experiential Learning The Role Of Learning Versus Performance Orientations When Reacting To Negative Outcomes In Simulation Games: Further Insights An Analysis Of The Interaction Of Firm Demand And Industry Demand In Business Simulations Consistency Of Participant Simulation Performance Across Simulation Games Of Growing Complexity Assessing And Incentivizing Learner Contribution And Performance Excellence: Creating Win-Win Evaluation Options The Technological Impact Analysis: A Research-Based Exercise To Heighten Learners' Technological Sensitivity Beginnings: How We Start the Semester and Individual Classes : A Roundtable Discussion Simulation Performance and Its Effectiveness As A PBL Problem: A Follow-Up Study The Use of Multimedia Learning Tools to Facilitate Online Learning of Business Statistics Forecasting Accuracy And Learning: A Key To Measuring Business Game Performance Panel Discussion: Alternative Ways of Using the Internet for Business Simulations to Input Decisions, Process, and Present Financial and Economic Data Output Absel Research -- One Additional Perspective On Where We Are And Where We Have Come From Assessing participant learning in a business simulation From Case Presentation To Case Facilitation: How Assessment Changed The Capstone Course The Use Of Computer-Assisted, Interactive Role-Play Simulation In Hong Kong Corporate Positioning: A Business Game Perspective Demonstration Of A Computer-Assisted Global Business Simulation Applying .NET Remoting To A Business Simulation Assessing Emotional Intelligence: The EQ Matrix Exercise Outcomes And Observations Of An Extended Accounting Board Game The Use Of Learning Styles Questionnaire In Hong Kong Forming Teams For Classroom Projects Online Budgeting And Marketing Control With The Proforma Analysis Package The Effect Of Experiential Learning Experiences On Management Skills Acquisition Students' Perceptions on the Individual Managerial Performance Orderness: A New Definition Of Alignment An Exploratory Study Of The Efficacy Of Servsafe® Online Team Behavior And Team Success Results From A Board Game Simulation Telecommuting Internships - Do They Work? The WEE GAME: A Pre-Game Effects On Learning When Students Have Information About Games And Their Outcomes When Playing Them Assessment And Simulations: Measuring The Academic Learning Compacts Within Dangerous Business: An Interactive Ethics Case Activity The Paradise Islands Assessing And Applying fiAppropriatefl Conflict Management Styles Distance Learning In Communication: Blended Or On-Line? Developing An On-Line Advanced Communication Course World Café: Simulating Seminar Dialogues in a Large Class The Programming Game: An Exploratory Collaboration Between Business Simulation And Instructional Design The Canary Principle: An Alternative Model for Providing Real-Time Coaching in an On-Line Discussion Environment Assessing Character Development Using A Direct-Approach Business Ethics Exercise Persistence Of Decisions By Simulation Game Participants The Invalidity of Profit = f(Product Quality) PIMS Validation of Marketing Games A Study of The Applicability of The Perceptions of Organizational Politics Scale (POPS) For Use in The University Classroom The ROI Of Connected Projects: A Payroll Example Initiation of Research on Gaming Simulation in Japan Inappropriate Use of Citations and Corrupting the Body of Knowledge: Accepting Urban Legends as Truth An Exploration Of The Perceived Value Of Highly Socio-Inductive Learning How "Whole" Is Whole Person Learning? An Examination Of Spirituality In Experiential Learning Modeling Outsourcing and Strategy Alignment into a Business Game Successful Integration Of Webct Into A Small Business School The Thinking Steps Model In Game Theory: A Qualitative Approach In Following The Rules The Application Of Means-End Theory To Understanding The Value Of Simulation-Based Learning Armchair Travel: An active learning approach to increasing global awareness and participant self-efficacy Goldratt's Thinking Process: Is there a place for it in the Total Enterprise Simulation Assessing And Developing Student Skills Using a Group Exercise Computer Business Simulation Design: Novelty & Complexity Issues