ASSESSING STUDENT PERFORMANCE ON A BUSINESS SIMULATION EXERCISE Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 241 ASSESSING STUDENT PERFORMANCE ON A BUSINESS SIMULATION EXERCISE Philip H. Anderson and Leigh Lawton College of St. Thomas St. Paul, Minnesota ABSTRACT The problem of evaluating student performance on a business simulation exercise is discussed. Possible learning Objectives and a variety of evaluation methods are presented. The ability of these methods to measure desired learning objectives is postulated. INTRODUCTION One of the often discussed issues among instructors using business simulation exercises is the question of how to best evaluate a student’s performance. Few, if any instructors look at one sole measure when determining the learning a particular student has achieved. t variety of measures are typically employed to aid in making this assessment. Earlier research has focused on comparing learning in simulation exercises to other pedagogies. Reviews of this research can be found in Greenlaw and Wyman (1913), Keys (1976), and Wolfe (forthcoming). The focus of this paper is limited to discussing measures of learning possible in a business simulation exercise. It will outline various measurement options available to instructors and propose a relationship between these methods and possible learning objectives for the simulation exercise. DESIRED LEARNING OBJECTIVES Before the performance of a student can be assessed, a determination must be made of the desired learning objectives. Bloom et al. (1979) developed a system to classify learning into six levels. These levels are arranged in a hierarchical order to reflect progressively higher levels of learning. They are ascending order, basic Knowledge, comprehension. application, analysis, objective synthesis, And objective evaluation. A summary of the levels, as presented by Gentry and Burns 1981), is shown in Figure Gentry and Burns (1981) used Blooms system in developing a research design for surveying users of simulation games and experiential exercises. As they noted, the operationalization of learning presents difficult measurement. Problems. Performance measures often attempt to assess learning at more than one level simultaneously. Distinguishing which levels the instructor desires to assess is key to accurate measurement. Of more serious concern, teaching methodologies tray be used with an objective in mind, but the testing methodology used may not relate to that objective. This creates problems for both the teacher and the student. The teacher is unable to make the assessment desired and the student feels unfairly tested, Simulation exercises provide the opportunity for learning to occur a: all levels in Blooms taxonomy. This breadth of learning only increases the likelihood of measurement problems occurring. Identifying the desired learning objectives, recognizing the possible methods for measuring that learning, and the relationships between them would serve to minimize the above problems. Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 242 METHODS FOR EVALUATING PERFORMANCE Inherent in any simulation exercise, is the notion of learning being reflected in performance emulating that found in the simulated environment. It is this ability to implement what has been intellectualized, that is at the heart of the simulation experience. However, as noted above, there are many facets to any simulation. As a consequence. successful performance in a simulation requires students to learn at multiple levels of Blooms taxonomy. Given this broad spectrum, there are many ways to evaluate performance on a simulation exercise. Exams on Simulation Rules and Procedures Exams on riles and procedures of the simulation exercise to be used serves three primary purposes, they: (1) provide the instructor with feedback on a students knowledge of the simulations framework, (2) provide incentive for the student to learn how the simulation operates and to be in a position to contribute to his or her team, and (3) provide the student feedback on his/her knowledge of the mechanical’ aspects of the simulation. Given the above, the maximum gains from exams of this type are found by giving them at the beginning of the simulation exercise. Early timing alludes all three of the above purposes to be accomplished. Administering the test later in the exercise will provide insight into a student’s comprehension of the simulation at that point in time (point #1, above), however, active involvement in the exercise typically addresses the other two points. Later exams do have the advantage of identifying individuals and have chosen not to get involved and are attempting to ride on the efforts of the other team members. Peer evaluations (discussed below) also address this issue, but with certain handicaps. Since the purpose of these exams is quite limited, objective (true: false or multiple choice) questions are typically used. Exams on Reading Simulation Output Another form of examination consists of providing students with output from a hypothetical company in their simulation exercise. From the output alone, the students are asked to recreate the decision inputs made by that company. The purpose is to measure the student’s understanding of how the inputs are reflected in the output. This type of exam can be conducted at various times during the simulation. Early exams should test the student’s ability to decipher the typical decisions a company must make and where they will be reflected in the output statements. Later exams should focus on outputs of more complex decision inputs. This provides feedback on a student’s more comprehensive understanding of broader aspects of the simulation, at the same time showing the student the impact these decisions can have on an organization. Exams on Conceptual Issues A traditional method for evaluating performance is to determine the conceptual issues involved in the simulation and to test the students on their understanding of these issues. Used in its strictest form, this method would treat a simulation exercise no differently than a lecture course. It would test a student’s intellectual comprehension of the concepts the simulation exercise was trying to present, but would not measure the ability to implement their intellectualized vision of the issues involved, This method of testing can exist in many forms; ‘objective (multiple choice) exams, essay exams, or case analysis. Evaluation can occur at various times throughout the course to measure understanding of issues relative to the course that the simulation will demonstrate. Another form of this type of exam could be to measure the student’s ability to transfer the concepts learned In the simulation to other educational pedagogies, articulating the integrative elements of the simulation, An exam used for this Purpose could usually be given toward the end of the simulation exercise and would consist of case analysis or broad, integrative essay questions. Evaluation of a Written Plan Requiring teams to develop written plans for their simulation company allows the instructor to begin to assess their ability to apply their base of conceptual knowledge. The written plan reflects how they intend to operate. Of course, actually being able to perform the planned actions is a separate, and considerably more difficult, issue. As discussed here, this method is a close relative of the exam testing conceptual issues described above. This assignment car fall at the beginning or middle of the simulation exercise. Delaying the plan until after a few decision sets have beer, made allows the students to develop their plan based on a more thorough understanding of their environment. However, this can pert it students to operate without clear direction until the plan is written. This concern can be addressed by requiring a clan at the beginning of the simulations process and then having the students turn Ii a modification of the initial plan at a later stage of the exercise. The instructor evaluates the quality of’ the plan, including such things as measurability of objectives, the realism and achievability of the objectives, internal consistency of the strategies, and the compatibility of the strategy with the objectives and the company mission. Ability to Predict Results A demonstration of a team’s ability to implement its conceptual knowledge, as well as its written plan, is its acuity to accurately predict the results of its actions. In a competitive business simulation, accurately forecasting market share, profit as a percent of sales, and P01 reflects a team’s ability to translate its decisions into simulation outputs in the midst of a dynamic environment. Without this ability, efforts bi the team to plan its future activities are of little value. Measures which teams have to predict should be those which summarize the results of various other measures and are predictive of the overall success of the team. To the extent the instructor is trying to simulate reality, these measures should be those utilized by practitioners in environments the simulation is attempting to replicate. Performance Relative to Implementation of the Team’s Plan This approach rewards a team’s performance relative to its own predetermined plan rather than performance Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 243 relative to its competition. It requires a team to decide what type of organization it wishes to be (i.e., determine the company’s mission), set objectives for their company consistent with its mission, and then implement a plan which will lead to the accomplishment of it~ company’s objectives. It does not require a team to outperform other teams, only to meet the performance standards ~t set for itself. It requires the team to successfully assess the environment it operates in and to learn now to apply that assessment to actions the am cuts into operation. It also requires the instructor to ensure that the objectives set are both challenging yet achievable. In this respect, the instructor plays a key role in the “success” of the team, since le or she influences the degree of difficulty of the standards of performance for the team. Identification of and Recovery from Mistakes A team’s ability to identify its mistakes, develop a plan of corrective action, and successfully implement it provides a measure of both conceptualization and implementation skills. Without any intellectual understanding of the issues involved as a problem develops, a team is left with making a “seat of the pants” response. While this may, over time, provide the knowledge necessary for effective operation, competitive performance will suffer in the interim. A ‘lucky guess” may result in good short term performance. but this can seldom be sustained over multiple iterations. In fact, initial lucky success can lead a team into complacence;’ and poor utilization of its knowledge base, If a team possesses knowledge, but fails to use it, its results are likely to differ little from those teams having little knowledge. Use of this method of evaluation requires the teams to turn in a form identifying the mistakes made in the last quarter of operation, the corrective action that has been planned, and the expected results of that action. Relative Ranking on Simulation Results Perhaps the most common form of evaluating performance on a simulation exercise is to compare the ranking of a student’s team to other teams on a number of predetermined measures generated by the simulation. This provides the instructor with an objective assessment of the team’s performance abilities within the parameters of the simulated environment. This approach rewards competitive performance; the ability to conceptualize how the environment operates (i.e., what it takes to beat the other teams on the measures involved), analyze and validate or revise that conceptualization, and then to implement that conceptualized vision. Analysis Paper After completion of the simulation exercise, teams can be required to complete a paper analyzing their performance on the simulation. While it is too late for them to take any corrective action, it does allow them, outside of the ‘heat” of the simulated environment to evaluate their company’s performance. In essence, they perform a case analysis on themselves. it demonstrates their intellectual understanding of the simulation experience, if not their ability to operationalize that knowledge. Their evaluation should be a comprehensive analysis of all dimensions of the simulation; behavioral as well as financial, qualitative as well as quantitative. This paper allows them the final opportunity to demonstrate the intellectual (if not behavioral) insight they have gained through the simulation exercise. Oral Presentation An oral presentation of a team’s performance can be used ~n lieu of. or in tandem with, the analysis paper described above. Its content and purpose would be essentially the same as the paper. The oral presentation allows one students the opportunity to embellish their position, while allowing the instructor to assess the ability of the students to think on their feet. This method is most effective when an open question and answer session follows the formal presentation. This question and answer session provides the instructor with the opportunity to probe a team’s level of understanding more deeply than is possible with an analysis paper alone, When used in conjunction with the analysts paper, the presentation can consist of an overview of the paper’s contents. Peer Evaluations Peer evaluations provide the instructor insight into the existing group dynamics and the extent of the team’s development as a cohesive unit. The roles and responsibilities of’ :individual members can also be identified. While the accuracy and reliability of the reporting of any one individual may be in question, by combining the perceptions of all team members regarding any one individual on the team, a good judgement can, often be made of that person’s contribution to the team’s performance. A legitimate complaint by individuals receiving poor evaluations from their peers is one subjective nature of these assessments, Their perception of their contribution to the team may easily differ from that of the remaining team members. These differing perceptions could be the result of individual differences in what constitutes acceptable performance, misunderstandings of performance expectations, and/or personality conflicts/power struggles. This problem is increased if there is no evaluation until the end of the simulation exercise. At that point they have no opportunity to make changes in their level of contribution to the team, even if they desired to do so. This problem can be ameliorated if multiple midstream evaluations are required of each team. This forces the team to. at a minimum, identify the problems developing within the team and, preferably, to take actions to correct the situation. To the extent the team is able to improve its internal operation, the multiple peer evaluations provide the instructor insight into team members’ ability to implement concepts relating to leadership, discipline, performance evaluation, and other managerial concepts. COMBINING METHODS AND LEARNING OBJECTIVES Exhibit 1 shows the postulated ability of the various evaluation methods discussed to measure student accomplishment of the learning objectives presented in Bloom’s Taxonomy. This is presented as hypothesis for discussion, not as a definitive statement. Basic Knowledge Basic Knowledge is best measured through the use of exams. Questions can be developed to focus on the area of knowledge where testing is desired, The analysis paper also allows the instructor to measure the basic knowledge level of the students. Failure to adequately address basic issues in the paper indicates either a lack of understanding Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 244 of the issues or a misperception as to their level of importance. Comprehension A student’s comprehension of simulation issues can be focused on through the use of conceptual exams. They force the student to reorganize the material into different forms and restate them in their own words, rather than just “parrot them back” as in a multiple choice exam. Application A student’s ability to apply the concepts presented in a simulation exercise can be assessed through a variety of measures. The ability to write a plan, implement it, and accurately predict the results clearly indicates a level of learning has occurred beyond simply acquiring conceptual knowledge. Relative rankings on simulation performance measures would also reflect this ability to apply knowledge gained. Analysis Learning at the analytical level can be measured through the use of the following methods: accurately predicting results, implementation of a plan, successful management of mistakes, an analysis paper, and an oral presentation. Each of these methods reflects the student’s ability not just to act, but also to analyze what has happened, why :t has happened, and what to do about it. Objective Synthesis The students’ ability to objectively synthesize what has been learned is best measured by evaluating their ability to implement a plan, manage mistakes that occur while implementing their plan, and their assessment of their team’s performance through their analysis paper or oral presentation. These measures reflect the learning which has occurred beyond simple analysis and to apply original, creative thinking. Objective Evaluation The analysis paper and oral presentation provide the best measures of the students’ ability to objectively evaluate. They include all the measures of creating standards of judgement, and the weighing and analysis of personal performance against those standards. SUMMARY AND CONCLUSIONS No one of these methods provides a comprehensive measure of how completely a student has mastered the learning objectives for the course. Some learning objectives are easier to measure than others, just as some knowledge is easier to demonstrate than others. Exams focus on the intellectual capabilities of the student without fully addressing implementation skills. They emphasize the knowledge objectives of the exercise over the other learning objectives. Exams do have the advantage of providing feedback on individual performance (as opposed to team performance). However, one of the learning objectives may be to affect attitude (i.e.; develop the perspectives of a manager), using the simulation to set a nontraditional classroom environment. The use of exams would likely mitigate against successful accomplishment of this objective given their relationship to traditional testing methods. Many of these methods evaluate performance rather than individual performance. Unless the instructor chooses to give the same grade to all team members regarding performance on the simulation Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 245 exercise, some method(s) of individual performance evaluation must be utilized. These would include the various forms of examinations as well as the use of peer evaluations. A cafeteria approach to utilization of the methods discussed would provide the most comprehensive appraisal of a student’s performance. Exactly which methods to use should be determined by the learning objectives desired, the overall focus of the course the simulation is a part of. arc the preferences/style of the instructor. REFERENCES Bloom, B. S.. M. D. Englehart E. J. Furst, W. H. Hill, and D. R. Drathwohl (1959), Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook 1: Cognitive Domain. New York: David McKay Company, Inc. Gentry, J. W. and A. C. Burns (1981), “Operationalizing a Test of a Model of the Use of Simulation Games and Experiential Exercises”, in W. D. Biggs and D. J. Fritzsche (eds.) Proceedings of the Eighth Annual Conference of the Association for Business Simulation and Experiential Learning. Greenlaw, P. S. and F. P. Wyman (1973) “The Teaching Effectiveness of Games in Collegiate Business Courses.” Simulation and Games 4: 259 - 294. Keys, B. (1976) “A Review of Learning Research in Business Gaming”, in B. H. Sord (ed.) Proceedings of the Third Annual Conference of the Association for Business Simulation and Experiential Learning Wolfe, J. (forthcoming) “The Teaching Effectiveness of Games in Collegiate Business Courses: a 1973 - 83 Update.” Simulation and Games. Table of Contents Volume 15, 1988 The Role of Experiential Knowledge and Human Information Processing in Decision Making A Semantic Differential Instrument to Evaluate Experiential Teaching Methods A Comparison of Two Approaches to Management Skill-Building in an Organizational Behavior Course: A Replication Integrating Simulations: A Model for Business Policy Success Capstone Renaissance = Simulation + Interaction + DSS A Hybrid Method of Executing a Management Simulation: Combining the Best of Mainframes and Microcomputers Providing an Experiential Dimension to Cost/Managerial Accounting Courses Utilization of Computerized Tax Research Services in the Tax Research Curriculum Using and Expert System Based Decision Aid in Accounting Information Systems Event-Extended Entity-Relationship Diagrams for Understanding Simulation Model Structure and Function Multiple Objectives in the Development of the Gordon Macro Game A Comparative Study of Strategic Performance Factors in Actual and Simulated Business Environments An Empirical Investigation of Integrated Spatial-Proximity MCDM-Behavioral Problem Solving Technology Group Decision models Computer Simulation of Human Interaction The Role of Experiential Learning and Simulation in Teaching Management Skills Expert Systems - The New Business Simulation Tool Integrating Prolog into and Undergraduate Logistics Course Simulating Material Requirements Planning on Lotus 1-2-3 Innovation in Management Education: The Impact of the AACSB Experiential Learning in the International Environment Educational Testing with the Microcomputer A Simulation of Investment Analysis, Portfolio Management and Reporting Using Lotus 1-2-3 The Use of an Expert System to Develop Strategic Scenarios Two Exercises for Teaching about Motivation Sex Roles and the Good Manager A Form and Process for Nonconfidential Peer Evaluations Simulation and the Recalcitrant Student Employee Rights-Student Rights: A Classroom Exercise Computer Simulated Competition: An Alternative to Team Play Management Simulation The Relationship of Locus of Control and Vividness of Imagination Measures to Simulation Performance Formal Planning, Simulation Team Performance, and Satisfaction: A Replication Experimental Analysis of Magnitude and Source of Students' Inequitable Classroom Perceptions in Three Reward Conditions Strategy Design, Process and Implementation in a Stable/Complex Environment: An Exploratory Study Matching a Strategy Simulation to the Business Policy Literature: A Black Box Approach to Simulation Development An Evolutionary Classroom Experiential and Computer Simulation Model of a Corporate Strategic Planning System Collective Bargaining in the City of Elson: A Public Sector Experience Should Students Play Games in Labor Relations? Applying Cognitive Educational Objectives to Business Management Cases Grading as a Teaching and Feedback Mechanism: Involving Students in the Grading Process Teaching Controversies: A New Approach to Computer-Assisted Instruction and Simulation Packages Simulating Demand in and Independent-Across-Firm Management Game Advertising Response in the Gold and Pray Algorithm: A Critical Assessment A Model for Pricing Decisions in First Period Marketing Simulation Games Jog Your Right Brain: An Exercise for the Classroom and for Research Six Thinking Hats: An Exercise to Combat Confusion and Develop Thinking Skills Communicating in Context: A Simulation for Learning Business Communication A Simulated Consulting Service for the Compete Marketing Simulation Game Action Exams in the Consumer Behavior Class Using Focus Groups to Teach Problem Definition in Basic Marketing Research The Use of Journals in Management Simulations: A Literature Review and an ABSEL Response An Initial Step Towards Developing and Using an Expert System with a Business Simulation Self-Managed Learning: An Experiential Course Design Using the QWL Paradigm A Review of Current Developments in Experiential Learning Bring the Real World into the Classroom Assessing Student Performance on a Business Simulation Experience Minimizing Startup Anxiety: Case Studies of Simulation Experiences A Tale of Two Shepards Or Using Simulation in a Class Without Walls Enhancing Business Simulations Through the Utilization of Experiential Activities Involving Local Community Executives Simulation with Integrated Spreadsheets: The Design and Development of a Conversational Marketing Concepts Decision Game Introducing INMART: An International Marketing Simulation Using a Computer-Based Business Plan Assistant in Conjunction with a Marketing Simulation Game Overview of the ABSEL Guide to Experiential Learning and Simulation Learning