METHODS FOR EVALUATING PERFORMANCE ON BUSINESS SIMULATIONS: A SURVEY Developments In Business Simulation & Experiential Exercises, Volume 17, 1990 177 METHODS FOR EVALUATING PERFORMANCE ON BUSINESS SIMULATIONS: A SURVEY Phillip H. Anderson and Leigh Lawton College of St. Thomas INTRODUCTION Evaluating and measuring student performance is a major concern for Instructors using business simulation exercises. Teach (1987) suggests that too much weight has been placed on profits as a measure of simulation performance. He states that measures of managerial ability should include the ability to accurately forecast events, operate within a budget and allocate limited resources. Gently and Burns (1981) suggested that instructors administering a simulation exercise should use a system for classifying learning developed by Bloom et al (1959). This system identifies six learning objectives: basic knowledge, comprehension application, analysis, Synthesis, and evaluation. Gentry and Burns argued the learning objective(s) sought should be determined early in the design of course content. These objectives should have a strong influence on measurement techniques used. THE STUDY The purpose of this study was to measure the use of different methods for evaluating student performance on a simulation exercise. Anderson and Lawton (1988) described several methods that can be used, alone or in combination, to assess student learning on a simulation exercise. Simulation instructors were surveyed to assess the frequency of use for eight of these methods. Those surveyed were also asked to use the system developed by Bloom et al (1959) to classify the learning objectives they were pursuing. A total of 374 questionnaires were mailed out to potential simulation users. Two mailing lists were used. One list was provided by a college textbook publisher. The second was a list of the ABSEL membership. A total of 149 questionnaires were returned. Three of these questionnaires were unusable yielding a response rate from the mailing of 40%. This figure underrepresents the true response rate for simulation users since nonusers receiving the questionnaire undoubtedly discarded it. THE RESULTS Demographics Over half the respondents indicated that they used the simulation exercise in the business policy area. Management and marketing were the two other areas where simulations were used with relative frequency. The simulation was used in an advanced course by eighty-live percent of the respondents. The remaining 15% used it in an introductory level course. Sixty-seven percent used the simulation only in an undergraduate level course. 20% in a graduate level course and 13% used it at both levels. Most (86%) simulations ran from 8 to 16 periods, with an average of 12.3 periods. The average number of decision variables players decided on each period was 3L2, with over half (52%) ranging from 20 to 60 decision variables per period. The average number of students per team was four (3.8). Eighty-six percent of the teams ranged from 3 to 5 members. The grade received on the simulation comprised an average of 30% of a students course grade. Number Frequency and Weight of Methods Used Few of the respondents relied on just one method of evaluation. Almost ninety percent (88%) of the respondents used from 2 to 6 different evaluation methods. Table 1 shows the frequency of use of assessment methods by respondents of the survey. The table also Indicates the percentage of the simulation grade allocated to each evaluation method for those instructors who used the method. The mean and the distribution of the grading weights for each method is reported. Learning Objectives Measured Information was also requested on the learning objective(s) the instructor was attempting to measure with the evaluation method(s) used. The Bloom et a! (1959) system mentioned above, was used to classify the evaluation methods. Table #2 shows the percentage of respondents who used an evaluation method to measure a particular learning objective. For example. 8% of the survey respondents said they used oral presentations to measure student’s basic knowledge while 28% said they used presentations to measure comprehension. Percentages sum to over 100% because multiple learning objectives were mentioned for a given evaluation method. DISCUSSION AND CONCLUSIONS Results of this survey dearly show that while relative performance on a simulation exercise is Important to instructors for assessing student performance, it is rarely the only evaluation method used. Survey respondents saw the evaluation methods as measuring different learning objectives. The use of multiple evaluation methods allowed Instructors to measure learning at a variety of levels. Further research efforts should attempt to ascertain whether the evaluation methods do, in fact, measure the learning objectives as perceived by the survey respondents. References furnished on request Table of Contents Volume 17, 1990 The Impact of Decision Support Systems on the Effectiveness of Small Group Decisions - Revisited The Relationship Between Financial Performance and Other Measures of Learning on a Simulation Exercise Use and Effectiveness of an Analogy-Based Expert System Suggestions for Computerized Business Authors Dealing with Power: An Experiential Exercise Using Movie and Personal Diary Analysis Techniques A Model for Developing Student Skills and Assessing Outcomes Through Outdoor Training Computer-Aided Exercises Versus Workbook Exercises as Learning Facilitator in the Principles of Marketing Course An Exposition of Guilford's Si Model as a Means of Diagnosing and Generating Pedagogical Strategies in Collegiate Business Education Formal Planning and Simulation Team Performance: A Cross Sectional Approach Cases: Real Organizations in Real Time in the Classroom An Empirical Investigation of the Internal Validity of Marketing Simulation Game An Empirical Evaluation of the Pedagogical Value of Playing a Simulation Game in a Principles of Marketing Course Factors Affecting Effective Teaching of Strategic Planning: Some Preliminary Evidence An Experiential Exercise for Learning About the Relationship Between Organizational Form & the Project Management Process Accounting Communication Skills can be Taught in the Auditing Course Modeling Cost Functions in Computerized Business Simulation: An Application of Duality Theory and Sheppard's Lemma A Life Cycle Analysis of Decision Making for a Strategic Management Team What's the Problem? A Dynamic Model for Teaching Problem Solving Skills Experientially International Currency Fluctuations: Money$im, A Simulation Superstores: A Specialized Retailing Simulation Within a Specialized Marketing Curriculum Factors Affecting Student Perceptions of Learning in a Business Policy Game VC + EL = VL The Name Game: An Experiential Exercise in Intergroup Relations The Effects of Experiential Accounting Work Experience on Student Performance in Intermediate Accounting Courses The Results of Using the Experiential Activity Group Performance Evaluation in a Business Policy Setting Using a Legal Database to Describe the Legal Environment of Marketing (and Business) Matching Environmental Uncertainty and Organizational Configuration An Instructional Computer Simulation of Tampering in QC Executive Evaluation of Student Learning in the Looking Glass Simulation Group Personality Composition and Total Enterprise Simulation Performance An Expert System for Selecting Analytical Techniques for Analyzing Marketing Research Data Effects of Cognitive Styles on Responses in an In-Basket Simulation A Psychometric Analysis of Kolb's Revised Learning Style-Inventory Selecting and Developing Experiential Exercises Using Movies Application of a Real-World Strategic Management Model in the Classroom Demand Equations which Include Product Attributes Consumption as the Objective in Computer-Scored Total Enterprise Simulations The Effects of Decision Format and Evaluation on Simulation Performance, Decision Time, and Team Cohesion The Effects of Computer Related Assignments on Student Performance in Business Administration The Money Game: A Dynamic Simulation Including Random Shocks for Money and Banking Courses Methods for Evaluating Performance on Business Simulations: A Survey The Effects of Synergogy on the Policy Course: Significant Improvements in Student Learning and Teacher Evaluation Conditions and Outcomes of Trust in a Two-Person Bargaining Exercise Bankgame Enhancing Computer Business Simulation with the Use of VGA Graphics An Experiential Approach to Entrepreneurship An Advanced Simulation Method (ASM) for Multiple Objective Problems The Influence of Experiential learning Techniques on Student Recognition of Non-Primary Learning Styles Negotiating Mergers and Acquisitions: A Cocktail Napkin Approach Identification of Unintended Effects in Experiential Laboratory Exercises An Experimental Comparison of Paper and Pencil and Computer Aided Decision Support Tools Porting a Simulation from the IBM World to the Macintosh World A Transaction Cost Analysis of Experiential Learning The Development of Experiential Exercises for Courses in Entrepreneurship and Small Business Management The Assessment Center as and Experiential Classroom Exercise Pricing Strategy Algorithms for Playing Business Simulations Organizational Structures for International Operations: An Experiential Activity Simulation Emphasis in the Business School Capstone Course An Integrated Approach to Computerizing the Business Curriculum Cross-Cultural Business Negotiations Exercise Organizational Socialization and Gender Differences in Students at Work Understanding Student Work Experience: A Content-Analytic Approach Introducing Executive MBA Programs with Management Games An Analysis of Improvement in Business Decision Outcome with Sequential Use of Two Simulation Games Teaching Business Policy Utilizing Mass Lecture and Individual Case Labs Potholes Along the Road to Evaluating Learning Outcomes: The Case of Outdoor Management Training Experiential Learning for Interior Design Students: Using CADD, Lotus 1-2-3, and Wordperfect Cognitive Learning Using a Computer-Based, Qualitative Interactive Business Simulation Sex Discrimination: Does the Woman get the Job or Does the Best Man Win? A Search for Visual Aids to Support Experiential Learning Through the 1990's An Experimental Analysis fo the Effectiveness of Student Role-Playing in Sales Training How to Have Students Learn from their Term Projects Self-Evaluation Exercise (SEE): An Assessment of Class Contribution A Study of the Influence of Team Formation on Attitudes and Performance in Management Games A Hardware Based CIM Simulation Laboratory Model Test Substance Abuse in Organizations Micro Computer Training Models Teaching Forecasting, Cash Budgeting and Inventory Model Building Using SBTools A Comparison of the Effects of Experiential Learning Activities and Traditional Lecture Classes An Adjunct Writing Instruction Assistant, The Computer; With an Illustration Classroom Software for ABC Analysis Utilizing Information Processing Technology to Enhance the Business Policy Simulation Experience Progressive Cases Realistic Job Previews Vs. Traditional Job Previews: Experiencing the Differences and Understanding the Consequences Investment Analysis Using the Pragmatic Multiplier Approach A Computer Simulation Interface for Competitive and Firm Analysis Self-Assessment of Ethical Decision Making Predispositions Preparing Managers for Overseas Assignments An Inquiry into Japanese Marketing: Workshop on Teaching Japanese Marketing The Performance Appraisal Feedback Interview: A Role Play for Human Resources Management Teaching Counselor Selling techniques Using Experiential Techniques to Teach International Topics International Management Simulation Gaming: Current Status and Future Developments A Realism Comparison of Simulation Technologies/Methodologies A Time-Efficient Game to Illustrate Concepts Taught in Management Courses and Management Development Programs Teaching the Management of Technology The Concept of Face and the Applicability of Experiential Exercises in an Oriental Culture's