EVALUATING STUDENT PERFORMANCE IN THE USE OF COMPUTER SIMULATION Development In Business Simulation & Experiential Exercises, Volume 21, 1994 190 EVALUATING STUDENT PERFORMANCE IN THE USE OF COMPUTER SIMULATION Thomas F. Penderghast, Pepperdine University ABSTRACT Computerized business games have been an integral part of the pedagogue of business schools. With the widespread use of microprocessors, these games are growing in popularity and use. This paper investigates the evaluation process connected with the use of a computerized business game in the classroom, and presents several recommendations for grading the simulation experience. The grading techniques discussed incorporate both the qualitative and the quantitative aspects of a business game. INTRODUCTION Computer simulations have been used in business schools as games since the 1950’s. In the 1970’s, it was estimated that over 200 computerized business games had been developed. More than 95% of the undergraduate and graduate business schools today have adopted one or more games in their curriculum. The simulation of a business environment in the form of a game provides one of the highest rated forms of classroom pedagogy in the opinion of students who participate in them. With the advent of the microprocessor, computer games have attained a much wider scope of potential use within the classroom environment. No longer is the professor dependent on the institution’s mainframe computer to run a game. The game can how be conducted totally within the classroom. Students can get faster feedback on their decisions and more cycles can be processed over the period of time the students are in the class. Types of Business Simulations There are two general types of simulations in use in business schools today. One pits the individual student against a computer program in much the same mode as a video game. The student competes against the algorithm and tries to attain the highest possible score. In this mode of simulation, the process is deterministic and the student’s strategy is often one of figuring out the algorithm and, in that way, gaming the game. Success in this mode often has little similarity with the competitive world of business. Its value is greatest when the simulation is used to teach the mastery of a skill that will be valuable later in the career of the student. Success is often measured in terms of the student’s ability to master the skill in a timely fashion and to know when to apply that skill when faced with a similar situation. The second type of business game is one that allows a student to compete against other students individually or in teams. This form of game is not deterministic since the results of the decisions of each participant or team depends on the decisions of all the other participants. These games usually have a greater breadth of possibilities and are probabilistically driven by a Monte Carlo random number generator. This paper will focus on the second of these two types of business simulations. The first type is usually self-scoring in that the stated objective is the mastery of some measurable skill. The administrator of the simulation can easily establish an objective standard for success and failure, or a continuous scale for grading purposes. For example, if the student is expected to put together an investment strategy using stocks and other forms off monetary investments, return on investment could be a straightforward measure of the effectiveness of the decision variables entered into the computer simulation. Competitive Business Games An assumption made here is that the game is played by individual students or teams of students running businesses in direct competition with each other. Within the game, the students must make decisions involving such aspects of business management as purchasing, pricing, advertising, hiring, financial management and growth. The game establishes the rules that must be followed and defines the costs that are incurred through the decisions made. For example, prices of raw material are clearly stated and applied equally to each company. Taxes and other government fees are automatically paid when incurred. Banks and other sources of funds charge an established rate for the use of their money. These games are played over a specified time period with decision cycles occurring either monthly, quarterly or annually. Each simulated business is given feedback on its prior decisions before the next set of decisions are submitted. Each student or team is expected to have and follow a strategy of action, but adjustments to the strategy may be made based on feedback obtained. Since all the businesses are competing in the same market, results for any one business depend on the integrated strategies of the competition as much as the decisions entered for that business. EVALUATION TECHNIQUES There is a tendency to apply the same evaluation approach used with single participant simulations to games involving competition. Under this criteria, the student or team with the greatest total net worth at the end of the simulated period wins and grades are assigned accordingly. Such an approach ignores the fact that a game, played over the course of a semester, cannot fully replicate businesses that operate over a much longer period of time. Some highly successful business strategies take longer to mature than the time allowed in the game. Take for example the long-term strategies applied by many Japanese firms. The promotion of a shortsighted approach as a measure of degree of academic success may do more harm than good by rewarding short term harvesting as a business strategy for the game. Two approaches will be taken to describe the means by which student performance in a business game can be adequately evaluated. These two approaches are quantitative and qualitative, respectfully. The difference between them is the quantitative approach is based on data generated in the playing of the game while the qualitative approach is based on the ability of the students to analyze the situation and to document either a solution to a problem or an approach to effective decision making. Qualitative Evaluation Techniques One qualitative technique is to have each student or team document the strategy that will be used in the simulation exercise. This assignment is best given around the third cycle of the simulation. Placing it at that period of time, rather than at the beginning of the game, allows participants to develop a better understanding of the simulated conditions and the decision variables required. This assignment could be graded in terms of the student’s ability to understand the objectives of the game and to formulate an effective Development In Business Simulation & Experiential Exercises, Volume 21, 1994 191 and comprehensive approach to making the required set of decisions. An additional side benefit of this exorcise is that it helps to firm up the details of the simulation exercise in the minds of the participants. When making this assignment, it should be emphasized that the student or teem is not required to stick to the strategy described if and when a bettor one can be adopted Another qualitative technique is to have the students or teams document their decision-making algorithm. This is most effectively used near the end of the simulation exercise. For example, if the game requires monthly inputs over a period of one year, this exercise should be used around the tenth month. The students are told that they will leave the country for several months and must turn the decision making over to someone else. They will not be able to contact this person over the time they are gone, but want to ensure that the decisions will be made as if they had made them. Hew each student document the decision-making algorithm to be used when evaluating their company’s feedback from the prior decision period. This exercise will formalize the decision-making technique developed over the periods already played. This assignment can be evaluated and graded based upon the student’s ability to describe a logical and effective decision making algorithm, one that could be used by someone else to make the required decisions in their absence. Quantitative Evaluation Techniques On the quantitative side, there are several techniques that could be employed to analyze the date base developed over the course of playing the game. Together with total net worth, they can be used at the end of the game to evaluate student success in the simulation exercise. In the techniques described below, it is assumed that the game is run over • one year time period end that the decision variables are input on a monthly cycle. The first of these is to count the unit sales of products throughout the simulated period. This could be used as a measure of inventory flow that is independent of pricing. While not as realistic as using dollar figures. it provides another way to look at success in playing the game that is not affected by either income or cost figures. The second of those techniques is to rank the total net worth or the contribution to retained earnings of the businesses each month. Each business could be ranked each period, based upon either total net worth or contribution to retained earnings. The rankings would then be added each month and the business with the highest cumulative total would have the best score. A variation to this adds or subtracts the amount of movement up or down the ranking each time a business advances or declines on the monthly ranking scale. This evaluation technique takes into account monetary success on a monthly basis as well as positional advancement or decline within the simulated industry. The third technique is based on the analysis of several selected ratios each month. One of these could be sales to cost of inventory. A second could be net profit after taxes to sales. A third could be the debt to equity ratio. Those ratios could be scored in much the same way as the technique described above using the cumulative sum of position rank each period end a score based on movement within the ranking. Even if a business begins on a bed footing because of an early poor decision, it could score well if it shows improvement over the period of the simulation. A fourth approach is to evaluate the dollar sales earned each month loss the direct costs spent to obtain those sales. Examples of these are cost of goods sold, specific advertising and direct inventory expenses. These results could be summed each month to get a total for the year, or could be evaluated with the ranking technique described above to take into account improvements in decisions made over the year. A final approach is to take into account the lost sales due to either insufficient inventory or not enough sales personnel. This evaluation could only be conducted if the simulation provides a tally of lost sales by product each month. One approach could be a total count of lost sales over the period of the simulation. The fewer the lost sales, the better the decisions mode. Improvement in decisions made could also be taken into account, as with the approach-described tow, by ranking the businesses based on lost sales each month. The monthly ranking of each business would be added together and the business with the lowest total would get the highest grade in this aspect of the evaluation. CONCLUSION The use of computerized business simulations in the classroom can odd greatly to the learning process. Students like business games because they allow them to put theory into practice. Business games provide •n active learning pedagogy that simulates the students through a spirit of competition that prevails during their use. When the simulation has been completed, the professor must evaluate the results. Th. bottom line profit, or total net worth, is not a complete picture of the performance of the students. Other factors should be employed to get a more accurate assessment of individual or team performance during the entire exorcise. This paper has identified several additional evaluations that could be conducted for the purpose of measuring performance end grading success. They pertain more to complex dynamic business games where students, acting as individuals or as teams, compete against each Other. Success in the game is determined not only by their individual decisions, but also by the decisions mode by all the other businesses in the simulation. These evaluation techniques can be both qualitative and quantitative in nature. It is up to the professor to select the techniques to be used and to determine how they will be weighted in the assignment of the final simulation grade. The selection of a computerized business game is also important since not all games on the market today provide the same detailed database from the decision variables input. The use of several evaluation techniques will give students the assurance that the complexity of the simulation was taken into consideration, and that there is more to business decision making than just the acquisition of the most money over the short term. Table of Contents Volume 21, 1994 ABSEL: The Way We Were and Need to Be The Intellectual Structure of ABSEL: A Bibliometeic Study of Author Cocitatons Over Time Activity-Driven Time in Computerized Gaming Simulations An Assessment Framework for Determing the Effectiveness of Total Enterprise Simulations Attributes of Learning Organizations: Simulating the Relationships Business Policy/Strategy Case Extension using Pro-Forma Planning: A Computer Based Model Complexity: Is it really that Simple? A Random-Strategy Criterion for Validity of Simulation Game Participation Enhancing a Computer Simulation with a Structured Reporting Environment Experiencing a Foreign Culture: A Cross-Cultural Simulation Group Cognitive Style and Computerphobia in Functional Business Simulations Human Issues in Technology Implementation Management Simulator Incorporating Advertising Strategy into Computer-Based Business Simulations: A Validation Study Increasing the Effectiveness of Performance Evaluation Through the Design and Development of Realistic Finance Algorithms The Packer-Feeder Game: A Commodity Market Simulator Relationships Between R&D and Profitability: An Exploratory Comparison of Two Business Simulations with Two Real-World, Technology Intensive Industries Simulation of the Predictive Value of Mammography Simulation Performance and Learning Revisited Strange Bedfellows: Competency Models and ACBSP Accreditation Standards Using a Business Simulation to Study the Determinants of Ethical Behavior What Simulation Users Think Players Should be Learning from the Simulations ADA and its Implications for Experiential Training Boss/ Subordinate Perceptions of Instrumental and Supportive Leadership Behaviors in Relation to Myers-Briggs Thinking Type Cluster Analyses of American Universities' Business Core Curricula Structures Utilized to Satisfy Fifteen Curriculum Areas Cooperative Learning or Learning to Cooperate Experiential Learning: Constraining Students with Time Budgets How Different Workplace Experiences Affect Different Worker Values Implications of the Trend Toward Relationship Marketing for Experiential Learning The Increasing Cultural Diversity of the American Workforce: Management's Challenge of the 21st Century Information and Uncertainty as Strange Bedfellows: A Model and Experiential Exercises Leadership as a Medium: It's Emergence and Effect on Performance in Small Leaderless Groups Speed, Depth, and Breadth: Assessing Learning in Learning Organizations Teaching Strategic Planning, Problem Solving, and Decision Making with Envisionary Experiential Exercises Validating an Instrument for Student Evaluation of Teachers: Some Noteworthy By-Products Don't Teach Ethics to Business Students Emotional Reactions Toward a Simulated Layoff: Before and After the Manipulation Enhancing Communication Using a Presentation Package Implementing Marketing Policies through a Business Management Simulation Integrating Action-Based Learning into Executive Development Programs On the validity of using the Microsegmentation Principle in Media Simulationsm Participatory Systems Analysis Some Relationships between Cultural, Organizational, and Educational Experience and Perceptions of Influence The Use of Decision Support Systems with a Marketing Simulation: The Future is Now Astute Business Policy: A Simulation of the Automobile Industry The Business Policy Game CEO II: A Gaming Simulation for Assessment Computer Paced Project Management Simulation Computerized Tutor Support Systems DEAL & GEO: Progressively Integrating Gaming Simulations for Entrepreneurship and International Business An Interactive Simulation Game for Competitive Decision-Making International Operations Simulation/Mark 2000 (INTOPIA) Multimedia Simulation Cuts Training Costs for Anderson Consulting Concepts of Total Quality Management: An Active Learning Exercise Cooperative Learning: The Extended Jigsaw Managing Diversity--Values and Attitudes: An Experiential Exercise in Awareness Navigating the Shoals of International Management Development Evaluating Student Performance in the Use of Computer Simulation Entrepreneurial Simulation Program