SENSITIVITY OF PERFORMANCE SCORES IN BUSINESS SIMULATIONS Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 227 SENSITIVITY OF PERFORMANCE SCORES IN BUSINESS SIMULATIONS Frank McLaughlin, University of North Florida ABSTRACT Administrators of business games are often faced with the problem of evaluating or rank ordering the performance of participating firms. One procedure for accomplishing this is to obtain a composite index of weighted rank orderings of performance factors such as earnings per share, unit Cost of production, and debt ratios. There is no general agreement, however, on what weights should be assigned to the various factors. The primary purpose of this paper is to determine how much of a difference the choice of weights actually makes. In other words, how sensitive is a set of rank orderings to a change in the weights placed on each performance factor. In this research, seven sets of weights were used to determine rank orderings of performance of teams in a business simulation in an MBA class. Rank order correlation analysis was used to test for the statistical significance of the relationships among the seven rank orderings. The findings indicate that the scores are not highly sensitive to the weighting system chosen. The underlying reasons for these correlations are explored. PURPOSE During the past decade, business games and simulations have become increasingly popular in collegiate schools of business and in management development programs. In these simulations, computer output is normally used to make a judgment about the performance of each firm. There are several methods of evaluating the performance of individual groups. One procedure is to base the scores on a single criterion such as earnings per share. Another popular method is to obtain a composite index of weighted rank orderings of performance factors such as earnings per share, unit Cost of production, and debt ratios. There is no general agreement, how- ever, on what weights should be assigned to various performance factors [1; 2;4;5]. The primary purpose of this paper is to question if the weighting system used in a particular situation has a major effect on the final rank orders of total performance. In other words, how 9ensitive is a particular set of rank orderings to a change in the emphasis (weight) placed on each performance factor. A secondary objective is to explain the degree of sensitivity by exploring the correlation between scores on various performance factors. PROCEDURE The results reported in this paper are taken from business game simulations performed in MBA classes in an urban oriented state university. The simulation used was The Business Management Laboratory [3] In each simulation, there were five participating firms, each with four individuals per group. With a few exceptions, all of the participants had 3 or more years of business experience. Early in the term, the participants were told that they would be evaluated using a form of the composite index method. Each group was asked to provide a suggested set of weights which would be used in the evaluation process. These sets of weights, along with the set suggested by the game administrator, provided alternative results which were used as a basis for sensitivity analysis. Participants were given a list of performance factors and told that many or all of the factors would be used in determining their final evaluation score. They were also told that weights would be applied to the factors so that some items would count more heavily than others in determining the final score. The actual weights were not revealed because of the concern that participants would ‘play the factors’ rather than operate the firm with good business practices. The list of factors used in this research Is given in Table 1. All firms were ranked from first to last in each of the categories. In some cases, such as earnings per share, the rankings exactly reflect the numerical results obtained from the simulation. That is, the firm with the largest earnings per share is ranked first, the group with the second largest is ranked second, etc. In other cases, such as consistency of dividends, there is more judgment involved. The factor ranks are multiplied by their weights and summed. The group with the lowest summation receives the ‘best overall performance score. Sensitivity of Performance Scores The question of interest is how will the composite rank orderings change as different weighting factors are applied to the scoring system described above. There are obviously several ways to approach this problem. One is to mathematically solve for the amount of change a particular weight could undergo (with everything else held constant) before the rank orderings would change. A more practical approach is to determine how different sets of weights, provided by different, but knowledgeable and interested parties, will effect the final rank orderings. This latter procedure was followed in this research. One set of weights used was supplied by the game administrator. Five sets of weights were supplied by the participants. During the third week of instruction, Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 228 each firm was asked to provide a set of weights for the factors shown in Table 1. They were told that the sum of the weights must equal 24. Any factor could be assigned a zero weight which would effectively remove it from the scoring process. Each of these weighting systems was used with the scoring procedure described above to obtain a set of rank ordering for the 5 participating firms. The final set of weights was derived from the single criterion method of evaluation. Under this scoring system, 100 percent of the weight was placed on earnings per share and no weight was given to any of the other performance factors. RANK ORDERINGS The rank orderings obtained with the seven Sets of weights are given in Table 2. The system labeled “ADM” was supplied by the administrator. "GPI" to "GP5" represent the five weights supplied by the five participating groups, and “EPS” is the system based solely on earnings per share. The first step In analyzing this data is to determine which of the above are significantly different and which can be attributed to chance. A Spearman rank order correlation coefficient can be used to test for a significant difference between scores obtained from two different scoring systems. Table 2 shows that six of the seven weighting systems yield similar rank orderings. The lone exception is the set of weights supplied by group 3. In fact, the rank orderings obtained from the set of weights supplied by the game administrator, group 1, and the EPS system are all identical. The same is true for the rank orderings obtained from the weighting systems of groups 2 and 5. These five rank orderings, plus the rank orderings obtained with the weights of group 6 all have correlation coefficients which are significant at the .05 level. Thus, except for system 3, we can argue that final rank orderings are not highly sensitive to the weighting system used. Analysis of Results In five of the seven cases, it is easy to explain the high degree of correlation among scores. Weighting systems ADN, GP1, GP2, GP5, and EPS assigned 50%, 62%, 62%, 41%, and 100% of the total weight to earnings per share. It is Interesting to note that there was a marked difference in the weights assigned to other performance factors, but these differences did not significantly effect the rank orderings obtained. The results obtained from the weights supplied by teams 3 and 4 require more analysis. Each assigned a relatively small, but similar percentage of the total weight to earnings per share (19% vs. 25%). However, the rank orderings obtained from the GP4 weights correlated very well with the others while those obtained with GP3 weights did not. An explanation of this disparity can be obtained by examining the rank order correlations of a firm’s position in total EPS and its relative position in the 12 other performance factors. Group 4 put 50% of its total weight on performance factors that have a significant positive correlation with EPS. Thus, we would expect the rank orderings obtained from this set of weights to correspond closely to the rank orderings obtained from earnings per share data only. Group 3 put even more weight (54%) on factors that have significant positive correlation with earnings per share. However, they also put a major degree of weight on factor E, consistency of dividends per share. This is the only factor that correlated negatively with earnings per share, and this correlation was significant. SUMMARY The purpose of this research was to determine if the final rank order scores of participating groups is highly sensitive to the weighting system chosen by the game administrator. In order words, do the weights really make a difference, or will the ‘cream come to the top as long as any reasonable weighting system is used. The results indicate that the weighting System chosen is not particularly important. Six of the seven rank orderings investigated correlated significantly with each other. The seventh deviated because of a “strange” negative correlation between earnings per share and consistency of dividends. The data do not offer an explanation for this correlation. It is something which should be investigated in future replications of the simulation. The data presented here is limited by the use of a single form of the business game by the number of simulations performed, and by the type of participants. Additional research should be performed. The implication is clear, however, that the final evaluation of a group’s performance may not be highly sensitive to the evaluation procedure. In other words, “the best will be the best" regardless of the weights chosen. This should give game administrators more confidence in assigning grades and making awards for performance. REFERENCES [1] Deep, S. A., Bass, B. B., and Vaughn, J. A. “Some Effects on Business Gaming of Previous Quasi-T Group Affiliations,” Journal of Applied Psychology, 51 (October, 1967), pp. 426-431. [2] Hand, H. H. and Sims, H. P. “Statistical Evaluation of Complex Gaming Performance,” Management Science, 21 (February, 1975), pp. 708-717. [3] Jensen, R. L. and Cherrington, D. J. , The Business Management Laboratory (Dallas, Texas: Business Publications, Inc., 1977). [4] Raid, A. D. “A Study of the Educational Values of Management Games,’ Journal of Business, 39 (July, 1966), pp. 339-346. [5] Wolfe, J. and Guth, F. G., "The Case Approach versus Gaming in the Teaching of Business Policy,” The Journal of Business, 48 (July, 1975), pp. 349-364. Table of Contents Volume 8, 1981 The Promotion: Human Sexuality in Organization The Simulation of Chaos Leadership Development in a Simulated Urban/Suburban (U/S) Environment Tomed: A Computer Game Emphasizing Social Responsibility/or/why the Pop-Top Can? Integrated Brain Activity and the Manager's Job: Utilizing the Troika Model What does R2 Have to do with a Product Management Course? An Analysis of the Effects of Jungian Problem-Solving Style Dimensions on Marketing Decisions Bargaining Behavior in Personal Selling and Buying Exchanges Extending the Simulation Product Life Cycle A Generalized Algorithm for Designing and Developing Business Simulations Operationalizing a Test of a Model of the Use of Simulation Games and Experiential Exercises Pygmalion and Perception: An Experiential Exercise Behavioral Consequences of Reward Regarding Employee Absenteeism in an Industrial Setting: An Operant Conditioning Approach The Simlab Program: The Use of Experimental Simulation and Process Analysis for the Development of Management and Organizations Designs for Research on Simulation-Games, Cases, and Other Experiential Exercises The Role of Students in The Case Method Weaknesses in Research Design Critical Variables in Research on the Educational Value of Management Games Research Questions for Cases Research on the Learning Effectiveness of Business Simulation Games - A Review of the State of the Science The Effects of Valuation Techniques on Holding Cost During Inflationary Periods: A Simulation Exercise The Operations Simulation - A Study in Game Development Applying Guided Design to the Production/Operations Management Course: A Progress Report and Evaluation Terminal Data Entry and Retrieval Systems Simulations and Microprocessors Microcomputers and Related Technology for Simulation Gaming Microcomputers - A New Technology for Innovations in Business Simulations Microprocessor Controlled Interactive Video Simulation Business Game Design: From Theory to Practice The Success of a Computerized Simulation in Microeconomic Pedagogy The Test Preview Game: Applying the Game Show Format Providing a Real World View of the Personal Function: A Simulation Finding an Effective Means of Teaching Managerial Behavioral Skills: Two Different Experiential Teaching Methods Compared A Management Development Program Based on the Experiential Learning Model Participant Type Differences in Response to Experiential Methods: An Informal Look Preparing Student Groups to Participate in Experiential Group Projects: An Organizational Development Approach An Instrument for the Assessment of Learning Dimensions: A Progress Report of the Learning Dimension Scale (LDS) An Empirical Analysis Relating the Learning Style Inventory to Memory and Logical Ability Teaching Styles in Simulation Experiential Learning Versus Traditional Teaching Styles Student Perceptions of Effective Teaching Behaviors Problems in Evaluation of Experiential Learning in Management Education Students' Perceptions of Learning by Simulation A Relative Evaluation of Experiential and Simulation Learning in Terms of Perceptions of Effected Changes in Students Overview of Computer Based Business Games in Business Policy Classes Behavioral Decision Theory and Business Policy Giving Accounting Students Writing Experience as Job Preparation Getting to First Base with MBO: An exercise for Writing and Evaluating Objectives Dimensions of Conflict in Experiential Learning Consumer Alienation and Perceived Relevance of the Business Simulation Using the Self-Reference Criterion to Simulate Culture in Internationalized Business Course Experiential Learning in a Cross Cultural Setting- The Practice of Simulation Approach to Business Education in Japanese Universities Student' Perceptions of the Use of a Computerized Simulation in Teaching Management Information Systems Using the Case Study Approach to Develop a Microcomputer Bases, Fully Integrated, Data Base Driven, Management Information System The Case Study as a Tool for Organizational Change: Applying the Steel Ax to the Designers of Management Information Systems Toward a Theory of Teaching Business Policy A Case Study in the Use of Experiential Learning (A Management Game Simulation) to Enhance Student Understanding of Strategy Evaluation and Policy Formulation Suggestions for Integration of the Business Administration Core Publishing Opportunities and Requirements for Business Simulation and Experiential Learning Materials How do we Apply Experiential Learning Intercollegiate Case Competitions for M.B.A. Students: Initiation and Implementation Teaching Business Policy and Strategy Using the Incident Process The Learning Co-Op Approach to the Core Policy Course Meeting the Managerial Skill Shortage - Is Academia Up to It? An Empirical Analysis of Experiential Learning Reinforcement International Experiential Learning: Experience is the Best Teacher Encouraging Student Participation During International Academic Programs European Summer Study Program: Can you, Should you, What Does it Take? Travel Seminars in Europe: How can I Direct One? International Experiential Learning: Student Evaluation ABSEL: Empirical Findings on the State of the Association Sensitivity of Performance Scores in Business Simulations Improving the Learning of a Business Simulation Game by Increasing the Process Content Student Participation in Deciding Performance Criteria for Grading in an OB Course: An Exercise and a Case Study Markup for Profit: A Simulated Self-Administered Experience in Retail Pricing The Investment Decision Game: An Experiential Learning Approach to Stock Market Decisions Through Gaming CHIPS: A Marketing Channels management Game External Validation: An Experimental Approach to Determining the Worth of Simulation Games Teaching Performance Appraisal Skills: An Experiential Approach In Support of Experiential learning: Results of a Follow-Up Survey The Introductory Management Course: Taking Theory Application One Step Further Decision Efficiency and Effectiveness in a Business Simulation The Implications of Cognitive Processing Variables and the Complex Decision Simulating the Simulation for Enhanced Player Rationality Simulation/Experiential Learning Audit