PROBABILITY ASSESSMENT AND PERFORMANCE IN BUSINESS GAME SIMULATIONS Experiential Learning Enters the Eighties, Volume 7, 1980 121 PROBABILITY ASSESSMENT AND PERFORMANCE IN BUSINESS GAME SIMULATIONS Frank McLaughlin, University of North Florida ABSTRACT Business games and simulations have been widely adopted in collegiate business schools. The level of performance by a group in a business simulation has been attributed to many factors. The primary purpose of this paper is to test the hypothesis of a relationship between performance and the group’s ability to make good subjective probability assessments. A business game was used in three graduate level courses in an urban state university. At the end of the term, a performance rank was established for each group. Each group was also given a subjective probability questionnaire. Four measures of probability assessment ability were obtained. A rank correlation was obtained between performance scores and assessment ability for each class and each assessment measures. Statistical significance was found for a number of relationships METHOD A business game simulation, the Business Management Laboratory [2] was utilized for 3 quarters in graduate business courses in a state supported urban university, At the end of each class, a ranking was made on the performance of each firm. The ranking consisted of a composite index of rank orderings of such factors as earnings per share, unit cost of production and debt ratios. In addition, a subjective probability test was administered to each group participating in the simulation, The questionnaire was answered by each group, as a group. It was administered half way through the quarter so that students had worked together for approximately six t4eeks. The data obtained were used to calculate measurements for the four subjective probability scores given below: 1. “A resolution’ score which is a measure of the assessor’s knowledge of events, 2. A “calibration” score which is a measure of the “assessment ability” of the respondent. 3. A “total assessment” or “Brier” score which is the sum of the above two scores, 4. A “COOP” score which indicates how “conservative” or “optimistic” an assessor tends to be in making probability assessments, A rank ordering was made of each groups’ scores for each of the above probability assessment measures. A correlation was made between game performance ranks and the rank orderings of each of the four assessment measures. A Spearman’s rank correlation coefficient was used to test for statistical significance. SUBJECTIVE PROBABILITY QUESTIONNAIRE Subjective probability scores were obtained by administering a test which consisted of fifty dichotomous questions. The test was similar to one discussed by Brown, Kahr, and Peterson [1]. Groups were asked to select the best of two possible answers and then to subjectively estimate how certain they were that they had selected the proper answer. Probability assessments were restricted to one of six categories which had assessment values of .5 ,.6,.7,.8,. 9, and 1.0. The following represents a typical question: Which city is at a higher altitude? 1. Syracuse, New York 2. Salt Lake City, Utah Numerous scoring rules have been developed to assist in research on probability assessment [3]. In this research we utilized the Brier score [4] which is one of the earlier scoring rules developed. The Brier score has a desirable feature in that the total score may be divided into parts. Each of these parts has a distinct meaning. The first quantity, resolution, measures the degree of accuracy of the assessors. It is smallest when all predicted events actually occur. It is largest when only 50 percent of the predicted events actually occur. Thus, this quantity can be regarded as a measure of the assessors’ knowledge of the events on which the assessments are being made. The second quantity, calibration, is smallest when the actual percentage correct in any category is exactly equal to the assessed probability associated with that category. Thus, if ten items were each assessed at .80 and eight of the ten occurred, the scoring factor for this category would be zero. The magnitude of this quantity increases with increasing distance between the assessed value and the true percentage correct. This component of the Brier score can be thought of as the component which measures “assessment ability.” The calibration score is not useful in measuring the degree of conservatism or optimism in the assessment. It can be modified to represent a measure of assessment ability which takes into consideration the degree of conservatism or optimism of the assessor. We shall call it the COOP (conservatism-optimism) score. A positive score would indicate that ability is less than confidence. Caution should be taken in interpreting the absolute value of the score. A “perfect score” of zero could be obtained by having a positive score which canceled an equal negative score. DATA ANALYSIS AND RESULTS An analysis was made for each of the four types of assessment scores for each of the three classes. A Spearman’s rank correlation coefficient was used to test for statistical significance. The results of the correlation analysis are given in Table 1. As can be seen, several significant correlations were found. These include the total or Brier score for classes 1 Experiential Learning Enters the Eighties, Volume 7, 1980 122 and 2 and the calibration score for classes 1, 2, and 3. TABLE 1 Summary of Rank Correlations Class Score r p 1 Total .78 .01 Resolution .07 .43 Calibration .52 .09 Coop .57 .07 2 Total .80 .05 Resolution .60 .14 Calibration .70 .09 Coop .40 .25 3 Total .60 .14 Resolution .50 .19 Calibration .60 .09 Coop -.10 .44 The Brier Score Rank correlations between Brier scores and performance scores were statistically significant at the .10 level for two of the three classes. This indicates that groups that made better probability assessments tended to perform better in the business simulation. As stated, however, the Brier score can be divided into two components. Therefore, to interpret these results, we must analyze the component parts. Resolution No significant correlations were found for any of the classes when comparing resolution with performance scores. Resolution reflects the degree of accuracy of the assessors. It can be thought of as a measure of the events on which the assessments are being made. Thus, it would be highly unlikely to find any significant correlation between resolution scores and performance scores. Any such significance would imply a relationship between performance and a knowledge of the correct answers to the questions used on the assessment instrument. Calibration Significant correlations were found at the .10 level in all three classes when comparing calibration scores to performance scores. The calibration score can be thought of as a measure of assessment ability. Thus, these results support the idea that those groups that can best deal with the uncertain aspects of the simulation tend to have the best performance scores. This should not surprise us. Courses in business schools continually tell students that business takes place in an uncertain environment. The successful entrepreneur is often the one who can best deal with these conditions of risk. COOP The score is a measure of assessment ability that takes into consideration the degree of conservatism of the assessor. It was included in the analysis to test the thought that firms which tend to be particularly aggressive or conservative will perform better in the business simulation. No support was given to this hypothesis. SUMMARY Several significant relationships were found on correlations between a group’s probability assessment score and their performance in a business simulation The significant correlations primarily involve the calibration assessment score. This finding should not be totally unexpected. The simulation is interactive. There are random elements involved and uncertainty plays a major role. The groups that perform the best are the one that react well to this uncertainty. The problem for the researcher, of course, is to determine what type of training, knowledge, etc., will help individuals improve their ability in probability assessment REFERENCES [1] Brown, R. V., Kahr, A. S., and Peterson, C. Decision Analysis for the Manager. New York: Holt, Rinehardt, and Winston, 1974. [2] Jensen, R. L., and Cherrington, D. 3. The Business Management Laboratory. Dallas, Texas: Business Publications, Inc. 1977. [3] Murphy, A. H. and Winkler, R. L. “Scoring Rules in Probability Assessment and Evaluation.” Acta Psychology (December, 1970), pp. 273-286. [4] Sanders, F. “On Subjective Probability Forecasting.” Journal of Applied Methodology (April, 1963). Table of Contents Volume 7, 1980 Symbol Recognition and Correlation for Evaluating Decision Making in Computer Aided Business Simulations Polanal: An Experiential Approach to Decision Support The Use of Time Contracts in Formal Education Indexing Simulation Model Response for Gaming Flexibility Moving Toward A Theory of the Use of Simulation Games and Experiential Exercises Use of Simulation Administration to Achieve Pedagogical Objectives Experiential Learning on the Job - A Business Internship Program Toward the Ultimate Experiential Exercise A Learning Through Managing Program Workshop Using Experiential Materials in Industry Training Sponsored Experiential Learning - An Opportunity Problems and Pitfalls of Externally Sponsored Field Research Projects Viewed form an Experiential Learning Perspective A Modular Approach to Experiential Learning: Classroom & Consulting Using Simulation & Experiential Learning In Industrial Settings Terminations: An Experiential Review Agenda Items -- Board of Supervisors' Meeting - Town of Jori The Objective-Setting Interview in MBO: An Experiential Approach The All-Star World Series Team Exercise: an Experiential Learning Exercise Dealing with Various Organizational Behavior Issues Progress Report on global, A Rich Multinational Gaming Environment Computer Simulation: A Tool to Teach Queuing Theory New Technology for Business Games Technological Frontiers in Computer Simulations for Business Education Simulating the Product Life Cycle on Interactive Terminals On Compensatory Demand Functions in Marketing Simulations The Use of Games at Different Levels in a Single Marketing Course to Increase Game Participation Sun Airlines: A Heterogeneous Consumer demand exercise Incorporating a Group Selection Test An Organization Development Approach to Teaching Organization Behavior CBID: Cognitive, Behavioral, and Interpersonal Development - A Skill Development/Social Learning Approach to Management Development Using a Live Case Via Video Tape A Town and gown Approach Development of Student Generated Cases Using Computerized Text Editing and Database Technology Interdisciplinary Approached to Problems in Utilizing Experiential Techniques To Use or Not To Use Experiential Techniques, That's is the Question Forming Participant Teams in Simulation Gaming The Problems of Motivating Students and Clients in Live-Case Projects Problems of Teaching Leadership Skills Through Experiential Techniques Conflict Style Measurement: Antecedent to Change - A Proposal for an Experiential Exercise Demonstration Fundamentals of Simulation for Newcomers An M.B.A. Orientation Simulation for Managing Time and the Areas of One's Life SimNet Workshop: The International Simulation Network Demonstrates Three New Business Games The Use of a Simulation Model in the Planning and Evaluation of Commercial Bank Operations Probability Assessment and Performance in Business Game Simulations WageSim: A Wage and Salary Administration Simulation Grading as a Teaching and Feedback Mechanism: Modifying Student's Self-Perceptions of Performance Can Business Games Effectively Teach Business Concepts? Development of Multiple Value Orientations in Conflict Resolution Behavior: An Experiential Teaching Paradigm in Labor-Management Relations Course Designing a Competency-Based Peer Assessment Scale for the Evaluation fo Teaching in Higher Education A Method for Evaluating Information for the Equipment Replacement Decision: An Application of Monte Carlo Simulation Simulation: A Method of Appraising Communication Networks in Managerial Decision Making An Evaluation of In-Class Student Involvement Evaluation of the SBI Program from an Experiential Viewpoint: Focus on the Student Differential Predictors of Academic Performance for White and Non-White Samples The Manager's Dilemma: An Unobtrusive Measure of the managerial Sex-Role Stereotype Are Computer Simulations Sexist? The Effect of Group Size on Attitudes Toward the Simulation Associations Between Individual Cognitive Processing Variables and Business Game Performance and Play Students' Perceptions on Managerial Functions After Exposure to Either the Case Method or a Simulation What Business Students Learn from Finance Simulations Attitude Toward Experiential Exercises, The Student-Teacher Relationship, Student Psychological Types, and Performance An Example of How to Design a Research-Based and Classroom-Effective Organizational behavior Exercise Some Issues in Game Design Untested Hypotheses: An Approach to Experiential Learning Evaluation of Simulation Games: A Critical Look at Past Efforts and Future Needs Is Self-Perception Predictable? - Some Laboratory Results An Exercise in Conflict-Handling Behavior The Relationship Between Group Size and the Learning Curve Effect in a Gaming Environment Learning About Organizational Management Through Organizational Management: Closing the Gap Strategies, Managerial Approaches, and Decision Making in a General Management Simulation A Comparison and Evaluation of Similar and Dissimilar Group Scenarios Generated Using Manual Simulation Games Weaknesses of Research Methods in Experiential and Simulation Studies