AN ANALYSIS OF PERFORMANCE IN SIMULATION GAMES COMPARED TO PERFORMANCE ON STRUCTURED COURSE CRITERIA: A CASE STUDY Exploring Experiential Learning: Simulations and Experiential Exercises, Volume 5, 1978 32 AN ANALYSIS OF PERFORMANCE IN SIMULATION GAMES COMPARED TO PERFORMANCE ON STRUCTURED COURSE CRITERIA: A CASE STUDY Tom F. Badgett, Texas Christian University Daniel C. Brenenstuhl, Arizona State University William J. Marshall, Texas Christian University INTRODUCTION Simulation games are used extensively today at many colleges of business administration. The widespread use of simulation games suggests that business faculty view the use of games as being valuable learning vehicles and that the use of simulation games stimulates student interest in course content. A study was performed in order to investigate the use of a typical simulation game by 104 undergraduate business students at a large university. The broad issues addressed by this study include: (1) What factors influence the performance of students participating in a simulation exercise? (2) Can these factors be used for prediction purposes? (3) How do the factors which influence student performance in simulation games compare to those factors which seem to influence performance in other types of course activities and course scenarios? DATA One of the interesting features of this study stems from the rather unique sample of data which was collected. The subjects in this study were 104 college juniors and seniors from two classes. The same subjects were enrolled in both a three semester-hour course in organizational behavior and a three hour integrative management simulation course using INTOP. The organizational behavior course was a highly structured, lecture course whereas the management simulation was highly unstructured. Because the students were required to take these two courses concurrently the authors were able to collect a matched sample of data on the students’ performance in two quite different course environments. Performance Variables The following performance variables were evaluated: (1) two 100-point objective exams covering text and lecture content from the organizational behavior course, (2) a total of 100 points from ten unannounced quizzes from the organizational behavior course, (3) final course grade in the organizational behavior course, (4) two announced quizzes covering the simulation game rules, (5) total points in the simulation game, (6) peer ratings of performance within the groups in the simulation game, and Exploring Experiential Learning: Simulations and Experiential Exercises, Volume 5, 1978 33 (7) final team ranks in the simulation exercise. Predictor Variables A review of the literature in the general topic area of predicting academic performance suggested the use of a wide variety of predictor variables. Cohn, for example, used motivation and maturity variables such as age, sex, and martial status; ability and achievement factors such as grade point average (CPA); background factors such as the number of semester hours completed in the major and whether or not the students had the prerequisites for a certain course; and course-related variables such as class attendance, whether or not the student purchased the text, and other factors such as the reason given for taking a course [1]. Other studies have made use of constructs such as Locus of Control (LOC) and Interpersonal Trust (ITS). Massari and Rosenblum concluded that a complex relationship between academic performance and LOC and ITS exists [5] . The general stream of research from the behavioral sciences has been oriented toward the use of similar personality constructs to predict academic performance [2], [3] Thus previous research suggested most of the predictor variables investigated in this study, and the authors’ judgments suggested the rest. Other variables which seemed, a priori, to have value for prediction and explanatory purposes were identified. From these two sources, then, the following predictor variables were included: (1) Internal-External scores. Roter’s LOC scale was administered to the subjects in order to measure the strength of two conflicting orientations, Internal vrs. External [6]. An Internal Control suggests that the subject perceives that there is a relationship between what the individual does and what happens to that individual. An External Control refers to the perception that fate, luck, or the environment largely determines the consequences of one’s behavior. (2) ITS scores. Rotter’s ITS scale was administered in order to measure the trust dimension [7] . A “low” truster is assumed to be suspicious of others in his or her interpersonal relationships. A “high” truster implies the opposite orientation. (3) Ability and achievement factors. From official university records the subjects’ cumulative college CPA scores were obtained and used in the analysis as both a measure of ability and achievement. Scholastic Aptitude Test scores (SAT) were obtained in the same manner. (4) Maturity, motivation, and background factors. The subjects were required to complete a brief questionnaire at the beginning of the semester which measured age, socio-economic status, expected grade in the course, and other information which was not used in this study. Class attendance records were maintained as part of the usual course procedures of the instructors involved. Exploring Experiential Learning: Simulations and Experiential Exercises, Volume 5, 1978 34 ANALYSIS Regression and correlation analysis were used to investigate the relationship among each performance variable and the predictor variables. The first step in the analysis was to derive the Pearson correlation coefficients for all of the variables included in the study. Table 1 gives the correlation coefficients for the ability and achievement factors. Ability and Achievement Factors The correlations between some of the performance variables and the ability-achievement variables are quite strong. Subjects who had high SAT scores and high CPA scores performed better on the rather structured performance criteria associated with the organizational behavior course. The high levels of significance reported in Table 1 for these correlation coefficients suggests that the relationship is too great to be attributable to chance. A similar, but less pronounced, finding was obtained for the two structured quizzes which covered the simulation game. The unexpected result found here occurred for the unstructured performance variables associated with the simulation game. Note that the relationship among SAT, CPA, and the simulation game results are much weaker than the relationships found for structured performance criteria. More important is the observation that many of Exploring Experiential Learning: Simulations and Experiential Exercises, Volume 5, 1978 35 the correlations are in the opposite direction from that which might be expected. That is, subjects with high SAT scores tended to perform worse on the simulation game criteria. That few of the correlations are significant detracts little from the surprising nature of the finding. LOC and ITS Table 2 gives the Pearson correlation coefficients for LOC, ITS, and the performance variables. Note in Table 2 the rather weak relationships among these variables as evidenced by the small correlation coefficients. This finding merits attention in view of the repeated use of these constructs to predict academic performance in studies reported in the literature. Another important observation concerning these correlations is that many of them are in the opposite direction from that which would usually be expected. High scores on LOC are associated with high scores on the performance criteria and vice versa. Thus externals performed slightly better on all performance criteria except two (Quiz 1 in the management simulation course and the unannounced quizzes in the organizational behavior course). Previous research suggested the hypotheses that internals would perform better on unstructured criteria (such as a simulation game or unannounced quizzes) and Exploring Experiential Learning: Simulations and Experiential Exercises, Volume 5, 1978 36 that LOC would yield no differences on structured criteria. Therefore it is surprising that the coefficients for the simulation performance variables and LOC are all positive. Some rather interesting results were obtained from the correlations of the maturity, motivation, and background variables with the set of performance variables. Note from Table 3 that those subjects who did not attend class regularly tended to perform more poorly on all the structured criteria in the organizational behavior class than those who attended class. Exactly the opposite result was observed for the structured quizzes given to the subjects in the management simulation course--those who were absent more often scored higher on the two quizzes. In addition, low attenders out performed high attenders on two of the important simulation game criteria. These findings are somewhat difficult to explain. One possible interpretation is that attendance was important in the organizational behavior class because a great deal of the test material was covered in class. The same was not true in the simulation course. Age and socio-economic status were not significantly related to any of the performance measures. The subjects’ expected grades yielded several significant correlations; however, additional unanticipated directional findings were observed. Subjects expecting higher grades earned better grades on four out of five structured criteria. This result was as expected. The one negative result Exploring Experiential Learning: Simulations and Experiential Exercises, Volume 5, 1978 37 was not significant (for Quiz 1 in the management simulation course). Subjects expecting higher grades received lower scores, however, on the unstructured performance criteria associated with the simulation. This finding is also difficult to explain. It should be noted here that if this finding is typical--if students expecting high grades usually obtain lower grades in simulation games--then games will likely be a frustrating experience for this group of students. Regression Analysis In order to provide additional insights about the relationships observed in the data, stepwise regression was performed on subsets of the data. The stepwise regression failed to produce useful regression equations as the coefficients of determination were not significantly greater than zero. The overall fit of the equations to the data was too weak to merit attention here. CONCLUSIONS This study did not produce significant findings about the factors which influence the performance of participants in a simulation exercise. However, the findings reported here suggest that some of the factors which are ordinarily considered important determinants of success in a simulation game really have very little influence at all. None of the variables studied here (LOC, ITS, SAT, CPA, socio-economic status, age, etc.) can be reliably used to predict performance in a simulation game. The study also revealed that the factors which influence performance on structured course activities are not the same as those which influence performance in a simulation exercise as measured here. Thus it is concluded that great care must be exercised in evaluating student performance in a simulation game. Moreover, it is reasonable to argue on the basis of this study that users of simulation games in business schools should heavily discount performance in a simulation game in determining individual grades. REFERENCES 1. Cohn, E., “Predicting, Performance: A Case Study of Students Success in An Economics Course,” Business and Economic Review, 21 (1975), pp. 3-6. 2. Crandall, V. C., W. Katkovsky, and V. J. Crandall, “Childrens Beliefs in Their Own Control of Reinforcements in Intellectual- Academic Achievement Situations,” Child Development, 36 (1965), pp. 91-109. 3. Hjelle, L. A., “Internal-External Control as a Determinant of Academic Achievement,” Psychological Reports, 26 (1970), p. 326. Exploring Experiential Learning: Simulations and Experiential Exercises, Volume 5, 1978 38 4. Imber, S., “Relationship of Trust to Academic Performance,” Journal of Personality and Social Psychology, 28 (1973), pp. 145-150. 5. Massari, D. J., and D. C. Rosenblum, “Locus of Control, Interpersonal Trust and Academic Achievement,” Psychological Reports, 31 (1972), pp. 355-360. 6. Rotter, J. B., “Generalized Expectancies for Internal versus External Control of Reinforcement,” Psychological Monographs, 80, No. 1 (1966), (whole no. 609). 7. Rotter, J. B., “A New Scale for the Measurement of Interpersonal Trust,” Journal of Personality, 35 (1965), pp. 651-665. Table of Contents Volume 5, 1978 Debriefing: The Key to Effective Experiential Learning A Kiss Before Debriefing Debriefing with Serialized Theory Development for TaskTeam Learning Insights Into Debriefing Experiential Learning Exercises The Two-Step Flow of Experiential Learning: A Preliminary Investigation Programmatic Experienced-Based Learning in an MBA Program The New Research Focus: An Analysis of the Simulation Game User An Analysis of Performance in Simulation Games Compared to Performance on Structured Course Criteria: A Case Study Le's Simplify the Administrative Requirements of Computerized Educational Simulations Assessment of Sex Stereotypes Within Task Group Simulation The Use of Program BAYAUD in the Teaching of Audit Sampling The Effectiveness of Progressively Complex Accounting Simulations in Increasing Decision-Making Performance The Natural Learning Project The Application of a US-USSR Trade Simulation to the Reaching of Business Russian The Use of Semi-Autonomous Work Group in Graduate Management Education: An Australian Experience Evaluation of an Experimental Course in Organizational Behavior for Managers of Japanese Banks Berkshire II: An Experiential Decision Making Exercise: An Exercise in Hegelian Inquiry Creating a Responsible Managerial Experience A Framework for Determining the Pedagogical Value of Simulation Gaming: Implications for Future Simulation Gamin Research Emergent Simulations in Administration Courses Measurement of Administrator Role for Feedback on Structure and Goals An Experiment in Assessing the Theoretical Recognition and Application Leaning Skills of Student Case Writers Some Game Information Systems Experience Using Job Control Language (JCL) for Group and Individual Simulation Considerations for the Use of Computerized Business Simulations Assessing the Effectiveness of Learning Styles as Predictors of Performance within Three Distinct Pedogogic Methodologies Learning Style and Performance A Psychometric Analysis of Kolb's Learning Styles Inventory Learning to Identify and Satisfy consumer Wants: A Classroom Game An Experiential Exercise in Product Benefit Segmentation Sales and Sales Management: A Case/Simulation Approach The SBI Case as Experiential Learning The Use of a Live Case in Teaching Organization Theory and management Principles to Graduate Students The Residency in Hospital and health Administration as Experiential Learning The Extended Live Case Approach to Teaching Marketing Research Experiential Learning in Marketing: Student Consultants The Brazil-Columbia Coffee Marketing Negotiations: An Experiential Exercise Aimed at Highlighting the Dynamics Involved in the International Business Negotiation Process Educational Values of Simulation Gaming Business Simulation Re-Revisited Simulation and Experiential Learning as Human Subject Research An Experiential Evaluation of a Didactic - Experiential Apporach for Teaching personnel Management Experiential Learning in Wage and Salary Administration Integrating Structured Experiences into Personnel Simulation Locus of Control and Performance in a Management Simulation Correlations Between Academic Achievement, Aptitude, and Business Game Performance Suggestions for Student Input of Decisions in General Management Simulations An Exploratory Investigation of Student Perceptions of Computer Simulation as an Educational Tool Time to Sell?: An Experiential Learning Approach to Stock Market Decisions through Interactive Gaming