OBSERVING GENERAL ABILITY IN A TOTAL ENTERPRISE GAMING SIMULATION Developments in Business Simulation and Experiential Learning, Volume 25, 1998 OBSERVING GENERAL ABILITY IN A TOTAL ENTERPRISE GAMING SIMULATION Sharma Pillutla Precha Thavikulwat Dept. of Management, Towson University, Towson, MD 21252-7097 ABSTRACT Historically, group performance scores of total enterprise (TE) gaming simulations administered as group exercises for between 4 and 12 decision cycles have failed to correlate with measures of general ability, such as overall grade point averages. This study differs from earlier ones in that the gaming simulation used (a) supplied individual performance scores even though it is administered as a group exercise, and (b) was designed to move through 30 decision cycles in a single semester. Results show that individual performance scores did correlate significantly with overall grade point averages and with grades in Quantitative Methods for Business. The finding implies that the individual performance score of this TE gaming simulation is a defensible basis for assigning grades. The suggestion is made that improvements in the design of gaming simulations will allow them to be more prominent in management development and assessment. Keywords: Grade Point Averages, Total Enterprise Gaming Simulations, Validity. INTRODUCTION Studies of the correlation between performance in TE gaming simulations and standard measures of general ability, such as course grades and scores on the SAT and ACT exams, have often failed to “yield consistent conclusions” (Gosenpud, 1987). Statistically significant correlations generally have been found when the simulations are administered as individual exercises, but not when they are administered as they are designed to be administered, as group exercises (Anderson & Lawton, 1997; Wolfe, 1990; Wolfe & Roberts, 1986). The disturbing suggestion of these studies is that when business gaming simulations are administered as they are designed to be administered, the resulting scores do not reflect general ability, for if they did, statistically significant correlations should be evident. To say that a gaming simulation experience does not reflect general ability is not to say either that participants enter random decisions, or that they do not learn from the experience. Thus, in a recent study with a traditional TE gaming simulation with a marketing focus, Dickinson and Faria (1997) showed that participant decisions were superior to random decisions after the first three periods. The point, however, is that a business gaming simulation must reflect general ability to be a reasonable candidate for the assessment of business education, irrespective of what participants may learn from the experience. Traditional TE gaming simulations suffer from two limitations that apparently hinder the manifestation of general ability. First, they have no provision for objectively measuring individual performance when administered as group exercises. Second, they are designed to be played over only a few decision cycles, generally ranging from 4 (Rollier, 1992) to 12 (Anderson and Lawton, 1997). The question remains as to whether a TE gaming simulation that objectively measures individual performance and is administered over a much larger number of periods might not allow general ability to manifest itself to the point where statistically significant correlations with established measures of general ability would be observed. The study reported here sought to answer the question with data obtained from a nontraditional TE gaming simulation that objectively measures 224 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 individual performance, although administered as a group exercise in the traditional way. This same gaming simulation is designed to progress through 30 decision cycles in a single semester. Our main hypothesis is the following: H1: Individual performance scores of the nontraditional TE gaming simulation will correlate with overall grade point averages. Like traditional TE gaming simulations, the nontraditional one used in this study had a strong production-operations component. In this case, the production-operations component was so extensive that a subset of the TE gaming simulation containing only that component constituted a separate functional game. This led to our second hypothesis: H2: Individual performance scores of the nontraditional TE gaming simulation will correlate more strongly with grades in Production and Operations Management than with grades in any other course. METHODOLOGY Subjects Subjects were 141 senior business students of a comprehensive university enrolled, between spring of 1992 and fall of 1995, in seven semesters of a required capstone course on business strategy and policy. All semesters were taught by the same instructor, and all used approximately a 50-50 combination of cases and a gaming simulation. The Gaming Simulation The simulation used was CEO, a progressive- difficulty TE gaming simulation configured to advance through 30 periods over five phases: from a single-product, single-plant, single-market proprietorship managed by one student to a three- product, three-plant, two-market public corporation managed by three to five students. Like the classical game of MONOPOLY, CEO gives each participant an inheritance and a periodic income. Unlike MONOPOLY, however, it allows each to invest in the stock of all companies, and it scores individual participants not on their monetary accumulations, but on the points they receive when they expend their monies to purchase products made by companies in competition with their own. To perform well, the participant must both invest and consume with intelligence. On the production side, however, CEO functions like traditional TE gaming simulations in that participants form teams to manage their companies. Thus, the gaining simulation supplies a tradition profit score for each team, and a nontraditional consumption score for each individual participant. The gaming simulation was administered on a local area network (LAN), an arrangement allowing students to enter their decisions directly and get results immediately, without administrative mediation. The program coordinated decisions asynchronously (Thavikulwat, 1996), with a maximum of 6 periods of decisions (one phase) allowed in any one sitting, and with progression from phase to phase controlled by the instructor. Generally, students executed between two and three periods at each sitting. For each period, students entered at least three types of company decisions: those valid for an entire period, those valid for one subperiod (one- sixth of a period), and those valid for one unit of output. Decisions valid for an entire period concerned advertising and production capacity, notably the acquisition of building space and equipment, the employment of labor and related expenses, and the purchase of raw materials; decisions valid for a subperiod concerned work assignment and scheduling; decisions valid for one unit of output concerned product inspection- students could decide to inspect or not to inspect each unit as it was produced, animated, “before their eyes.” 225 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 A fourth type of company decision, valid until whenever the students should change them, was added after the first phase. These were product pricing decisions, time sensitive because the major market for the products were the students themselves, who, to receive points towards grades, used play money to buy products made by competitors. A fifth type, valid for the phase, was added after the third phase. These were strategy decisions wherein companies were allowed to change by up to 100 absolute percentage points any combination of 73 simulation parameters, with the limitation that no single parameter could be changed by more than 20% in either direction. Counting only those company decisions valid for an entire period, the kind shared by traditional TE gaming simulations, the number of decisions per period ranged from 35 at the start to 65 at the finish. Although these numbers in combination with the other types of decisions allowed may suggest that the game was inordinately complex (Wolfe, 1978), in actual operation much of the complexity was oblivious to the students because of the high degree of automation permitted by the LAN installation, and because options were added progressively over the course of each semester in which the game was administered. Dependent Variable Two individual performance measures were considered as dependent variables: earned points and total points (PTS). Earned points are those each student received when that student actually bought products. Total points are earned points plus credit for the student’s net worth at the conclusion of the exercise. Credit was assigned by exponentially smoothing (a = 0.01) actual market prices over the first four phases of the exercise and selecting the highest peak price, to assure that students who bought products with their money generally had the advantage over students who merely held on to it. Because both measures were highly correlated (r =.968), only total points are reported in the results. A company performance measure, accumulated worth (capital plus earning accumulated over the course of the exercise), also was available and affected grades, but this measure was not used in the study. Grade weight was based on an either-or scheme, with each student allotted 20% grade-weight for the higher of the student’s relative individual performance (total points) or the student’s company performance (accumulated worth), and 10% grade-weight for the lower of the two. Thus, if a student receiving an 80% score on individual performance was associated with a company that received a 60% score on company performance, that student’s individual performance would have had a grade-weight of 20%; that student’s company performance, a grade weight of 10%. But if the student’s company received a 90% score on its performance, then the grade-weight for the student’s individual performance would have been 10% and the grade-weight for the student’s company performance, 20%. Thus, the objective scores of the gaming simulation directly affected grades, and each player could “win” either by association with a winning team or by individual efforts independent of the team. Independent Variables The independent variables were overall grade point averages (GPA) as of spring 1997, and grades received in the following core business courses: 1. Management and Organization Theory (MOT) 2. Quantitative Methods for Business I (QMB) 3. Principles of Production and Operations Management (POM) 4. Management Information Systems (MIS) 5. Business Environment and Public Policy (EPP) 6. Principles of Financial Management (FIN) 7. Principles of Marketing (MKT) 226 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 Control Variables Control variables were the seven semesters over which the study took place. For educational and administrative reasons, the gaming simulation was modified from semester to semester, and modifications also were made in the cases and assignments that constituted the balance of the course. Dummy variables control for these differences among semesters. RESULTS Means and standard deviations of the variables are shown in Table 1. A correlation matrix of the independent variables is shown in Table 2. These results appear to be ordinary in all respects. The multiple regression results of individual performance scores on dummy-variable semesters and grade point averages are given in Table 3. As hypothesized (H1), individual performance scores correlate significantly with grade point averages, after controlling for generally strong intersemester differences. Table 3 Multiple Regression of Individual Performance Scores on Semesters and Grade Point Averages The multiple regression results of individual performance scores on dummy-variable semesters and course grades are given in Table 4. Contrary to hypothesis (H2), individual performance scores correlate significantly with course grades only in Quantitative Methods for Business (QMB), after controlling for intersemester differences. DISCUSSION The results support the main hypothesis: that individual performance scores of the gaming simulation would correlate with overall grade point averages. This may be the first instance of such a correlation has been observed in a TE gaining simulation administered as a group exercise. The lack of correlation between individual performance scores and grades in POM may be due to the greater variability with which that course and the other noncorrelated courses are taught, as compared to QMB, where the content is more standardized, and where a correlation is observed. Moreover, because the gaming simulation allows 227 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 companies to sell products to participants as soon as they are produced, it gives a compelling advantage to those who act quickly, thereby reducing the advantage of competence in POM methods, which generally are time-consuming. Table 4 CONCLUSION This study establishes, apparently for the first time, that the objective individual score of a TE gaming simulation administered as a group exercise measures a domain that overlaps with overall grade point averages. Thus, general ability does play a major role in determining performance in this particular total TE simulation, even if Anderson and Lawton (1997) should be correct in concluding that it does not play a role in determining performance in other such simulations. The implication of our finding is that the individual performance score of this TE gaming simulation is a defensible basis for assigning grades. Because the individual performance score also is the easiest basis, requiring absolutely no additional efforts on either student or instructor, other methods (Anderson & Lawton, 1992; House & Napier, 1988; Smith & Golden, 1989; Teach, 1990, 1993a, 1993b; Wolfe, 1990, 1993a, 1993b) must tend at best to be second best. This study did not examine the possible contribution of the company performance score, another objective score that this gaming simulation supplied. Although the students who participated in the study were given grade-credit for team performance as measured by the company performance score, the credit may be unnecessary. Its use makes it more difficult for participants to tell, overall, where they stand, but it may help sustain motivation by giving participants more than one way to win in the competition. Finally the positive finding of this study demonstrates that changes in the design of gaming simulations can make them better instruments for the assessment of business education. As Teach (1993a) and Wolfe (1994) have observed, many of the business gaming simulations in use today are not much different from those created in the late 1950s. The stagnation in the design of these simulations may account for the concomitant stagnation in usage (Keefee, Dyson, & Edwards, 1993). It may be that Keys and Wolfe’s (1990) prediction that “management games will play a more significant role in management development and assessment efforts” (p. 324) will not come to pass until those who specialize in this field take issue with Wolfe’s (1994) contention that “lack of change is not seen as a vice, but instead attests to the originality, creativity, and basic integrity of the original gaming concepts and teaching environments established at that time” (p. 276). 228 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 REFERENCES Anderson, P.H., & Lawton, L. (1992). A survey and methods used for evaluating student performance on business simulations. Simulation & Gaming, 23, 490-498. Anderson, P. H., & Lawton, L. (1997). Performance on a TE simulation: What does it represent? Developments in Business Simulation & Experiential Learning, 24, 19-24. CEO: A business simulation for policy and strategic management, Thavikulwat, P. (1991). New York: McGraw-Hill. Dickinson, J. R., & Faria, A. J. (1997). A random strategy criterion for validity of simulation game participation. Simulation & Gaming, 28, 263-275. Gosenpud, J.. (1987). Research on predicting performance in the simulation, Developments in Business Simulation & Experiential Exercises, 14, 75-79. House, W. C., & Napier, H. S. (1988). A comparative study of strategic performance factors in actual and simulated business environments. Developments in Business Simulation & Experiential Exercises, 15, 50-5 5. Keys, B., & Wolfe, J. (1990). The role of management games and simulations in education and research. Journal of Management, 16, 307-336. Keeffe, M. J., Dyson, D. A., & Edwards, R. R. (1993). Strategic management simulations: A current assessment. Simulation & Gaming, 24, 363-368. Rollier, B. (1992). Observations of a corporate facilitator. Simulation & Gaming, 23, 442- 456. Smith, J. R., & Golden, P. A. (1989). Strategic planning and organizational performance in a business simulation: An empirical study. Developments in Business Simulation & Experiential Exercises, 16, 188-190. Teach, R. D. (1990). Profits: The false prophet in business games. Simulation & Gaming, 21, 12-26. Teach, R. D. (1993a). Forecasting and management ability: A response to Wolfe. Simulation & Gaming, 24, 63-72. Teach, R. D. (1993b). Forecasting as a performance measure in business simulations. Simulation & Gaming, 24, 476-490. Thavikulwat, P. (1996). Activity-driven time in computerized gaming simulations. Simulation & Gaming, 27, 110-122. Wolfe, J. (1978). The effects of game complexity on the acquisition of business policy knowledge. Decision Sciences, 9, 143- 155. Wolfe, J. (1990). The evaluation of computer- based business games: Methodology, findings, and future needs. In J. W. Gentry (Ed.), Guide to business gaming and experiential learning (pp. 279-300). East Brunswick, NJ: Nichols/GP. Wolfe, J. (1993a). Forecasting and management ability: A rebuttal to Teach’s response. Simulation & Gaming, 24, 73-75. Wolfe, J. (1993b). On the propriety of forecasting accuracy as a measurement of team management ability: A preliminary investigation. Simulation & Gaming, 24, 47-62. Wolfe, J. (1994). Recollections on 25 years of simulation/gaming. Simulation & Gaming, 25, 274-278. Wolfe, J., & Roberts, C. R. (1986). The external validity of business games: A five-year longitudinal study. Simulation & Gaming, 17, 45-59. 229 Table of Contents Volume 25, 1998 Marketing Goes to the Movies Bringing Experiential Learning to a Principles of Marketing Course Investment Analysis Application Using In-house Spreadsheet Models SugarCoated Statistics: An Exercise for the First Day of Class Improving Undergraduate Student Involvement in Management Science and Business Writing Courses Using the Seven Principles in Action Establishment and Funding for Interuniversity / Multidisciplinary Student experiences The Prospects of Creative Teaching: A Discussion with Patricia Sanders The Simulation and Classroom Assessment Techniques Developments of Management Skill Assessment Games as Instruments of Assessment: A Framework for Evaluation The Role of Artificial Intelligence in Business Curricula Threshold Solo Competitor: A Management Simulation (V1.0) a Windows-Based. Play Alone, Total Enterprise Simulation and Assessment Instrument Toward An Understanding of One's self-concept Total Enterprise Simulations and the Internet: Improving Student Perceptions and Simplifying Administrative Workloads The Expatriate an Assignment Orientation Game An Expatriate's Nightmare: An Experiential Exercise in Coping with Overseas Assignments Analyzing Experiential Exercise: Using the Scientific Method for Problem Solving The Second Component to Experiential Learning: A Look Back at How ABSEL has handled the Conceptual and Operational Definitions of Learning Predictive Models of Learning: Participant Satisfaction of Experiential Exercises in Business Education Accelerating Moral Development through Use of Experiential Ethical Dilemmas Ethical Dilemmas to use with Business Simulations to Teach Ethics The Class Approach in Behavioral Simulation in a Business Policy/Strategic Management course: A Progression toward Greater Realism An Exploration of the Emergence of Process Prototypes in a Management Course Utilizing a Total Enterprise Simulation How Organizations Are Improving Their Performance Utilizing Electronic Commerce: Examples From The Internet The Market Access Planning System (Maps): A Computer-Based Decision Support System For Facilitating Experiential Learning In International Business An Excel Workbook For Student Planning And Interface With A Simulation Game Design Of Multi-Media Based Pedagogy For Leadership Training Panel Discussion On Using The Internet For Courses Valuing And Enhancing Teaching: Sharing Tips Via The Web The Buddy Project: A Semester Long Project Aimed At Developing An Appreciation For Diversity Enhancing The Excitement And Learning Retention In The Classroom: The Power Of Magic The Supervised Management Internship: A Job Or Learning Experience Team Ware™ An Online Moderated Class Discussion Facility And Beyond A Neophyte Distance Educator's Experience Learning Management By Practicing Management: A Report Of Significant Student Service In 1997 Integration Of Academic And Service Learning: Students' Perceptions About Its Effects And Outcomes The Value Of Incorporating A Service Learning Component Into Course Content: A Presentation And Roundtable Discussion Business Games Teach: Thoughts on the Sources of Conflicting Conclusions on their Effectiveness Antecedents Of Learning In The Simulation: A Replication Using Student Journals To Enhance Learning From Simulations Technological Change And Intertemporal Movements In Consumer Preferences In The Design Of Computerized Business Simulations With Market Segmentation Integrating The Marketing Curriculum Using Collaborative Learning Teaching Time Management In A Sales Program: The Application Of A Computer Simulation Game Adapting Interactive Computer Simulations For Content Based Esl Instruction Multimedia And Student Expectations Synthesizing Data For Media Simulations Composing A Team Health Promoting Behaviors-A Decision Making Exercise Does it really Work? An Application of the Group Interaction Framework Administering the MIT Beer Game: Lessons Learned A Paperless Economy? Instructing Students on the Aspects of Successful Electronic Commerce Maximizing Learning Gains in Simulations: Lessons from the Training Literature Observing General Ability in a Total Enterprise Gaming Simulation FReach Teach: A Computer-Based System for Teaching Advertising Media Planning An Integrated Business Instruction System An Experiential Exercise you can Tinker With Experiential Exercises or Computer Simulations? Cash Flow Statements: Are They Important in Business Simulations? Holistic Cognitive Strategy in a Computer-Based Marketing Simulation Game: An Investigation of Attitudes Towards the Decision-Making Process Barnga: A Game on Cultural Clashes The Many Faces of Culture: Understanding Country and Corporate Culture Students' View of the Use of Business Gaming in Hong Kong Assessing General Management Interest What is the Future of Business Gaming? Starting a Small Music Trivia Business Exercise and Other Innovative Icebreakers An Integrated Approach to Behavioral Skill Development Career Focus: A Student and Business Learning Experience The Use of Concept Mapping in Teaching Strategic Management An Experiential Approach to Developing Mission Statements Business Games in Brazil-Learning or Satisfaction A Simulation within a Simulation: Job Layoff's and Emotional Reactions