CONSISTENCY IN SIMULATION PERFORMANCE OVER TIME AND ACROSS SIMULALTION GAMES Developments In Business Simulation & Experiential Learning, Volume 24, 1997 CONSISTENCY IN SIMULATION PERFORMANCE OVER TIME AND ACROSS SIMULATION GAMES W.J. Wellington, University of Windsor A.J. Faria, University of Windsor T.R. Whiteley, West Virginia State College ABSTRACT The present study examines the relationship between simulation game performance over two separate simulation competitions. Students were exposed to two different simulation games over two semesters in two separate marketing courses. The study involving 27 students on 27 single participant teams found little relationship (correlation of .05 80, significant at .774) between rank order performance in The Marketing Management Simulation versus rank order performance in the COMPETE simulation game. Because of the small sample involved in this study, it was concluded that further research into simulation performance over separate competitions is warranted to resolve the conflict between this finding and past research which shows a relationship. INTRODUCTION Poor performing students in simulation competitions often attribute results in such competitions to luck rather than skill. While luck may play a part in any simulation competition, if simulation games are a meaningful educational experience, skill must be the most important factor in explaining good performance. This study seeks to provide some academic research on this topic and to support the notion that good performance in simulations is not the result of luck. Past research has examined the relationship between student performance in simulation competitions and a wide range of variables. Among the variables examined have been numerous personality characteristics, locus of team control, achievement motivation, previous academic performance, time pressure, ethnic origin of team members, gender, team size, previous business experience, team organizational structure, method of team formation, and grade weighting (see for example Anderson and Lawton 1992; Brenenstuhl and Badgett 1977; Butler and Parasuraman 1977; Chisholm, Krishnakuman and Clay 1980; Edge and Remus 1984; Faria 1986; Gentry 1980; Gosenpud 1989; Gosenpud and Miesing 1992; Hergert and Hergert 1990; Hsu 1984; Moorhead, Brenenstuhl and Catalanello 1980; Newgren, Stair and Kuehn 1980; Patz 1990; Roderick 1984; Walker 1979; Washbush 1992; Wheatley, Anthony and Maddox 1988; and Wolfe, Bowen and Roberts 1989). Summarizing much of the past research have been major review articles by Greenlaw and Wyman (1973), Keys (1976), Wolfe (1985), Miles, Biggs and Shubert (1986), Wolfe and Keys (1990) and Randel, Morris, Wetzel and Whitehall (1992). The present study examines whether good simulation performance is repeatable and thus attributable to the differing skills and abilities of the simulation participants as opposed to being due to some element of luck. This study builds upon repeat performance results reported in an earlier study (Wellington and Faria 1995). LITERATURE REVIEW While only one previous study has specifically addressed the issue reported in this paper, 146 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 several related areas of research will be briefly discussed. Several factors may explain good performance in a simulation competition. For example, it is possible that good students will consistently outperform poor students. To test this, a number of studies have examined the relationship between grade point average (GPA) and simulation performance. While some studies have reported a positive relationship to exist (Hsu 1989; Wolfe and Keys 1990; and Wolfe and Chanin 1993) many others have found no such relationship (Faria 1986; Gosenpud 1987; Gosenpud and Washbush 1991; Norris and Niebuhr 1980 and Wellington and Faria 1994). Learning is another obvious factor that might lead to good simulation performance and several studies have examined this relationship. Learning is generally measured by performance on end of course examinations. While two studies have reported a relationship between simulation performance and performance on mathematical problems (Faria and Whiteley 1989 and Whiteley and Faria 1990), many more studies report no relationship between superior simulation game performance and performance on course final examinations (Anderson and Lawton 1992; Washbush and Gosenpud 1993; Wellington and Faria 1991; and Whiteley 1993). A number of studies have examined the personality traits of successful simulation game players and successful business executives (Babb, Leslie and VanSlyke 1966; Gray 1972; McKinney and Dill 1966; Vance and Gray 1967; and VanSlyke 1964). These studies have generally shown that the characteristics of successful game players conform to those of successful business executives. Additional studies have examined the decision-making styles of successful simulation participants and successful business executives (Babb and Eisgruber 1966 and Wolfe 1976). These studies have reported that the decision- making styles of successful executives and game players were similar. Several longitudinal studies have been undertaken in which a student’s business game performance is compared to some measure of subsequent business career success (e.g., number of promotions, job title, salary level, number of salary increases, management level in the company hierarchy, etc.). Good simulation performance might suggest something about an individual’s managerial skills and, hence, serve as a predictor of later career success. One early longitudinal study (Norris and Snyder 1982) did not find a correlation between business game performance and later career success but two more recent, and more comprehensive, studies have reported such a correlation (Wolfe and Roberts 1986 and Wolfe and Roberts 1993). Four studies have reported that successful business simulation game firms practice strategic management (Gosenpud, Miesing and Milton 1984; Gosenpud and Wolfe 1988; Miesing 1982; and Wolfe and Chanin 1993). In these studies, strategic management was considered to exist when the team developed clear goals, analyzed the external environment in which they were operating, understood their strengths and weaknesses, developed clear strategies as part of a formal plan, monitored their performance, and took corrective action when needed. The research studies cited above have suggested that good simulation performance might be related to student grade point average, student learning in the simulation competition, the personality characteristics of the simulation participants, the decision-making style of the participants, or the degree of formal planning of the superior performing teams. As well, several longitudinal studies have suggested that good simulation 147 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 performers will be more successful in later business careers. If any, or all, of the above is true, this would suggest that good simulation performers should be consistently good over time in repeated simulation competitions. To test this idea, Wellington and Faria (1995) studied 555 students in two rounds of a simulation competition. The students played The Marketing Management Simulation (Faria and Dickinson 1995) in a Principles of Marketing course. Student teams made six decisions, at which point the competition was restarted. The teams were randomly reassigned so that new industries were created with each new industry having a first, second, third, fourth, fifth and sixth place team from the first competition. From this new start, a second round of six decisions were made. The results reported by Wellington and Faria (1995) indicated that there was a strong correlation between performance in the second round of the competition with performance in the first round of the competition. The conclusion reached was that good simulation performers continue to be good performers in repeated competitions. PURPOSE AND HYPOTHESIS The purpose of the present study is to determine whether, in fact, good performers continue to be good performers as stated by Wellington and Faria (1995). While the Wellington and Faria study was well designed, one difficulty with it was that participants played the same simulation in both rounds of the competition. Hence, good performers in the first round likely repeated similar decisions/strategies in the second round. The present study examines if participants can repeat their performance in two different marketing classes, with two different teachers, and using two different simulation games. Based on the findings from previous research and, where previous research is lacking, based on what would seem to be intuitively logical, the following hypotheses have been formulated for testing purposes. H1: In a second round of a simulation competition using a different simulation game, players exhibiting higher rank order performance in the first round of the competition will outperform players exhibiting lower rank order performance. H2: Performance in one round of a simulation competition will be related to performance in the second round of a simulation competition even with the use of a different simulation game. METHODOLOGY The subjects for the research to be reported here were 58 students who took a Marketing Problems and Applications course in which COMPETE: A Dynamic Marketing Simulation (Faria, Nulsen and Roussos 1994) was used, and in an earlier semester, took a Principles of Marketing course in which The Marketing Management Simulation (Faria and Dickinson 1995) was used. In addition to the simulation game being different, the instructor in the two courses was different. The Marketing Management Simulation was designed to be used in introductory marketing courses while COMPLETE is designed for upper level marketing courses. In the Marketing Problems and Applications course all 58 course participants played the COMPETE simulation as single person companies in order that performance could be tracked from the previous simulation competition. In the previous Principles of Marketing course they may have played as individuals or in teams. Only students who had played The Marketing Management Simulation as individuals were included in the study, producing a usable sample of 27 teams. 148 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 Participants were divided into industries of six single person companies for The Marketing Management Simulation competition and participated in an eight period game, while they were divided into industries of five single person companies for the COMPETE competition and played a twelve period game. As the two simulation games were clearly different, the one criticism of the Wellington and Faria (1995) study was overcome. In the Principles of Marketing course, in addition to making decisions in the simulation competition, the participants were required to set sales and earnings objectives. Furthermore, the participants were required to complete a self-report attitude survey to be submitted with each decision. Among other things, the attitude survey measured time spent making each decision; expected company ranking at the end of the competition; simulation enjoyment; simulation learning rating relative to lectures, cases, and readings; perceived appropriateness of the simulation evaluation method being used; and the degree to which the participants felt that their simulation performance reflected their managerial abilities. Finally, simulation performance was measured in terms of final earnings per share with the participants ranked from first to sixth place within their industries. In the Marketing Problems and Applications course, in addition to the simulation competition, the participants were required to follow a predesigned Decision Submission Sequence, which prescribed their market activity. They were also required to compile and complete cash flow statements at the end of periods 4 and 8 but also complete an initial starting report, a period 4 ending report, period 8 ending report, a game ending report and a future marketing plan report. Finally, simulation performance was measured in terms of final earnings per share with the participants ranked from first to fifth within their industries. H1 was tested using the KRUSKAL-WALLIS One-Way Analysis of Variance by Ranks Test (Siegel 1956, pp. 184-194) comparing The Marketing Management Simulation rank order performance and collapsed rank order performance as factor variables versus the COMPETE rank order performance and collapsed rank order performance. H2 was tested by computing a nonparametric rank-order correlation between The Marketing Management Simulation cumulative performance versus the COMPETE cumulative performance. The use of a collapsed ranking measure was instituted because both The Marketing Management Simulation and COMPETE, like most simulations, can produce wide variations in earnings from industry to industry. It was felt that uncovering differences between good and poor simulation performers might require a broader measure to describe good, medium and poor performance but still be based on rankings. Further, there may be little actual difference between a first and second place, third and fourth place, or fifth and sixth place industry ranking position. FINDINGS The overall findings from the KRUSKAL- WALLIS One-Way Analysis of Variance by Ranks Test and the nonparametric rank-order correlation analyses are reported in Tables 1 and 2. The findings, support rejection of both H1 and H2. To test Hi, the simulation teams were divided into rank order groups based on their order of finish (from first to sixth) in The Marketing Management Simulation competition and their order of finish (from first to fifth) in COMPETE. Because of the small sample size, their were no representative finishers who finished sixth in The Marketing Management Simulation so this comparison 149 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 could not be analyzed. A collapsed set of three rankings, good performers (first or second), medium performers (third or fourth in The Marketing Management Simulation and third in COMPETE), and poor performers (fifth or sixth in The Marketing Management Simulation and fourth or fifth in COMPETE), was also used. The nonsignificant KRUSKAL-WALLIS results reported in Tables l.A and 1 .B support the rejection of Hi. Participants who were highly ranked in The Marketing Management Simulation did not outperform less highly ranked performers in the COMPETE competition. H2 examined the relationship between each simulation player’s performance in The Marketing Management Simulation competition and the COMPLETE competition. The findings from the nonparametric rank-order correlation analysis indicate that performance in the two different simulation competitions is not related, hence, H2 is rejected. DISCUSSION AND CONCLUSIONS The research reported here sought to examine how consistent participant performance would be over play in two different marketing simulation competitions involving two different simulation games. The findings indicate there is little relationship between rank order performance in one simulation game versus rank order performance in a second simulation, different game. These findings must, however, be viewed with some caution owing to the small sample size employed (only 27 students). Despite the small sample, which is typical of much simulation research, this study supports the notion that simulation performance is not consistent over time in two different simulations and could be based on a factor such as “luck”. These results are in direct conflict with those reported by Wellington and Faria (1995) who concluded that good decision-making ability carries over from competition to competition, as does poor decision-making or managerial ability. The differences in findings in this study compared to the findings reported by Wellington and Faria (1995) and others beg explanation and further research. By way of possible explanations, there were important differences between this study and earlier studies, which may account for the differences in findings. For example, the findings in this study were based on comparisons of performance in two different simulations, in two different courses offered by two different instructors, with considerable elapsed time between competitions (in some cases, nearly 6 months elapsed between completing the principles course and taking the applications and decisions course). In addition, this study involved examination of individual team players. The work by Wellington and Faria (1995) involved comparisons of simulation performance of “groups" of players in the same course, taught by the same instructor, in the same semester as well as using the same simulation. Aside from this unit of analysis issue, the most likely reason why the results varied could have been due to the "nature" of the COMPLETE competition in comparison to The Marketing Management Simulation competition and the differences in the manner in which the simulations were employed. With respect to the differences in the simulations, The Marketing Management Simulation is less complex than the COMPETE simulation. The Marketing Management Simulation has two products in two markets and asks players to make decisions on pricing, advertising, sales promotion, shipping requests, salesforce size, 150 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 salesforce salaries, salesforce commissions, product research and development and market research. In contrast; COMPETE offers a more complex environment. It has three products in three markets. Players are also asked to make decisions on pricing, advertising, shipping requests, salesforce size, salesforce salaries, salesforce commissions, and market research. However, respondents are also asked to make decisions on salesforce selling time allocations by product and by region and the research and development expenditures can be allocated to reducing product costs, improving product quality, or a combination of these. As such, the decision making demands on competitors in COMPETE are greater and a bit different from those of The Marketing Management Simulation. In light of these facts, the findings could be considered encouraging. It would appear that the COMPETE competition presented a different challenge and learning experience from that of The Marketing Management Simulation for students and, consequently, they responded differently to it and produced different results. As far as how the simulations were employed, in The Marketing Management Simulation competition students were simply asked to demonstrate profit maximization behavior and were not constrained as to the marketing strategy they could pursue. In the COMPETE competition the students were also asked to demonstrate profit maximization but their marketing strategy behavior was constrained to following a scripted approach. It is likely that this constraint caused students to adjust their decision-making approaches since they were asked to profit maximize but they had to follow a more strictly regulated decision-making sequence to do so. Still, it could be argued that this is simply one more “new” rule in a “new” simulation environment that a good decision-maker has to adjust to. Based on the findings from this research. It would seem that good performance in one marketing simulation game is not a good predictor of performance in a second, different marketing simulation. Although past research on playing the same simulation, learning, decision-making style, and later career success research indicates that good performing simulation participants may have a skill that consistently separates them from poor performers, this research indicates that this may not always be true. There are several implications from this research for marketing educators who have been using, or are considering using, marketing simulation games in their classes. Most importantly, if a marketing educator is convinced that marketing simulation gaming is valuable as a pedagogical exercise (e.g., having students deal with uncertainty, provide teamwork experience, etc.) but not sure if playing more than one simulation game will add to the student’s experience. This study indicates that different games do provide different experiences. In conclusion, good and bad simulation performance in one marketing simulation game do not appear to carryover to a different marketing simulation game. Hence, using different simulation games in different courses can produce different learning experiences and different learning outcomes. (References and Tables available on request.) 151 Table of Contents Volume 24, 1997 Incorporating Computer Telephony into the MIS Course A Learner Oriented Infrastructure for videoconferencing Based Distance Education Courses The Identification of Temporally Related Structural Elements of the Experiential Component of Electronic Spreadsheet Tasks Does Involvement Influence Learning from Simulation Participation? Some Relationships with Helpfulness and Performance Outcomes Musings on Business Garne Performance Evaluation Performance on a TE Simulation: What does it represent? The Business Policy Game: An International Simulation - An Assessment Tool Students in Free Enterprise as Experiential Learning The Role of Computer Models in Wargames: A Practitioner's view Using Focus Groups as a Tool to Research Learning: A Demonstration and a Discussion Using Medical Simulations to Teach Multi-Cultural Diversity Using the World Game to Internationalize International Exchange Game Community Services Needs Assessment: An Innovative Approach Introducing Students to Potential Total Quality Management Ethical Dilemmas Privacy in the Workplace: A Situational Analysis Antecedents of Learning in Simulations Demonstrating the Learning Effectiveness of Simulation: Where we are and Where we need to go The Use of Computer Simulations as a Pedagogical Aid in Teaching Management Information Systems Evaluating Simulation Learning in a Distant Learning Instructional Model Financial Engineering of Global Investments The Labor History Game: Playing with the Past Perks Participation Assessing Negotiator's Proficiency with a Negotiation role-play The Art of Negotiating Enhancing Learning and Employee Development through the Assessment of Learning Pedagogy Preferences across selected Dimensions of Culture: A Preliminary Investigation Cooperative Learning: What are we Learning? Effective Use of Mastery Based Experiential learning in a Project Course to improve skills in system analysis and Design Simulations and Learning: Can we prove a relationship? (Seminar) Predicting and Reviewing NYSE Stock Prices by use of Basic Statistical Analysis and Logic Toy Car Depreciation Exercise ABSEL as Home Community: An Interactive Exploration Service Learning: Linking Academic Study to Community Development and Business Enhancement Ability of Efficient Evaluation of Knowledge-Based Management Strategies Corporate Ethics Training Programs Modeling Attributes in Demand Functions of Computerized Business Simulations: An Extension of Teach's Gravity Flow Algorithm Expert Systems Combined with Neutral Networks: Tools to Benefit the Marketing Researcher Computer Game Design: New Directions for Intercultural Simulation Game Designers Consistency in Simulation Performance over Time and Across Simulation Games The Impact of an Artificial Market Leader on Simulation Competitor's Strategies Business Plans, Case Studies, and Total Enterprise Simulations: A Natural Co-existence An Exploration into the Non-Use of Business Simulations The Market Game: Interactive Learning Through Market Simulation Plotting Brand Trajectories with the COMPLETE PPM package: A Market Segmentation Analysis and Positioning Tool Me and Mine Inc: An Exercise in Management Theory Learning to Differentiate Leadership from Managerial Position Business Ethics Survey: A Perspective from the Retail Industry Experiential Learning in Demand Analysis for an Agricultural Commodity Marketing on the Internet: A Pedagogical Exercise College Students Need Simple Computer Simulations, Especially for International Business Courses An Analysis of Student Attitudes, Performance, and Strategies in a Simulation Competition Based on a Controlled Product-Market-Entry Game Structure The Crystal Enterprise: Application of a Non-Computerized Simulation Model in a Process of Organizational Change Threshold: A Windows-Based Behaviorally Oriented Total Enterprise Simulation Coaching Business Game Teams Using a Decision Variable Optimizer Rock and Roll is here to Stay: Enhancing Experiential Pedagogy with Musical Exercises An Example of Business Process Analysis Simulation for Customer Software Support The Use of Business Gaming in Hong Kong Academic Institutions Using the Integrative International Simulation INTOPIA Mark 2000 in a Concentrated MBA Curriculum A Group Experience as an Integrated Part of the Core Management Course Empowered Learning in the Classroom The Inter-Group Interaction: An Innovative Approach to Cooperative Learning Courses that Utilize Student Team: An Approach to enhancing their Effectiveness Thoughts about the Measurement of Learning: The Case for Guided Learning and Associated Measurement Issues Measuring Student Learning Using Business Simulations: A Theory Based Perspective On the Use of PC Fingame in an Undergraduate Finance Course How Managers get Things Done: A Virtual Soundbite Internet Experiential learning in the Principals of Marketing Classroom: A Pedagogical Approach Current Student Perceptions Relative to Business Simulations Beyond Capitalism: Designing Business Policy and Social Justice The Use of Boards of Directors to Evaluate Reports and Presentations in an Undergraduate Business Policy Course Beacon Lumber: An Experiential Introduction to Financial Accounting The Application of Organizational Motivation Principles: The Experiential Business Simulation Motus Manufacturing Avoiding a Bogey: Grading Case Discussions Scientifically Communicating Consumer Behavior II: A Modified Exercise Using Personal Consumption Journals in Condensed Courses Designing Instruments for Assessing the Effectiveness of Simulations, (Seminar) 360º Performance Feedback: Appraisal vs. Assessment The Cafeteria Approach to Managing an Academic Career: Remaining Non-Perishable while Doing Your Own Thing An Experiential Exercise Related to Person-Organization Fit and it's Consequences For Today's Dynamic and Changing World, Business Simulations need to be expanded to encompass Much Greater Complexity: A Demonstration The Incident Process: A Case in Reverse Contextually-Anchored Business Simulations The Energy Factor: Building Motivation in the Simulation Gaming Environment The Design of an Internet Game