AN INVESTIGATION OF THE ENVIRONMENTAL AWARENESS ATTAINED IN A SIMPLE BUSINESS SIMULATION GAME Developments in Business Simulation and Experiential Learning, Volume 28, 2001 AN INVESTIGATION OF THE ENVIRONMENTAL AWARENESS ATTAINED IN A SIMPLE BUSINESS SIMULATION GAME William Wellington, University of Windsor R87@UWINDSOR.CA A. J. Faria, University of Windsor AD9@UWINDSOR.CA ABSTRACT Past research examining participant adaptability to game parameters in computerized business simulation games has examined the degree to which game participants understand the environment into which they are placed. Results suggest that participants only moderately understand their environments. It is felt that the complexity of the simulations used in these studies contributed to the lack of significant findings. This study uses a very simple simulation game in which the game administrator can only manipulate two game parameters. Decision responses were gathered from 331 single player competitive companies assigned to fifty-nine six team industries for a nine period competition. As with past studies, only moderate learning of the simple environment was found. INTRODUCTION Good business managers make good decisions. To make good decisions business managers must understand the marketplace environment in which they are competing. If business simulation games are to provide good learning opportunities for business students, students must be able to understand the environment created for the simulation competition and develop strategies that adapt to that environment. This would serve as clear evidence of learning in the simulation environment (Gentry, Stoltman & Mehlhoff 1992; Hatton, Hatton & Mecord 1992; Washbush & Gosenpud 1994; Washbush & Gosenpud 1995; Wolfe & Roberts 1992). This study adds to an ongoing stream of research pursuing the concept of simulation participation validity predicated on the extent to which participants understand and respond to the simulation environment in which they are placed. Game administrators assume that active participation in the simulation provides participants with the opportunity to learn from their experiences and improve their decision-making skills. Traditionally, game performance outcomes, such as earnings per share or return on investment, are used as measures of game performance success and learning. When a participant outperforms a competitor, it is assumed that the winner has better understood the simulation environment and has translated that learning into better decisions. Rather than simply measuring performance outcomes, asking participants to articulate their understanding of the simulation environment is another way to measure learning. PAST RESEARCH Learning theory (Schiffman & Kanuk 1987) would suggest that underlying the behavioral decisions made by a simulation participant is a learning process that leads to the determination of what types of decisions work (e.g., low price in a price sensitive market). Several studies have examined participant decision-making response to artificially manipulated game parameters (Faria & Dickinson 1990; Faria, Whiteley & Dickinson 1990; Whiteley, Faria & Dickinson 1990). In each of these three studies, simulation participants were randomly assigned to "push" responsive or "pull" responsive marketplaces. Push and pull strategies are well documented in the marketing literature. Push strategies focus on channel middlemen while pull strategies focus on the household consumer. The results reported in these studies suggested that the participants' decisions only moderately reflected the importance weightings of the game parameters which were manipulated to create the push and pull environments. A study undertaken by Dickinson and Faria (1994) utilized an administrator created company. In this study, an artificial competitor company was created, and inserted into each competitive industry, using a randomly generated set of decisions. The decisions of the administrator created company were controlled to be within the upper and lower limits of the real competitors in each industry. The purpose of the study was to determine if the real competitors, developing strategies based on the what they learned during the competition, could defeat an artificial competitor utilizing a randomly generated strategy. Overwhelmingly, 239 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 the real companies outperformed the random strategy companies. Two previous studies examined the impact of an artificial industry leader (Wellington, Dickinson & Faria 1991; Wellington & Faria 1997). The decisions of the artificially created industry leader were designed to be perfectly in tune with the industry environment. Student participants could learn from the industry leader and, accordingly, better adapt their decisions to the manipulated environment. Only moderate decision-making adaptation to the environment was reported. While a number of studies have focused on the behavioral side of decision-making, research examining the cognitive decision-making process is light (Whiteley, Dickinson & Faria 1992; Wellington, Faria, Whiteley & Nulsen 1995; Wellington, Faria & Whiteley 1998). These studies reported limited cognitive understanding of the simulation environment and limited correct behavioral response to the manipulated simulation environment. Further, even in cases where participants understood their environment (cognitive learning) they often made incorrect decisions (behavioral learning). PURPOSE AND METHODOLOGY The results from past research suggest that game participants have been only moderately successful at understanding and adapting to their simulation environments. However, past research studies have utilized relatively complex simulation games. The present study utilizes a very simple marketing simulation. In addition, the participants have a good decision support system in the simulation that allows them to identify and track environmental variables rather easily. The simulation used for this study is PAINTCO V (Galloway, Evans, Berman & Wellington 1997). The game administrator is able to manipulate only two environmental variables in the competition: level of demand and raw material cost. Participants operate companies that manufacture paint for the retail and organizational markets. Only five marketing decisions are made each period: product quality, distribution, advertising, personal selling, and price. The subjects for this study were 331 students in two sections of a Principles of Marketing course. The simulation competition amounted to 20 percent of the course grade. The 331 participants were divided into 59 industries of six teams each. Each participant operated as a single person company in the competition. The competition covered nine decision periods comprised of one trial period and eight real periods. During each decision period, participants submitted their decisions which included an estimate of demand level and the raw materials index - the two manipulated environmental variables. HYPOTHESES The general hypothesis for this study, as with previous studies in this series, is that if simulation games are to be a meaningful learning tool, participants must exhibit some learning from the simulation experience. In this case, the learning measure used was the participants' ability to track and correctly forecast the demand level and raw material index in the PAINTCO V simulation. As learning occurs during the play of the simulation, the participants' accuracy in forecasting these environmental variables should improve. As well, top performing teams should exhibit a greater awareness of the actual game parameters than lower performance competitors. The specific hypotheses formulated for testing were: H1: The variance between the actual and estimated raw material index will decrease from competition Period 1 through Period 8. H2: The variance between the actual and estimated seasonality in the demand index will decrease from competition Period 1 through Period 8. H3: The variance between the actual and estimated raw material index will be smaller for top ranked competitors (first or second in their industry) than for medium ranked (third or fourth in their industry) and lower ranked competitors (fifth or sixth in their industry). H4: The variance between actual and estimated seasonality in the demand index will be smaller for top ranked competitors (first or second in their industry) than for medium ranked (third or fourth in their industry) and lower ranked competitors (fifth or sixth in their industry). Hypotheses H1 and H2 were tested using SPSS Reliability Analysis to compare the change in variance between the actual and forecasted game parameters throughout the competition. Hypotheses H3 and H4 were tested by comparing the average variances between the top performing companies (first or second place) on the two manipulated game parameters and the medium ranked (third or fourth place) and lower ranked simulation competitors (fifth and sixth place). RESULTS The actual values of the seasonality and raw material indices are reported in Table 1 along with the mean estimates made by the top, medium and low performers. The results of the reliability analyses for H1 and H2 are presented in Tables 2 and 3. The results of the MANOVA analysis for H3 and H4 are reported in Tables 4 and 5. 240 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 The reliability analysis supports the partial acceptance of H1. The average variance of the estimates of the raw material index declined from Period 1 through to the middle of the competition. However, by the end of the competition, the variance had increased. An ANOVA analysis of the repeated measures of this index indicated that the change in value between measures was significant. This supports H1 but only partially because the expectation was that the absolute value of the difference would continue to decline throughout the competition. With respect to H2, the pattern that emerged was the opposite of what was hypothesized. The mean of the absolute value of the variance between the actual and expected seasonality index increased from period to period. The changes were significantly different but they were in the wrong direction. As such H2 is rejected. The MANOVA analysis of H3 indicates that the better performing teams understood the raw material index better than the poorer performing teams over time. At the outset the mean absolute variance of the raw material index was the same for both groups. However, as the simulation progressed, the top performers had a smaller variance than the poorer performers. Overall the MANOVA results were significantly different indicating the top performing teams were able to predict the raw material index better than the poorer performers. As such, H3 is accepted. With respect to the seasonality index, H4 is also supported as shown by the significant MANOVA results. The analysis of variance results of the individual periods indicate two significant differences in the middle and at the end of the competition. In both instances the top performers' predictions of the seasonality index were superior to the medium and low performers. DISCUSSION AND CONCLUSIONS As has been the case in past studies, the results from this study are mixed. The acceptance of H1, H3 and H4 indicates that participants were able to perceive changes in the raw material index over time and that top performers were better able to identify the raw material index and the seasonality of demand than were poorer performers. As such, there is some indication that participants were able to discern an uncontrollable variable in their environment. However, even in this very simple simulation and with a decision support system to help them, participants did not become more accurate in their forecasts of seasonality over the course of the competition. While the participants were able to determine that changes were occurring, they were not always able to determine the scoop of those changes. The results from this research are very similar to the results reported in earlier studies. Once again, a group of introductory marketing students could not fully understand the nature of their simulation environment. The participants were able to perceive that their simulation environments were changing but they could not always identify the true nature of the changes. The fact that participants did not fully understand their marketplace environment, even in a very simple simulation and with a decision support system to help them, is surprising and disturbing. If simulation participants cannot recognize the true nature of their business environments, even in very simple settings, one must ask what is being learned and how are decisions being made? It is significant to note that the participants were not oblivious to their environments, they did note changes, but they were unable to accurately identify them. The study findings indicate that top performers had a better understanding of their environment than weaker performers - this would be expected. Regardless, it is also true that top performers did not have a "true" understanding of the environment in which they were operating. The first conclusion that can be drawn from this research is that top performers adapt faster and more appropriately to their simulation marketplace environment than poorer performers. This is not a particularly surprising finding. This finding suggests, as has previous research that a better understanding of the game environment will lead to superior performance and, secondly, one can rule out "luck" as a determining factor in top performance. Top performance is the result of better understanding. A second conclusion from this study is that participants do not truly understand the actual nature of their simulation environments. This is a very uncomfortable finding for simulation game users. REFERENCES Dickinson & Faria (1994), "A Random Strategy Criterion for Validity of Simulation Game Participation," Developments in Business Simulation & Experiential Learning, Volume Twenty-One, 35-40. Faria & Dickinson (1990), "Extant Measures of Simulation Validity and an Addition," Society for Computer Simulation, Volume Twenty-Two, Number Two, 66- 71. Faria, Whiteley & Dickinson (1990), "A Measure of the Internal Validity of a Marketing Simulation Game," Proceedings of the Southwest Decision Sciences Institute, Volume Thirty-Five, 133-141. Galloway, Berman, Evans & Wellington (1997), PAINTCO V, Prentice Hall Canada Inc.: Scarborough, Ontario. Gentry, Stoltman & Mehlhoff (1992), "How Should We Measure Experiential Learning?" Developments in Business Simulation & Experiential Learning, Volume Nineteen, 54-58. 241 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 Hatton, Hatton & Mecord (1992), "Key Determinants and Decision Making Performance in a Business Simulation and Experiential Learning Environment," Developments in Business Simulation & Experiential Learning, Volume Nineteen, 77-82. Schiffman & Kanuk (1987), Consumer Behavior, Prentice- Hall, Inc.: Englewood Cliffs, New Jersey. Washbush & Gosenpud (1994), "Simulation Performance and Learning Revisited," Developments in Business Simulation & Experiential Exercises, Volume Twenty- One, 83-87. Washbush & Gosenpud (1995), "Simulation Performance, Learning, and Struggle," Developments in Business Simulation & Experiential Exercises, Volume Twenty- Two, 1-4. Wellington & Faria (1997), "The Impact of an Artificially Created Market Leader on Simulation Competitors' Strategies," Developments in Business Simulation & Experiential Learning, Volume Twenty-Four, 152-158. Wellington & Faria (1998), "Holistic Cognitive Strategy in a Computer-Based Marketing Simulation Game: An Investigation of Attitudes Towards the Decision- Making Process," Developments in Business Simulation & Experiential Learning, Volume Twenty- Five, 246-253. Wellington, Faria, Whiteley & Nulsen, "Cognitive and Behavioral Consistency in a Computer-Based Marketing Simulation-Game Environment," Developments in Business Simulation & Experiential Exercises, Volume Twenty-Two, 12-19. Wellington, Dickinson & Faria (1991), "The Impact of a Market Leader on Simulation Participant Strategies," Society for Computer Simulation, Volume Twenty- Three, Number Two, 33-43. Whiteley, Faria & Dickinson (1990), "The Impact of Market Structure, Versus Competitor Strategy, on Simulation Outcomes," Proceedings of the Academy of Marketing Science, Volume Thirteen, 279-288. Wolfe & Roberts (1992), "Peer Group Indicators of the External Validity of Business Games: A Five Year Longitudinal Study," Developments in Business Simulation & Experiential Exercises, Volume Nineteen, 194-199. TABLE 1 242 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 ACTUAL RAW MATERIAL AND SEASONALITY INDICES AND ESTIMATED RAW MATERIAL AND SEASONALITY INDICES BY PERFORMANCE GROUP Period 1 - Raw Material index Estimated Values Mean Standard Deviation N Top Ranked 1.093 .602 82 Med Ranked 1.068 .519 68 Low Ranked 1.288 1.005 34 Actual Value 1.500 Period 3 - Raw Material index Estimated Values Mean Standard Deviation N Top Ranked 1.060 .354 82 Med Ranked 1.096 .305 68 Low Ranked 1.212 .806 34 Actual Value 1.300 Period 6 - Raw Material index Estimated Values Mean Standard Deviation N Top Ranked 1.035 .300 82 Med Ranked 1.106 .396 68 Low Ranked 1.174 .673 34 Actual Value 0.900 Period 8 - Raw Material index Estimated Values Mean Standard Deviation N Top Ranked 0.984 .270 82 Med Ranked 1.009 .300 68 Low Ranked 1.206 .841 34 Actual Value 0.700 243 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 TABLE 1 (Continued) Period 1 - Seasonality Index Estimated Values Mean Standard Deviation N Top Ranked 23.756 2.553 86 Med Ranked 24.235 2.623 81 Low Ranked 22.938 2.470 48 Actual Value 22.000 Period 3 - Seasonality Index Estimated Values Mean Standard Deviation N Top Ranked 27.628 1.511 86 Med Ranked 26.975 1.533 81 Low Ranked 26.938 2.067 48 Actual Value 30.000 Period 6 - Seasonality Index Estimated Values Mean Standard Deviation N Top Ranked 22.453 2.380 86 Med Ranked 23.012 2.658 81 Low Ranked 23.792 3.383 48 Actual Value 26.000 Period 8 - Seasonality Index Estimated Values Mean Standard Deviation N Top Ranked 27.023 1.841 86 Med Ranked 26.309 2.183 81 Low Ranked 26.271 2.421 48 Actual Value 34.000 244 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 TABLE 2 RELIABILITY ANALYSIS FOR H1 MEAN ABSOLUTE VALUE OF THE VARIANCE OF RAW MATERIAL INDEX MEAN STANDARD DEVIATION N PERIOD 1 .5750 .5108 184 PERIOD 3 .3620 .3432 184 PERIOD 6 .2387 .3867 184 PERIOD 8 .3734 .4144 184 Analysis of Variance Results Sum of Mean Source of Variation Squares DF Square F Between People 91.16 183 .50 .000* Within People 46.59 552 .08 Between Measures 9.66 3 Residual 36.93 549 .07 Total 137.75 735 .19 Grand Mean .3923 ___________________ * significant at < .05 245 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 TABLE 3 RELIABILITY ANALYSIS OF H2 MEAN ABSOLUTE VALUE OF THE VARIANCE OF THE SEASONALITY INDEX MEAN STANDARD DEVIATION N PERIOD 1 2.5535 1.8101 215 PERIOD 3 2.7814 1.8173 215 PERIOD 5 3.6512 1.8806 215 PERIOD 7 7.4140 2.1314 215 Analysis of Variance Results Source of Sum of DF Mean Square F Variation Squares Between People 641.40 214 2.99 .000* Within People 5788.00 645 8.97 Between Measures 3292.54 3 Residual 2495.46 642 3.89 Total 6429.40 859 7.48 Grand Mean 4.1000 ___________________ * significant at < .05 246 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 TABLE 4 MEAN COMPARISONS AND ANALYSIS OF VARIANCE RESULTS BY PERIOD FOR RAW MATERIAL INDEX AND SEASONALITY INDEX BY PERFORMANCE GROUP Absolute Value of the Variance of the Raw Material Index Period 1 - Raw Material index Mean Standard Deviation N F-Value Sig. Top Ranked .568 .451 82 .37 .690 Med Ranked .550 .390 68 Low Ranked .641 .795 34 Period 3 - Raw Material index Mean Standard Deviation N F-Value Sig. Top Ranked .362 .226 82 1.82 .166 Med Ranked .316 .184 68 Low Ranked .453 .668 34 Period 6 - Raw Material index Mean Standard Deviation N F-Value Sig. Top Ranked .204 .258 82 1.51 .224 Med Ranked .303 .326 68 Low Ranked .303 .660 34 Period 8 - Raw Material index Mean Standard Deviation N F-Value Sig. Top Ranked .318 .228 82 2.50 .085 Med Ranked .374 .213 68 Low Ranked .506 .841 34 247 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 TABLE 4 (Continued) Absolute Value of the Variance of the Raw Seasonality Index Period 1 - Seasonality Index Mean Standard Deviation N F-Value Sig. Top Ranked 2.547 1.707 86 2.92 .056 Med Ranked 2.852 1.924 81 Low Ranked 2.062 1.630 48 Period 3 - Seasonality Index Mean Standard Deviation N F-Value Sig. Top Ranked 2.372 1.511 86 3.76 .025* Med Ranked 3.025 1.936 81 Low Ranked 3.104 2.003 48 Period 6 - Seasonality Index Mean Standard Deviation N F-Value Sig. Top Ranked 3.919 1.689 86 1.67 .191 Med Ranked 3.556 1.817 81 Low Ranked 3.333 2.253 48 Period 8 – Seasonality Index Mean Standard Deviation N F-Value Sig. Top Ranked 6.977 1.971 86 3.47 .048* Med Ranked 7.691 2.183 81 Low Ranked 7.729 2.421 48 ___________________ * significant at < .05 248 Developments in Business Simulation and Experiential Learning, Volume 28, 2001 TABLE 5 MULTIPLE ANALYSIS OF VARIANCE RESULTS FOR H3 AND H4 H3 H4 RAW MATERIAL INDEX SEASONALITY INDEX MANOVA RESULTS Pillai's Value .09536 .07457 Degrees of Freedom 8 8 Approximate F 2.24059 2.01864 Significance .024* .041* ___________________ * significant at < .05 249 Table of Contents Volume 28, 2001 Using A Web Based Tutorial Program To Enhance Student Learning Reflecting On Reflection: An Examination Of Reflective Learning And Assessing Outcomes Overcoming Obstacles To A Cross-Global Collaborative Classroom Panel Discussion On Building And Maintaining Trust In The Abselesque Classroom Cosmopolitan-Based Cross National Segmentation In Global Marketing Simulations Strategic Management And The Case Method: Survey And Evaluation In Hong Kong Designing Interactive Self-Learning Modules Using Macromedia Director Teaching Financial Markets Using The Wall Street Journal Core Course Outcomes Assessment And Program Continuous Improvement Using An Integrative Business Plan Œ An Empirical Evaluation Using A Mock Trial Activity For Interdisciplinary Learning Fidelity, Verifiability, And Validity Of Simulation: Constructs For Evaluation 2001 International Collegiate Business Strategy Competition: 37 Years Of Collegiate Competition Motivating Students: An Initial Attempt To Operationalize The Curiosity Gap Model E-Learning: The Next Wave Of Experiential Learning Helping New Game Adopters: Four Perspectives An Initial Validity Invesitigaion of A Test Assessing Total Enterprise Simulation Learning Administering A Marketing Simulation - Common And Varied Practices Among Instructors Issues In Case Method Instruction: Class Discussion Leadership Sex Composition, Cohesion, Consensus, Potency And Performance Of Simulation Teams Integrating Management Curriculum: Linking Courses With Frontpage Absel: The Way We Talk! Teaching Strategic Management: A Case Study In The Diffusion Of Innovation In Education A Demonstration Of Bussim Strategy A Total Enterprise Simulation Teaching Methods: Etic Or Emic Antecedents Of Work Team Performance In A Business Simulation: Personality And Group Interaction When Experiential Educators Reflect Upon Their Craft: A Qualitative Research Report Strategic Sensemaking In Organizations: Model Formulation And Operationalization Panel Discussion: A Model For Initiative Development: Discussion And Ideas Of Individual Spirit Harmonics Assessing The Efficacy Of Experiential Learning In A Multicultural Environment Are Business Schools Producing 21st Century Managers? Blue Horizon Cruises: An International Simulation Exercise To Illustrate Conflicting Economic And Cultural Models Different Perceptions Of The Same Thing: An Experiential Exercise Current Use Of Agribusiness Simulation Games: Survey Results Of University Agribusiness And Agricultural Economics Programs Team Building 101 For Accountants An Online Evaluation Of The Compete Online Decision Entry System (Codes) Total Enterprise Simulation Winners And Losers: A Preliminary Study Gilding The Lily, Enhancing The Pedagogical Experience, Or Fixing What Isn™t Broke: The Dilemma Of New Simulation Versions The Absel Endnote Database: The Perfect Tool For The Bernie Keys Library Decision-Making Exercises (A), (B), And (C): Examining The Role Of Conflict In Group Decision-Making Group Decision Simulation (A)-(F): Examining Procedural Justice In Group Decision-Making Processes Evaluation Of Performance In Business Games: Financial And Non Financial Approaches A Framework For Evaluating Simulations As Educational Tools Congruence II™: A Strategic Business Board Game The Tournament Concept: Extending The Idea Of Using Simulations As Instruments Of Assessment The Economic And Political Transition Of The Mexican State In The Threshold Of Twenty First Century: From The Entrepreneurial State To The State Of Entrepreneurs. Experiential Learning For Accountants: The Not For Profit Project An Investigation Of The Environmental Awareness Attained In A Simple Business Simulation Game A Daily Exercise In Theory Application: Is This Why I Had To Take Marketing, Management, Finance –? An Alternative View to Scoring Selected Simulation Games and Experiential Exercises