SIMULATION PERFORMANCE AND FORECAST Developments in Business Simulation and Experiential Learning, Volume 30, 2003 SIMULATION PERFORMANCE AND FORECAST ACCURACY—IS THAT ALL? John B. Washbush University of Wisconsin-Whitewater washbusj@mail.uww.edu ABSTRACT This study evaluated the proposition that there is a cor- relation between forecasting accuracy and total enterprise simulation performance. Using Mean Absolute Deviation (MAD) over the periods of play as a measure of forecasting accuracy (predicting unit sales), three significant correla- tions were found for the five course sections studied. The study found, therefore, that there is a general correlation between forecasting accuracy over the periods of play and simulation standing. Additionally examined were simula- tion performance compared to a pre-play knowledge-based examination and an applied knowledge final course exami- nation. Not surprisingly, no relationships were found for these latter evaluations. The paper contends that finding a correlation between forecasting accuracy and simulation performance is to be expected and of limited use in assess- ing what can and should be gained from simulation partici- pation. BACKGROUND One of the more common research topics in the use of business simulations has been an effort to determine the individual and group factors that correlate with simulation performance. While there are many factors that have been identified, there is no clear set of findings that consistently predict how participants will perform. Gosenpud and Meis- ing (1983) found that gpa, major, teammate prior acquaint- ance, affection for teammates, the extent of working to- gether, formalized decision-making, degree of organization, and desire to play were positively related to performance. Gosenpud et al. (1984) found that forecasting accuracy, formal planning, strategic stability, having strategies ori- ented to price, strategic clarity, and group cohesion were positively related to performance. Gosenpud et al. (1985) identified positive relationships to group cohesion and the personality factors of self-esteem, need for achievement, and internal locus-of-control. Gosenpud (1987), reviewed research on the extent to which academic ability, major, personality, motivation, team cohesion and organizational formality predicted simulation performance and concluded that performance varies with combinations of variables and that some relationships are conditional. Wellington and Faria (1990) found that committee-decision format groups outperformed regional-decision format groups and reported more learning benefits. Gosenpud and Washbush (1991) identified choosing teammates carefully, grade point aver- age and major. Wellington and Faria (1992) found a strong relationship between beginning team cohesion and perform- ance expectations and final game performance. Patz (1990,1992, and 1999) has argued that teams possessing intuiting and thinking strengths establish and sustain supe- rior performances. Anderson and Lawton (2002) found that the application of previously-learned marketing concepts was positively associated with simulation performance. Recent research has done little to clarify the clutter docu- mented by Burns et al. (1990), however, the power of attrac- tion of the search, despite years of inconclusive research, is reflected in Cannon and Burns (1999) argument for deter- mining underlying competencies that can be assessed by simulation performance. For purposes of the research reported here, of singular importance is the research of Teach (1992) who found con- sistent relationships between forecasting accuracy (using forecasts of market share and product sales and forecasts of cash flow and profits) and profit performance of simulation teams. This finding should not be surprising since the abil- ity to predict what the firm can and will do in the market- place makes business decision making easier and, from a pro forma perspective, more accurate. One would expect that, over time, business organizations that forecast well, improve the likelihood of success. Motivated by this re- search, Washbush and Gosen (2002) examined the relation- ship between cumulative forecasting error, measured by Mean Absolute Deviation, and simulation performance. Results found were inconclusive, however, they provided the motivation to continue this line of investigation. METHOD This study was conducted during the fall and spring semesters of the 2001-2002 academic year using five sec- tions (two in the fall and three in the spring) of the required undergraduate BBA capstone administrative policy course at the University of Wisconsin-Whitewater. Students are assigned to sections of this course on a non-random manner. The simulation used was MICROMATIC (Scott, et al., 1992). The research hypotheses, stated in null form, were: 250 mailto:washbusj@mail.uww.edu Developments in Business Simulation and Experiential Learning, Volume 30, 2003 H1 Simulation performance and forecasting accuracy do not co-vary. H2 Simulation performance does not vary with pre- play knowledge of the simulation. H3 Simulation performance does not vary with post- play applied knowledge of strategic concepts and actions. Hypothesis 1 was motivated as described above. Hypothe- ses 2 and 3 were suggested, in a roundabout way, by previ- ous research into simulation performance and learning (Washbush and Gosen, 2001, 2002). Students were self-selected into groups of three or four for purposes of simulation play. Simulation play was a pri- mary focus during approximately the last 1/2 of each course. Other aspects of the course included strategic management concepts, case analyses and an overview and introduction to the simulation. Simulation play began with a practice deci- sion round. In addition to play, students had to write and submit brief periodic performance analysis reports and a final, overall performance assessment. The grading system included pre-play knowledge of MICROMATIC, simulation standing, simulation reports, and peer evaluations. The final exam reflected both strategic management concepts and issues relating to decision making in the simulation. During both semesters the courses were essentially identical except for minor changes in the weights of various components of the courses in final grade determination Simulation performance was measured using the nor- malized scoring routine that is a component of the simula- tion software. The factors used to determine simulation performance within the game’s scoring routine were total profits (40%), return on sales (30%), and return on assets (30%). Forecast accuracy was determined by requiring each group to prepare and turn in a forecast for sales in each market area for each decision round. Total demand for each period was determined by summing actual sales and lost sales for each area. Forecast error for each round of play was calculated by subtracting forecast sales from actual de- mand and converting to the absolute value. For all periods of play, the mean absolute deviation (MAD) was calculated for each group by summing the absolute errors for each pe- riod of play and dividing that total by the number of periods of play. A smaller MAD indicated greater forecast accu- racy. Data were analyzed using linear regression. For Hy- pothesis 1, simulation performance, a simple linear regres- sion was performed using forecast accuracy (MAD) as the independent variable. Similarly, for Hypothesis 2, student performance on a mid-term examination (given before the start of play and testing knowledge of MICROMATIC) was the independent variable. Finally, for Hypothesis 3, final examination performance served as the independent vari- able. Correlations were also calculated for all variable pairs. RESULTS Table 1 shows results of regression and correlation analyses for the five sections. One regression indicated a significant relationship between MAD and performance (p = .0318). Four of the five correlations were negative, and three were strongly so. This would be expected because lower MAD would indicate more accurate forecasting and, if forecasting effectiveness is related to better performance, the correlation should be negative. Three correlations were statistically significant beyond the 0.01 level. On balance, Null Hypothesis 1 was not confirmed. Table 1 Regression and Correlation MAD vs Performance Academic Term & Section # Students # Groups Play Periods MAD Beta Slope Sig. Adj R2 Correl. Corr. Sig. Fall 2001, Sect 1 31 9 14 -0.0487 0.0318 0.4178 -0.7004 <0.01 Fall 2001, Sect 2 26 8 14 -0.0052 NS -0.1645 -0.0428 NS Spr. 2002, Sect 3 37 8 16 0.0040 NS -0.1399 0.1541 NS Spr. 2002, Sect 4 33 8 16 -0.0157 NS 0.3867 -0.6887 <0.01 Spr. 2002, Sect 5 26 7 17 -0.0277 NS 0.2760 -0.6298 <0.01 Table 2 shows the results of regression analyses and correlations comparing student scores on the pre-play MICROMATIC test to simulation standing. Table 3 shows similarly calculated analyses comparing student scores on the final exam to simulation standing. Because the re- searcher used these as knowledge measures, he analogized the test scores to learning scores developed in previous re- search (as noted above). In general, there was no systematic and consistent correlation between measures of knowledge and simulation performance, and Null Hypotheses 2 and 3 were accepted. While these analyses lack depth and rigor, they are consistent with prior findings (Washbush and Go- sen, 2001, 2002). There is no compelling evidence of any knowledge/learning-performance relationship. 251 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 Table 2 Regression and Correlation Micromatic Test vs Performance Academic Term & Section # Students Micromatic Test Beta Slope Sig. Adj R2 Correl. Corr. Sig. Fall 2001, Sect 1 31 1.1206 NS -0.0073 0.1623 NS Fall 2001, Sect 2 26 2.1831 0.0342 0.1385 0.4159 <0.05 Spr. 2002, Sect 3 37 0.8210 NS 0.0669 0.3046 NS Spr. 2002, Sect 4 33 0.5360 NS -0.0256 0.0801 NS Spr. 2002, Sect 5 26 0.6559 NS -0.0103 0.1735 NS Table 3 Regression and Correlation Final Exam vs Performance Academic Term & Section # Students Final Exam Beta Slope Sig. Adj R2 Correl. Corr. Sig. Fall 2001, Sect 1 31 1.1866 NS 0.0766 0.3276 NS Fall 2001, Sect 2 26 2.2253 NS 0.0989 0.3673 NS Spr. 2002, Sect 3 37 0.8962 0.0418 0.0876 0.3361 <0.05 Spr. 2002, Sect 4 33 0.0130 NS -0.0323 0.0020 NS Spr. 2002, Sect 5 26 -0.4816 NS -0.0294 0.1084 NS DISCUSSION The major findings in this study are consistent with Teach’s (1992) findings that forecasting accuracy correlates with simulation performance. This is scarcely surprising since, from a pro forma perspective, reasonably accurate forecasts must lead to reasonably accurate projections of revenues, expenses and cash flows. Additionally, it is rea- sonable to expect a learning curve effect over time (seen here to a degree) encouraging better forecasting. However, there are legitimate questions that can be raised about Teach’s assertion that forecasting is a critical measure of managerial competence and therefore uniquely important to evaluation of participants in simulations. Such an assertion shortchanges both managers and students of management. Clearly, managing requires far more than developing some facility with forecasting. Moreover, there is far more that students can learn about management (and themselves) from the simulation. Like it or not, real strategic managers are held to rather rigorous profit performance standards. This may not be fair or even completely logical, but it is reality. For the student, the simulation provides an opportunity to do a number of things that are generally missed in most of the non-simulation experiences they encounter in they typical business course of study. Among these are: • Taking responsibility for the OUTCOMES of deci- sions • Problem finding • Identification of key strategic concerns through analysis of performance data • Testing aptitude for and desire to manage • Assessing willingness to take risks • Assessing personal performance under risk-stress conditions • Developing other abilities relevant to an organiza- tional career Washbush and Gosen (2002) have argued that simula- tions are consistent with and complement the learning envi- ronment of the traditional policy course. There continues to be a need to explore that that issue. There exist, no doubt, a plethora of issues, outcomes, behaviors and potentials. Looking for a magic bullet for assessment is tempting, but the richness of the simulation-learning environment de- mands more. It is also important to note that Wolfe and Roge’ (1997) have identified a number of important elements of learning common to most popular total enterprise simulations. These include strategy, environmental analysis, forecasting, mar- ket development and penetration, cost and differentiation strategies, and performance measures. Assessing learning in all such measures would seem to be relevant information for students who participate in the simulation experience. However, their paper also implies that another important aspect of the simulation environment is instructor intent. Instructors using simulations should clearly determine what they want the simulation to do. Those intentions may be unique to the individual instructor, but they are nonetheless valid to that specific situation. It is therefore important for the instructor to not only be aware of the learning potential of a given simulation, but that person should consciously determine how that simulation best complements the course and its objectives. Assessment of student learning should reflect all of these realities. As researchers continue to examine learning potential and learning methods in the simulation, the likelihood is that a complex array of findings will emerge, as it has to an ex- tent already. Such results should not be viewed as problems to be transcended. Rather they should be seen as reflective of the complexity inherent in organizations and organiza- 252 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 tional studies. The user of a simulation should employ it with wide-open eyes, armed with the best information avail- able about the simulation to be used and its potential. No “one size fits all” approach is desirable or even possible. In the end, instructors must do what they have always done, use best efforts to design and assess the learning environ- ment. Efforts aimed at helping instructors exercise those unavoidable responsibilities should continue, but with ac- ceptance and appreciation of the clutter which will inevita- bly result. REFERENCES Anderson, P. & Lawton, L. (2002). Is simulation perform- ance related to application? An exploratory study. De- velopments in Business Simulation and Experiential Learning, Vol. 29, 108-113. Burns, A, Gentry, J. & Wolfe, J. (1990). A cornucopia of considerations in evaluating the effectiveness of experi- ential pedagogies. In J.W. Gentry (ed.), A Guide to Business Gaming and Experiential Learning. East Brunswick, NJ: Nichols/GP, 253-278. Cannon H. & Burns, A. (1999). A framework for assessing the competencies reflected in simulation performance. Developments in Business Simulation and Experiential Exercises, Vol. 26, 40-44. Gosenpud, J. (1987). Research on predicting performance in the simulation. Developments in Business Simulation and Experiential Exercises, Vol. 14, 75-79. Gosenpud, J. & Meising, P. (1983). Determinants of per- formance in the simulation. Developments in Business Simulation and Experiential Exercises, Vol. 10, 53-56. Gosenpud, J., Meising, P. & Milton, C. (1984). A research study on the strategic decisions in a business simula- tion. Developments in Business Simulation and Experi- ential Exercises, Vol. 11, 161-165. Gosenpud, J., Meising, P. & Larson, A. (1985). Predicting performance over the course of the simulation. Devel- opments in Business Simulation and Experiential Exer- cises, Vol. 12, 5-10. Gosenpud, J. & Washbush, J. (1991). Predicting simulation performance: differences between groups and individu- als. Developments in Business Simulation & Experien- tial Exercises, Vol. 18, 44-48. Patz, A. (1990). Group personality composition and total enterprise simulation performance. Developments in Business Simulation and Experiential Learning, Vol. 17, 132-137. Patz, A. (1992). Personality bias in total enterprise simula- tions. Simulation & Gaming: An International Journal, Vol 23, 45-76. Patz, A. (1999). Overall dominance in total enterprise simu- lation performance. Developments in Business Simula- tion and Experiential Learning, Vol. 26, 115-116. Scott, T., Strickland, A., Hofmeister, D., & Thompson, M. (1992). MICROMATIC: A Management Simulation. Boston: Houghton-Mifflin. Teach, R. (1992). Using forecasting accuracy as a measure of success in business simulations. Developments in Business Simulation & Experiential Exercises, Vol. 16, 103-107. Washbush, J. & Gosen, J. (2001). An exploration of game- derived learning in total enterprise simulations. Simula- tion and Gaming: An International Journal. Vol. 32, No. 3, 281-296. Washbush, J. & Gosen, J. (2002). Total enterprise simula- tion learning compared to traditional learning in the business policy course. Developments in Business Simulation and Experiential Exercises, Vol. 29, 281- 286. Wellington, W. & Faria, A. (1990). The effect of decision format and evaluation on simulation performance, deci- sion time, and team cohesion. Developments in Busi- ness Simulation and Experiential Learning, Vol. 17, 170-174. Wellington, W. & Faria, A. (1992). An examination of the effect of team cohesion, player attitude, and perform- ance expectations on simulation performance results. Developments in Business Simulation and Experiential Learning, Vol. 19, 184-189. Wolfe, J. & Roge′, J. (1997). Computerized general man- agement games as strategic management learning envi- ronments. Simulation & Gaming: An International Journal, 28, 423-441. 253 Table of Contents Volume 30, 2003 The Optimal Timing For Introducing Business Simulations Can Handicapped Students Access Your Class Web Site? The Competition Game: Decision Making In A Dynamic Environment Pan-Pacific Enterprises: Strategic Decision Making Simulation Study Of Stochastic Channel Redistribution The Impact Of Business War Games: Quantifying Training Effectiveness Experiential Learning: Introducing Faculty And Staff To A University Leadership Development Program The Feasibility Of The Balanced Scorecard For Business Games A Misuse Of Pims For The Validation Of Marketing Management Simulation Games Incorporating Technology Into The 21st Century Classroom: Are We Facilitating Academic Dishonesty? Improving The Effectiveness Of Peer Evaluations The Use Of A Simulation In An Integrated Mba Curriculum Student Portfolios In Business Education Student Portfolios In Business Education At Ashland University Using SAP ERP Technology To Integrate The Undergraduate Business Curriculum Board Games And Teaching Textile Marketing And Finance Blogging: A New Threat To Student Research? The Way We Talk! Take II Strategic Management: An Evaluation Of The Use Of Three Learning Methods In Hong Kong Adoption Of Discussion-Based Teaching And Assessment In Teaching Strategic Management In Hong Kong The Tobin Q As A Company Performance Indicator Using Representative Nominal Group Technique For Course Review And An Interactive Solicitation Of Ways To Enhance Absel's Image Making Teaching Matter: The Art And Science Of Teaching Business Communication Beyond Sex, Age, And Race: Exploring The Deeper Contents Of Diversity Teaching & Learning The Facilitation Process A Brief On Debriefing: What It Is And What It Isn't Changing Perceptions Of The Importance Of Leadership: The Contribution Of Individual Spirit Harmonics In Leadership Knowledge, Skills And Sustainable Values In Learning Organizations: Some Implications From The Multicultural Virtual Classroom Challenges Of Teaching Undergraduate Organizational Behavior In A Nontraditional Time Format Interactive Online Positioning With The Web-Based Product Positioning Map Graphics Package The Longitudinal Effects Of Entrepreneurship Training On Risk Tolerance: A Look At Similarities And Differences Between Male And Female Undergraduate Students Revisiting Strategy Learning In A Total Enterprise Simulation What Are Simulations For?: Learning Objectives As A Simulation Selection Device Gaming Agency Markets Cooperate For Profits Or Compete For Market? Study Of Oligopolistic Pricing With A Business Game The Design Of A Business Simulation Using A System-Dynamics-Based Approach Modeling The Product Development Function For An Entrepreneurial Firm Simulation Performance And Forecast Accuracy? Is That All? Business Manager Identification Of Competitors In Real World And Simulation Settings Monte Carlo Simulation Analysis On The Costs Reduction Argument Of Interest Rate Swaps Ebiz Game: A Scalable Online Business Simulation Game For Entrepreneurship Training A Model For Online Education Delivery And A Look At Online Delivery Effectiveness Incorporating "Company Reputation" into Total Enterprise Simulations The Genesis And Future Of The Absel "Classicos" Initiative