TOTAL ENTERPRISE SIMULATION LEARNING COMPARED TO TRADITIONAL LEARNING IN THE BUSINESS POLICY COURSE Developments in Business Simulation and Experiential Learning, Volume 29, 2002 TOTAL ENTERPRISE SIMULATION LEARNING COMPARED TO TRADITIONAL LEARNING IN THE BUSINESS POLICY COURSE WASHBUSH, JOHN UNIVERSITY OF WISCONSIN-WHITEWATER washbusj@uww.edu GOSEN, JERRY UNIVERSITY OF WISCONSIN-WHITEWATER gosenpuj@uww.edu ABSTRACT This paper reports a study that examined the relationship between total enterprise simulation learning and learning developed in traditional components of the undergraduate business policy course. Also examined were the relation- ships between learning and simulation performance, be- tween learning and forecast accuracy, and between simula- tion performance and forecast accuracy. Learning was measured using researcher-developed tests administered at the beginning, after the completion of traditional compo- nents (theory and cases), and after the simulation-only components of the courses. Significant learning was found for both the traditional and the simulation components of the sections studied. There was no relationship found be- tween simulation learning and simulation performance or for simulation learning and forecast accuracy. Inconsistent results were found regarding simulation performance and forecast accuracy. These results suggest that the simulation effectively complements traditional approaches to the policy course and that forecast accuracy may be a proxy for simu- lation performance. BACKGROUND This paper continues a series of research studies explor- ing business undergraduate student learning associated with participating in the play of a total enterprise (TE) simula- tion. Based on our research efforts, we have concluded that simulation participants learn what the simulation has to teach (i.e., begin to master the skills and concepts presented in the simulation used) and that the simulation is a valid learning methodology. Additionally, while documenting that participants learn, we have also consistently found that simulation performance and learning do not co-vary (Washbush & Gosen, 1993, 1994, 1995, 2001; Gosen & Washbush, 1996). This study is then a logical extension of our previous efforts to document the nature, process and extent of learn- ing accompanying simulation play by asking increasingly complex research questions. It is also faithful to the charge we made several years ago (Gosen & Washbush, 1999) that there now exist many issues that should be researched to improve the understanding of the impact of teacher behavior on simulation learning. Our studies have consistently been designed to analyze and evaluate learning that resulted from playing the game itself rather than evaluating the degree to which game play obtained a particular course's learning objectives. All other studies on the teaching effectiveness of business games have focused on identifiable skills or particular subject matter learning, such as the domain of strategic management, asso- ciated with a particular course. For a review of this litera- ture see Wolfe (1997). Because there exist twin focuses (learning in the simu- lation vs. learning in the course), it is reasonable to assess whether or not our methods of learning evaluation are perti- nent to both domains. Such a study could attempt to answer these questions: • Is simulation learning documented from the tradi- tional part (strategic management theory and case analysis) of the capstone policy course? • Does learning occur only in the traditional or simu- lation components of the course, or does it occur in both? • Does there exist complementary learning between traditional and simulation components of the course? • Is there a relationship between a group’s ability to forecast sales (in units of product) and (a) simula- tion learning and (b) performance in the simula- tion? The latter question was suggested for addition because Teach (1989) found that profitability-forecasting accuracy, a measure of learning about the firm's competitive environ- ment, correlated with measures of profits. While Teach used profitability forecasting, this study evaluated sales-unit forecasting accuracy given that the simulation used is a sin- gle product game. 281 mailto:washbusj@uww.edu mailto:gosenpuj@uww.edu Developments in Business Simulation and Experiential Learning, Volume 29, 2002 METHOD This study was conducted during the fall and spring semesters of the 2000-2001 academic year using three sec- tions (two in the fall) of the undergraduate business admini- stration degree capstone administrative policy course at the University of Wisconsin-Whitewater. Within the con- straints of day and time of class, individual preference and class capacity, students were assigned to these courses in a non-random manner. The primary research hypotheses, stated in null form, were: • H1. No simulation-related learning occurs during the traditional part of the course. • H2. No simulation-related learning occurs during the simulation part of the course. • H3. Learning and simulation performance do not co-vary. • H4. Learning and forecasting accuracy do not co- vary. • H5. Simulation performance and forecasting accu- racy do not co-vary. Learning was measured using three parallel forms of an objective-item and short-answer examination developed by the researchers during previous simulation learning studies. These tests were specifically designed to measure learning that resulted from playing the game itself (Washbush & Gosen, 2001). The examination was constructed to reflect the below listed learning objectives (based on the content and procedures found in MICROMATIC (Scott, et al., 1992), the simulation used in these studies. Learning in these studies also reflected some of the elements in the stra- tegic management domain as identified by Wolfe and Roge’ (1997) including strategy, environmental analysis, forecast- ing, market development and penetration, cost and differen- tiation strategies, and performance measures. Additionally reflected was the description of decision making in total enterprise simulation games as provided by Keys and Biggs (1990) involving business functional areas and their integra- tion. Simulation participants were expected to be able to: • Effectively make decisions integrating the market- ing, production, and financial aspects of a business. • Evaluate periodic performance with respect to profits, cost control and strategic impact. • Improve business performance with respect to cost reduction, profitability, and strategic potential. • Pose and implement effective and efficient solu- tions to problems encountered or opportunities that arise. • Analyze effectively market conditions and the be- havior of competitors. • Recognize needs for strategic and tactical change. • Develop and demonstrate a mature ability to read and interpret financial statements. • Understand and manage cash flow with respect to sources, needs, and uses. • Properly allocate costs on a per-unit-sold basis. • Develop basic skills in forecasting product de- mand. • Use pro forma statements and "what if" analysis to evaluate the probable impact of decisions and stra- tegic options. Students completed one form of the examination at each of three points in the course: 1. A pre-test given during the first week of the course 2. A mid-test given at the completion of the tradi- tional (theory and case) portion of the course 3. An end-test given as part of a final examination at the completion of the course. Students were self-selected into groups of three or four for purposes of simulation play. Simulation play was the sole focus of the course during approximately the last 1/3 of each course. The previous parts of the course were devoted to strategic management theory (3 weeks), case analysis (6 weeks) and an overview and introduction to the simulation (1 week). Simulation play began with a practice decision round. In addition to play, students had to write and submit brief periodic performance analysis reports and a final, overall performance assessment. The grading scale was based on 500 possible points and simulation related compo- nents were graded as follows: Simulation Reports 100 possible points Group (20% of course grade) Simulation Final Standing 75 possible points Group (15% of course grade) Peer Evaluations 25 possible points Individual ( 5% of course grade) Final Exam 100 possible points Individual (20% of course grade) As noted above, the last exam was part of the course fi- nal. In addition to questions from the simulation-learning examination item pool, approximately 40% of the final exam addressed cases studied in the course. No case-related items were used in calculating learning scores. The simulation learning tests used were forms devel- oped in our research intended to (eventually) create an ex- amination suitable for use in evaluating learning in any simulation environment (Gosen, Washbush et al, 1999; Go- sen, Washbush & Scott, 2000; Gosen and Washbush, 2001). These were scored using standardized answer keys. Each exam raw score was recalculated as a percentage score by dividing the raw score by points possible for the form used. Simulation performance was measured using the normalized scoring routine that is a component of simulation software. The factors used to determine simulation performance within the game’s scoring routine were total profits (40%), return on sales (30%), and return on assets (30%). This weighting system has been used in all of our previous simu- lation and learning research. Specific results of the pre and 282 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 mid-tests were not provided to the students nor were those administrations debriefed. Forecast accuracy was determined by requiring each group to prepare and turn in a forecast for sales in each market area for each decision round. Total demand for each period was determined by summing actual sales and lost sales for each area. Forecast error for each round of play was calculated by subtracting forecast sales from actual de- mand and converting to the absolute value. For all periods of play, the mean absolute deviation (MAD) was calculated for each group by summing the absolute errors for each pe- riod of play and dividing that total by the number of periods of play. A smaller MAD indicates greater forecast accu- racy. Consistent with our procedures in all related prior stud- ies, learning was defined, test to test, as the difference in percentage score of the latter test minus the percentage score of the prior test. As in most of our previous studies, learn- ing was measured individually, simulation performance was measured on a group basis. Learning was evaluated by comparing between-test percentage scores using paired, two-sample t-tests for means to determine whether or not significant differences existed. Learning was compared to simulation performance by regressing learning on perform- ance and by regressing learning on MAD. Finally, perform- ance, as measured above, was regressed on forecast accu- racy (MAD). RESULTS Table 1 displays results for t-tests conducted for each of the groups studied. Learning scores were developed com- paring test 2 to test 1, test 3 to test 2, and test 3 to test 1. In all sections studied, significant learning occurred between tests, and test 3-2 learning was generally greater than test 2- 1 learning. Thus, learning occurred in both the traditional (strategic theory and case analysis) and simulation-only parts of the courses. This suggests that the simulation val- idly represents the content of the theory and case analysis components of the policy course. This is consistent with the findings of Wolfe (1976) that simulations are externally valid, and this argues strongly that simulations are effective complements to other learning activities in business policy courses. Null hypotheses 1 and 2 were rejected because significant learning occurred during both components of the course and overall. Table 1 Learning Test Results Fall 2000 Sect 1 Test Means Variance N Test 1 54.136 83.096 26 Test 2 59.653 49.880 26 Test 3 66.667 75.663 26 t tests t Prob. (2-tail) Test 2 – Test 1 3.366 0.0025 Test 3 – Test 2 3.565 0.0015 Test 3 – Test 1 6.006 0.0000 Fall 2000 Sect 2 Test Means Variance N Test 1 50.993 124.212 38 Test 2 57.382 65.082 34** Test 3 65.015 56.013 38 t tests t Prob. (2-tail) Test 2 – Test 1 3.675 0.0008 Test 3 – Test 2 5.087 0.0000 Test 3 – Test 1 8.338 0.0000 **4 students did not complete the second test 283 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 Spring 2001 Sect 3 Test Means Variance N 1 49.346 106.878 30 2 54.843 147.048 30 3 62.549 54.714 30 t tests T Prob. (2-tail) Test 2 – Test 1 2.736 0.0105 Test 3 – Test 2 4.027 0.0004 Test 3 – Test 1 7.662 0.0000 Table 2 displays results for analyses regressing per- formance on learning. The regressions performed compared performance to test 2 - test 1 learning, test 3 - test 2 learn- ing, and total learning (test 3 - test 1). For Section 2 (fall 2000), there were two cases (performance regressed on test 2 – test 1 learning, and performance regressed on total learn- ing) where there occurred significant, positive relationships between learning and performance. In these cases R2 was 0.15 (adjusted R2 was 0.12) and 0.18 (adjusted R2 was 0.15) respectively. In no other instances over the nine re- gressions was there any significant Beta relating perform- ance to learning. In general, these results were consistent with those of our previous studies. Null hypothesis 3 was therefore accepted because there was no consistent relation- ship between learning and simulation performance. Table 2 Regressions: Performance on Learning Fall 2000, Sect 1 Learning Learning Beta Slope Sig. R2 Test 2 – Test 1 -0.641 ns 0.0556 Test 3 – Test 2 -0.344 ns 0.0230 Test 3 – Test 1 -0.701 ns 0.1078 Fall 2000, Sect 2 Learning Learning Beta Slope Sig. R2 Test 2 – Test 1 0.964 0.0231 0.1506 Test 3 – Test 2 0.064 ns 0.0004 Test 3 – Test 1 1.179 0.0083 0.1775 Spring 2001, Sect 3 Learning Learning Beta Slope Sig. R2 Test 2 – Test 1 -0.414 ns 0.0035 Test 3 – Test 2 -0.369 ns 0.0025 Test 3 – Test 1 -1.018 ns 0.0155 Table 3 displays results for analyses regressing fore- casting accuracy (measured by unit-sales forecast MAD) on learning in the same manner as noted above for perform- ance. Compared to the learning-performance results, there was one significant relationship between learning and MAD (MAD regressed on test 2 – test 1 learning), and in this case R2 was 0.13 (adjusted R2 was 0.10). In no other instances over the nine regressions was there any significant B relat- ing MAD to learning. Null hypothesis 4 was therefore ac- cepted because there was no consistent relationship between learning and forecast accuracy. 284 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 Table 3 Regressions: MAD on Learning Fall 2000, Sect 1 Learning Learning Beta Slope Sig. R2 Test 2 – Test 1 6.928 ns 0.0109 Test 3 – Test 2 -6.406 ns 0.0134 Test 3 – Test 1 -1.422 ns 0.0007 Fall 2000, Sect 2 Learning Learning Beta Slope Sig. R2 Test 2 – Test 1 -26.119 0.0350 0.1312 Test 3 – Test 2 21.380 ns 0.0561 Test 3 – Test 1 -12.759 ns 0.0271 Spring 2001, Sect 3 Learning Learning Beta Slope Sig. R2 Test 2 – Test 1 1.571 ns 0.0012 Test 3 – Test 2 -2.238 Ns 0.0023 Test 3 – Test 1 -0.624 Ns 0.0001 Table 4 displays results of analyses regressing perform- ance on MAD. Here, the results were mixed. For the two fall 2000 sections, there were no significant relationships between standing and forecast accuracy, but for spring 2001 the MAD coefficient was significant. However, definitive conclusions about simulation and forecast accuracy were not found, and therefore null hypothesis 5 was accepted. The fact that the spring 2001 semester included 13 periods of play versus 8 and 9, respectively, for the fall 2000 sections may explain the inconsistencies found. One suspects that a larger number of periods of play will permit players to gain experience-based expertise in forecasting. These results would seem to be consistent with those found by Teach (1989). Table 4 Regressions: Performance on MAD Academic Term & Section N Periods of Play MAD Beta Slope Sig. R2 Fall 2000, Sect 1 26 8 -0.008 Ns 0.0353 Fall 2000, Sect 2 38 9 0.005 Ns 0.0161 Spring 2001, Sect 3 30 13 -0.101 0.0001 0.4142 DISCUSSION The consistent and significant learning documented over the traditional and simulation components of these course sections was the most interesting finding. These results strongly suggest that the policy course is, at least, a natural complement to theory and case-based studies in the policy course. In general, users of TE simulations have acted on an assumption that traditional instruction in strate- gic management and the use of simulations are complemen- tary pedagogically. Additionally, Wolfe and Roge’ (1997) have documented that there is a substantial amount of com- mon ground covered by both traditional and simulation mo- dalities, but that some games are more effective in covering typical strategic management subject matter and concepts. The results of the analyses presented here provide empirical evidence that complementary learning does in fact occur, and they argue for studies examining this phenomenon more expansively and precisely. Secondly, these results suggest that the simulation may be more equivalently useful than is assumed by policy course instructors in terms of learning compared to the tradi- tional policy-course format. The simulation may be, in fact, a learning method that may stand alone (or nearly so) in the policy-course setting. If this is true, instructors might be well advised to build the policy course experience with a focus on the simulation as the primary activity of the course, supplementing it with some traditional components as might be appropriate or necessary. Studies exploring this ap- proach seem warranted. Thirdly, as expected, these results confirmed our previ- ous consistent findings that simulation-based learning and simulation performance do not co-vary. These persistent findings indicate that that question need not be a primary focus of continuing research. Instead, they do argue that the ongoing development of a standardized generic test of simu- lation learning is still a worthwhile project. Additionally, such a validated instrument would be invaluable in assess- 285 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 ing the broader implications for the simulation suggested in the previous paragraph. Fourthly, there is also the possibility that learning im- provement over three test cycles may be, in part, a result of repeated testing. However, that need not be seen as unde- sirable. If, in fact, the process of measuring intended learn- ing contributes to such learning, that could be seen to be a valuable and desirable consequence. Thus, studies that at- tempt to evaluate simulation learning might well be de- signed to assess whether or not repeated examination-based learning assessments contribute to and enrich the learning environment of the simulation. Finally, continuing to assess the extent to which meas- ures of forecasting accuracy may have learning implications are warranted. Unfortunately, the analyses documented here did not consistently indicate a correlation between learning and forecasting accuracy. At best, the analyses suggest that forecasting accuracy may be a proxy for other measures of simulation performance. One would like to believe that people who learn to forecast accurately (whether in terms of sales units or profitability) in TE simulation environments are in fact learning valuable strategic analysis and synthesis competencies. One would like to believe that such learning would occur on a regular basis, but the results here suggest otherwise. Thus, in the end, we ought ask, why do such outcomes not occur? REFERENCES Gosen, J. & Washbush, J. (1996). The relationship between total enterprise simulation performance and learning. Developments in Business Simulation and Experiential Exercises, Vol. 23, 43-48. Gosen, J. & Washbush, J. (1999). As teachers and research- ers, where do we go from here? Simulation and Gam- ing: An International Journal, Vol. 30 No. 3, 292-303. Gosen, J & Washbush, J. (2001). An initial validity investi- gation of a test investigating total enterprise simulation learning. Developments in Business Simulation and Experiential Learning, Vol. 28, 92-95. Gosen, J., Washbush, J., Patz, A., Scott, T., Wolfe, J., & Cotter, D. (1999). A testbank for measuring total en- terprise simulation learning. Developments in Business Simulation and Experiential Learning, Vol. 26, 87-92. Gosen, J, Washbush, J., & Scott, T. (2000). Initial data on a test bank assessing total enterprise simulation learning. Developments in Business Simulation and Experiential Learning, Vol. 27, 166-171. Keys, B. & Biggs, W. (1990). A review of business games. In J.W. Gentry (ed), A Guide to Business Gaming and Experiential Learning. East Brunswick, NJ: Nich- ols/GP. Scott, T.W., Strickland, A.J., Hofmeister, D. L., & Thomp- son, M. D. (1992). MICROMATIC: A Management Simulation. Houghton-Mifflin Company, One Beacon Street, Boston, MA 02108. Teach, R. (1989). Using forecasting accuracy as a measure of success in business simulations. Developments in Business Simulation & Experiential Exercises, Vol. 16, 103-107. Washbush, J. & Gosen, J. (1993). The relationship between total enterprise simulation performance and learning. Developments in Business Simulation and Experiential Exercises, Vol. 20, 141. Washbush, J. & Gosen, J. (1994). Simulation performance and learning revisited. Developments in Business Simu- lation and Experiential Exercises, Vol. 21, 83-86. Washbush, J. & Gosen, J. (1995). Simulation performance, learning, and learning. Developments in Business Simulation and Experiential Exercises, Vol. 22, 1-4. Washbush, J. & Gosen, J. (2000). Simulations and learning: dialog and directions. Developments in Business Simu- lation and Experiential Learning, Vol. 27, 3-4. Washbush, J.B. & Gosen, J. (2001). An exploration of game-derived learning in total enterprise simulations. Simulation and Gaming: An International Journal. Vol. 32, No. 3, 281-296. Wolfe, J. (1976). Correlates and measures of the external validity of computer-based business policy decision making environments. Simulation & Games: An Inter- national Journal, Vol. 7, 411-438. Wolfe, J. (1997). The effectiveness of business games in strategic management course work. Simulation & Gaming: An International Journal, Vol. 28, No. 4, 360-376. Wolfe, J. & Roge′, J. (1997). Computerized general man- agement games as strategic management learning envi- ronments. Simulation & Gaming: An International Journal, Vol. 28, 423-441. 286 Table of Contents Volume 29, 2001 Threshold Marketer A Family Of Marketing Simulations: Basic Marketer And Advanced Marketer Team Mode And Solo Mode Globalization As An Extended Experiential Exercise The Benefits And Planning Considerations Of Short Term Study Abroad Programs How 2 Setup Your Office Computer To Run Linux For Teaching E-Commerce Without Messing Everything Else Up Incorporating Cosmopolitan-Related Focus-Group Research Into Global Advertising Simulations Demonstration Of Advanced Features In Computer-Assisted Gaming Of International Business A Comparison Of Discrimination-Based Versus Conventional Simulation Game Scoring A Universal Mathematical Law Criterion For Algorithmic Validity The Impact Of Public Policy On Innovation: A Simulation Project For Research And Teaching Participant Identification Of Competitors In A Marketing Simulation Competition Simulation Research In The Hospitality Industry "Computer Simulation, Games And Roleplay: Drawing Lines Of Demarcation" Simulation Distribution Alternatives: Author/User Considerations Managing The Curiosity Gap Does Matter: What Do We Need To Do About It? Use Of External Interventions In A Computer Based Simulation Putting Service Learning Into Orbit It's A Wonderful Life: Simulating The Golden Years Adventures In Creating An Outdoor Leadership Challenge Course For An Emba Program Vbotz: A Pedagogical Cross-Disciplinary, Multi-Academic Level Manufacturing Corporate Simulation Use Of Computer Modeling In Management Accounting Is Simulation Performance Related To Application? An Exploratory Study Learning Cooperatively May Not Be Learning Collaborately! Perception Is Reality: Sharing Frames International Management Virtual Teamwork: A Simulation Financial Plan For Your Life And Career Goals Using Project-Based Experiential Learning Groups In The Principles Of Marketing Course Futures Course: Learning How To Anticipate The Future Of Business Interactive Online Strategic Market Planning With The Web-Based Boston Consulting Group (BCG) Matrix Graphics Package Strategy Learning In A Total Enterprise Simulation Investigation Of The Impact Of Decision Parameters For A Dutch Auction Simulation For Ipo Issues Integrating In-Class Learning With Out-Of-Classroom Experiences Through A Managerial Competency Development Framework War And Peace: Managing Students Learning Experience In A Competitive Simulation Game Virtually Experiential Classrooms Exercise: Conducting Role Plays Using Student Generated Cases Procedural Justice And Acceptance In Group Decision Making The E-Commerce Game: A Strategic Business Board Game Does Student Preparation Matter In A Simulation? A Comparison Of Pedagogical Styles The Game Of Business - A Weekend MBA Course Volume-Dependent Money Exchange Model For Gaming Simulations Implementing Service Learning For Accountants: The Not For Profit Project To Teach Vikings To Behave Among Mandarins: Lessons From Teaching With A Simulation Model Of Applied Business Ethics In International Management Maze Bright Teachers In The Classroom The Validity Investigation Of A Test Assessing Total Enterprise Simulation Learning What Makes Strategy Possible: An Illustration Using Paper And Scissors Learning Micro-OB Skills While Making Top Management Decisions In A Multinational Industrial Firm The Absel Research Heritage And The Bkl: Leveraging Their Value For Future Research New Product Development (Npd) Simulations: Some Challenging Questions And Tough Modeling Issues The Biofeedback Stress Test The Power Circle Exercise Total Enterprise Simulation Learning Compared To Traditional Learning In The Business Policy Course A Business Game Distance Education Application: Learning Outcomes And Experiences Is the Tobin's Q a good Indicator of a Company's Performance? A Critical Examination of the "Experiential" Premise Underlying BUsines Simulation Usage