WHAT ARE SIMULATIONS FOR?: LEARNING OBJECTIVES AS A SIMULATION SELECTION DEVICE Developments in Business Simulation and Experiential Learning, Volume 30, 2003 WHAT ARE SIMULATIONS FOR?: LEARNING OBJECTIVES AS A SIMULATION SELECTION DEVICE E. Brian Peach The University of West Florida bpeach@uwf.edu Martin Hornyak The University of West Florida mhornyak@uwf.edu ABSTRACT A longstanding topic of discussion among simulation users is what types of learning objectives are appropriate. Research has shown mixed results at establishing a relationship between simulations and learning, and the construct of learning has been a contentious topic as well. Various streams of thought on the topic of learning objectives are reviewed and the authors then propose several fundamental objectives they personally have accepted as critical in adopting a simulation for classroom use. Two versions of a leading simulation are evaluated on how well they support these learning objectives. The conclusion is that users must clearly identify desired learning objectives not only prior to adopting a new simulation, but before progressing to the latest revision of a simulation currently in use. INTRODUCTION The widespread popularity of simulations is supported by an increasing number of studies. Over a decade ago, Faria (1989) estimated that more than 5,000 U.S. companies used business simulations in corporate training and development programs. In surveys of AACSB business schools, Faria (1987) estimated 95% used simulations in the mid-1980s, and for the mid-1990s, he increased the estimate to 97% (Faria & Nulsen, 1976). More recently, Joldersma & Guerts (1998) indicated use of simulations is also widespread in Europe and other parts of the world. Presumably, the use of simulations by corporate universities and business schools is predicated on the belief that they assist in achieving desired learning objectives (Gentry & Burns, 1981). Learning objectives have been a consistent topic of discussion among developers and users since the introduction of simulations to the learning environment. Whether simulations are effective at (or even capable of) helping students achieve learning objectives has been a rich field for research and publication. A number of major reviews have discussed the topic (Greenlaw & Wyman, 1973; Keys, 1976; Wolfe, 1985; Miles, Biggs & Schubert, 1986, Malik & Howard, 1996). Anderson & Lawton (1997a) summarized four streams of research or schools of thought on the association between simulations and learning, indicating widespread disagreement on both the possible relationship, and learning as a construct. Nevertheless, we presume prospective as well as current users of simulations believe some useful learning is taking place. The substantial commitment of time and resources required of both instructors and students mandates that there is some perceived benefit. SELECTING A SIMULATION Many users select a simulation based on what a colleague uses and/or recommends. Others may base the selection on publisher support, attractive marketing, or other factors not directly related to their specific course learning objectives. Poor selection technique compounds the problem that once a simulation is adopted there is substantial pressure to continue using it. Most simulations today, especially the Total Enterprise Simulations favored for policy courses, are increasingly complex and sophisticated. Although ease of use has often been improved. from both a game administrator and student user viewpoint, there remains a steep learning curve to become familiar with a simulation’s operating characteristics. For most instructors, it takes several semesters of use to become comfortable at fielding questions from students. Thus, there is a very strong tendency to continue with a simulation once it is adopted. As new “improved” versions are introduced, it is easier to continue with the known rather than initiate a search for a new simulation. 220 mailto:bpeach@uwf.edu mailto:mhornyak@uwf.edu Developments in Business Simulation and Experiential Learning, Volume 30, 2003 CHANGING SIMULATIONS - MINIMIZING THE PAIN The policy course is a mixture of new concepts, and a ‘pulling together’ of previous knowledge. Thus, consistent with Bloom’s Taxonomy, some of our objectives might be for comprehension, and some for application or analysis. The Anderson & Woodhouse (1981) model is comparable, except that from this model we could have attitude objectives as well. Recently, the authors of this paper experienced increasing discomfort with the simulation they were using. The many years of experience with that simulation made any contemplation of changing almost painful. But, as any organizational change agent knows, when the pain of remaining where you are exceeds the pain of change, you are forced to contemplate and then accept change. However, prior to selecting a new simulation, we decided to assess what was causing our discomfort with the current simulation. This necessitated assessing what it was we were trying to accomplish with the simulation, i.e., what were our learning objectives. Some researchers have embarked on a more explicit identification of simulation learning objectives. An example is the stream of research by Gosen & Washbush (Gosenpud & Washbush, 1994; Gosen, et. al., 1999; Gosen, Washbush & Scott, 2000). Using surveys, they identified forty learning objectives (Gosen, et. al., 1999). This list varied from the rather specific “attend to detail such as ordering raw material,” to the somewhat more general “correctly use game rules,” to the rather broad “appropriately apply strategic concepts” (1999, p88-89). This list was a rich source of ideas, but in some cases was game-specific, and was somewhat lengthy for our purposes. A number of authors (e.g., Gagne, 1968; Gartner, 1993; Gentry, McBain & Burns, 1979) recommend identifying clear learning objectives prior to picking a pedagogy or selecting a simulation. Certainly this advice has a great deal of face validity, but it is not clear how many users actually identify desired learning objectives prior to selection. In this case, however, we felt that identifying learning objectives would have a double benefit. First, it would help to assess the strengths and weaknesses of our present simulation. Second, it would provide a basis for assessing potential candidates for adoption. As a first step to defining our learning objectives, and to ensure a practical outcome of this process, we identified what we considered to be the fundamental learning objectives we expected to achieve through the simulation. Although there might be many more desirable learning objectives, we decided that for a simulation to have sufficient value to justify the resources required, it would have to support our fundamental simulation objectives. As it turned out, they are a mixture of knowledge acquisition, attitude change, application, and analysis. IDENTIFYING SIMULATION LEARNING OBJECTIVES Discipline Application The first category of objectives we considered is the application of previously acquired knowledge. Primarily this knowledge would be financial in nature. The simulation must provide sufficient financial data that students have practical experience interpreting financial statements, and they must be required to make financing decisions of debt or equity with consequences that reflect real world experience. This may be a bias of the authors, but we find that most, if not all, students arrive at the capstone course believing debt (any debt) is bad. Thus, the ideally designed simulation would ensure that for two teams that were otherwise equal, a properly leveraged firm would be ranked higher than one that was under-leveraged or over- leveraged. Teaching the importance of access to capital markets must be reinforced by the simulation. Examples of applying other prior knowledge would be marketing principles such as relationship of advertising, product characteristics, and sales, or production operations concepts such as unit cost, break-even analysis, optimal ordering quantities, etc. A number of authors have devised frameworks for devising learning objectives. Ramsey & Woodhouse (1981) advised determining the purpose of the course: was it to introduce new theoretical concepts or to pull together previous learning? They cited Anderson & Woodhouse (1980) in identifying three categories of objectives: acquisition of knowledge, development of attitudes, and development of skills. Thus, the simulation user needs to identify which areas of the course are exposing the student to new knowledge and attempt to determine the extent to which the simulation reinforces the acquisition of new knowledge. Similarly, the user should identify where attitudes and the application of previously acquired knowledge are important, because simulation support for these would then be critical. In a more widely cited paper, in the same year Gentry & Burns (1981) suggested using Bloom’s taxonomy. The six levels of learning identified by Bloom (1959) were: basic knowledge, comprehension, application, analysis, objective synthesis, and objective evaluation. Gentry & Burns stated that “the level of learning taught should be determined very early. If, in fact, the purpose of the course is to provide an awareness of the general topic area, then methodologies aimed at higher levels of learning may be counter-productive” (1981, p49). Marginal Return The second category of objectives is a combination of application and analysis. The simulation should be sufficiently sophisticated that production and marketing variables are interrelated, and react in a way that reflects the real world. We find students are prone in case analyses to recommend increasing sales revenue through raising quality, spending more on advertising and 221 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 ASSESSING SIMULATIONS simultaneously cutting prices. The simulation must show that all decisions have costs – price, quality, revenues, profit – and they all must be realistically interlinked. The concept of marginal revenue benefit in the business world is critical for firms. One should not cut costs just because profits are low - one should assess the benefit of each cost center and in a world of limited resources apply those resources in a manner that maximizes their return to the firm. We decided to apply our learning objectives to the simulation we were using. The two simulations to be assessed using our learning objectives are the 6th and 7th versions of the Business Strategy Game (Thompson & Stappenbeck, 1999, 2001). Version 4 had some minor computational bugs that could easily have been fixed, but other than that, the underlying performance was clear and strong. The 5th and 6th versions added features that appeared to diminish the clarity of variable relationships. As Elgood (1984) points out, complexity and cleverness can create problems in simulations. Version 7 added a whole new set of variables as well as a revamped presentation. On the positive side, the administrative functions were significantly improved. On the negative side, we experienced student dissatisfaction, and had serious concerns about some procedural changes. However, we felt the bottom line was how well either or both of the simulations supported our learning objectives. Competitive Analysis & Planning The final, and very important, category of objectives we considered is a combination of knowledge acquisition from the current policy course, and attitude change engendered by the policy course. The policy course promotes leaning how to accomplish a thorough environmental analysis, developing or honing the ability to anticipate future competitive moves, and developing strategic plans that position the firm to effectively compete in the expected future environment. As we moved to explicitly identify our desired learning objectives, we realized that it was not realistic to expect every student to become proficient at accurately anticipating competitive moves and developing effective plans to combat the competitors’ moves. After some soul searching, it became clear that the minimum learning we wanted was an attitudinal change – an acceptance on the students’ part that more effective plans are likely to win against effective plans. We wanted them to believe that success is not a random walk, and that good competitive analysis and development of an effective plan was the path for ‘winning’ in the business world. Discipline Application In Version 6, financial analysis and application were well supported. All relevant financial statements were calculated and presented to the student. One of the six factors determining team standing was ROE. The weight of ROE could be adjusted by the instructor for desired emphasis on debt/equity financing decisions. There were seven variables identified as competitive factors that students manipulated to affect sales and performance. In Version 7, ROE was removed as a factor and ROI substituted. The rationale provided by the game developer was that students could drive equity to zero and distort the effect of ROE on standings. This was a known problem, and as game administrators we compensated for this by publishing a rule where teams that went below a minimum equity ratio were punished through fines that increased with diminishing equity. Rather than design a penalty within the game to accomplish this, the game developer switched to ROI. Using ROI instead of ROE effectively removed the debt/equity consideration from the game and this could not be easily compensated for by the administrator. Over the course of operating a simulation, any instructor learns that some industries are very unprofitable, some are very profitable, and most fall somewhere in between. In the real world, a spectrum of profitability performance is acceptable. In a simulation, students are obsessed with ‘winning’. All industries have a personality based on the teams involved. Thus, for simulations, comparisons are between teams in a specific industry more than to an objective standard. Certainly we want students to develop as much as possible skill at competitive analysis and planning, but the vagaries of team composition and industry personality preclude arbitrary preset standards. In a well-designed simulation, it should be clear that the better performers are better planners. Thus, regardless of their own particular individual success in the simulation, the learning objective is for students to accept that competitive success in a business environment (as well as life itself) comes through effective application of the course concepts. Marginal Return Version 4 was a significant revision over 3 and essentially established the baseline for future versions. Versions 5 and 6 added variables (e.g., advertising was changed from just advertising to current and total, then celebrities were added for Version 6). The increased complexity made the relationships between variables less clear, and occasionally results would be unexplainable by the instructor - i.e., the apparent ‘best’ strategy did not win. The simulation still was generally usable but less effective for this learning objective. Version 7 added a new regional market (Latin America), new distribution channels (megastores and the Internet) and other variables that overwhelmed the player with complexity. Overall, the number of variables and increased complexity made it very difficult to assess the relative impact of individual variables. This means that a well-designed simulation should provide both a mechanism and a basis for meaningful planning. Students may not anticipate competitive moves, but post hoc analysis should confirm the rationality of events that transpired. In other words, firms with lower prices or higher quality should sell more. It should not be a puzzle to students as to why one firm did well and theirs did not. 222 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 Environmental Analysis & Planning Version 6 provided a 5-year planning template. In practice it was initially difficult for any team to accurately predict events five years out, but over the course of the game, the better teams became fairly good at predicting sales and revenues. Teach (1992) contends that forecasting is a good measure of assessing performance. Peach & Platt (2000) used forecasting accuracy to assess learning. One of our learning objectives connected with planning was students should learn to adjust capacity to roughly meet expected demand. Excess capacity that was not used increased overhead costs on units produced. Production in excess of sales resulted in inventory. Thus the incentive was to match capacity expansion to anticipated sales. VERSION CHANGE EFFECT ON LEARNING OBJECTIVES Once the learning objective assessment was complete, the reason for our unease with version 7 was clear. Support for every single one of our critical learning objectives had been eliminated from Version 7. • Financial analysis was no longer critical, and there was no debt/equity learning. Our fundamental goal of applying financial skills and learning the importance of using debt versus equity for financing was not achieved. • Gaining a clear understanding of marginal return was impossible. The complexity of the game itself tended to mask the relationships, but the addition of the demand screen eliminated any interest in identifying and understanding relative costs and benefits. The simulation became instead an exercise in spending time manipulating various variables one by one on the demand screen to maximize sales and profit. Version 7 reduced the planning template to three years. This reduced the effective time horizon students had in planning plant construction to meet anticipated demand changes. However, it turned out that the reduction to three years was a secondary consideration for Version 7. Version 7 has so many computational bugs in the three year planning template we considered it as effectively unusable. • Planning became irrelevant and long term planning was impossible. The three year template was unusable. More importantly, each week was treated by students as a stand- alone event, where strategies were introduced to respond to the demand screen. Most importantly, it was no longer clear that the team with the best competitive analysis, or the team with the best plan, would do better. The basic premise of our course was no longer supported. In Version 7, what turned out to be of absolutely critical importance was the integration of a screen where students could manipulate variables and predict sales. This screen uses the actual game demand function. The BSG does not use an algorithm, but relies on lookup tables. Because Version 7 relies on Excel, and this screen required access to the actual lookup tables to provide demand predictions, one could fairly easily access the underlying lookup tables and see the exact effect of various changes in individual variables. Few students discovered this as use of the demand prediction screen was enough to substantively change their decision making behavior. Students quickly shifted from planning capacity and production to match expected demand (the desired learning objective) to running current production at existing capacity (the failed Roger Smith/General Motors strategy in the late 80s and early 90s) and then manipulating variables (price, quality, etc.,) until the demand screen predicted all product would be sold. In effect, the screen introduced a Prisoner’s Dilemma for students. Although it was poor strategy for all teams to use this approach (over- production was a natural outcome), if only one team did it, it would have an advantage and thus the others had to follow. CONCLUSION The wisdom of the advice of previous authors that instructors should clearly define desired learning objectives prior to selecting a simulation was conclusively established for us after we applied our learning objectives to Version 7 of the BSG. It may well be that for some other instructor with different learning objectives, this would be a satisfactory aid to learning. For the accomplishment of our learning objectives, it had transmogrified from an effective experiential learning aid to an unsuitable and counterproductive tool. The result of this demand function screen was that all teams discarded any attempt at a long-term strategy and for each decision period used whatever strategy the demand screen indicated was best. Thus, teams switched weekly from high quality to low cost or high service, and the industry became totally unpredictable. Any team trying to establish a long term strategy was destined to frustration and likely failure. 223 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 224 REFERENCES Anderson, Philip H., & Leigh Lawton (1997a) Simulations and learning: Can we prove a relationship? Developments in Business Simulation & Experiential Exercises, 24, 109-110. Anderson, Philip H., & Robert H. Woodhouse (1980) The integration of objectives and pedagogy for the business policy course. Unpublished manuscript. Bloom, B.S., N. D. Englehart, E. J. Furst, V. H. Hill, & D. R. Drathwohl (1959) Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive Domain. New York: David McKay Company, Inc. Elgood, C. (1984) Handbook of Management Games, 3rd Ed. Gower, Aldershot. Faria, A. J. (1989) Business gaming: Current usage levels. Journal of Management Development, 8(2), 207-223. Faria, A. J. (1987) A survey of the use of business games in academia and business. Simulation & Games, 18, 207- 224. Faria, A. J., & Nulsen, R. O. (1976) Student reactions and evaluations towards differing assignment levels in a marketing simulation competition. Developments In Business Simulation and Experiential Learning, 3, 26- 35. Gagne, R. M. (1968) Learning hierarchies. Educational Psychologist, 6, 1-9. Gartner, W. B. (1993) Dr. Demming goes to class. Journal of Management Education, 17, 143-150. Gentry, James W., & Alvin C. Burns (1981) Operationalizing a test of a model of the use of simulation games and experiential exercises. Developments in Business Simulation & Experiential Exercises, 8, 48-72. Gentry, James W., Kenneth C. McBain & Alvin C. Burns (1979) Relating teaching methods with educational objectives in the business curriculum. Developments in Business Simulation & Experiential Exercises, 6, 196- 198. Gosen, Jerry, John Washbush, Alan Patz, Joseph Wolfe, & Dick Cotter (1999) A testbank for measuring total enterprise simulation learning. Developments in Business Simulation & Experiential Exercises, 26, 82- 92. Gosen, Jerry, John Washbush & Tim Scott (2000) Initial data on a test bank assessing total enterprise simulation learning. Developments in Business Simulation & Experiential Exercises, 27, 166-171. Gosenpud, J., & Washbush, J. (1994) What simulation users think players should be learning from simulations. Developments in Business Simulation & Experiential Exercises, 21, 96-99. Greenlaw, P.S., & Wyman, F. P. (1973) The teaching effectiveness of games in collegiate business courses. Simulation and Games, 4(3). 259-293. Joldersma, C., & Geurts, J. L. A. (1998) Situation/gaming for policy development and organizational change. Simulation and Gaming, 29(4), 391-399. Keys, J. B. (1976) A review of learning research in business gaming. In B. H. Sord, (Ed.), Computer simulation and learning theory. 173-184. Malik, S. Dolly, & Barbara J. Howard (1996) How do we know where we are going if we don’t know where we have been: A review of business simulation research. Developments in Business Simulation & Experiential Exercises, 23, 49-53. Miles, W. G., Jr., William D. Biggs, & J. N. Schubert (1986) Students perceptions of skill acquisition through cases and a general management simulation: A comparison. Simulation and Games, 1, 7-24. Peach, E. Brian, & Richard Platt (2000) Total enterprise simulations and optimizing the decision set: assessing student learning across decision periods. Developments in Business Simulation & Experiential Exercises, 27, 242-262. Ramsey, V. Jean & Robert H. Woodhouse (1981) Toward a theory of teaching business policy. Developments in Business Simulation & Experiential Exercises, 8, 179- 183. Teach, Richard (1992) Using forecasting accuracy as a measure of success in business simulations. Developments in Business Simulation & Experiential Exercises, 19, 103-107. Thompson, A. A. & Gregory J. Stappenbeck (1999) The Business Strategy Game. 6th ed. Boston, MA: Irwin/McGraw-Hill. Thompson, A. A. & Gregory. J. Stappenbeck. (2001) The Business Strategy Game. 7th ed. Boston, MA: Irwin/McGraw-Hill. Wolfe, J. (1985) The teaching effectiveness of games in collegiate business courses: A 1973-1983 update. Simulation and Games, 16(3), 251-288. Table of Contents Volume 30, 2003 The Optimal Timing For Introducing Business Simulations Can Handicapped Students Access Your Class Web Site? The Competition Game: Decision Making In A Dynamic Environment Pan-Pacific Enterprises: Strategic Decision Making Simulation Study Of Stochastic Channel Redistribution The Impact Of Business War Games: Quantifying Training Effectiveness Experiential Learning: Introducing Faculty And Staff To A University Leadership Development Program The Feasibility Of The Balanced Scorecard For Business Games A Misuse Of Pims For The Validation Of Marketing Management Simulation Games Incorporating Technology Into The 21st Century Classroom: Are We Facilitating Academic Dishonesty? Improving The Effectiveness Of Peer Evaluations The Use Of A Simulation In An Integrated Mba Curriculum Student Portfolios In Business Education Student Portfolios In Business Education At Ashland University Using SAP ERP Technology To Integrate The Undergraduate Business Curriculum Board Games And Teaching Textile Marketing And Finance Blogging: A New Threat To Student Research? The Way We Talk! Take II Strategic Management: An Evaluation Of The Use Of Three Learning Methods In Hong Kong Adoption Of Discussion-Based Teaching And Assessment In Teaching Strategic Management In Hong Kong The Tobin Q As A Company Performance Indicator Using Representative Nominal Group Technique For Course Review And An Interactive Solicitation Of Ways To Enhance Absel's Image Making Teaching Matter: The Art And Science Of Teaching Business Communication Beyond Sex, Age, And Race: Exploring The Deeper Contents Of Diversity Teaching & Learning The Facilitation Process A Brief On Debriefing: What It Is And What It Isn't Changing Perceptions Of The Importance Of Leadership: The Contribution Of Individual Spirit Harmonics In Leadership Knowledge, Skills And Sustainable Values In Learning Organizations: Some Implications From The Multicultural Virtual Classroom Challenges Of Teaching Undergraduate Organizational Behavior In A Nontraditional Time Format Interactive Online Positioning With The Web-Based Product Positioning Map Graphics Package The Longitudinal Effects Of Entrepreneurship Training On Risk Tolerance: A Look At Similarities And Differences Between Male And Female Undergraduate Students Revisiting Strategy Learning In A Total Enterprise Simulation What Are Simulations For?: Learning Objectives As A Simulation Selection Device Gaming Agency Markets Cooperate For Profits Or Compete For Market? Study Of Oligopolistic Pricing With A Business Game The Design Of A Business Simulation Using A System-Dynamics-Based Approach Modeling The Product Development Function For An Entrepreneurial Firm Simulation Performance And Forecast Accuracy? Is That All? Business Manager Identification Of Competitors In Real World And Simulation Settings Monte Carlo Simulation Analysis On The Costs Reduction Argument Of Interest Rate Swaps Ebiz Game: A Scalable Online Business Simulation Game For Entrepreneurship Training A Model For Online Education Delivery And A Look At Online Delivery Effectiveness Incorporating "Company Reputation" into Total Enterprise Simulations The Genesis And Future Of The Absel "Classicos" Initiative