AN ASSESSMENT FRAMEWORK FOR DETERMINING THE EFFECTIVENESS OF TOTAL ENTERPRISE SIMULATIONS Development In Business Simulation & Experiential Exercises, Volume 21, 1994 16 AN ASSESSMENT FRAMEWORK FOR DETERMINING THE EFFECTIVENESS OF TOTAL ENTERPRISE SIMULATIONS Stephen J. Snyder. The University of West Florida ABSTRACT The current state of total enterprise strategy simulations is assessed within a framework of four dimensions for adopters to consider. As such, simulations are viewed in terms of their user friendliness. comprehensiveness, theoretical grounding, and adaptability. This view builds on Snyder’s (1993) work on providing a theoretical means for assessing total enterprise simulations. The paper concludes with a proposed questionnaire to assess the current mix of strategy simulations on the market. INTRODUCTION Recent writings on total enterprise simulations have focused on the external validity of simulations (Wolfe & Roberts, 1993; 1986), algorithmic designs (Wolfe & Jackson, 1989), types of courses using total enterprise games (Faria, 1987). and dimensions of impact in simulations (Miller & Leroux-Demers, 1992). By “total enterprise games” I refer to simulations that model the functional areas of production, marketing, finance, and personnel (Keys, 1987). These simulations are pervasive in most capstone courses in business curriculums, yet assessments of such simulations lack empirical or theoretical grounding. This paper further refines Snyder’s (1993) model for assessing the usefulness of total enterprise simulations by adding an additional construct and presents a questionnaire that will assist adopters in assessing the worthiness of existing business strategy simulations. TOTAL ENTERPRISE SIMULATIONS A wealth of literature exists on the worthiness of simulations as an experiential tool in the learning process (Gentry, 1990). Yet, such experiential tools are only as good as 1) the lines of code in software programs and 2) the ability of instructors in using the simulation. Currently there are over fifteen different total enterprise simulations on the market for instructors to select. Yet only anecdotal data exists (usually from publishers) for adopters to use when deciding on an existing simulations. Consider the range of available simulations: Thompson & Stappenbecks’s Business Strategy Game, Smith & Golden’s Airline, Scott & Strickland’s Tempomatic IV, Henshaw & Jackson’s The Executive Game, Keys & Leftwich’s The Executive Simulation, Cotter & Fritzsche’s Business Policy Game, Priesmeyer’s Strategy!: A Business Unit Simulation, and Eldridge & Bates’ The Business Strategy and Policy game all claim to be top-notch total enterprise games. Yet, these simulations differ in focus, content and complexity. This makes the learning curve for switching from one simulation to another steep, and necessitates picking a satisfactory simulation the first time. Adopters can little afford to switch from simulation to simulation in the search for one that fits their needs. A need exists for adopters and potential adopters to have a means to screen what simulations they will consider. Similarly, researchers need a means for assessing the worth of existing simulations so that future developments can be focused on improving rather than reinventing “the wheel” of software programming. This fact makes any assessment of currant total enterprise simulations invaluable. Faria (1987), building from Biggs’ (1979) work on the use of games in schools, estimated that approximately 1,914 schools and 3,287 courses use business simulations. This estimate was deemed conservative in 1987. The extent of this use makes simulations a force in our educational system. As such, the use of simulations in the classroom can be considered a resource that cannot be wasted. Yet Keeffe at al. (1993) found that overall use of strategic management simulations went down from 48.4% in 1985 to 46% in 1990 (not significant at the .05 level, n=63). In fact, Keeffe at al. found that the percentage of professors who have never used a simulation on the course increased from 1 6% in 1985 to 22% in 1990. This downturn in use may be due to a general confusion in the market relating to the usefulness of specific simulations. In other words adopters may have found that the use of the simulation had serious flaws or was either too complex or simple to successfully use in the classroom. A means to assess existing simulations may provide valuable information for those considering adopting simulations, or for those who seek to switch from a simulation they currently use to one that better fits their needs. A FRAMEWORK FOR ASSESSMENT Snyder (1993), building on Keys (1987), developed a framework for assessing total enterprise simulations. He identified the importance of comprehensiveness, user-friendliness and theoretical grounding. To this list I have added an additional variable; adaptability. Each of these four dimensions will be discussed with the goal of providing a means for assessing existing total enterprise simulations. If successful, the survey will serve as a ready reference to adopters and developers when considering existing simulations. The need for developing an instrument stems from the fact that, like adopters researchers are hard-pressed to learn the workings of more than one simulation at a time. User-Friendliness Total enterprise simulations typically come with both player’s and instructor’s manuals. User friendliness, for the adopter, is a concept that relates to ease in learning the simulation, the degree of on-line computer or telephone help, the ability of the simulation to correct problems -- such as corrupted programs, and ease of use for students using the simulation. While on-line help features exist for students and adopters, the addition of a toll-free hot line to the game developers allows new problems to be handled immediately. Part of the user-friendliness assessment ignored by Snyder (1993) is the specifications for running the game. For example, some simulations can be run on IBM-XT machines, but the slowness in running the game on such arcane machines obviates any real consideration. In addition, certain simulations cannot be run on a networked personal computer due to memory requirements. These are user-friendliness issues typically addressed in the beginning of the instructor’s manual. Unfortunately, other problems, such as bugs in the software are only uncovered once the game has been adopted and used. Comprehensiveness Snyder specified that “the degree of complexity of a given simulation need not correlate with the level of comprehensiveness” (1993, 138). By comprehensiveness, Snyder referred to the ability of the game to include a degree of rigor in its modeling of all functional areas of the total enterprise, including management, finance, accounting, marketing, economics, and production/operations. The degree of rigor in modeling each functional area of the total enterprise is critical to display realism. Users are quick to discover any shortcomings in simulations that allow an advantage or a way to “beat the system”. This destroys both realism and the learning experience. Development In Business Simulation & Experiential Exercises, Volume 21, 1994 17 In 1989 Wolfe & Jackson conducted a study on the need for algorithmic validity. They concluded that games range in the degree of realism in algorithms. The logical or conceptual model of each game developer is transformed into game format through the modeling of algorithms that link variables together. Any game is only as good as the algorithms that have boon written for it. For instance, some simulations emphasize the global arena by including manufacturing and distribution capabilities in overseas operations. Taking the step into the global marketplace involves another level of difficulty to the software developer, since variables such as exchange rates, differing rates of inflation, and tariffs must be built into the game. However, it is useless to model such variables unless the algorithms correctly match real world occurrences. For instance, one simulation The Business Strategy Game models the effects of exchange rate fluctuations by multiplying such changes by ten for a given decision period of one year. Using such e formula con give drastic changes in exchange rate effects from year to year. Using exchange rates in the model isn’t incorrect, but the algorithm formula creates substantial financial gains and losses due to exchange rate fluctuations, sometimes severe enough to absorb whatever margin existed for a given team. Garners are adept at figuring out how algorithms work. If they can find a way to manipulate stock price or productivity they will, since the object of the game is winning. Game developers face the dilemma of providing realism in an environment where realism is growing in complexity. Adopters of a particular game typically hove the option of setting up a given industry’s parameters, such as growth rote, tariffs, and productivity. To simulate real world conditions, other inputs are made for each decision by tracking current business markets, such as S&P 500 or Dow Jones composites. Any algorithm magnifies incremental changes in such variables to simulate how such changes would affect markets given longer time horizons. Therefore, sensitivity analyses are incorporated into the writing of the algorithms. Any simplification of real-world conditions, such as algorithms, is prone to defects. Increasing the number of variables, and therefore increasing the number of cause-effect relationships between variables, is bound to increase flaws in the simulation. This challenge is more critical than the number of variables in the simulation since any minor algorithm flaw can destroy the purpose of the simulation. Theoretical Grounding The underlying theory linking the algorithms together is as important as the software itself. Tying conceptual models of the total enterprise to the simulation lends support f or the theoretical models taught in the classroom. The importance of theoretical grounding was mentioned by Snyder when he specified that lock of theoretical justification provides no real understanding of cause-effect relationships”. These cause effect relationships in simulations should have o basis in real world market dynamics. For instance, current trends in management include chaos theory, reengineering, and just-in-time inventory control. Such dynamics need to be modeled in simulations to lend credence to the outcomes of a given game (e.g. performance indicators). Without considering process models of strategy simulations tend to become esoteric, not fitting the organizations it was designed to simulate. Unfortunately, simulation realism--measured in terms of its theoretical grounding--can only be assessed once the simulation is conducted. Therefore, a means for assessing this dimension before a simulation is chosen would be equally valuable and would ovoid much of the guinea- pig” use of simulations in determining effectiveness. Adaptability The last dimension to be considered in assessing the value of a total enterprise simulation is adaptability. The term “strategy” has been used as a catchall word for any gem relating to the types of decisions that top managers make. This has resulted in the marketing of simulations that may appear on the surface to be true comprehensive strategy games, but in fact focus on one functional area. For example. Strategy & Competition (Pitta, 1989) takes a more marketing oriented approach, whereas Airline (Smith & Golden, 1991) focuses on cash flow concerns. Emphasis in a given simulation has direct influence to the game’s utility to adopters. Strategy & Competition is more appropriate for a marketing audience than individuals seeking a working knowledge of strategy, since the game’s algorithms center on the product life cycle. Adaptability refers to the ability of a given simulation to be used for different audiences end in different contexts. There is a wide range of strategy-related courses at the college and university level that would be enhanced by the inclusion of a game. Few games currently on the market ore appropriate or sufficient for application in international business classes. Strategy simulations that are highly adaptable would be appropriate for any strategy or policy course, decision-making course, marketing strategy course, production or operations course, or finance course. Since strategy is en integral component of any functional area, simulations must identify whether their focus is solely on the area of strategic management, or if the game is appropriate for applications in other functional areas. Merely labeling the game as “strategic” does not specify the parameters within which the game holds relevancy. The ability of a total enterprise simulation to be used in multiple course in the business curriculum allows for a more integrative approach in teaching in business, something the AACSB is recently encouraging. An assessment of a simulators adaptability would further this endeavor. A SURVEY Appendix 1 outlines a survey to be used to assess the mix of total enterprises on the market. The survey blends questions relating to the four dimensions discussed above. Provision is also mode f or open-ended questions. A 7-point liken format was used in developing the questions. The questionnaire was export-reviewed by 5 simulation users to determine proper question wording. The goal of the survey is to build a sufficiently large poof of responses for each of the major total enterprise simulations so that comparisons can be made. Interrater reliability is of concern. It would be more appropriate to have several researchers analyze o range of simulations on the four dimensions. However, correct evaluation of simulations con take 2 or 3 semesters of use, by which time successive generations of simulations come to market. What is sought is realistic approach to assessing total enterprise simulations. The life e current total enterprise simulation is approximately 2 years. Any assessment made after that time period is worthless to someone considering adopting the simulation, since another edition will be developed or the simulation will be discontinued. CONCLUDING REMARKS Norris (1986) and Wolfe and Roberts (1993. 1986) studied the external validity of business simulations to determine the ability ate simulation to prepare managers for real-world demands. Gold & Pray (1982, 1984) looked inside simulations to analyze internal validity of specific variables, such as demand functions, elasticity of prices, and stockouts. This paper has concerned itself with a multi-dimensional approach to assess internal validity of total enterprise simulations. In this vein, I consider the four dimensions presented above the key to determining the validity of any findings relating to external validity. Development In Business Simulation & Experiential Exercises, Volume 21, 1994 18 Moving on to notions of external validity before assessing the strengths of existing total enterprise simulations is putting the cart before the horse. The logical next step, now that a means has been developed for assessing the mix of existing total enterprise simulations is to poll adopters and researchers on their experiences in using existing simulations. Once that has been completed an empirically driven assessment will exist for adopters end potential adopters to view before the selection of a simulation is made. Such an assessment also holds value to developers as they seek to improve on existing simulations. APPENDIX I A SURVEY ON TOTAL ENTERPRISE GAMES Please respond to the questions below concerning the computer simulation(s) you use/have used. 1. Name of Simulation ___________________ 2. Publisher of Simulation ___________________ 3. Version 1 _______ 4. Release Date ______ 5. Years you have used a computer simulation in the classroom ________ 6. Approximately how many different computer simulations have you used in the classroom? 7. What are the course titles in which you use computer simulations? Please rate the simulation you currently use on the following dimensions: Quality Rating Low high 8 How would you rate the simulation on the quality/accuracy of how it models real-world organizations? 1 2 3 4 5 6 7 9. To what degree does the simulation model the total enterprise or take into consideration all functional areas of the organization? 1 2 3 4 5 6 7 10. How would you rate the level of user-friendliness of the simulation? 1 2 3 4 5 6 7 11. What is your assessment of the quality of theoretical grounding or the modeling of current trends in your field in the simulation? 1 2 3 4 5 6 7 12. What is your assessment of the simulation’s ability to be used in classes/environments outside of your functional area (i.e. marketing classes as wall as management classes)? 1 2 3 4 5 6 7 13. Will you be using your current simulation next semester/quarter? If no, what is the primary reason for not using the current simulation? REFERENCES Arellano, F.E., & Hopkins W.E. (1992). “Modeling economic environments in business simulations: some comparisons and recommendations”. Developments in Business Simulation & Experiential Exercises. 1 9, 7- 10. Biggs W.D. (1979) “Who is using computerized business games? A view from publisher’s adoption lists”, in Proceedings of the Association for Business Simulation and Experiential Learning, 202-206. Cotter, R.V. & Fritzsche, D.J. (1991) The business policy game. Third Edition, Englewood Cliffs NJ: Prentice-Hall. Eldridge, D.L. & Bates, D.L. (1984) The business strategy & policy game. Neadham, Mass: Simon & Schuster. Faria, A.J. (1987) “A survey of the use of business games in academia and business” Simulation & Games. 18:2, 207-224 Gentry, J. W. (1990) Guide to Business Gaming and Experiential Learning. East Brunswick: Nichols/GP Publishing. Development In Business Simulation & Experiential Exercises, Volume 21, 1994 19 Gentry, J.W., Stoltman, J.J., & Mehlhoff, C.E. (1992) “How should we measure experiential learning?” Developments in Business Simulation & Experiential Exercises. 19, 54-57 Keefe, M. J., Dyson, D.A., & Edwards, R.R. (1993) “Strategic management simulations: a current assessment”. Simulation & Gaming 24:3, 363-372 Keys J.B. (1987). “Total enterprise games”. Simulation & Gaming 18:2, 225-241 Keys, J.B,, & Wolfe, J. (1990). “The role of management games and simulations in education and research”. Journal of Management 16, 307- 336 Klein, R.D. & Fleck, RA (1 990). “International business simulation/gaming: an assessment and review”. Simulation and Gaming, 21-2 147-165. Miller, R. & Leroux-Demers (1992) “Business simulations: validity and effectiveness”. Simulation/Gaming for Learning, 22:4, 261-285. Norris, D.R. (1987) “External validity of business games” Simulation & Gaming. 17:4, 447-459 Pitta, D.A. (1989) Strategy & Competition Boston: Allyn & Bacon. Pray T.F. & Gold, S.C. (1982). “Inside the black box--an analysis of underlying demand functions in contemporary business simulations”. in Developments in Business Simulation and Experiential Exercises, 9, ads. David Fritzsche and Lee A. Graf., Normal: Illinois State University, 110- 115. Pray T.F. & Gold, S.C. (1984). “Two algorithms for redistribution of stockouts in computerized business simulations”, in Developments in Business Simulation and Experiential Exercises, 11, ads. David M. Currie and James W. Gentry, Stillwater: Oklahoma State University, 247-252 Priesmeyer, R.H. (1987) Strategy! a business unit simulation. Cincinnati, Oh: SouthWestern Scott, T.W., Strickland, A.J Hofmeister, D.L., & Thompson, M.D (1992) Micromatic: a strategic management simulation. Boston: Houghton Mifflin Smith, J.R., & Golden, P.A. (1991) Airline: a strategic management simulation. Englewood Cliffs, NJ: Prentice-Hall. Snyder, 5. (1993). “Strategy simulations in context: An evaluation of key dimensions in development”. Proceedings of the Association for Business Simulation and Experiential Learning , 138. Stanislaw, H. (1986) “Tests of computer simulation validity: what do they measure”. Simulation & Games 17:2, 173-191 Teach, R.D. (1990) “Designing business simulations”. in Guide to business gaming and experiential learning. East Brunswick: Nichols/GP Publishing Thatcher, D.C. & Robinson, M.J. (1986) “A simulation in the design of simulations” Simulation & Gaming. 21:4, 256-261 Thavikulwat, P. (1991) CEO: a business simulation for policy and strategic management. New York: McGraw-Hill. Thompson A.A., & Stapenbeck, G.J. (1990) The business strategy game. Home wood, IL: Irwin. Wolfe, J. & Roberts, C.R. (1993) “A further study of the external validity of business games: five-year peer group indicators”. Simulation & Gaming 24:1, 21-33 Wolfe, J. & Jackson, A. (1989) “An investigation of the need for algorithmic validity” Simulation and games. 20:3, 272-291 Wolfe, J. & Roberts, C.R. (1986) “The external validity of a business management game: a five-year longitudinal study”. Simulation & Games 17~1 45-59. Wolfe, J. (1985) “The teaching effectiveness of games in collegiate business courses”. Simulation & Games 26, 251-288 Yeo, G. K. (1991) “A framework for developing simulation game systems” Simulation & Gaming. 22:3, 308-327 Table of Contents Volume 21, 1994 ABSEL: The Way We Were and Need to Be The Intellectual Structure of ABSEL: A Bibliometeic Study of Author Cocitatons Over Time Activity-Driven Time in Computerized Gaming Simulations An Assessment Framework for Determing the Effectiveness of Total Enterprise Simulations Attributes of Learning Organizations: Simulating the Relationships Business Policy/Strategy Case Extension using Pro-Forma Planning: A Computer Based Model Complexity: Is it really that Simple? A Random-Strategy Criterion for Validity of Simulation Game Participation Enhancing a Computer Simulation with a Structured Reporting Environment Experiencing a Foreign Culture: A Cross-Cultural Simulation Group Cognitive Style and Computerphobia in Functional Business Simulations Human Issues in Technology Implementation Management Simulator Incorporating Advertising Strategy into Computer-Based Business Simulations: A Validation Study Increasing the Effectiveness of Performance Evaluation Through the Design and Development of Realistic Finance Algorithms The Packer-Feeder Game: A Commodity Market Simulator Relationships Between R&D and Profitability: An Exploratory Comparison of Two Business Simulations with Two Real-World, Technology Intensive Industries Simulation of the Predictive Value of Mammography Simulation Performance and Learning Revisited Strange Bedfellows: Competency Models and ACBSP Accreditation Standards Using a Business Simulation to Study the Determinants of Ethical Behavior What Simulation Users Think Players Should be Learning from the Simulations ADA and its Implications for Experiential Training Boss/ Subordinate Perceptions of Instrumental and Supportive Leadership Behaviors in Relation to Myers-Briggs Thinking Type Cluster Analyses of American Universities' Business Core Curricula Structures Utilized to Satisfy Fifteen Curriculum Areas Cooperative Learning or Learning to Cooperate Experiential Learning: Constraining Students with Time Budgets How Different Workplace Experiences Affect Different Worker Values Implications of the Trend Toward Relationship Marketing for Experiential Learning The Increasing Cultural Diversity of the American Workforce: Management's Challenge of the 21st Century Information and Uncertainty as Strange Bedfellows: A Model and Experiential Exercises Leadership as a Medium: It's Emergence and Effect on Performance in Small Leaderless Groups Speed, Depth, and Breadth: Assessing Learning in Learning Organizations Teaching Strategic Planning, Problem Solving, and Decision Making with Envisionary Experiential Exercises Validating an Instrument for Student Evaluation of Teachers: Some Noteworthy By-Products Don't Teach Ethics to Business Students Emotional Reactions Toward a Simulated Layoff: Before and After the Manipulation Enhancing Communication Using a Presentation Package Implementing Marketing Policies through a Business Management Simulation Integrating Action-Based Learning into Executive Development Programs On the validity of using the Microsegmentation Principle in Media Simulationsm Participatory Systems Analysis Some Relationships between Cultural, Organizational, and Educational Experience and Perceptions of Influence The Use of Decision Support Systems with a Marketing Simulation: The Future is Now Astute Business Policy: A Simulation of the Automobile Industry The Business Policy Game CEO II: A Gaming Simulation for Assessment Computer Paced Project Management Simulation Computerized Tutor Support Systems DEAL & GEO: Progressively Integrating Gaming Simulations for Entrepreneurship and International Business An Interactive Simulation Game for Competitive Decision-Making International Operations Simulation/Mark 2000 (INTOPIA) Multimedia Simulation Cuts Training Costs for Anderson Consulting Concepts of Total Quality Management: An Active Learning Exercise Cooperative Learning: The Extended Jigsaw Managing Diversity--Values and Attitudes: An Experiential Exercise in Awareness Navigating the Shoals of International Management Development Evaluating Student Performance in the Use of Computer Simulation Entrepreneurial Simulation Program