DEMONSTRATING THE LEARNING EFFECTIVENESS OF SIMULATIONS: WHERE WE ARE AND WHERE WE NEED TO GO Developments In Business Simulation & Experiential Learning, Volume 24, 1997 Demonstrating the Learning Effectiveness of Simulations: Where We are and Where We Need to Go Philip H. Anderson, University of St Thomas Leigh Lawton, University of St Thomas ABSTRACT This paper focuses on the learning effectiveness of simulation exercises. It identifies a myriad of possible Learning objectives for simulations. Most of these objectives have been studied using perceptions as the dependent variable; few have been investigated using a more objective dependent variable. It is clear that objective measures have been used for a very limited range of learning objectives. Much of the reason for the limitations of our present state of knowledge can be attributed to our lack of attention or inability to develop appropriate, objective dependent measures of learning. Until appropriate objective variables are identified for a broader range of learning objectives, solid support for the learning effectiveness of simulations will be lacking. INTRODUCTION Since the early days of gaming, there has been a call for hard evidence to support the teaching effectiveness of simulations (see, for example, Neuhauser, 1976; Snow, 1976). The purpose of this paper is to identify where the deficiencies in our knowledge exist and to suggest directions for future research. An abundance of research has been conducted analyzing various dimensions of the simulation experience. The length of the bibliography in Keys’ and Wolfe’s 1990 review of the state of simulation is awe-inspiring. Despite the extensive literature, it remains difficult, if not impossible, to support objectively even the most fundamental claims for the efficacy of games as a teaching pedagogy. There is relatively little hard evidence that simulations produce learning or that they are superior to other methodologies. Much of the reason for the inability to make supportable claims about the efficacy of simulations can be traced to the selection of dependent variables and to the lack of rigor with which investigations have been conducted. If we are to increase our knowledge of what student’s gain from participating in a simulation, we must be more systematic in our research efforts. Can We Generalize the Learning Outcomes of Simulations? Much of the research on simulations deals with “the generic” simulation. While acknowledging that there are functional and TE simulations, conclusions are drawn as if all simulation experiences are alike. Is it justifiable to discuss the “simulation experience,” or must we treat each simulation experience as a unique case? The outcomes to be expected from a simulation depend upon several factors. Different games are designed to model different disciplines and to emphasize different learning outcomes. In addition, a simulation experience rarely consists exclusively of a game. Most game administrators use the game as the hub for a broader set of activities. For example, simulations may be run as team competitions; game participants may write plans and strategies; and the participants may deliver oral presentations and/or submit written reports on the performance of their team. The set of activities selected by the game administrator will almost certainly influence the learning that occurs over the course of the simulation. Given the diversity of simulation experiences that students may encounter, it may seem fruitless to attempt to generalize about the learning outcomes of a business simulation. However, the situation may not be as hopeless as it first appears. By the nature of a simulation, virtually all games, regardless of the 68 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 discipline for which they are developed, require the participants to make decisions under conditions of uncertainty. Almost all (over 90%) simulation users run their games as team competitions, and over 75% require the teams to prepare written plans (Anderson & Lawton, 1990). As a result, we should be able to draw some general conclusions about the learning effectiveness of simulations in their most commonly used form. What is needed to demonstrate the teaching effectiveness of simulations? Clearly specify learning outcomes Virtually all research designed to measure the outcomes produced by engaging in an activity requires, by necessity, assumptions concerning the expected outcomes of performing that activity. We cannot construct an assessment instrument without knowing what it is we wish to measure. For example, most simulations attempt to put the student in the role of a manager. Managing requires a broad range of knowledge and skills - such things as knowledge of the business discipline, interpersonal skills, problem- solving and decision-making skills, etc. Consequently, participating in a game may produce a wide range of outcomes. To assess whether a simulation is successful at teaching managerial skills, we must first identify which specific skills we wish to assess and then accurately measure them. The list presented in Table 1, identifies learning outcomes instructors may adopt as they strive to educate business students. These learning outcomes have been advanced in the simulation literature, as targeting the skills and knowledge needed by practicing managers. The items on the list were compiled from a review of simulation literature. The sources were: Miles, et al. (1986); Teach and Govahi (1988); Teach (1990); Hemmasi and Graf (1992); and Klabbers (1996). Miles, et al. drew items from two 1979 studies. Teach and Govahi state, “The literature was searched to define the skills and attributes that ‘managers’ need and the tasks they employ in plying their trade. A set of 41 tasks, skills and/or attributes was developed (Waters, 1980) (Livingstone, 1971) (Mintzberg, 1973).” Hemmasi and Graf drew several items from the Miles, et al. study and added a number of their own. Simulation users have speculated, and in many cases claimed, that game playing is an effective pedagogy for achieving many of these outcomes. However, there is evidence that some of these Learning objectives can be developed more effectively by pedagogies other than games. For example, Teach and Govahi (1988) conclude that, based on student perceptions, lectures are superior for learning to listen reflectively; cases are best for learning a set of nine skills including analyzing problems, conceptualizing, and writing effectively; and experiential exercises excel for a set of 17 skills. But it is not the purpose of this paper to review the literature and draw conclusions about the effectiveness of the various pedagogies. Several such reviews already exist [for example, Wolfe (1985); Keys and Wolfe (1990)]. Rather, it is the purpose of this paper to question what claims attributed to simulations can be supported by the objective evidence presented in the literature. 69 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 70 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 Note: There are some additional ‘non-learning’ objectives having to do with improved attitudes toward the course or toward business which may influence the choice of a pedagogy. These objectives include: Adding realism to the course 1 adding enjoyment to class 1 adding entertainment value to course1; and gaining a positive mental attitude -essential for a happy, successful career in life3. 1 Miles et al. (1986); 2Teach and Govahi (1988); 3Teach (1990); Hemmasi and Graf (1992); 5 Klabbers (1996) Note: A “p” in the first column indicates that at least one study exists with participant perceptions used as the dependent variable. An “o” in the second column indicates that at least one study exists with some objective measure used as the dependent variable. The Choice of Dependent Variables Many are Based on Perceptions There is a reasonably extensive literature concerned with the perceived effect of participating in a simulation. [See, for example, Hemmasi and Graf (1992); Anderson and Lawton (1989); Teach, Richard and G. Govahi, (1988); Miles, et al. (1986)] Many of the studies compared the perceived learning for simulations versus that for other pedagogies -particularly cases. The findings of these studies have not been consistent, particularly for the perceived learning from simulations versus cases. Even a replication by Anderson and Lawton (1989) found contradictory findings from the original Miles, et al. study (1986). In general, however, simulations have fared quite well, especially when compared to lectures. But, of course, these results are based on perceptions rather than on objective measures of learning. Table 1 is presented in an attempt to identify the gaps the current state of literature. The “p” in Column 1 of Table 1 denotes that at least one study exists where participant perceptions were used as the dependent variable for a learning outcome. The “o” in Column 2 of Table 1 indicates that at least one study exists where some form of objective measure was used as the dependent variable for a learning outcome. (So, for example, there have been studies measuring student perceptions of the extent to which simulations help them learn to resolve conflicts (item B-4 in Table 1 “p”), but there have been no such studies using some objective measure (no “o”). For the outcome, “Increase the student’s knowledge of basic principles and concepts of the discipline,” (item A in Table 1) there have been studies using student perception as the dependent variable and there have been studies using objective tests - thus, both “p” and “o”.) Even a glance at the table reveals that our review of the literature found studies dealing with perceptions for many of the learning outcomes, but objective studies have attempted to examine far fewer of these learning outcomes. While objectives involving improved attitudes toward a course or toward business (e.g., adding realism or enjoyment to a course) necessitate using attitudes and perceptions as the dependent measure, perceptions alone should not be taken as a sufficient indicator of merit for most of the other learning objectives. As far back as 1981, Parasuraman called 71 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 for a movement away from perceptions to more rigorous measures of learning. Yet, a decade and a half later, we still speak anecdotally of the effectiveness of simulations with few studies using rigorous, objective dependent variables to support our claims. A relatively recent departure from straight self- evaluations has been the move to solicit the opinions of team members. Instruments like the modified Stumpf’s Strategic Management Skills Questionnaire (see Wolfe, 19xx) and the Human Synergistics’ modified Managerial Effectiveness Profile System (see Anderson and Lawton, 1990) have been used in an effort to gather more objective information than a simple self-report. While the aim of this approach is laudable, these evaluations are still based on subjective perceptions. Further, the validity and reliability of these instruments have not been established for a classroom setting. Until such instruments have been validated, they must be viewed with the same suspicion as self-evaluations. Few Studies are Rigorous A second factor interfering with our ability to make definitive statements about the efficacy of simulations is that many of the studies purporting to demonstrate the prowess of simulations have employed weak designs. As Keys and Wolfe state, “...Many of the claims and counterclaims for the teaching power of business games rest on anecdotal material or inadequate or poorly implemented research designs. These research defects have clouded the business gaming literature and have hampered the creation of a cumulative stream of research.” (Keys and Wolfe, 1990). Many studies lack control measures to ascertain the influence of moderating variables. In addition, there has been a lack of replication to determine whether results are generalizable, or merely artifacts of a particular simulation. Yet this hasn’t prevented us from making claims as to the power of simulations. External Validity Studies are Inadequate to Assess Learning A handful of studies have been conducted to in an effort to establish the external validity of simulations. (See, Norris and Snyder, 1982; Wolfe and Roberts, 1986; and Wolfe and Roberts, 1993). While these studies make a useful contribution to our knowledge of the association between successful performance in a simulation and successful performance on-the-job, they are not substitutes for measures of what is, or is not, learned in simulations. Although Wolfe and Roberts’ (1993, p 22) contend that “external validity studies must be conducted because it is the ultimate test of this learning technique’s value,” these studies can not establish the learning effectiveness of simulations. Performance in a simulation exercise may, as Wolfe and Roberts state (1993, p. 25), serve “as a device for assessing potential managerial talent.” However, if we see a strong relationship between successful game performance and on-the-job performance, have we demonstrated that learning from the simulation caused better on-the-job performance or that performance in both arenas is dependent upon similar skills and knowledge? External validity studies cannot determine whether simulation performance is a cause or merely reflects some other, unmeasured variable or variables. Until we design experiments to control (and test) for these effects, we will not know if there is a causal relationship between simulation performance and later career success. DISCUSSION The Guide to Business Gaming and Experiential Learning (1990) has two excellent articles that address problems in evaluating the educational value of simulations - Burns, et al. and Wolfe. Neither of the articles emphasizes what is perhaps the single most difficult task in conducting research to establish the value of simulations - the development of appropriate dependent measures. Even if researchers are assiduously attentive to good experimental design, useful research results will not be achieved if the measure of learning is invalid. Bloom’s Taxonomy provides a useful guide to selecting and assessing learning outcomes, but as Bums et al. (1990, p. 262) point out, “...the Bloom et al. classification scheme does not solve all the various problems in the assessment of learning .. [even though] it does provide a framework within which to begin systematically working on these difficulties.” Looking at Table 1 it is clear that many of the skills are aimed at high levels of Bloom’s Taxonomy. Few 72 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 of the objectives are cognitive or affective. It will require an enormous amount of ingenuity to develop appropriate objective measures for most of these learning objectives; and for most of these objectives, we haven’t even begun to develop objective instruments. Only after we have valid instruments can be begin to investigate meaningfully the relative merits of different pedagogies. Even if valid instruments are developed, there are at least two other factors, which further complicate our ability to measure: (1) Relatively few instructors base their entire course on a simulation. Most use the game in combination with lectures, cases, etc. So, if learning is observed, how can we attribute the learning to the simulation rather than some other part of the course? (2) Many, if not most, of the learning objectives listed in Table 1 involve very broad skills. Even if a pedagogy is an extremely powerful learning technique, how much change can we expect to observe in such a deeply ingrained skill as, say, decision-making just as a result of a single course? These two factors suggest that we will not only need valid measures, but will we need extremely sensitive measures. The upshot of paper is that the reason for the lack of objective support for the learning effectiveness of simulation games cannot simply be attributed to sloth or ignorance. Perceptions and attitudes have been over- used because we know how to measure them. However, even with the best of intentions and the greatest of rigor, we will not succeed in assessing the educational merits of simulations until we develop better dependent measures; measures which are objective. Table 1 highlights the many areas where dependent measures must be identified or developed if we are to examine the broader effects of using a simulation. How Should we Proceed? An issue that has arisen in past ABSEL meetings when discussing the assessment of learning, concerns the disagreement between those who emphasize the need to measure learning versus those who advocate the need to measure mastery. Perhaps the difference between these two camps is more a matter of semantics than a real difference in opinion. Nevertheless, in an effort to clarify the situation, we offer the following: One model for establishing learning involves using a pre-, post-design. The difference between the score on the first assessment and the second assessment is taken to be the amount of learning that has taken place. This approach yields the most direct measure of learning since it yields a measure of change in cognition or behavior for each individual participant. A second model for establishing learning is an after-only, with control group design. If we take two randomly selected groups and expose them to different pedagogies (e.g., one group is taught using cases while a second group uses a game, we take the difference in scores between the two groups as our measure of the relative effectiveness of the two pedagogies. This approach to establishing learning is less direct than the first model, since we get no measure of learning at the individual level. We contend that the after-only, with control group provides a more useful measure. As Burns et al. (1990, p. 253) note, “implicitly we should be seeking to discover and adopt the ‘best’ pedagogical climate for our students, for, in the absence of this concern, the educator is continually plagued with opportunity cost and efficient time use questions.” Using this second model does not require that we measure learning at the individual participant level. Rather we can measure the mastery of one group against the mastery of a second group and draw conclusions about the relative effectiveness of the two pedagogies. This is not to say that only one of these models is correct. But we need to be clear about what we intend to measure and then use a model that is appropriate for our purpose. Regardless of which of these two models a researcher employs, It is imperative that the validity and reliability of the measurement instrument is established. We can continue to regale ourselves with anecdotal evidence about the ‘wonders of simulation,’ but we are preaching to the converted. If we truly want others to respect and adopt simulations, we need to provide objective evidence as to its efficacy. Reference available upon request. 73 Table of Contents Volume 24, 1997 Incorporating Computer Telephony into the MIS Course A Learner Oriented Infrastructure for videoconferencing Based Distance Education Courses The Identification of Temporally Related Structural Elements of the Experiential Component of Electronic Spreadsheet Tasks Does Involvement Influence Learning from Simulation Participation? Some Relationships with Helpfulness and Performance Outcomes Musings on Business Garne Performance Evaluation Performance on a TE Simulation: What does it represent? The Business Policy Game: An International Simulation - An Assessment Tool Students in Free Enterprise as Experiential Learning The Role of Computer Models in Wargames: A Practitioner's view Using Focus Groups as a Tool to Research Learning: A Demonstration and a Discussion Using Medical Simulations to Teach Multi-Cultural Diversity Using the World Game to Internationalize International Exchange Game Community Services Needs Assessment: An Innovative Approach Introducing Students to Potential Total Quality Management Ethical Dilemmas Privacy in the Workplace: A Situational Analysis Antecedents of Learning in Simulations Demonstrating the Learning Effectiveness of Simulation: Where we are and Where we need to go The Use of Computer Simulations as a Pedagogical Aid in Teaching Management Information Systems Evaluating Simulation Learning in a Distant Learning Instructional Model Financial Engineering of Global Investments The Labor History Game: Playing with the Past Perks Participation Assessing Negotiator's Proficiency with a Negotiation role-play The Art of Negotiating Enhancing Learning and Employee Development through the Assessment of Learning Pedagogy Preferences across selected Dimensions of Culture: A Preliminary Investigation Cooperative Learning: What are we Learning? Effective Use of Mastery Based Experiential learning in a Project Course to improve skills in system analysis and Design Simulations and Learning: Can we prove a relationship? (Seminar) Predicting and Reviewing NYSE Stock Prices by use of Basic Statistical Analysis and Logic Toy Car Depreciation Exercise ABSEL as Home Community: An Interactive Exploration Service Learning: Linking Academic Study to Community Development and Business Enhancement Ability of Efficient Evaluation of Knowledge-Based Management Strategies Corporate Ethics Training Programs Modeling Attributes in Demand Functions of Computerized Business Simulations: An Extension of Teach's Gravity Flow Algorithm Expert Systems Combined with Neutral Networks: Tools to Benefit the Marketing Researcher Computer Game Design: New Directions for Intercultural Simulation Game Designers Consistency in Simulation Performance over Time and Across Simulation Games The Impact of an Artificial Market Leader on Simulation Competitor's Strategies Business Plans, Case Studies, and Total Enterprise Simulations: A Natural Co-existence An Exploration into the Non-Use of Business Simulations The Market Game: Interactive Learning Through Market Simulation Plotting Brand Trajectories with the COMPLETE PPM package: A Market Segmentation Analysis and Positioning Tool Me and Mine Inc: An Exercise in Management Theory Learning to Differentiate Leadership from Managerial Position Business Ethics Survey: A Perspective from the Retail Industry Experiential Learning in Demand Analysis for an Agricultural Commodity Marketing on the Internet: A Pedagogical Exercise College Students Need Simple Computer Simulations, Especially for International Business Courses An Analysis of Student Attitudes, Performance, and Strategies in a Simulation Competition Based on a Controlled Product-Market-Entry Game Structure The Crystal Enterprise: Application of a Non-Computerized Simulation Model in a Process of Organizational Change Threshold: A Windows-Based Behaviorally Oriented Total Enterprise Simulation Coaching Business Game Teams Using a Decision Variable Optimizer Rock and Roll is here to Stay: Enhancing Experiential Pedagogy with Musical Exercises An Example of Business Process Analysis Simulation for Customer Software Support The Use of Business Gaming in Hong Kong Academic Institutions Using the Integrative International Simulation INTOPIA Mark 2000 in a Concentrated MBA Curriculum A Group Experience as an Integrated Part of the Core Management Course Empowered Learning in the Classroom The Inter-Group Interaction: An Innovative Approach to Cooperative Learning Courses that Utilize Student Team: An Approach to enhancing their Effectiveness Thoughts about the Measurement of Learning: The Case for Guided Learning and Associated Measurement Issues Measuring Student Learning Using Business Simulations: A Theory Based Perspective On the Use of PC Fingame in an Undergraduate Finance Course How Managers get Things Done: A Virtual Soundbite Internet Experiential learning in the Principals of Marketing Classroom: A Pedagogical Approach Current Student Perceptions Relative to Business Simulations Beyond Capitalism: Designing Business Policy and Social Justice The Use of Boards of Directors to Evaluate Reports and Presentations in an Undergraduate Business Policy Course Beacon Lumber: An Experiential Introduction to Financial Accounting The Application of Organizational Motivation Principles: The Experiential Business Simulation Motus Manufacturing Avoiding a Bogey: Grading Case Discussions Scientifically Communicating Consumer Behavior II: A Modified Exercise Using Personal Consumption Journals in Condensed Courses Designing Instruments for Assessing the Effectiveness of Simulations, (Seminar) 360º Performance Feedback: Appraisal vs. Assessment The Cafeteria Approach to Managing an Academic Career: Remaining Non-Perishable while Doing Your Own Thing An Experiential Exercise Related to Person-Organization Fit and it's Consequences For Today's Dynamic and Changing World, Business Simulations need to be expanded to encompass Much Greater Complexity: A Demonstration The Incident Process: A Case in Reverse Contextually-Anchored Business Simulations The Energy Factor: Building Motivation in the Simulation Gaming Environment The Design of an Internet Game