SIMULATIONS AND LEARNING: CAN WE PROVE A RELATIONSSHIP? Developments In Business Simulation & Experiential Learning, Volume 24, 1997 SIMULATIONS AND LEARNING: CAN WE PROVE A RELATIONSHIP? Philip H. Anderson, University of St Thomas Leigh Lawton, University of St Thomas SUMMARY Assessing what students learn on a business simulation exercise has been a focus in the literature and at ABSEL conferences since simulation exercises were first introduced (Greenlaw and Wyman, 1973; Keys, 1976; Wolfe, 1985; Whiteley and Faria, 1989; Burns, Gentry, arid Wolfe, 1990; Wolfe, 1990; Gosenpud, 1990; Wellington and Faria, 1991; Anderson and Lawton, 1992; Gosenpud and Washbush, 1993, 1994; Anderson and Lawton, 1995; Washbush and Gosenpud, 1995). Debate continues as to what is learned and can it be proven with hard data versus anecdotal evidence. A seminar on this topic organized by Jim Gentry for ABSEL’s 1996 conference resulted in a crowed room, much debate, but little resolution. The focus of this proposed seminar is to continue to push this debate further. As pressures to demonstrate outcomes increase, educators find themselves increasingly required to present evidence of the merits and validity of their pedagogies. Beyond external pressures, we must understand the learning outcomes of a simulation if we are to integrate it with other teaching methodologies to optimize learning. The design of the seminar will center around 5 -10 minute presentations by four teams of researchers that have made continuing contributions to ABSEL on this topic. Their presentations will be followed by a overview and reflection of the presentations by Joe Wolfe. This will be followed by an open discussion. The following outline identifies the four teams of researchers, and the principal points in their position on the topic. OUTLINE Gentry and Burns • “Simulation gaming” and “learning” are not measurable constructs, at present. • There is a large variation in simulation gaming experience. More precision in pedagogical assessments of simulations is needed. • Learning is improperly operationalized. • Learning is not performance, it is value-added. • The relationship between simulations and learning cannot be properly assessed until the constructs are operationalized adequately. Faria and Wellington • Relationships between simulation participation and learning have been established. • Good simulation performers score better on final examinations. • Students learn to identify different marketing environments (e.g., push vs. pull). • Good simulation performers are able to repeat their performance. Gosenpud and Washbush • No relationship has been established, yet. • Due to measurement problems, a relationship has not been found, but one exists. • There is an association between what one learns from an experience and how one behaves during that experience. • Capturing the association between learning arid performance will be very difficult with “single score” measures. Multiple measures of each are needed. 109 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 Anderson and Lawton • Only a very weak link between participation in a simulation and learning has been shown. • Valid, reliable instruments to assess “mastery” are rare, and valid measures of “higher level” learning objectives are almost non-existent. • There is a need to understand the kinds of learning simulations are capable of producing. • Each simulation user must identify clearly the kinds of learning he or she hopes to produce through use of a simulation. • Valid instruments need to be developed for each kind of learning we identify. • Instruments must be used to assess whether mastery of learning objectives exists. Joe Wolfe • Overview response to presentations and reflections. REFERENCES Anderson, P.11 and L. Lawton (1992), “The Relationship Between Financial Performance and other Measures of Learning on a Simulation Exercise”, Simulation & Gaming, 23, 326-340. Anderson, P.H. and L. Lawton (1995), “The Problem of Determining an “Individualized” Simulation’s Validity as an Assessment Tool”, Developments in Business Simulations and Experiential Exercises, 22, pp.43-48. Burns, A.C., J.W. Gentry, and J. Wolfe (1990), “A Cornucopia of Considerations in Evaluating the Effectiveness of Experiential Pedagogies,” in Guide to Business Gaming and Experiential Learning, J.W. Gentry (ed.), New York: Nichols/GP Publishing, 25 3-278. Gosenpud, J. (1990), “Evaluation of Experiential Learning,” in Guide to Business Gaming and Experiential Learning, J.W. Gentry (ed.), New York: Nichols/GP Publishing, 301-329. Gosenpud, J. and J. Washbush (1993), “The Relationship Between Total Enterprise Simulation Performance and Learning.” Developments in Business Simulations and Experiential Exercises 20, 141. Gosenpud, J. and J. Washbush (1994), “Simulation Performance and Learning Revisited.” Developments in Business Simulations & Experiential Exercises, 21, 83-86. Greenlaw, P.S. and F.P. Wyman (1973), “The Teaching Effectiveness of Games in Collegiate Business Courses.” Simulation and Games, 4 (3), 259-293. Washbush, John and Jerry Gosenpud (1995), “Simulation Performance, Learning, and Struggle,” Developments in Business Simulation & Experiential Exercises, 22, 1-4. Wellington., W.J. and A.J. Faria (1991), “An Investigation of the Relationship Between Simulation Play, Performance Level, and Recency of Play on Exam Scores”, Developments in Business Simulations and Experiential Exercises, 18, 177. Whiteley, T.R. and A.J. Faria (1989), “A Study of the Relationship between Student Final Exam Performance and Simulation Game Participation.” Developments in Business Simulations & Experiential Exercises, 16, 78-82. Wolfe, J. (1985), “The Teaching Effectiveness of Games in Games in Collegiate Business Courses: A 1973-83 Update.” Simulation and Games, 16, 25 1-288. Wolfe, J. (1990), “The Evaluation of Computer- based Business Games: Methodology, Findings, and Future Needs,” in Guide to Business Gaming and Experiential Learning, J.W. Gentry (ed.), New York: Nichols/GP Publishing, 279- 300. 110 Table of Contents Volume 24, 1997 Incorporating Computer Telephony into the MIS Course A Learner Oriented Infrastructure for videoconferencing Based Distance Education Courses The Identification of Temporally Related Structural Elements of the Experiential Component of Electronic Spreadsheet Tasks Does Involvement Influence Learning from Simulation Participation? Some Relationships with Helpfulness and Performance Outcomes Musings on Business Garne Performance Evaluation Performance on a TE Simulation: What does it represent? The Business Policy Game: An International Simulation - An Assessment Tool Students in Free Enterprise as Experiential Learning The Role of Computer Models in Wargames: A Practitioner's view Using Focus Groups as a Tool to Research Learning: A Demonstration and a Discussion Using Medical Simulations to Teach Multi-Cultural Diversity Using the World Game to Internationalize International Exchange Game Community Services Needs Assessment: An Innovative Approach Introducing Students to Potential Total Quality Management Ethical Dilemmas Privacy in the Workplace: A Situational Analysis Antecedents of Learning in Simulations Demonstrating the Learning Effectiveness of Simulation: Where we are and Where we need to go The Use of Computer Simulations as a Pedagogical Aid in Teaching Management Information Systems Evaluating Simulation Learning in a Distant Learning Instructional Model Financial Engineering of Global Investments The Labor History Game: Playing with the Past Perks Participation Assessing Negotiator's Proficiency with a Negotiation role-play The Art of Negotiating Enhancing Learning and Employee Development through the Assessment of Learning Pedagogy Preferences across selected Dimensions of Culture: A Preliminary Investigation Cooperative Learning: What are we Learning? Effective Use of Mastery Based Experiential learning in a Project Course to improve skills in system analysis and Design Simulations and Learning: Can we prove a relationship? (Seminar) Predicting and Reviewing NYSE Stock Prices by use of Basic Statistical Analysis and Logic Toy Car Depreciation Exercise ABSEL as Home Community: An Interactive Exploration Service Learning: Linking Academic Study to Community Development and Business Enhancement Ability of Efficient Evaluation of Knowledge-Based Management Strategies Corporate Ethics Training Programs Modeling Attributes in Demand Functions of Computerized Business Simulations: An Extension of Teach's Gravity Flow Algorithm Expert Systems Combined with Neutral Networks: Tools to Benefit the Marketing Researcher Computer Game Design: New Directions for Intercultural Simulation Game Designers Consistency in Simulation Performance over Time and Across Simulation Games The Impact of an Artificial Market Leader on Simulation Competitor's Strategies Business Plans, Case Studies, and Total Enterprise Simulations: A Natural Co-existence An Exploration into the Non-Use of Business Simulations The Market Game: Interactive Learning Through Market Simulation Plotting Brand Trajectories with the COMPLETE PPM package: A Market Segmentation Analysis and Positioning Tool Me and Mine Inc: An Exercise in Management Theory Learning to Differentiate Leadership from Managerial Position Business Ethics Survey: A Perspective from the Retail Industry Experiential Learning in Demand Analysis for an Agricultural Commodity Marketing on the Internet: A Pedagogical Exercise College Students Need Simple Computer Simulations, Especially for International Business Courses An Analysis of Student Attitudes, Performance, and Strategies in a Simulation Competition Based on a Controlled Product-Market-Entry Game Structure The Crystal Enterprise: Application of a Non-Computerized Simulation Model in a Process of Organizational Change Threshold: A Windows-Based Behaviorally Oriented Total Enterprise Simulation Coaching Business Game Teams Using a Decision Variable Optimizer Rock and Roll is here to Stay: Enhancing Experiential Pedagogy with Musical Exercises An Example of Business Process Analysis Simulation for Customer Software Support The Use of Business Gaming in Hong Kong Academic Institutions Using the Integrative International Simulation INTOPIA Mark 2000 in a Concentrated MBA Curriculum A Group Experience as an Integrated Part of the Core Management Course Empowered Learning in the Classroom The Inter-Group Interaction: An Innovative Approach to Cooperative Learning Courses that Utilize Student Team: An Approach to enhancing their Effectiveness Thoughts about the Measurement of Learning: The Case for Guided Learning and Associated Measurement Issues Measuring Student Learning Using Business Simulations: A Theory Based Perspective On the Use of PC Fingame in an Undergraduate Finance Course How Managers get Things Done: A Virtual Soundbite Internet Experiential learning in the Principals of Marketing Classroom: A Pedagogical Approach Current Student Perceptions Relative to Business Simulations Beyond Capitalism: Designing Business Policy and Social Justice The Use of Boards of Directors to Evaluate Reports and Presentations in an Undergraduate Business Policy Course Beacon Lumber: An Experiential Introduction to Financial Accounting The Application of Organizational Motivation Principles: The Experiential Business Simulation Motus Manufacturing Avoiding a Bogey: Grading Case Discussions Scientifically Communicating Consumer Behavior II: A Modified Exercise Using Personal Consumption Journals in Condensed Courses Designing Instruments for Assessing the Effectiveness of Simulations, (Seminar) 360º Performance Feedback: Appraisal vs. Assessment The Cafeteria Approach to Managing an Academic Career: Remaining Non-Perishable while Doing Your Own Thing An Experiential Exercise Related to Person-Organization Fit and it's Consequences For Today's Dynamic and Changing World, Business Simulations need to be expanded to encompass Much Greater Complexity: A Demonstration The Incident Process: A Case in Reverse Contextually-Anchored Business Simulations The Energy Factor: Building Motivation in the Simulation Gaming Environment The Design of an Internet Game