DESIGNING INSTRIJI’6ENTS FOR ASSESSING THE EFFECTIVENESS OF SIMULATIONS Developments In Business Simulation & Experiential Learning, Volume 24, 1997 DESIGNING INSTRUMENTS FOR ASSESSING THE EFFECTIVENESS OF SIMULATIONS Philip H. Anderson University of St. Thomas Leigh Lawton University of St. Thomas PURPOSE: Using a panel/workshop approach, the purpose of this seminar is to identify alternative instruments designs that would provide objective assessment of learning outcomes achieved through the use of simulation exercises. METHOD USED: A Delphi technique will be utilized. The first stage will be to brainstorm possible instrument designs. Session participants will then work in small groups to identify the major components of the individual designs. The second stage will consist of a full group discussion of the small group sessions. The focus of this stage will be to (1) crystallize the major constructs to be included in each instrument and (2) identify the most promising designs. OBJECTIVE: The objective of the workshop is to begin to develop 2 or 3 instruments that have possible assessment capabilities for measuring learning outcomes at the higher levels of Bloom’s Taxonomy. These instruments would then be used by ABSELers at various institutions in conjunction with differing simulation exercises. By so doing, we should foster a systematic move away from perceptions toward a more objective basis for assessing the learning value of simulations. Using these instruments at multiple institutions should also help build a database for determining the validity and reliability of the instruments. RATIONALE FOR SEMINAR Assessing what students learn on a business simulation exercise has been a focus in the literature and at ABSEL conferences since simulation exercises were first introduced (Greenlaw and Wyman, 1973; Keys, 1976; Wolfe, 1985; Whiteley and Faria, 1989; Burns, Gentry, and Wolfe, 1990; Wolfe, 1990; Gosenpud, 1990; Wellington and Faria, 1991; Anderson and Lawton, 1992; Gosenpud and Washbush, 1993, 1994; Anderson and Lawton, 1995; Washbush and Gosenpud, 1995). Debate continues as to what, if anything, is learned and whether there is objective versus anecdotal evidence to support the conclusions reached. Jim Gentry organized a seminar on the relationship between learning and simulation exercises for ABSEL’s 1996 conference. While there was considerable debate, there was also little resolution. The focus of this proposed seminar is to begin the process of developing instruments that collect valid, reliable, and objective data, which will facilitate resolution of the debate. As “scientists”, we need to develop insights based on objective rather than subjective data. This seminar will facilitate that process. REFERENCES Anderson, PH. and L. Lawton (1992), “The Relationship Between Financial Performance and other Measures of Learning on a Simulation Exercise”, Simulation & Gaming, 23 (3), 326-340. Anderson, P.H. and L. Lawton (1995), "The Problem of Determining an "Individualized" Simulation’s Validity as an Assessment Tool”, Developments in Business Simulations and Experiential Exercises, 22, pp.43-48. Bums. AC., J.W. Gentry, and J. Wolfe (1990), “A Cornucopia of Considerations in Evaluating the 300 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 Effectiveness of Experiential Pedagogies,” in Guide to Business Gaming and Experiential Learning, J.W. Gentry (ed.), New York: Nichols/GP Publishing. 253-278. Gosenpud. J. (1990), "Evaluation of Experiential Learning,” in Guide to Business Gaming and Experiential Learning, J.W. Gentry (ed.), New York: Nichols/GP Publishing, 301-329. Gosenpud, J. and J. Washbush (1993), “The Relationship Between Total Enterprise Simulation Performance and Learning.” Developments in Business Simulations and Experiential Exercises, 20, 141. Gosenpud, J. and J. Washbush (1994), "Simulation Performance and Learning Revisited.” Developments in Business Simulations and Experiential Exercises. 21, 83- 86. Greenlaw, P.S. and F.P. Wyman (1973), “The Teaching Effectiveness of Games in Collegiate Business Courses.” Simulation and Games, 4 (3), 259-293. Keys, B. (1976), “A Review of Learning Research in Business Gaming.” In B.H. Sord (ed.), Computer Simulations and Learning Theory, 173-184. Washbush, John and Jerry Gosenpud (1995), “Simulation Performance, Learning, and Struggle,” Developments in Business Simulation & Experiential Exercises, VoL 22, 1-4. Wellington, W.J. and AJ. Faria (1991), “An Investigation of the Relationship Between Simulation Play, Performance Level, and Recency of Play on Exam Scores”, Developments in Business Simulations and Experiential Exercises 18, 177. Whiteley, T.R. and A.J. Faria (1989), “A Study of the Relationship between Student Final Exam Performance and Simulation Game Participation.” Developments in Business Simulations and Experiential Exercises, 16, 78- 82. Wolfe. J. (1985), "The Teaching Effectiveness of Games in Games in Collegiate Business Courses: A 1973-83 Update.” Simulation and Games, 16. 251-288. Wolfe, J. (1990), "The Evaluation of Computer- based Business Games: Methodology, Findings, and Future Needs,” in Guide to Business Gaming and Experiential Learning, J.W. Gentry (ed.), New York: Nichols/GP Publishing, 279- 300. 301 Table of Contents Volume 24, 1997 Incorporating Computer Telephony into the MIS Course A Learner Oriented Infrastructure for videoconferencing Based Distance Education Courses The Identification of Temporally Related Structural Elements of the Experiential Component of Electronic Spreadsheet Tasks Does Involvement Influence Learning from Simulation Participation? Some Relationships with Helpfulness and Performance Outcomes Musings on Business Garne Performance Evaluation Performance on a TE Simulation: What does it represent? The Business Policy Game: An International Simulation - An Assessment Tool Students in Free Enterprise as Experiential Learning The Role of Computer Models in Wargames: A Practitioner's view Using Focus Groups as a Tool to Research Learning: A Demonstration and a Discussion Using Medical Simulations to Teach Multi-Cultural Diversity Using the World Game to Internationalize International Exchange Game Community Services Needs Assessment: An Innovative Approach Introducing Students to Potential Total Quality Management Ethical Dilemmas Privacy in the Workplace: A Situational Analysis Antecedents of Learning in Simulations Demonstrating the Learning Effectiveness of Simulation: Where we are and Where we need to go The Use of Computer Simulations as a Pedagogical Aid in Teaching Management Information Systems Evaluating Simulation Learning in a Distant Learning Instructional Model Financial Engineering of Global Investments The Labor History Game: Playing with the Past Perks Participation Assessing Negotiator's Proficiency with a Negotiation role-play The Art of Negotiating Enhancing Learning and Employee Development through the Assessment of Learning Pedagogy Preferences across selected Dimensions of Culture: A Preliminary Investigation Cooperative Learning: What are we Learning? Effective Use of Mastery Based Experiential learning in a Project Course to improve skills in system analysis and Design Simulations and Learning: Can we prove a relationship? (Seminar) Predicting and Reviewing NYSE Stock Prices by use of Basic Statistical Analysis and Logic Toy Car Depreciation Exercise ABSEL as Home Community: An Interactive Exploration Service Learning: Linking Academic Study to Community Development and Business Enhancement Ability of Efficient Evaluation of Knowledge-Based Management Strategies Corporate Ethics Training Programs Modeling Attributes in Demand Functions of Computerized Business Simulations: An Extension of Teach's Gravity Flow Algorithm Expert Systems Combined with Neutral Networks: Tools to Benefit the Marketing Researcher Computer Game Design: New Directions for Intercultural Simulation Game Designers Consistency in Simulation Performance over Time and Across Simulation Games The Impact of an Artificial Market Leader on Simulation Competitor's Strategies Business Plans, Case Studies, and Total Enterprise Simulations: A Natural Co-existence An Exploration into the Non-Use of Business Simulations The Market Game: Interactive Learning Through Market Simulation Plotting Brand Trajectories with the COMPLETE PPM package: A Market Segmentation Analysis and Positioning Tool Me and Mine Inc: An Exercise in Management Theory Learning to Differentiate Leadership from Managerial Position Business Ethics Survey: A Perspective from the Retail Industry Experiential Learning in Demand Analysis for an Agricultural Commodity Marketing on the Internet: A Pedagogical Exercise College Students Need Simple Computer Simulations, Especially for International Business Courses An Analysis of Student Attitudes, Performance, and Strategies in a Simulation Competition Based on a Controlled Product-Market-Entry Game Structure The Crystal Enterprise: Application of a Non-Computerized Simulation Model in a Process of Organizational Change Threshold: A Windows-Based Behaviorally Oriented Total Enterprise Simulation Coaching Business Game Teams Using a Decision Variable Optimizer Rock and Roll is here to Stay: Enhancing Experiential Pedagogy with Musical Exercises An Example of Business Process Analysis Simulation for Customer Software Support The Use of Business Gaming in Hong Kong Academic Institutions Using the Integrative International Simulation INTOPIA Mark 2000 in a Concentrated MBA Curriculum A Group Experience as an Integrated Part of the Core Management Course Empowered Learning in the Classroom The Inter-Group Interaction: An Innovative Approach to Cooperative Learning Courses that Utilize Student Team: An Approach to enhancing their Effectiveness Thoughts about the Measurement of Learning: The Case for Guided Learning and Associated Measurement Issues Measuring Student Learning Using Business Simulations: A Theory Based Perspective On the Use of PC Fingame in an Undergraduate Finance Course How Managers get Things Done: A Virtual Soundbite Internet Experiential learning in the Principals of Marketing Classroom: A Pedagogical Approach Current Student Perceptions Relative to Business Simulations Beyond Capitalism: Designing Business Policy and Social Justice The Use of Boards of Directors to Evaluate Reports and Presentations in an Undergraduate Business Policy Course Beacon Lumber: An Experiential Introduction to Financial Accounting The Application of Organizational Motivation Principles: The Experiential Business Simulation Motus Manufacturing Avoiding a Bogey: Grading Case Discussions Scientifically Communicating Consumer Behavior II: A Modified Exercise Using Personal Consumption Journals in Condensed Courses Designing Instruments for Assessing the Effectiveness of Simulations, (Seminar) 360º Performance Feedback: Appraisal vs. Assessment The Cafeteria Approach to Managing an Academic Career: Remaining Non-Perishable while Doing Your Own Thing An Experiential Exercise Related to Person-Organization Fit and it's Consequences For Today's Dynamic and Changing World, Business Simulations need to be expanded to encompass Much Greater Complexity: A Demonstration The Incident Process: A Case in Reverse Contextually-Anchored Business Simulations The Energy Factor: Building Motivation in the Simulation Gaming Environment The Design of an Internet Game