MEASURING STUDENT LEARNING USING BUSINESS SIMULATIONS: A THEORY BASED PERSPECTIVE Developments In Business Simulation & Experiential Learning, Volume 24, 1997 MEASURING STUDENT LEARNING USING BUSINESS SIMULATIONS: A THEORY BASED PERSPECTIVE Paul K. McDevitt, University of Illinois at Springfield ABSTRACT Many experiential educators believe that computer simulations are an effective pedagogical tool for enhancing student learning. Proving this belief has been difficult because of a number of difficult conceptual issues as well as some notoriously challenging measurement issues. This paper addresses the subject of measuring student learning using business simulations in the classroom. The broader justification for this research is the ongoing effort to investigate the relationship, if any, between student learning and simulation performance. Beginning with established learning theory, the author proposes a means and a process for measuring student learning using computer simulations. INTRODUCTION An issue of acute concern to proponents of experiential learning is the relationship, if any, between computer simulation performance and learning in the business classroom. The pedagogical technology has become quite sophisticated; the question is: “What is it good for?” Trying to measure learning is a major challenge to educators who, after many decades, still cannot agree even on how to measure good teaching. Measuring learning is a challenge because of the complexity of the process and the number of factors that influence outcomes (Thatcher, 1990). While measuring the learning in experiential learning has proven difficult, researchers should be comforted to know that practitioners of alternative pedagogies, i.e., case method and Lecture/test approaches, have fared little better. Nevertheless, the effort to relate simulation use to Learning forces educators to be clear about their learning objectives, disciplined and thoughtful with their pedagogy and realistic in their evaluation. As a result the process may ultimately prove more important than the outcome. This paper addresses the subject of measuring student learning using business simulations in the classroom. The broader justification for this research is the ongoing effort to investigate the relationship, if any, between student Learning and simulation performance. We begin by briefly reviewing earlier research. We then examine some fundamentals of experiential learning theory to help sharpen some key concepts and to identify appropriate opportunities for measurement. We conclude with a conceptual proposal for measuring learning. LITERATURE REVIEW Research reviews of educational and training applications of business simulations have appeared periodically in past decades and have been widely cited. Keyes and Wolfe (1990) concluded in their review that, among other things, “Research is needed to evaluate the relationship between learning in a management game and performance in a game’ (p. 318). A substantial body of research was forthcoming. A full review of the ensuing literature is beyond our scope here, but in their review Malik and Howard (1996) concluded that: “The research, to date, has not been unanimously encouraging with respect to the effectiveness of simulations as a learning tool” (p. 52). Fortunately, those authors edged away from the precipice, concluding that: “There are a number of issues that researchers need to focus on before simulations are completely dismissed in terms of their effectiveness’ (Malik & Howard, 1996, p. 52). Some important methodological issues are involved in this research stream. There exists a certain urgency among proponents of simulations to statistically relate learning (some aspect ... any 247 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 aspect) to simulation performance. This is tantamount to searching for an end to satisfy a means. In this effort, educators share with many others the rather widespread predicament of trying to determine how best to use the fruits of the information revolution. Illustrating the learning value of simulations could perhaps be more appropriately accomplished by restating the research question to read: “For a given learning objective, does simulation use facilitate student learning?” This question is based on the premise that a simulation provides a learning platform or environment within which numerous learning objectives and corresponding activities may be pursued. The research challenge should be to state the objectives a priori and then to establish whether simulation use and learning are related. Difficult measurement problems are often problems of definition. Therefore, before addressing the issue of measuring learning, we review some basic fundamentals of experiential learning theory. Our purpose is to understand clearly the conceptual foundations of learning and perhaps to identify viable measurement opportunities. LEARNING THEORY “Learning is the process whereby knowledge is created by the transformation of experience” (Kolb, 1984). Kolb’s model of experiential learning identified four steps of effective learning: Concrete experience (CE), reflective observation (RO), abstract conceptualization (AC), and active experimentation (AE). Active experimentation and concrete experience together constitute the concrete dimension of learning while reflective observation and abstract conceptualization form the abstract dimension. The emphasis upon experience in this model differentiates it from other more cognitive models of learning (Kolb, 1976). Students using a computer simulation receive concrete experience in the form of a printout of results (CE), they then think systematically (RO) about the results, trying to assess their meaning and to relate them to previous knowledge; then they reinterpret the market environment in light of their new understanding (AC) and make a new set of decisions (AE). A]l four steps are necessary for effective learning, and, in a continuing learning environment such as a simulation, the learning process includes a succession of related learning loops (Thatcher, 1990). If; in fact, simulations are an educative learning pedagogy, then students are continuously receiving outputs (CE), analyzing (RO), reorganizing reality (AC), and making decisions (AE). The Centrality of Reflective Observation A long line of distinguished learning theorists argue that the key step in learning is the reflective observation step, RO. Dewey (1993) described reflective thought as: Active, persistent, and careful consideration of any belief or supposed form of knowledge in the light of the grounds that support it and further conclusions to which it leads ... It includes a conscious and voluntary effort to establish belief upon a firm basis of evidence and rationality (p. 9). Similarly, Kelly personal construct theory (1955) and Boud, Keogh, and Walker’s reflective learning model (1985), among others, focus on the centrality of reflection in the learning process. Boud, et al (1985) suggest three key considerations that are relevant to this discussion. First, only learners can learn from reflection; as educators we can at best provide a favorable learning environment within which reflection can occur. Second, learning must be pursued with intent, which is to say it must be purposeful. Clearly formulated learning objectives assure that reflection is purposeful. Third, the reflective process is a complex and highly individualistic one in which the cognitive, affective and behavioral dimensions are all involved. Students’ feelings and 248 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 behaviors, of both a negative and positive nature, influence learning as well as do cognitive considerations. Learning Levels Within this theoretical context, experiential educators talk of three levels of learning, including participation, debriefing and writing. Consider the first level of experiential learning, participation. Participation is the personal involvement of the student with a business simulation. Participation spans all four steps of the Kolb learning model. If the next two learning levels are to be productive, student participation must be enthusiastic and fully engaging at this first level, since debriefing and writing build directly on student participation. Debriefing is an activity in which people who have had an experience are led through a purposive discussion of that experience (Lederman, 1992). Debriefing challenges students at the key RO stage of learning (Lederman, 1984; Petranek, Corey & Black, 1992; Thatcher, 1990). Its use is based on the assumptions that a simulation has affected participants in some meaningful way and that a processing of that experience is necessary to provide insight into it and its impact (Lederman, 1992). The goals of debriefing are to facilitate understanding of what happened, to find out what the participant learned and to test the latter against the instructor’s learning objectives (Lederman, 1992). Debriefing is considered higher order learning than participation because it forces students to acknowledge, organize and interpret the outcomes of their experiences rather than merely living them. Usually conducted orally and during the RO stage of leaning, debriefing seeks to enable students to articulate a clear and logical basis for the abstract conceptualization (AC) that normally follows. Writing is a third (and some argue the richest) experiential learning level. Writing commits the learner to actively and formally organize and express the results of the RO stage. The process of writing has long been thought to produce a critical learning experience that exceeds the oral debriefing and participation activities. Francis Bacon (1883) observed long ago that "Reading maketh a man full, conference a ready, and writing an exact man” (p. 14). Contemporary experiential learning theorists would restate Bacon as follows: Simulation maketh a student fill, debriefing a ready, and writing an exact student.” Wollman - Bonilla (1989) also argued persuasively for writing: Researchers have shown that journals validate self expression and personal response, encourage understanding, imaging, speculation questioning, and the shaping of ideas, and provide students with information relevant to their concerns and problems in the content of their own entries and teachers response (P. 113). More recently, Charles Petranek (1994) writes that “Playing a simulation is the adolescent step in learning, whereas oral debriefing is the young adult stage and written debriefing is the older adult stage of learning” (p. 518). Writing includes a variety of activities, including journals, logs, portfolios, verbatims, record books (Walker, 1985) and written concepts (Petranek, 1994). Journalling enjoys a very modest level of use among, for example, marketing educators. Recent applications include using journals in the personal selling curriculum (Donoho, Swenson & Taylor, 1995) and using journals in international semester programs (Moncrief, Shipp & Lamb, 1995). The content of journals may be classified as unfocused, loosely focused or focused. Petranek uses what might be characterized as a tightly focused approach by asking students to address: events, emotions, empathy, explanations, everyday, and employment (Petranek, 1994). His ninth principle of experiential learning: “The stratagem of the Six Es of Debriefing propels learning to a greater level” (p. 519). Certainly one approach that might be used by 249 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 instructors who are neophytes at using writing to document learning is to begin with a relatively unfocused journalling effort to establish what the scope of current learning includes. As learning objectives become more clear and simulation choice more driven (see below), the scope of the journals is likely to become more focused. Appendix A includes a handout used by a neophyte instructor outlining a minimally focused set of journalling requirements for a group of MBA students using the COMPETE (Faria, Nulsen and Roussos, 1994) simulation. Taken together, the experiential learning literature suggests the following for those interested in measuring learning. Firstly, learning is a highly individualized process that includes abstract and concrete learning components. Secondly, the key activity in learning is reflection, the effort to consider new information, experience, etc. and to “reorganize reality" (Piaget, 1973) accordingly. Thirdly there exists a hierarchy of learning levels with specific activities appropriate at each level. MEASURING LEARNING Our goal is to measure learning, if any, that results from simulation usage. As we conceptualize them, the challenges are: (1) determine what students are actually learning when using business simulations; (2) establish what we wish them to learn; (3) select simulations that offer learning opportunities that match our learning objectives; and (4) measure the learning, if any, taking place with those simulations. What are students learning? Given the individualistic nature of experiential learning, understanding the scope of learning is a challenge. Nevertheless, learning theory provides some guidance. We take as a given that debriefing and writing are the richest learning activities and offer the greatest opportunities for measuring student learning. Practically speaking, however, the requirements of the oral debriefing process proposed by Lederman (1992) and the extended time frame of most simulation play (Anderson and Lawton, 1992) seem to rule out debriefing as a viable discovery approach. Consequently, we cite the learning theory literature discussed above and propose the following: student writing is a preferred method for establishing what simulation students are learning. Of the written debriefing options, journals seem to offer the greatest promise for measurement purposes. Learning objectives may span the spectrum of the cognitive, affective or behavioral dimensions of experiential learning. They may include content and/or process components. They may be individual or group level objectives. Presumably these will be driven, in part, by the role/level of the course in the overall curricular structure. The key consideration is that objectives be clearly stated so that students are able to chronicle learning appropriately in their journals, i.e., that reflection is purposeful. Selection of a simulation which offers Learning opportunities that coincide with learning objectives is straightforward. When the scope of e.g., a full firm or functional level simulation is not ideal, faculty can and often do augment or narrow the application accordingly. The final activity is measurement of learning. In classes where learning objectives are relatively narrowly defined, e.g., a measurable skill such as forecasting, measurement is likely to be simpler, in classes where the objective is more ambitious, e.g., the development of strategic thinking skills, measurement may be more difficult. We propose the following process: 1. Prepare journal instructions directing students' efforts at the desired learning objective. 2. Collect a database of student journal entries. 3. Analyze the data to understand fully the learning process by which students accomplish the learning objective. 4. Identify (and perhaps order) the key junctures involved in the process. 250 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 5. Formulate a “model” that identifies the key learning experiences. 6. Evaluate the extent to which students master the key learning experiences on the basis of their journal entries. 7. Develop an instrument, i.e., a scale, for measuring student learning. 8. Assess the reliability and validity of the scale. APPENDIX I THE COMPETE JOURNAL “TO RECORD, TO REFLECT, TO ASSESS” FALL 1996 Description Your journal is a written record how you are learning in the COMPETE simulation. It is an opportunity for you to reflect on your awareness of what and how you are Teaming to manage a marketing strategy and how you feel about this process. Purpose The purpose of writing this journal is to make you and the instructor more aware of your cognitive and affective development as you progress through COMPETE. Learning to manage effectively is a process, and you will record in this journal your progress through twelve weeks. Grade Your journal will be collected on random Thursdays at the beginning of class. You should bring your journal, including all previous entries, to each class. (They will not be accepted late and please don’t plan on writing them in class.) The journals will be returned to you at the beginning of the next class with a grade of 0 through 10. These grades will be based on the thoughtfulness of the content rather than a notion of right or wrong. Students who make a good faith effort to record, reflect and assess what they are learning and what they are feeling in the COMPETE simulation should expect to receive grades of 9 -10. These journal entries, along with your COMPETE quiz score, weekly COMPETE decisions, group earnings performance, and peer evaluations at semesters end will all count towards the COMPETE grade which constitutes 25 per cent of your course grade. Format Your weekly journal entry should be one to two dated pages clearly written or typed. The maximum length for 10 points is two pages and the minimum is one page. Please be aware that the course instructor grew up in a family of 12, has raised two children and has taught in the MBA Program for 13 years. He is aware of the difference between a fabricated and a serious one- page effort. Please keep your entries in the right side of a two pocket folder, with the most recent entry on top and the least recent on the bottom. Your name should be clearly visible on the front. Topics Feel free to discuss anything regarding what you are learning and what you are feeling in the process. Remember: the purpose of COMPETE is to enable you to learn about formulating and executing a product level marketing strategy —the purpose of the journal is to allow you to record, to reflect upon, and to assess that learning. Timing Journal entries should ideally be recorded immediately after each week’s decisions, e.g., before you leave L Building or when you arrive home. If impossible, try to complete this as soon as possible the next morning. You will be surprised at how quickly you forget even some of the most intense COMPETE experiences. 251 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 REFERENCES Anderson, P. & Lawton, L. (1992) A survey of methods used for evaluating student performance on business simulations. Simulation & Gaming 2.1. (4), 490-498. Bacon, F. (1983) The essays or, counsels, civil and moral of Francis Bacon London: Chesterfield Society. Boud, D., Keogh, K & Walker, D. (eds.). 1985. Reflection: Turning experience into leaning. London: Koan Page. Dewey, J. (1933) How we think (Rev ed.). Boston: D C Heath. Donoho, C. L., Swenson, M. J. & Taylor, G. R. (1995) Journal writing in the personal selling curriculum Journal of Marketing Education 17, (2), 5 - 16 Faria, A. J., Nulsen, R. 0 & Roussos, D. S. (1994) COMPETE: A dynamic marketing simulation. Burr Ridge, II: Richard D. Irwin. Kelly, G. A. (1955) The psychology of personal constructs, I and 2 New York: Norton. Keys, B. and Wolfe, J. (1990). The role of management games and simulations in education and research Journal of Management I 6 (2), 307- 336. KoIb, D. A. (1984) Experiential learning Experience as the source of learning. Englewood Cliffs, NJ: Prentice-Hall. Kolb, D. A. (1976) Management and the learning process California Management Review. 18. 21- 31. Lederman, L. C. (1984) Debriefing: A critical re- examination of the post-experience analytic process with implications for its effective use. Simulation and Games 15, 415-431 Lederman, L. C. (1992) Debriefing: Toward a systematic assessment of theory and practice. Simulation and Gaming 23 (2) 145-160 Malik D. S. & Howard, B. J. (1996) How do we know where we are going if we don’t know where we’ve been? A review of business simulation research. Developments in Business Simulation and Experiential Exercises, 49-53 Moncrief, W. C., III, Shipp, S.H., & Lamb, C. H. Jr. (1995) Student journal writing in an international setting. Journal of Marketing Education, 17, (2), 71-80. Petranek, C. F. (1994) A maturation in experiential learning: principles of simulation and gaming. Simulation and Gaming 25, (4), 513-523. Petranek, C. F., Corey, S. & Black, R. (1992) Three levels of learning in simulations: participating, debriefing, and journal writing Simulation and Gaming 23 (2), 174-185 Piaget, J. (1973) Psychology and epistemology New York: The Viking Press. Thatcher, C. (1990) Promoting learning through games and simulation. Simulation and Gaming.. 21. (3), 262-273. Walker D. (1985) Writing and Reflection In D Boud et al (Eds) Reflection: Turning experience into learning. New York: Nichols Publishing Company. Wollman-Bonilla, J.E.(1989). Reading journals: Invitations to participate in literature. The Reading Teacher 43. (2), 112-120. 252 Table of Contents Volume 24, 1997 Incorporating Computer Telephony into the MIS Course A Learner Oriented Infrastructure for videoconferencing Based Distance Education Courses The Identification of Temporally Related Structural Elements of the Experiential Component of Electronic Spreadsheet Tasks Does Involvement Influence Learning from Simulation Participation? Some Relationships with Helpfulness and Performance Outcomes Musings on Business Garne Performance Evaluation Performance on a TE Simulation: What does it represent? The Business Policy Game: An International Simulation - An Assessment Tool Students in Free Enterprise as Experiential Learning The Role of Computer Models in Wargames: A Practitioner's view Using Focus Groups as a Tool to Research Learning: A Demonstration and a Discussion Using Medical Simulations to Teach Multi-Cultural Diversity Using the World Game to Internationalize International Exchange Game Community Services Needs Assessment: An Innovative Approach Introducing Students to Potential Total Quality Management Ethical Dilemmas Privacy in the Workplace: A Situational Analysis Antecedents of Learning in Simulations Demonstrating the Learning Effectiveness of Simulation: Where we are and Where we need to go The Use of Computer Simulations as a Pedagogical Aid in Teaching Management Information Systems Evaluating Simulation Learning in a Distant Learning Instructional Model Financial Engineering of Global Investments The Labor History Game: Playing with the Past Perks Participation Assessing Negotiator's Proficiency with a Negotiation role-play The Art of Negotiating Enhancing Learning and Employee Development through the Assessment of Learning Pedagogy Preferences across selected Dimensions of Culture: A Preliminary Investigation Cooperative Learning: What are we Learning? Effective Use of Mastery Based Experiential learning in a Project Course to improve skills in system analysis and Design Simulations and Learning: Can we prove a relationship? (Seminar) Predicting and Reviewing NYSE Stock Prices by use of Basic Statistical Analysis and Logic Toy Car Depreciation Exercise ABSEL as Home Community: An Interactive Exploration Service Learning: Linking Academic Study to Community Development and Business Enhancement Ability of Efficient Evaluation of Knowledge-Based Management Strategies Corporate Ethics Training Programs Modeling Attributes in Demand Functions of Computerized Business Simulations: An Extension of Teach's Gravity Flow Algorithm Expert Systems Combined with Neutral Networks: Tools to Benefit the Marketing Researcher Computer Game Design: New Directions for Intercultural Simulation Game Designers Consistency in Simulation Performance over Time and Across Simulation Games The Impact of an Artificial Market Leader on Simulation Competitor's Strategies Business Plans, Case Studies, and Total Enterprise Simulations: A Natural Co-existence An Exploration into the Non-Use of Business Simulations The Market Game: Interactive Learning Through Market Simulation Plotting Brand Trajectories with the COMPLETE PPM package: A Market Segmentation Analysis and Positioning Tool Me and Mine Inc: An Exercise in Management Theory Learning to Differentiate Leadership from Managerial Position Business Ethics Survey: A Perspective from the Retail Industry Experiential Learning in Demand Analysis for an Agricultural Commodity Marketing on the Internet: A Pedagogical Exercise College Students Need Simple Computer Simulations, Especially for International Business Courses An Analysis of Student Attitudes, Performance, and Strategies in a Simulation Competition Based on a Controlled Product-Market-Entry Game Structure The Crystal Enterprise: Application of a Non-Computerized Simulation Model in a Process of Organizational Change Threshold: A Windows-Based Behaviorally Oriented Total Enterprise Simulation Coaching Business Game Teams Using a Decision Variable Optimizer Rock and Roll is here to Stay: Enhancing Experiential Pedagogy with Musical Exercises An Example of Business Process Analysis Simulation for Customer Software Support The Use of Business Gaming in Hong Kong Academic Institutions Using the Integrative International Simulation INTOPIA Mark 2000 in a Concentrated MBA Curriculum A Group Experience as an Integrated Part of the Core Management Course Empowered Learning in the Classroom The Inter-Group Interaction: An Innovative Approach to Cooperative Learning Courses that Utilize Student Team: An Approach to enhancing their Effectiveness Thoughts about the Measurement of Learning: The Case for Guided Learning and Associated Measurement Issues Measuring Student Learning Using Business Simulations: A Theory Based Perspective On the Use of PC Fingame in an Undergraduate Finance Course How Managers get Things Done: A Virtual Soundbite Internet Experiential learning in the Principals of Marketing Classroom: A Pedagogical Approach Current Student Perceptions Relative to Business Simulations Beyond Capitalism: Designing Business Policy and Social Justice The Use of Boards of Directors to Evaluate Reports and Presentations in an Undergraduate Business Policy Course Beacon Lumber: An Experiential Introduction to Financial Accounting The Application of Organizational Motivation Principles: The Experiential Business Simulation Motus Manufacturing Avoiding a Bogey: Grading Case Discussions Scientifically Communicating Consumer Behavior II: A Modified Exercise Using Personal Consumption Journals in Condensed Courses Designing Instruments for Assessing the Effectiveness of Simulations, (Seminar) 360º Performance Feedback: Appraisal vs. Assessment The Cafeteria Approach to Managing an Academic Career: Remaining Non-Perishable while Doing Your Own Thing An Experiential Exercise Related to Person-Organization Fit and it's Consequences For Today's Dynamic and Changing World, Business Simulations need to be expanded to encompass Much Greater Complexity: A Demonstration The Incident Process: A Case in Reverse Contextually-Anchored Business Simulations The Energy Factor: Building Motivation in the Simulation Gaming Environment The Design of an Internet Game