THOUGHTS ABOUT THE MEASUREMENT OF LEARNING: THE CASE FOR GUIDED LEARNING AND ASSOCIATED MEASUREMENT ISSUES Developments In Business Simulation & Experiential Learning, Volume 24, 1997 THOUGHTS ABOUT THE MEASUREMENT OF LEARNING: THE CASE FOR GUIDED LEARNING AND ASSOCIATED MEASUREMENT ISSUES James W. Gentry, University of Nebraska-Lincoln Alvin C. Burns, Louisiana State University First of all, let us make it clear that we have no qualms whatsoever in declaring that we should measure learning (and not just mastery) well. Mastery is not unimportant, but it is easier to capture. Learning relates to the value-added nature of a class environment, and reflects whether any movement in the “correct” direction has taken place. Even more clearly, we need to separate "learning" from ‘background.” Students come to us with a variety of skills and aptitudes; those with well developed analytical and communication skills will likely perform better on any assessment test we provide. “Learning” may not be present if “good" students do well; even excellent students need to do better in order for learning to have taken place. In short, experiential learning is the affective, cognitive, and behavioral change in a student pursuant to some structural experience designed to facilitate these changes. A major philosophical issue is who constitutes the proper unit of evaluation -- who determines if learning has taken place? Who determines what is “correct”? While we concur that the instructor has greater expertise in terms of the content to be learned, we believe that it must be the student who determines if learning has taken place. If the student is not cognitively aware of that learning [a situation implied by Wellington, Faria, and Nulsen (1996) in their discussion of the observed appropriate use of push (or pull) strategies in reaction to the simulated environment, even though there was no explicit recognition of the situation faced], then observations of the student’s behavior are required. Ideally, the debriefing sessions conducted at the end of the experiential exercises will remove the lack of awareness and generate the “Aha” experience so commonly observed in experiential exercises. Another philosophical issue deals with the use of multiple measures. Given the limited amount of high quality work done in ABSEL on the measurement of the learning aspect of experiential learning, the intent of the paper is to discuss issues related to potential measures, starting with a discussion of the proper domain. DOMAIN ISSUES Any attempt ax measurement necessarily begins with a definition of the domain of the constructs being measured. We have claimed that learning relates to the value-added nature of a class environment. That is, there is something that the student “comes away with” over and above what was initially brought to the experience or otherwise acquires as a result of the learning experience. To simply inventory the value-added cognitions, attitudes, or performance skills of any learning experience is relatively easy. However, complications arise when one realizes that different expectations or standards are invariably in place. To be more exact, there are four learning domains: (1) instructor expected; (2) student expected; (3) instructor perceived; and (4) student actual. The instructor establishes his/her expectations about the content and skills to be learned and selects a pedagogy to facilitate the student’s learning of them. Parallel to this, the student has expectations about what he/she will learn. Some, of course, are low or vague (just as some of the content in a discipline may be noisy in nature), while others might be very precise. The instructor institutes the pedagogy and selects a measurement method to assess the degree to which the student has achieved the instructor’s expected learning. This measurement, such as the performance on an examination., constitutes the instructor’s perception of what has been learned. The term, “perception,” is used intentionally as it strongly implies that there is some variance between what is reported by the 241 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 measure and what is actually learned. In other words, measurement error can now come into play as the instrument may or may not be consistent with the domain of the instructor expected. The instructor assesses learning by the degree to which the measurement (instructor perceived) indicates that the student has achieved the instructor expected. If careful measurement development is not applied, or if the measure is not scrutinized for consistency with the instructor's expectations of what is to be learned, there exists a danger that the measure is a false indicator of learning. At the same time, the student actual learning may or may not be what was measured. As noted by Frazer (1996), Gosenpud (1996), and Teach (1996), students often learn things which instructors did not foresee nor intend, even if they had been foreseen. Interestingly, this conceptualization allows students to feel they have learned a lot when the measure says they have learned little or nothing. Student actual-to-student expected is the student’s measure, while instructor perceived-to-instructor expected is the instructor's measure of learning. In general, we have no formal measures of student expected unless it happens to be identical or similar to the instructor expected. The failure to address student actual learning in the grade assessment process potentially has great impact on students’ learning strategies. To make this issue clearer, we draw upon the work of Dweck (1990), who has developed a research stream investigating the differences between those with learning orientations (which have the aim to increase competence) and those with performance orientations (which aim to gain favorable judgments of competence and to avoid unfavorable ones).1’ 1 ‘As an aside, this framework casts an interesting perspective on Dick Teach’s (1996) anecdote about having the worst performing team in a simulation game contribute money each period to a pot which would buy refreshments at the end of the semester. He noted that this (dis)incentive system created a different learning environment and in his observation, a more active one. When cast in Dweck’s framework, though, one wonders if it did not create an emphasis on performance as opposed to learning Learning-oriented students exhibit strong mastery orientations regardless of their confidence in their present ability, and failure does not keep them from the pursuit of knowledge. They do not perceive that intelligence is a fixed quantity; in fact, their continued growth proves otherwise. Performance-oriented students react very differently to failure, especially if they have little confidence in their abilities (in which case learned helplessness is a likely outcome). Those with performance orientations and high self-confidence may be mastery-oriented, but failure is not handled in the “trial and error” fashion implicitly assumed by most work on experiential learning. The point here is that students who know that they are learning, but perceive that the assessment process does not demonstrate that (or, worse, indicates that they are not learning the “correct’ material) may over time lose learning orientations and become performance oriented. Business schools teach well that one must adapt to the organizational culture; an unintended consequence may be that we shift learning orientations toward performance orientations. The criticism that the ‘exam did not measure what I know” may be indicative that the instructor expected and the student actual have little overlap. If the cause of this is more the bounded rationality of the instructor than it is the misperception of the student, the well-intentioned professor may be doing great harm. Our suggestion concerning “guided learning” may well address this phenomenon. One experiential extension would be to have students create their own measures of learning, which might serve to re-direct the instructor in terms of the nature of actual learning. GUIDED LEARNING How does one resolve all this? We offer an approach that could be called “guided learning” where the 242 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 instructor resolves to make his/her learning expectations the student’s expectations. That way, the target learning domain is the same for teacher and learner. The idea of asking students what they want to learn seems a reasonable way of finding out the degree of discrepancy between the two expectations at the onset. Assuming the teacher has the correct expectations, it is up to him/her to bring the student’s expectations to correctness, i.e., to guide the student’s expectations to agree with the teacher’s evaluation. For example, as a part of the first day of the semester activities, the first author has begun asking (in written form, as the last question on a student background information sheet), “What do you expect to learn in _______ this semester?” Three of 35 students in International Marketing and two of 26 Consumer Behavior students filled out the rest of the form but left the learning expectations question blank. Many of those who did respond did not provide a great deal more insight. Example responses indicate that most students’ first-day expectations are quite vague and, thus, instructors do have great freedom to structure the learning agenda. The first day asking of what is desired in terms of learning has at least two useful purposes. First it reminds the instructor that students often come to us with little pre-conceptions in terms of what they want. Second, it encourages students to think about what they want to learn. Burns and Gentry (1996) discuss the role of curiosity in motivating students, and base much of their Tension-to-Learn Theory on Loewenstein’s (1994) gap model of curiosity; the gap refers to the difference between what students know now and what they wish to learn. Loewenstein (1994) argues that the closing of gaps is pleasurable, that the key is the perception of manageable gaps (limited or non- existent gaps do not motivate the student, while gaps that are too large foster learned helplessness), and that those with more knowledge are able to perceive more manageable gaps. Asking students what they wish to learn may help them recognize the existence of gaps, and may also make them more attentive when you discuss what you perceive can be learned from this course. Guided learning would suggest that each class begins with asking students what they expect to learn on that topic or day, and then providing students with the instructor's expectations of learning for that topic or day. Another approach might be suitable if the instructor is seeking to understand the value-added aspect of learning phenomenologically. Here, it would be incumbent on the instructor to attempt to understand the true nature of the student’s actual learning. Accordingly, it would be necessary to devise or adopt a measurement that tapped the student’s actual learning. Asking students what they expect and what they actually learned may open the instructor’s eyes to relevant aspects of learning that he/she had not considered. Note that the student may be the guide for the instructor in this instance. Assessing the student perspective of what they are learning in our classes should aid in the development of more systematic instruments to assess learning in more traditional ways, as responses may shed light on the “customer” perspective as opposed to the “sellers” perspective. A summer Consumer Behavior class was asked, “What did you learn in this class?” and “Did you learn what you expected to learn? Please explain.” The responses to the former questions were disappointingly vague, in general, but they were relatively homogeneous and of the following nature: I learned that so many more factors go into determining how and why people behave as consumers the ways that they do. Diversity can have such an amazing impact on the values that people use to make choices. To a large extent, this statement reflects the rather abstract nature of the course as represented in the instructor expected. As it happens so often, there was also the unsatisfied customer. While most indicated that they learned 243 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 what they expected to learn, one indicated that he2 did not. In fact, his response to what he learned was "nothing.” To some extent, this student’s disenchantment could have been predicted. He was finance major who wanted to learn, “How the general public makes consumption decisions and what information requirements go into making decisions.” These macro expectations do not fit well with the instructors more micro expectations. In this case, better advising might have moved this student into an International Marketing or Marketing Strategy course instead of Consumer Behavior. A better implementation of guided learning by the instructor would have reconciled the differences in expectations. MEASUREMENT ISSUES Measurement issues are found in the degree to which the measurement accommodates all student actual learning plus the degree to which we want to accommodate all value-added learning. If we miss some of it that is relevant, we may underestimate true learning. If we include some of it that is irrelevant, we may overstate the amount of learning. Taking our previous comments into account, we have three measures: one of what the student learns relative to instructor expectations (instructor perceived-instructor expected), one of what the student learns relative to student expectations (student actual - student expected), and one of what the student actually learns relative to what the instructor measures (student actual - instructor perceived). The proper starting point for investigating and resolving differences is the third case. If no differences exist here, and if there is scant measurement error relative to the instructors expected learning, the only concern is the student's expectations, which will need to be somehow guided into the instructors expectations. Probably the most logical approach to these issues is evolutionary convergence in which the instructor seeks to understand the student's expectations and folds the relevant ones into his/her expectations for student learning. At the same time, it may be necessary to extinguish irrelevant or counter student expectations that drain student energies unnecessarily as they participate in the educational exercise or which otherwise lead students to believe they have/have not learned because they have/have not met those irrelevant learning expectations. 2 Names were required on the information sheet at the beginning of the semester, but not on the final course evaluation. This student’s identity was clear from his distinctive printing style as well as from having shared his opinions with the instructor after the return of graded exercises. This discussion is not meant to support a contention that only the student knows what is to be learned. The point is that learning takes place in the student and there has to be an understanding of his/her frame of reference in order to understand what changes are taking place. Probably, both frames of reference need to change but assuming that the instructor has a deeper understanding of the material encompassed in his/her course, the frame of reference of the student should change more. One example would be the issue of the global economy. Ethnocentric Americans (the majority of us, unfortunately) still think domestically first (and too often, foremost). Despite several decades of emphasis on internationalizing curricula, most business courses still have a predominantly domestic focus. If that were not true, why would we need courses in International Marketing or International Finance or International Whatever? A truly globalized curriculum would find such courses redundant. Yet many undergraduates come to our classes with a decidedly domestic focus. Many may not want to learn global perspectives, even though it is inconceivable that they will not be affected greatly by the global economy when they enter the workforce. Here, what some students come to class wishing to learn and what instructors know they must learn (eventually in order to be employable in the future) may differ greatly. The instructor should attempt to change the student’s learning agenda as soon as possible. Many other specific domains could be discussed in a similar vein. If students did not 244 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 come to us with limited horizons, there would be little justification for our professional existence. On the other hand, the fact remains that learning takes place within the student and it is incumbent upon instructors to monitor the content of that learning and to adjust their future expectations accordingly. ADDITIONAL MEASUREMENT ISSUES As noted by Wolfe (1996), the current AACSB emphasis on knowledge, values, and skills is reminiscent of the work done by Bloom et al. (1956) on the cognitive, affective, and psychomotor areas. As Lawton noted at last years ABSEL conference (Anderson and Lawton 1996), most simulations are used with the intent of engendering learning at the hi/her levels of Bloom’s cognitive hierarchy, especially the analysis and synthesis levels. Measures of whether students can perform these functions or, Less preferably, whether they think they can perform these functions are needed. For a learning instrument to have any universal appeal, it needs to have at least two types of content. First, it should assess general aspects of the analysis and/or synthesis processes which any business game or exercise should stimulate. Second, it should have sample items which are specific to the exercise used and which could be modified easily by the individual instructor to fit his or her own situation. Palia (1996) noted that one should not concentrate on the learning fostered by the experiential exercise alone, but that assessment should be broadened to capture the interfaces among the various pedagogies used. We concur, but would broaden this perspective to include the role the various pedagogies used in this course play within the student’s overall curriculum. Clearly a capstone course cannot be independent of the various courses which precede it. Courses within functional areas can engender learning in other courses within that functional area or in other functional areas. The student able to make those linkages is indeed performing higher level cognitive tasks. A possible way to assess the level of learning associated with the use of an experiential exercise would be to require students to describe the structure of the simulated learning environment. The nature of the beast is that most students cannot fathom how a simulation game could be structured. 1f at the end of the game play, they can describe the structure of the game environment -- the patterns of relationships that exist and the types of actions most likely to lead to success — learning has taken place. If they can make astute suggestions as to how the game can be made more realistic, they have taken learning a step further to interface the game experience with the rest of the course and the curriculum. We believe that concern has to be paid to the overall learning environment, and not just to the specific pedagogy. Business schools vary tremendously in terms of context; the many varying factors have been discussed at length by Burns and Gentry (1977, 1980) and Burns, Gentry, and Wolfe (1990). We argue that a person who is a very effective instructor in one environment may well have to adapt greatly in order to become an effective instructor in a second environment. For example, using a series of cases in an environment where the average class size is 20-25 may be a very effective strategy, but it may be infeasible when the same course is taught in an environment with class sizes of 100-150 students. Thus, we would like to see authors who report research on the effectiveness of various pedagogies also summarize the nature of the general learning environment: day or night program, what percent of the students work, time of day, class size, the level of experiential exercise usage in the curriculum, etc. At some point, researchers may want to do a meta-analysis of the effectiveness of various pedagogies (and hopefully the mix of pedagogies), and specification of the learning context may allow them to control for many sources of variance. We see the question, “Which form of pedagogy is the best in terms of generating learning?” as being meaningless. On the other hand, the question, “Which type of pedagogy will be best for a particular 245 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 instructor in a particular course in a specific organizational culture?” may be answered feasibly at some point. REFERENCES Anderson, Philip and Leigh Lawton (1996), “How Do We Measure the ‘Learning' in Experiential Learning,” in Alan L. Patz and John K. Butler Jr., Developments in Business Simulation and Experiential Learning, Vol. 23, p. 197 Bloom, B. S., M. D. Englehart, E. D. Furst, W. H. Hill, and D. R. Krathwohl (1956), Taxonomy of Educational Objectives: The Classification of Education Goals, New York: David McKay Company, Inc Burns Alvin C. and James W. Gentry (1977), “Some Thoughts on a ‘Theory of the Use of Games and Experiential Exercises,” in Carl C. Nielsen (Ed.), New Horizons in Simulation Gaming and Experiential Learning, Vol. 4, 187- 194. Burns, Alvin C. and James W. Gentry (1980), "Moving Toward a ‘Theory’ of the Use of Simulation Games and Experiential Exercises,’ in Daniel C. Brenenstuhl and William D. Biggs (Eds.), Experiential Learning Enters the Eighties, Vol. 7, 17-20. Burns Alvin C. and James W. Gentry (1996), "Motivating Students to Engage in Experiential Learning: A Tension-to-Learn Theory Based on Curiosity and Absorptive Capacity, ”Working Paper University of Nebraska-Lincoln. Burns, Alvin C., James W. Gentry, and Joseph Wolfe (1990), “A Cornucopia of Considerations in Evaluating the Effectiveness of Experiential Pedagogies,” in James W. Gentry (Ed.), Guide to Business Gaming and Experiential Learning, New York: Nichols/GP Publishing, 253-278. Dweck, Carol S (1990), “Self-Theories and Goals: Their Role in Motivation, Personality, and Development, “in R. Dienstbier (Ed.), Perspectives on Motivation: Nebraska Symposium on Motivation, Vol. 58, 199-235. Frazer, Ron (1996), ‘Show Do We Measure the Learning in Experiential Learning,” in Alan L. Patz and John K. Butler Jr., Developments in Business Simulation and Experiential Learning, Vol. 23, p 197. Gosenpud, Jerry (1996), “‘Learning in Total Enterprise Simulations,” in Alan L. Patz and John K. Butler Jr., Developments in Business Simulation and Experiential Learning Vol. 23, p 197. Loewenstein, George (1994), ‘The Psychology of Curiosity: A Review and Reinterpretation,” Psychological Bulletin, 116 (No. 1), 75-98. Palia, Aspy (1996), Comments made at the Session on research directions for ABSEL, Orlando ABSEL Conference. Teach, Richard (1996), “Measuring Success in Business Games," in Alan L. Patz and John K. Butler Jr., Developments in Business Simulation and Experiential Learning, Vol. 23, p. 197. Wellington, William, Anthony J. Faria, and Ray Nulsen (1996), "The Game Learning Process,” in Alan L. Patz and John K. Butler Jr., Developments in Business Simulation and Experiential Learning, Vol. 23, p. 197. Wolfe, Joseph (1996), Comments made at the session on the measurement of learning, Orlando ABSEL Conference. 246 Table of Contents Volume 24, 1997 Incorporating Computer Telephony into the MIS Course A Learner Oriented Infrastructure for videoconferencing Based Distance Education Courses The Identification of Temporally Related Structural Elements of the Experiential Component of Electronic Spreadsheet Tasks Does Involvement Influence Learning from Simulation Participation? Some Relationships with Helpfulness and Performance Outcomes Musings on Business Garne Performance Evaluation Performance on a TE Simulation: What does it represent? The Business Policy Game: An International Simulation - An Assessment Tool Students in Free Enterprise as Experiential Learning The Role of Computer Models in Wargames: A Practitioner's view Using Focus Groups as a Tool to Research Learning: A Demonstration and a Discussion Using Medical Simulations to Teach Multi-Cultural Diversity Using the World Game to Internationalize International Exchange Game Community Services Needs Assessment: An Innovative Approach Introducing Students to Potential Total Quality Management Ethical Dilemmas Privacy in the Workplace: A Situational Analysis Antecedents of Learning in Simulations Demonstrating the Learning Effectiveness of Simulation: Where we are and Where we need to go The Use of Computer Simulations as a Pedagogical Aid in Teaching Management Information Systems Evaluating Simulation Learning in a Distant Learning Instructional Model Financial Engineering of Global Investments The Labor History Game: Playing with the Past Perks Participation Assessing Negotiator's Proficiency with a Negotiation role-play The Art of Negotiating Enhancing Learning and Employee Development through the Assessment of Learning Pedagogy Preferences across selected Dimensions of Culture: A Preliminary Investigation Cooperative Learning: What are we Learning? Effective Use of Mastery Based Experiential learning in a Project Course to improve skills in system analysis and Design Simulations and Learning: Can we prove a relationship? (Seminar) Predicting and Reviewing NYSE Stock Prices by use of Basic Statistical Analysis and Logic Toy Car Depreciation Exercise ABSEL as Home Community: An Interactive Exploration Service Learning: Linking Academic Study to Community Development and Business Enhancement Ability of Efficient Evaluation of Knowledge-Based Management Strategies Corporate Ethics Training Programs Modeling Attributes in Demand Functions of Computerized Business Simulations: An Extension of Teach's Gravity Flow Algorithm Expert Systems Combined with Neutral Networks: Tools to Benefit the Marketing Researcher Computer Game Design: New Directions for Intercultural Simulation Game Designers Consistency in Simulation Performance over Time and Across Simulation Games The Impact of an Artificial Market Leader on Simulation Competitor's Strategies Business Plans, Case Studies, and Total Enterprise Simulations: A Natural Co-existence An Exploration into the Non-Use of Business Simulations The Market Game: Interactive Learning Through Market Simulation Plotting Brand Trajectories with the COMPLETE PPM package: A Market Segmentation Analysis and Positioning Tool Me and Mine Inc: An Exercise in Management Theory Learning to Differentiate Leadership from Managerial Position Business Ethics Survey: A Perspective from the Retail Industry Experiential Learning in Demand Analysis for an Agricultural Commodity Marketing on the Internet: A Pedagogical Exercise College Students Need Simple Computer Simulations, Especially for International Business Courses An Analysis of Student Attitudes, Performance, and Strategies in a Simulation Competition Based on a Controlled Product-Market-Entry Game Structure The Crystal Enterprise: Application of a Non-Computerized Simulation Model in a Process of Organizational Change Threshold: A Windows-Based Behaviorally Oriented Total Enterprise Simulation Coaching Business Game Teams Using a Decision Variable Optimizer Rock and Roll is here to Stay: Enhancing Experiential Pedagogy with Musical Exercises An Example of Business Process Analysis Simulation for Customer Software Support The Use of Business Gaming in Hong Kong Academic Institutions Using the Integrative International Simulation INTOPIA Mark 2000 in a Concentrated MBA Curriculum A Group Experience as an Integrated Part of the Core Management Course Empowered Learning in the Classroom The Inter-Group Interaction: An Innovative Approach to Cooperative Learning Courses that Utilize Student Team: An Approach to enhancing their Effectiveness Thoughts about the Measurement of Learning: The Case for Guided Learning and Associated Measurement Issues Measuring Student Learning Using Business Simulations: A Theory Based Perspective On the Use of PC Fingame in an Undergraduate Finance Course How Managers get Things Done: A Virtual Soundbite Internet Experiential learning in the Principals of Marketing Classroom: A Pedagogical Approach Current Student Perceptions Relative to Business Simulations Beyond Capitalism: Designing Business Policy and Social Justice The Use of Boards of Directors to Evaluate Reports and Presentations in an Undergraduate Business Policy Course Beacon Lumber: An Experiential Introduction to Financial Accounting The Application of Organizational Motivation Principles: The Experiential Business Simulation Motus Manufacturing Avoiding a Bogey: Grading Case Discussions Scientifically Communicating Consumer Behavior II: A Modified Exercise Using Personal Consumption Journals in Condensed Courses Designing Instruments for Assessing the Effectiveness of Simulations, (Seminar) 360º Performance Feedback: Appraisal vs. Assessment The Cafeteria Approach to Managing an Academic Career: Remaining Non-Perishable while Doing Your Own Thing An Experiential Exercise Related to Person-Organization Fit and it's Consequences For Today's Dynamic and Changing World, Business Simulations need to be expanded to encompass Much Greater Complexity: A Demonstration The Incident Process: A Case in Reverse Contextually-Anchored Business Simulations The Energy Factor: Building Motivation in the Simulation Gaming Environment The Design of an Internet Game