THE SECOND COMPONENT TO EXPERIENTIAL LEARNING: A LOOK BACK AT HOW ABSEL HAS HANDLED THE CONCEPTUAL AND OPERATIONAL DEFINITI... Developments in Business Simulation and Experiential Learning, volume 25, 1998 THE SECOND COMPONENT TO EXPERIENTIAL LEARNING: A LOOK BACK AT HOW ABSEL HAS HANDLED THE CONCEPTUAL AND OPERATIONAL DEFINITIONS OF LEARNING James W. Gentry, University of Nebraska-Lincoln Suraj R. Commuri, University of Nebraska-Lincoln Alvin C. Burns, Louisiana State University John R. Dickinson, University of Windsor ABSTRACT This paper reviews the coverage of “learning” in the twenty four years of ABSEL Proceedings, examining the both the conceptual and operational definitions of the three domains (cognition, affect, and behavior) of experiential learning. The paper ends with recommendations for future research. INTRODUCTION Recent ABSEL Conferences have had special sessions dealing with learning, trying to determine its conceptual and operational definitions and systematic attempts to measure it. Despite the fact that many of the participants in these sessions have been involved with ABSEL since its inception, their bounded rationality (not to mention their fast-fading memories) no doubt resulted in incomplete perspectives of ABSEL’s coverage of “learning”. Further, given the differing views of what is an acceptable measure of learning, the discussion in these sessions no doubt was biased in terms of one’s own stance. This paper will look at learning as it has been defined both conceptually and operationally in the 24 years of ABSEL Proceedings, and evaluate where ABSEL stands now on the topic. This is an extension of the work by Butler, Markulis, and Strang (1985), who also conducted a review and evaluation of the coverage of “learning,” only in the early years of ABSEL. GENERAL OVERVIEW Our take of ABSEL’s history is that the organization has spent a disproportionate amount of time covering “experiential” and too little dealing with “learning”. Our reading of the 24 years of Proceedings found that nearly 40% of the papers were presentations of new experiential exercises of some type, with no more than attitudinal responses given in support of the conclusion that the exercise was successful. [See Table 1 (available from the authors) for our summary of the coverage of “learning” in the 24 years of Proceedings.] While we support the development and presentation of new exercises, we believe that the attitudinally-based claims for successful learning have been at best divergent from the systematic effort to measure whether learning is taking place. The paper will first discuss the conceptual definitions of learning offered at ABSEL, with most emphasis on explicit ones but some discussion of implicit ones as well. Second, we will discuss the operational definitions that have been used. As a caveat, we are sure that we missed some definitions in our reading of the 24 proceedings, but that may be due in part to the failure of some authors to make their definition of learning explicit, either conceptually or operationally. Third we will offer directions in which we would hope that work at ABSEL will proceed. 62 Developments in Business Simulation and Experiential Learning, volume 25, 1998 CONCEPTUAL DEFINITIONS Many explicit definitions of learning have been given at ABSEL over the years, and an even wider variety of implicit definitions has been assumed. While authors have referred to the work of such historical figures as Confucius, Aristotle, Sophocles, John Dewey, and Piaget, the most frequently cited bases for the conceptual definitions of experiential learning in ABSEL have been the work of Bloom (Bloom et al. 1956) and Kolb (1971, 1974, 1984). Evolving from the Bloom’s taxonomy has been a focus on the multi-dimensional nature of “learning”: cognition, attitude, and behavior. These dimensions were incorporated in the early efforts by Hoover (1974) and Hoover and Whitehead (1975) to define experiential learning conceptually. They also noted the “whole-person” perspective, which is so pertinent to the distinction between “experiential” learning and other forms of learning. Numerous ABSEL papers have presented models incorporating the tripartite definition of learning, though some have used different labels. Gentry, Stoltman, and Mehlhoff (1992) and Macintosh, Gentry, and Stoltman (1993) extended the behavior/skills concept to include the Sternberg notion of tacit knowledge (that which is not typically expressed or stated openly, and not directly taught or spoken about -- i.e., “street smarts”). Mintzberg (1975, p. 60) stated this philosophy more succinctly: “Our management schools need to identify the skills managers use, select students who show potential in these skills, put the students into situations where their skills can be practiced, and give them systematic feedback on their performance.” Rather than three distinct dimensions, some work has advocated other frameworks for the learning process. For example, Burns, Gentry, and Wolfe (1990) argued that attitude moderates the relationship between the pedagogical manipulation and learning. Later, Burns (1992) offered an alternative model with learning as a mediating variable between the pedagogy and performance. For the purposes of this paper, we adopt the more traditional approach and specify that there are three distinct domains of learning (cognition, attitude, and behavior). Besides leading us to consider multiple learning domains, the Bloom’s taxonomy also specifies that there are different levels of learning, varying from basic knowledge and comprehension to synthesis and evaluation. Anderson and Lawton (1988) delineated 11 evaluation methods and rated them as to their appropriateness for the different levels of learning. The need to specify learning objectives has been stated frequently, most recently and perhaps most articulately by Anderson and Lawton (1997b). It makes little sense to attempt to measure learning if what is intended to be learned has not been clearly specified. However, the unique meaning of the experience to each student results in the observation that often students “learn” something that the instructor never expected them to learn (Frazer 1977, 1996; Gosenpud 1996; Teach 1996). This unplanned learning can occur in any of the three domains, and its existence requires that those attempting to measure learning must approach the task with a very open mind, One last issue is the need to measure the value- added aspect of the experiential exercise. Too many measures of “learning” reflect the student’s background and native aptitudes rather than the increased knowledge resulting from the experience. Learning implies change, requiring that students walk away from the course/exercise with something that has practical value to them (Gentry and Burns 1997; Parasuraman 1980). OPERATIONAL DEFINITIONS The relatively complex nature of experiential learning provides us with a plethora of measurement problems. Many “Act of Faith” 63 Developments in Business Simulation and Experiential Learning, volume 25, 1998 justifications have been made at ABSEL: the instructor just knows that learning has taken place. More disconcerting have been the views of fatalists, such as that espoused by Nulsen and Faria (1977, p. 219): “One wonders if, in fact, we haven’t been beating our heads against a brick wall for the last ten years trying to measure something that is non-measurable.” Despite the many problems involved, our review of the ABSEL literature suggests strongly that progress has been made. One overriding issue concerning the measurement of learning is the need to use a pseudo- experimental design in order to measure change in cognition, attitude, and behavior. A pre-post design can measure change within the individual, and while a post-only design may have less methodological confounding due to the elimination of testing effects, the inability to randomize the assignment of students to sections may result in performance differences due to factors besides the manipulation. For example, the same instructor teaching 8 AM and 11 AM sections will likely find that more talented and more motivated (though more quiet) students may self-select themselves into the earlier session , part of this problem can be controlled by the use of a covariate--i.e., The failings of experiential design in pedagogical research are well documented (Butler, Markulis, and Strang 1985; Wolfe 1977, 1981), but the fact remains that experimental designs still provide more credible comparisons than those that do not compare pedagogies or that do not use a control group (Anderson and Lawton 1997; Brenenstuhl and Catalanello 1976; Kelley 1982). Operational Definitions of Cognition As Parasuraman (1980) noted, the most typical measure of cognition is the common test, in which both those exposed and those unexposed to the pedagogical manipulation take the same written exam. Thus, the test grade represents the learning that has taken place. The standard assumption (though frequently violated due to the inability to assign students randomly to sections) is that the various sections will have similar ability levels prior to the class. Thus, differences in performances between the class exposed to the experiential exercise and the control group will be attributable to the added learning taking place. The use of grade as a dependent variable raises questions of validity; for example, Markulis and Strang (1997) asked if test scores measure learning, before skirting the issue. One consistent criticism of the common test approach is that different pedagogies have varying levels of effectiveness across the levels in Bloom’s taxonomy. Assessing learning associated with an experiential exercise using a Basic Knowledge oriented test would ignore the “whole-person” nature of the experience. In contrast, the traditional lecture/discussion format may not fit well with test items designed to measure learning at higher levels of Bloom’s taxonomy (such as those developed by Washbush and Gosenpud 1993, 1994). The interaction between levels of learning and the appropriateness of pedagogies results in a convoluted specification of the “what” that is to be measured. Further, the “what” issue is confounded by the instructor’s incomplete understanding of what may be learned. As noted earlier, one of the true benefits of experiential learning is that the student often learns something completely unforeseen by the instructor. Clearly, this poses measurement problems since the topic was not included in the learning objectives, which should provide the basis for the substance of the common test. Thus, the design of suitable test instruments must be a dynamic process in which continuous revision takes place as clearer perspectives of the scope of learning are achieved. In order for this to transpire, student feedback concerning perceptions as to what they have learned is necessary. Some feedback will come from the debriefing sessions, but that feedback will likely be limited to a few isolated 64 Developments in Business Simulation and Experiential Learning, volume 25, 1998 remarks. A more systematic approach is to require written feedback from each student after the end of the exercise. Besides the summary of what happened in the exercise, students should discuss what they perceived they learned from a “whole- person” perspective. Parasuraman (1980) suggested that a second general approach to the measurement of learning is the comparison of performance in the exercise to performance in other parts of the course. The intent is to see whether there is convergence between exercise performance and other, more traditional measures of learning. In general, little relationship has been found (Anderson and Lawton 1990; Parasuraman 1980; Washbush and Gosenpud 1993, 1994) between exercise performance and other measures of learning. In part, this may be due to the poor measures of learning in general and to the noise in exercise performance measures themselves (Thorelli 1997). Burns, Gentry, and Wolfe (1990) and Parasuraman (1980) conclude that exercise performance is not a surrogate for learning, noting that the student who learns the most may be the one who digs a hole for himself/herself and has to crawl out rather than the student who, possibly by chance, gets off to a flying start and coasts to victory. In fact, Gosenpud and Washbush (1996) found no relationship between performance and learning, but did find a relationship between learning and the fascinating construct, “struggling to perform,” which may capture the “trial and error” component so prevalent in “experiential learning”. The Windsor (Faria, Wellington, Whiteley, Dickinson, etc.) School of Thought (and this term is used loosely) assumes that high performance indicates higher levels of learning, despite all the evidence that has failed to substantiate any such relationship. The bottom-line perspective, as noted earlier, has many limitations, and many researchers have suggested that a monitoring of the decision processes would be a far superior approach. While most would agree with this, most would also acknowledge that the period-by-period monitoring of decision processes is extremely tedious and often yields ambiguous insights. Dickinson (1997) offers a possible solution that uses easily accessible performance measures to monitor progress over time: the plot of earnings per share data over the game play. His preliminary findings indicate that the plots for group performance in the use of the Dickinson and Faria (1995) Marketing Management Simulation resemble standard learning curves. Further, the structure of this simulation is not one which would assure that performance increases over time. If such a measure can be related to other measures of learning at different stages of the exercise, then the approach may provide an indicator that learning is taking place. If the intent of the use of the exercise is to have students learn as opposed to have students win, the existence of a learning-curve effect could serve as a component of the grading process that is independent of overall outcome. The third general approach to measuring cognitive learning mentioned by Parasuraman (1980) was the use of written feedback from the students. Typically, students are required to summarize the nature of the experience, providing a rich basis for evaluating whether they understand the underlying framework of the game. As noted above, students can also be asked to provide their own perceptions as to what they learned. In addition, they can be required to make suggestions as to how the exercise can be made more realistic; not only do instructors get more insight into the students’ understanding of the exercise framework, but they also may get new ideas as to how to improve the exercise in the future. Operational Definitions of Affect The measurement of “affect” represents an enigma. On one hand, we would argue that affect is the most commonly measured construct in the evaluation of experiential learning. On the other hand, we agree with the conclusion of 65 Developments in Business Simulation and Experiential Learning, volume 25, 1998 Butler, Markulis, and Strang (1985) that there has been little systematic work to measure change in attitudes. The key to this contradiction is the fact that little experimental work has used “affect” as a dependent variable; instead teacher-evaluation type comments, most of an anecdotal variety, have been used to support the “success” of the exercise. While such measures no doubt provide the instructor with reasons to reduce any cognitive dissonance about the use of the exercise, they do not provide scientific evidence to support the effectiveness of the exercise. We include the commonly used “perception of cognitive/learning” type of measure in this “feel good” category of measures. If the students enjoyed the experience (which we hope is one nearly universal benefit from using games), then there is likely to be a halo effect in terms of measuring perceptions of learning. While we advocate assessing “what” students perceive they learn, we concur with Parasuraman (1980) that too much effort has been given to perceptions of learning and that the overuse of attitude and perception measures occurs because we know how to measure them. We would instead encourage the measurement of attitudes concerning stimuli other than the exercise experience itself Did students’ perspectives of the value of “profit” change? Did perspectives of ethical dilemma expand? Our perspective is that the MBA degree became much more popular in the early 1970s due to the turmoil of the 1960s and to the desire by firms to hire people who would not sabotage their operations. Business graduates may not have been better trained than those from history, political science, and sociology, but at least they were able to accept “business values.” Attitudes toward business processes should be considered when determining the exercise’s learning objectives Operational Definitions of Behaviors! Skills We concur with Mintzberg’s (1975) statement (cited earlier) that we should determine the skills needed to be a successful manager and then to instill those through guided experiences. Many of these skills involve tacit knowledge, which makes determination of the needed skills problematic. Shanteau (1987) notes that true experts often do their tasks without awareness of just what they are doing; they cannot articulate the processes which they go through, while novices can. Nevertheless, we maintain that continued efforts need to be made to determine the critical skills, to teach them to students, and to measure whether they have been learned. Butler, Markulis, and Strang (1985) noted that little work had been done in ABSEL to measure skill issues. One pleasant surprise in our review of the literature was the finding that a number of attempts have been made to measure changes in skill levels (Anderson and Lawton 1988, 1997; Armstrong 1978; Fry, Kidron, and Schriesheim 1975; Gosenpud 1982; Kelley 1982; Savage 1979). For example, Gosenpud (1982) investigated changes in self concept in groups and tolerance for ambiguity as a result of the group experience, while Kelley (1982) measured changes in empathy, level of regard, and genuineness. Anderson and Lawton (1988, 1997b) offer perhaps the most extensive battery of learning measures, many of which would be categorized as “cognitive” measures. However, they also provide in-depth lists of skills as well. We believe that ABSEL has provided excellent starting points in terms of measuring behavior/skills, and we encourage future efforts to build on these. DIRECTIONS FOR FURTHER RESEARCH ON EXPERIENTIAL LEARNING It appears to us that the field has not generated processes that will yield valid measures of the amount of learning that has taken place. Further, while the 24 years of ABSEL conferences have not generated tight conceptual and operational definitions of learning, we believe that this literature base should not be ignored. Too much “re-invention of the wheel” occurs at each 66 Developments in Business Simulation and Experiential Learning, volume 25, 1998 ABSEL conference without reference to the systematic thinking that is available in print. Future research at ABSEL dealing with “learning” should demonstrate awareness of a number of classic pieces, including Anderson and Lawton (1988, 1997b); Burns and Gentry (1977, 1980); Butler, Markulis, and Strang (1985); Frazer (1978); Fritzsche (1976); Goosen (1986); Gosenpud (1982); Gosenpud and Washbush (1997); Hoover and Whitehead (1975); Kelley (1982); Nulsen and Faria (1977); Parasuraman (1980); and Ramsey and Woodhouse (1981). There is need for a Best of ABSEL publication dealing with the measurement of learning resulting from guided experience. We believe that ABSEL should adopt one conceptual definition of learning that will at least provide a common starting point for the subsequent operational definitions. This definition must focus on the student’s perspective as opposed to the teacher’s. Clearly, much thought must be paid to what we want students to learn. Also, much effort needs to be made to investigate just “what” was learned. Even if that which the instructor wanted to be learned was not communicated effectively via the experience, it may be that the student did-indeed learn perspectives of value. While we believe firmly that the expertise of the- instructor provides the best basis for determining what should be learned, we also admit that all instructors are fallible and believe that more concern with what students are actually learning will broaden instructors’ perspectives of what should be learned. Besides the focus on the learner rather than on the instructor, we also assert that the definition of learning should be quite broad in its scope Aspy Palia at an ABSEL session on the measurement of learning in 1995 made the excellent point that we need to evaluate our pedagogical techniques not only in terms of the specific exercise or in terms of the whole course, but also in the context of the whole curriculum. The most obvious case in point is the business policy simulation game which usually is intended to integrate all the business functions. If students do not learn respect for business functions other than their own, has not the game failed in one of its purposes? Is such an issue normally incorporated in our measures of “learning”? We have experienced the phenomenon in which several courses deal tangentially with the same conceptual base, and know the joy of getting a more holistic perspective of the topic. Given that the college student is forming his/her professional identity and is selecting courses that foster that identity, this phenomenon should be quite frequent. But a broader perspective as to what students should learn, one that includes the interface with other functional areas, will help instructors in their efforts to make this phenomenon even more frequent. After reviewing the coverage of “learning” in ABSEL, we choose the definition supplied by Hoover and Whitehead (1975, p. 25) as our starting point: Experiential learning exists when a personally responsible participant cognitively, affectively, and behaviorally processes knowledge, skills, and/or attitudes in a learning situation characterized by a high level of active involvement. We would add to that definition an emphasis on measurable value-added occurring as a result of that processing. Further, we would also like to acknowledge the dynamic nature of experiential learning, noted by Byrne and Wolfe (1974): the learner engages in some concrete experience, leading to reflective observations, and, subsequently, hypotheses which can be tested later in the experiential exercise. From our review of ABSEL’s history, there is little to criticize in terms of the “active involvement” and the “responsible participant” criteria. However, few, if any, works have dealt with the three distinct domains (cognitive, affect, and behavior) at all, much less systematically. As mentioned earlier, most supportive evidence of 67 Developments in Business Simulation and Experiential Learning, volume 25, 1998 the effectiveness of new experiential exercises has been affective in nature, and too often anecdotal. Few studies have used affect as a dependent variable in an experimental framework. Positive attitudes from only one class (the one with the new pedagogy) do not constitute scientific evidence supporting the conclusion that learning has taken place. We do not doubt that learning has taken place, but know that skeptical non- experiential colleagues will not be convinced. We subscribe to the philosophy that Schreier (1976) advocated in regard to the use of experiential learning, “If it feels good, do it.” However, we also cannot argue well that it is not the contagious enthusiasm of the instructor who devised this new exercise rather than the increased perspective generated from its use that is creating the positive response. Further, we assert that students tend to give “socially desirable” responses to efforts made by instructors to do things out of the ordinary, regardless of their effectiveness. Experimental designs have been more common when “cognitive” measures of learning have been investigated; we advocate the use of pseudo- experimental designs in all attempts to measure learning (whether cognitive, attitudinal, and/or behavioral), despite the many threats to internal and external validity that always arise. When the “learning” objectives are considered in the design and/or selection of an exercise, concern should be given to all three domains (cognition, attitude, and behavior). Both the exercise designer and the user should make these objectives explicit (Anderson and Lawton 1997). If the instructor wants to investigate the effectiveness of the exercise, he or she should develop a pseudo- experimental design in which all of the desired learning objectives are measured. For example, if one of the desired behaviors to be learned is the ability to work well in groups, measures of changes in communication style or of respect for others’ communication styles might be taken in a pre-post design. An example of such an approach is discussed in Gentry, Stoltman, and Mehlhoff (1992). If one objective is to increase the student’s attraction to the topic area, systematic measures of affect change should be taken. As noted earlier, one suggestion for measuring cognitive change at higher levels of Bloom’s taxonomy would be to have students write reports about the exercise experience, with instructions to discuss the pattern of relationships underlying the exercise structure, similar to the approach suggested by McDevitt (1997). The “whole-person” emphasis of our advocated conceptual definition requires a focus on the student’s background and concern for how the new knowledge will fit with the existing bank of abilities, attitudes, and aptitudes. Further, there should be concern with the role of this course and the exercise in the overall degree-attainment process. For example, the simulation-happy Faculty of Management at Windsor have carefully sequenced the use of games so that the complexity level grows with the course numbers. Finally, we must acknowledge that the design of measures to capture experiential learning must be developed through a dynamic process due to the unplanned nature of much of the learning associated with the experience. REFERENCES Available from the authors. 68 Table of Contents Volume 25, 1998 Marketing Goes to the Movies Bringing Experiential Learning to a Principles of Marketing Course Investment Analysis Application Using In-house Spreadsheet Models SugarCoated Statistics: An Exercise for the First Day of Class Improving Undergraduate Student Involvement in Management Science and Business Writing Courses Using the Seven Principles in Action Establishment and Funding for Interuniversity / Multidisciplinary Student experiences The Prospects of Creative Teaching: A Discussion with Patricia Sanders The Simulation and Classroom Assessment Techniques Developments of Management Skill Assessment Games as Instruments of Assessment: A Framework for Evaluation The Role of Artificial Intelligence in Business Curricula Threshold Solo Competitor: A Management Simulation (V1.0) a Windows-Based. Play Alone, Total Enterprise Simulation and Assessment Instrument Toward An Understanding of One's self-concept Total Enterprise Simulations and the Internet: Improving Student Perceptions and Simplifying Administrative Workloads The Expatriate an Assignment Orientation Game An Expatriate's Nightmare: An Experiential Exercise in Coping with Overseas Assignments Analyzing Experiential Exercise: Using the Scientific Method for Problem Solving The Second Component to Experiential Learning: A Look Back at How ABSEL has handled the Conceptual and Operational Definitions of Learning Predictive Models of Learning: Participant Satisfaction of Experiential Exercises in Business Education Accelerating Moral Development through Use of Experiential Ethical Dilemmas Ethical Dilemmas to use with Business Simulations to Teach Ethics The Class Approach in Behavioral Simulation in a Business Policy/Strategic Management course: A Progression toward Greater Realism An Exploration of the Emergence of Process Prototypes in a Management Course Utilizing a Total Enterprise Simulation How Organizations Are Improving Their Performance Utilizing Electronic Commerce: Examples From The Internet The Market Access Planning System (Maps): A Computer-Based Decision Support System For Facilitating Experiential Learning In International Business An Excel Workbook For Student Planning And Interface With A Simulation Game Design Of Multi-Media Based Pedagogy For Leadership Training Panel Discussion On Using The Internet For Courses Valuing And Enhancing Teaching: Sharing Tips Via The Web The Buddy Project: A Semester Long Project Aimed At Developing An Appreciation For Diversity Enhancing The Excitement And Learning Retention In The Classroom: The Power Of Magic The Supervised Management Internship: A Job Or Learning Experience Team Ware™ An Online Moderated Class Discussion Facility And Beyond A Neophyte Distance Educator's Experience Learning Management By Practicing Management: A Report Of Significant Student Service In 1997 Integration Of Academic And Service Learning: Students' Perceptions About Its Effects And Outcomes The Value Of Incorporating A Service Learning Component Into Course Content: A Presentation And Roundtable Discussion Business Games Teach: Thoughts on the Sources of Conflicting Conclusions on their Effectiveness Antecedents Of Learning In The Simulation: A Replication Using Student Journals To Enhance Learning From Simulations Technological Change And Intertemporal Movements In Consumer Preferences In The Design Of Computerized Business Simulations With Market Segmentation Integrating The Marketing Curriculum Using Collaborative Learning Teaching Time Management In A Sales Program: The Application Of A Computer Simulation Game Adapting Interactive Computer Simulations For Content Based Esl Instruction Multimedia And Student Expectations Synthesizing Data For Media Simulations Composing A Team Health Promoting Behaviors-A Decision Making Exercise Does it really Work? An Application of the Group Interaction Framework Administering the MIT Beer Game: Lessons Learned A Paperless Economy? Instructing Students on the Aspects of Successful Electronic Commerce Maximizing Learning Gains in Simulations: Lessons from the Training Literature Observing General Ability in a Total Enterprise Gaming Simulation FReach Teach: A Computer-Based System for Teaching Advertising Media Planning An Integrated Business Instruction System An Experiential Exercise you can Tinker With Experiential Exercises or Computer Simulations? Cash Flow Statements: Are They Important in Business Simulations? Holistic Cognitive Strategy in a Computer-Based Marketing Simulation Game: An Investigation of Attitudes Towards the Decision-Making Process Barnga: A Game on Cultural Clashes The Many Faces of Culture: Understanding Country and Corporate Culture Students' View of the Use of Business Gaming in Hong Kong Assessing General Management Interest What is the Future of Business Gaming? Starting a Small Music Trivia Business Exercise and Other Innovative Icebreakers An Integrated Approach to Behavioral Skill Development Career Focus: A Student and Business Learning Experience The Use of Concept Mapping in Teaching Strategic Management An Experiential Approach to Developing Mission Statements Business Games in Brazil-Learning or Satisfaction A Simulation within a Simulation: Job Layoff's and Emotional Reactions