PROBLEMS IN EVALUATION OF EXPERIENTIAL LEARNING IN MANAGEMENT EDUCATION Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 137 PROBLEMS IN EVALUATION OF EXPERIENTIAL LEARNING IN MANAGEMENT EDUCATION Lane Kelley, University of Hawaii Jeffrey Easton, University of Lancaster ABSTRACT A cursory examination of a management professor’s book shelves indicates an increasing number of textbooks that radically alter the traditional classroom. This approach is based upon the idea of experiencing the material to be learned and consequently is labeled experiential learning. During the seventies it received attention in both the academic conferences and Journals such as the Academy of Management Journal resulting in several research articles. The primary focus of the articles was to determine the effectiveness of “the’ experiential approach. The present paper reviews the major articles on experiential learning published in business periodicals the last several years and raises questions of their validity. The problems inherent in assessing learning experiences is addressed by a sequential research design. Finally, the research results are used to propose a tentative theoretical model for assessing experiential learning. INTRODUCTION A number of journals, especially the Academy of Management Journal and Journal of Experiential Learning, have published articles describing the effectiveness of experiential learning (4, 6, 8, 9, 11, 16, 21, 24). Of importance to the present paper is how most of the papers have operationally defined effectiveness. One research article in the Academy of Management Journal (24) reported that a business policy simulation computer game team which had gone through Kolb, et. al’s. Organizational Psychology: An Experiential Approach did not do better than a team which had not gone through the exercises. The readers should be reminded that Kolb’s book emphasizes individual psychology and not group dynamics, nor group problem solving and, therefore, is not especially relevant to business games. Teachers who have experience with both management policy games and Kolb’s exercises probably wouldn’t have expected any type of relationship. Another study of the experiential approach in a business policy course reported that the use of “the’ experiential approach to teaching business policy appears to be a very effective teaching device and that secret ballots of students the next semester chose the experiential approach (16). A third study of the effectiveness of ‘the experiential approach used in a business policy course concluded that the experiential method produced relevant learning about the human factors involved in the process of policy formulation, but for the bulk of the class, however, this experience was almost “too real’ (4). The experiential approach in this study was actually a required field study without reading lists, lectures, a reward system and standards, and a highly uncertain environment. The hypothesis formation, research design, etc. of this study, like the previous, does not allow for any conclusions about “experiential” learning. All of the above studies relied upon the student’s perceptions of course effectiveness rather than more objective criteria. Butler and Keys (6) studied the relative effectiveness of a simulation laboratory and a traditional lecture- discussion course. Two groups of participants in each course were pre- tested as to their knowledge of facts and concepts regarding human relations and human resource development. Post-test measurements of the two groups on the same instruments revealed gains that were not significant for the control (lecture-discussion) group. The experimental (simulation) group measured highly significant gains. Commensurate with the significant knowledge gains by the experimental participants, the responses from their subordinates over a a two-month period indicated significant changes in their perceptions of their supervisor’s behavior. This was not true of the control group. The authors conclude that the simulation method was more effective than the traditional method in improving the supervisory behavior of the participants. Their research and the following study used either cognitive and/or behavioral change criteria. Kelley (18) compared an integrated didactic-experiential approach for teaching a personnel management course to a lecture-discussion course and reported that students were more ‘satisfied” with the experiential course, and scored higher on an exam designed to measure knowledge of the subject matter. The students also reported that participation in the exercises was more interesting and stimulating than lectures but they didn’t feel that they necessarily learned more. This is important because the students in the experimental course scored higher on an exam designed to measure their knowledge of the subject which is just the opposite of their perceptions. A review of the research illustrates the importance of the student’s perception to determine course effectiveness. METHODOLOGY The research for this study is actually three separate studies conducted over the last five years. The studies are summarized in the table below. The first study compared the effectiveness of the experiential method via the students perception of its effectiveness, and cognitive learning, interest in the course, and performance in the course. Based on the results of the first study, the second study was undertaken to look at the relationship between the teacher- student relation and course effectiveness. This in turn suggested a more sophisticated study of the student- teacher dynamic, student personality, and course effectiveness. Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 138 The research was conducted in classes held at the University of Hawaii in the basic personnel management course. The source of experiential exercises was Whatley and Kelley’s Personnel Management in Action and the authors! files of exercises. The closures for the exercises emphasized the tie between the results and the students’ psychological set and environment. The smallest class was 36 students and the largest 80. Over one half of the class time was committed to exercises. There were three basic research instruments used: (1) an orthodox form of a student evaluation questionnaire administered at the end of the first and third study and also in the first third of the third study, (2) an objective examinations of course content, and (3) the California Psychological Inventory. RESULTS Study 1 was largely concerned with the relationships among student performance, teaching method, the professor teaching the course, and student attitudes. Overall it was clear that the experiential produced better performances on what was mainly a test of cognitive learning. What is even more important though for research design is the interaction of the teaching method with the professor teaching the course. Professor A (experiential) not only had had above average class evaluations in all of his classes but was also one of the designers of the experiential learning method used here. It can be reasonably assumed that this understanding of and enthusiasm for the method is likely to be greater than that of his Professor B (lecture). These factors interacting with the teaching method no doubt partially explain the better performance of his class. Students may perform well in tests but that does not necessarily mean that they value the experience. Table 2 shows student responses on rating scale (5= very interesting) to the question: How interesting did you find the content of the course? Overall the integrated method was judged no more interesting than the lecture method. Similar between method comparisons were possible for a number of other attitudinal variables. Students believed that the experiential course was more stimulating, that the assignments were better and that overall it was better than other courses taken. They did not believe it was more relevant. This at least indicates that the students treated the questions seriously. Clearly the objectives and content of the courses were similar. The relevance of the courses should therefore also be similar. Students did not fall into the trap of believing that because the method of teaching is Stimulating that automatically makes it relevant. These are the results averaging across the two professors teaching the experiential courses. From Table 2 it can be seen that this average hides a major difference. With Prof. A using the experiential method, student attitudes towards the course were much more positive. But in fact there was relatively little difference in student attitudes towards the experiential method as used by Prof. B and the lecture method. This confirms a result which occurs time and again in the literature on educational evaluation. A competent and enthusiastic instructor can use almost any method of instruction and still produce better results, in terms of student performance and attitudes than a less competent and unenthusiastic instructor. It is clear that to evaluate in more depth the experiential method as used here, the relationship between student attitudes towards the course and their attitudes towards the professor teaching it must be examined. STUDY 2 Two measures of attitude towards the teacher were used in Study 2. One was a question asking whether the student felt comfortable with the teacher, the second asked about the classroom atmosphere in general. Not surprisingly the measures were heavily correlated (Table 3). They are almost certainly measuring the same thing, i.e. an individual student’s feelings for his teacher. Five measures of attitude towards the course, or more accurately the exercises which make it distinctive, were used. These attitudes form a highly consistent pattern. Students who found the exercises interesting also thought that they were beneficial for understanding and would like to see more of them and fewer lectures. They believed that exercises like this should be used in other business courses. The interrelationship between the teacher attitude and course attitude variables is consistent with the idea that a halo effect was operating. Those students who felt positive about the teacher also felt positive about all aspects of the course. That is to say that they did not appear to evaluate different components separately. Students who were comfortable with the teacher might have thought the exercises interesting but not very beneficial to understanding. In practice, they did not. This raises at least two problems. Firstly, it is difficult to develop and improve courses if students are unwilling to comment adversely on the few problem areas that occurred in a course which they considered to be very good. Presumably a halo effect for a course evaluated as being disastrous would have a similar though more critical problem. The second problem has to do with causality. Attitudes to the teacher and the course are correlated. But Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 139 could this be due to the intervention of a third variable, i.e. performance. In this study attitude measures were made when students already knew their grades. Perhaps the students with high grades liked both the teacher and the course as a results and not the converse as proposed. In the last study an attempt was made to examine this relationship as well as to introduce another dimension into the evaluation. STUDY 3 In Study 3 attitudes towards the course and towards the teacher were measured early in the course before students had had any formal feedback about their performance. These early impressions should therefore be relatively free from contamination by the performance variable. Personality variables were also measured in this study to discover if certain personality types preferred and/or did well on this type of experiential course. These personality variables were those relating to dominance, self acceptance and need for achievement. It might be expected that these would be important in an experiential learning situation. Looking now at the correlations between sets of variables, the relationship between impressions of the teacher and Impression of the course are similar to those found in Study 2. Having a good first impression of the teacher correlates well with having good first Impressions of the course. The relationships between performance and the other variables are however much less straightforward and therefore more interesting. For example, those students who had good first impressions of the course did badly on both midterms. A similar result occurred for those students who have initially impressed by teacher/student relationships. This group of students were apparently "surprised" by the assessment procedure and could not translate their interest in and enjoyment of the course into good test results. A further clue is furnished by introducing personality variables into the analysis. The only large correlations they demonstrated are those with performance; negative with the first midterm and positive with the second. Clearly these dominant, self-accepting, achievement oriented students were also caught by the first midterm but remedied their mistake in time for the second. The intercorrelations among four sets of variables -attitude to teacher, attitude to course, performance (as measured by 2 midterm results) and personality-are shown in Table 4. Examining first the intra set correlations, it once again appears that being comfortable with the teacher and believing that teacher/student relationships are good measures are the sane variables. The personality variables are clearly inter- correlated supporting the view that they are all tapping an underlying "integrated" personality trait. Unusually, the midterm results show hardly any correlation. This is the first suggestion that there may have been major changes occurring during the course. The pattern of correlations among the attitude to course variables is rather different from that in Study 2. To emphasize that these measures were made early in the course relates to the impressions that follow. Those students who thought that the exercises were interesting did not necessary believe that they helped understanding. In addition, they would have preferred both fewer lectures and fewer exercises. Students who believed exercises beneficial for understanding wanted more exercises. Here is evidence that an overall halo has not yet begun to develop. Students were evaluating separate aspects of the course differently. All of this suggests a possible mismatch between a novel learning experience and a traditional method used to assess the resulting learning. It might initiatively be concluded that those students who were most impressed by the experiential approach were those who paid for it in terms of poor grades. The achievement oriented students quickly realized the problem and did what was necessary in the interim to meet their grade goals. If the terminal measurements in Study 3 paralleled those in Study 2, then the following scenario can be written. Those students who have good first impressions of the course and its teacher gradually become disillusioned when they get poor grades. Other students who are not so impressed, perhaps because they are uncertain and unhappy with non- traditional teaching methods, discover that the grading sy9tem is familiar and that they can cope. Their attitudes towards the course and the teacher then become more favorable. This In turn suggests major attitude changes during the course. This would be relatively simple to investigate and would form the subject of future research. CONCLUSIONS Above all the results of these three studies underscore the multidimensionality of evaluation of teaching and learning. The situation is probably more complex for novel trends of learning experiences as compared with traditional ones. This is particularly true since proponents of new teaching methods would like to be able to make general statements about where these methods are superior or inferior to existing methods. Study 1 indicates the near impossibility of this quest. The teacher clearly had more greater effect on the Outcome than the learning method. The results of Study 2 warn us to beware that it may not always be easy, or even pos8ible, to separately evaluate various features of a course if the main product variable is student attitudes. The halo effect created by a good or bad overall experience is the culprit here. Study 3, while unearthing a possible mismatch between learning and assessment methods, also hinted that measurements durings rather than just at the end of a course, may be necessary if we are to conduct satisfactory evaluation studies. It is tempting to conclude that there is little future in research aimed at making general statements about Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 140 The of one teaching method versus another. Such studies are possible but are complex, expensive and difficult to mount in an actual teaching situation. More importantly, in controlling for variables other than teaching method their results are necessarily conditional. These conditions which include the teachers involved, type of subject, educational objectives, type of students, etc., will be different for each situation in which the particular teaching method will be employed. The interrelationships identified in this paper can be described as depicted in Figure 1. The Presage Variable in this model is the teacher, his inclinations, and his attitude toward the course (students, classroom properties, course, as well as pedagogical method). The first study illustrated how important this is regardless of the method used. This phenomenon has also been documented in other studies (2, 10, 13, 20). This paper did not deal with the assessment of different teaching methods but the model and other research (12, 20) begs the question of the optimum method being based upon (1) the teacher characteristics, (2) the classroom context, i.e. class size, and (-3) the objective of the product variable, i.e. cognitive, behavioral or cognitive? It would appear that it would be difficult for any one method to be “the most effective’ given the variables to consider. This paper and others did deal with the two other factors - student characteristics (5, 7, 17) and the student-teacher relationship (2, 10, 18, 19, 22, 23). In terms of student personality, first that student attitude toward experiential exercises did not correlate with student personality variables, but did identify the importance of the student-teacher relationship. Lastly, the model asks the question of the Product Variable called student growth which can be defined in many ways but in the model illustrates the weakness of the most common form of course/teacher evaluation -the student evaluation (13, 14, 15). The Product Variable is the student’s growth which the student is asked to assess in most studies very subjectively - not objectively. The student enters pedagogical process at the classroom - unaware of the best of decisions that have taken place by the administration, curriculum committees, department and course committees concerning the course content, prerequisite, class size, time of day, and assignment of instructors. The model and the research reported in this paper illustrate the complexity of assessing pedagogical methods and the need to address this complexity in our research. REFERENCES [1] Alpin, J.C. and R.A. Cosier, “The Use of Intensive Simulation in Executive Development and Academic Settings, Insight into Experiential Pedagogy, (S.C. Certo and D.C. Brenenstuhl, Eds.), Bureau of Business and Economic Research, Arizona State University, Tempe, AZ, 1979, pp. 161-163. [2] Aspy, D.N. and W. Hadlock, “The Effect of Empathy, Warmth and Genuineness on Elementary Students Reading Achievement,” unpublished thesis. [3] Beatty, R.W., R. W. Johnson, T.E. Hendrick, C.P. Morgan, and E.J. Morrison, “The Next Decade of Management Education: How Different? Academy of Management Proceedings, 1973, pp. 271-278. [4] Belasco, J.A., A.M. Glassman, and J.A. Alutto, “Experiential Learning: Some Classroom Results, Academy of Management Proceedings, 1973, pp. 235-241. [5] Brenenstuhl, D.C. and R.F. Catalanello, “Assessing the Effectiveness of Learning Styles as Predictors of Performance Within Three Distinct Pedagogic Methodologies,” Exploring Experiential Learning: Simulations and Experiential Exercises, (D.C. Brenenstuhl, and S.C. Certo, Ed.), Bureau of Business and Economic Research, Arizona State University, Tempe, AZ, 1978, pp. 179-185. [6] Butler, J.L. and L.B. Keys, A Comparative Study of Simulation and Traditional Methods of Supervisory Training in Human Resource Development,’ Academy of Management Proceedings, 1973, pp. 302-305. [7] Byrne, E.T. and D.E. Wolfe, An Exploratory Study of Student Characteristics and Educational Processes in Programmatic Experiential Learning,” Insights into Experiential Pedagogy, (S.C. Certo and D.C. Brenenstuhl, Eds.) Bureau of Business and Economic Research, Arizona State University, Tempe, AZ, 1979, pp. 239-242. [8] Calás, Marta, “Experiential vs. Traditional Classroom Approach in the Basic Management Course: A Puerto Rican Experience,’ Insights into Experiential Pedagogy, (S. C. Certo and D. C. Brenenstuhl, Eds.) Bureau of Business and Economic Research, Arizona State University, Tempe, AZ, 1979, p. 266. [9] Carroll, S. J., F. T. Paine and 3.3. Ivancevich, “The Relative Effectiveness of Teaching Methods -Expert Opinions and Research,” Personnel Psycho- 1972, pp. 495-509. [10] Christensen, C. M., Relationships Between Pupil Affect-Need, Teacher Warmth, and Teacher Permissiveness," Journal of Educational Psychology, Vol. 51, 1960, pp. 169-174. [11] Connolly, E., W. J. Connolly, A. Pounds, and F. A. Wiebe, “Evaluation of an Experimental Course in Organizational Behavior for Managers of Japanese Banks,” Exploring Experiential Learning: Simulations and Experiential Exercises, (D. C. Brenenstuhl and S. C. Certo, Ed.), Bureau of Business and Economic Research, Arizona State University, Tempe, AZ, 1978, pp. 90-103. Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 141 [12] Gentry, T. W., K. G. McCain, and A. C. Burns, “Relating Teaching Methods with Educational Objectives in the Business Curriculum,” Insights into Experiential Pedagogy, (S. C. Certo and D. C. Brenenstuhl. eds.), Bureau of Business and Economic Research, Arizona State University, Tempe, AZ, 1979. pp. 196-198. [13] Hart, J. and J. Driver, Teacher Evaluation as a Function of Student and Instructor Personal!- Teaching of Psychology, 1979. pp. 198-199. [14] Haslett. 3.5.. “Dimensions of Teaching Effectiveness: A Student Perspective.” Journal. of Experimental Psychology, Vol. 44, Summer 1976, pp. 4-10. [15] Hawkes, G. R. and R. L. Egbert. ‘‘Personal Values and the Empathis Responses: Their Relationships.” Journal of Educational Psychology, Vol. 45. 1954. pp. 6-476. [16] Hicks M. O. is Popp, Experiential Approach Teaching Business Policy Learning Environment. Academy Management Proceedings, 1972, pp. 277-280. [17] Johnson, G. A. and W. E. Stratton. Learning Style and Performance. Exploring Experiential Learning: Simulations and Experiential Exercises. D. C. Brenenstuhl and S. C. Certo, Bureau of Business and Economic Research Arizona State University, Tempe, AZ. 1978, pp. 186-192. [18] Kelley, L., “And Experiential Evaluation of an Integrated Didactic - Experiential Approach for Teaching Personnel Management,” Journal Experiential Learning and Simulation. Vol. 1, No. 1 January 1979, pp. 39-44. [19] Kelley, A. Whatley, “Student-Teacher Relationship in Experiential Classes and The Student's Perception of Course Effectiveness." Journal of Experiential Learning and Simulation Vol, 2, No. -- March 1980. pp. 9-16. [20] Sherman, B. R. and R. T. Blackburn 'Personal Characteristics and Teaching Effectiveness of College Faculty. Journal of Educational Psychology, 1975, pp. 224-131. [21] Smith J. R. “Gaming and Attitudinal Change,” Insights into Experiential Pedagogy, S. C. Certo and D. C. Brenenstuhl, Eds. Bureau of Business and Economic Research. Arizona State University, Tempe, AZ. 979, pp. 231-234. [22] Thistlewaite D.L. and N. Wheeler, “Effects of Teacher and Peer Subcultures Upon Student Aspirations.” Journal of Educational Psychology, Vol. 57, 1966, pp. 35-47. [23] Truax, C. S. and R. R. Carkuff, Toward Effective Counseling and Psycho-Therapy: Training and practice, Marin Alexander. Ed. . (Chicago: Aldine Publishing Co., 1966). [24] Wolfe J.. "A Comparative Evaluation of the Experiential Approach as a Business Policy Learning Environment.” Academy of Management Journal, Vol. 18, September 1975, pp. 442-452. Table of Contents Volume 8, 1981 The Promotion: Human Sexuality in Organization The Simulation of Chaos Leadership Development in a Simulated Urban/Suburban (U/S) Environment Tomed: A Computer Game Emphasizing Social Responsibility/or/why the Pop-Top Can? Integrated Brain Activity and the Manager's Job: Utilizing the Troika Model What does R2 Have to do with a Product Management Course? An Analysis of the Effects of Jungian Problem-Solving Style Dimensions on Marketing Decisions Bargaining Behavior in Personal Selling and Buying Exchanges Extending the Simulation Product Life Cycle A Generalized Algorithm for Designing and Developing Business Simulations Operationalizing a Test of a Model of the Use of Simulation Games and Experiential Exercises Pygmalion and Perception: An Experiential Exercise Behavioral Consequences of Reward Regarding Employee Absenteeism in an Industrial Setting: An Operant Conditioning Approach The Simlab Program: The Use of Experimental Simulation and Process Analysis for the Development of Management and Organizations Designs for Research on Simulation-Games, Cases, and Other Experiential Exercises The Role of Students in The Case Method Weaknesses in Research Design Critical Variables in Research on the Educational Value of Management Games Research Questions for Cases Research on the Learning Effectiveness of Business Simulation Games - A Review of the State of the Science The Effects of Valuation Techniques on Holding Cost During Inflationary Periods: A Simulation Exercise The Operations Simulation - A Study in Game Development Applying Guided Design to the Production/Operations Management Course: A Progress Report and Evaluation Terminal Data Entry and Retrieval Systems Simulations and Microprocessors Microcomputers and Related Technology for Simulation Gaming Microcomputers - A New Technology for Innovations in Business Simulations Microprocessor Controlled Interactive Video Simulation Business Game Design: From Theory to Practice The Success of a Computerized Simulation in Microeconomic Pedagogy The Test Preview Game: Applying the Game Show Format Providing a Real World View of the Personal Function: A Simulation Finding an Effective Means of Teaching Managerial Behavioral Skills: Two Different Experiential Teaching Methods Compared A Management Development Program Based on the Experiential Learning Model Participant Type Differences in Response to Experiential Methods: An Informal Look Preparing Student Groups to Participate in Experiential Group Projects: An Organizational Development Approach An Instrument for the Assessment of Learning Dimensions: A Progress Report of the Learning Dimension Scale (LDS) An Empirical Analysis Relating the Learning Style Inventory to Memory and Logical Ability Teaching Styles in Simulation Experiential Learning Versus Traditional Teaching Styles Student Perceptions of Effective Teaching Behaviors Problems in Evaluation of Experiential Learning in Management Education Students' Perceptions of Learning by Simulation A Relative Evaluation of Experiential and Simulation Learning in Terms of Perceptions of Effected Changes in Students Overview of Computer Based Business Games in Business Policy Classes Behavioral Decision Theory and Business Policy Giving Accounting Students Writing Experience as Job Preparation Getting to First Base with MBO: An exercise for Writing and Evaluating Objectives Dimensions of Conflict in Experiential Learning Consumer Alienation and Perceived Relevance of the Business Simulation Using the Self-Reference Criterion to Simulate Culture in Internationalized Business Course Experiential Learning in a Cross Cultural Setting- The Practice of Simulation Approach to Business Education in Japanese Universities Student' Perceptions of the Use of a Computerized Simulation in Teaching Management Information Systems Using the Case Study Approach to Develop a Microcomputer Bases, Fully Integrated, Data Base Driven, Management Information System The Case Study as a Tool for Organizational Change: Applying the Steel Ax to the Designers of Management Information Systems Toward a Theory of Teaching Business Policy A Case Study in the Use of Experiential Learning (A Management Game Simulation) to Enhance Student Understanding of Strategy Evaluation and Policy Formulation Suggestions for Integration of the Business Administration Core Publishing Opportunities and Requirements for Business Simulation and Experiential Learning Materials How do we Apply Experiential Learning Intercollegiate Case Competitions for M.B.A. Students: Initiation and Implementation Teaching Business Policy and Strategy Using the Incident Process The Learning Co-Op Approach to the Core Policy Course Meeting the Managerial Skill Shortage - Is Academia Up to It? An Empirical Analysis of Experiential Learning Reinforcement International Experiential Learning: Experience is the Best Teacher Encouraging Student Participation During International Academic Programs European Summer Study Program: Can you, Should you, What Does it Take? Travel Seminars in Europe: How can I Direct One? International Experiential Learning: Student Evaluation ABSEL: Empirical Findings on the State of the Association Sensitivity of Performance Scores in Business Simulations Improving the Learning of a Business Simulation Game by Increasing the Process Content Student Participation in Deciding Performance Criteria for Grading in an OB Course: An Exercise and a Case Study Markup for Profit: A Simulated Self-Administered Experience in Retail Pricing The Investment Decision Game: An Experiential Learning Approach to Stock Market Decisions Through Gaming CHIPS: A Marketing Channels management Game External Validation: An Experimental Approach to Determining the Worth of Simulation Games Teaching Performance Appraisal Skills: An Experiential Approach In Support of Experiential learning: Results of a Follow-Up Survey The Introductory Management Course: Taking Theory Application One Step Further Decision Efficiency and Effectiveness in a Business Simulation The Implications of Cognitive Processing Variables and the Complex Decision Simulating the Simulation for Enhanced Player Rationality Simulation/Experiential Learning Audit