THE ROLE OF EXPERIENTIAL LEARNING AND SIMULATION IN TEACHING MANAGEMENT SKILLS Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 65 THE ROLE OF EXPERIENTIAL LEARNING AND SIMULATION IN TEACHING MANAGEMENT SKILLS Richard Teach, Georgia Institute of Technology Gita Govahi, University of Southern California ABSTRACT This is an empirical Study that questioned college alumni who were graduated during the years 1982, 1983 and 1984. To be included in this study, each subject was to have had an exposure to either simulation or experiential exercises in either their graduate or undergraduate program. Each person was asked to report the importance of a set of 41 attributes or skills to their current jobs. In addition, they were asked to rate various teaching methods on how well each method conveyed this set of predetermined skills. The analysis of the data showed the following results; teaching methods employing experiential exercises best taught how to develop consensus, how to appraise performance and how to resolve conflict, while the use of simulations best taught how to measure objectives, how to solve problems systematically and how to forecast. The use of the case method best taught how to conceptualize, how to put structure to unstructured problems and how to think creatively. The only skill or attribute that traditional lectures taught best was how to listen reflectively. THE CONCEPT Members at ABSEL meetings have consistently discussed the role of simulations and experiential learning techniques in conveying knowledge about a set of skills which are needed by the students when they enter the job market after graduation. Frequently these discussions compared the hands-on techniques of experiential learning and business simulations to the more traditional case methodology. The authors of this paper considered the various concepts, reviewed some of the literature (Whetten, 1984), (Cohen, 1984), (Rocklin, 1987) and put forth their own hypothesis: Each teaching technique has its own advantages." That is, one teaching method conveys a particular set of skills better than others and different teaching methods convey different sets of skills. Thus, a mixture of teaching techniques is able to leach the entire set of desired skills better than any single method(Tough, 1979). The question remained. Which skills are best taught by what teaching methods?” (Brush, 1983) In order to answer this question. ii was decided to go to those individuals who had been in the work force for three to five years after college and who had experienced at least one of these two teaching methods while enrolled in a college or university. The sampling frame was determined by a two stage process. First, a letter was sent to all attendees of the 1987 ABSEL meeting. (This letter was sent to 110 attendees.) The letter asked each person to go into their files and select 10 students per year from their class roles of 1 982, ‘83 and ‘84. Then, they were to obtain these previous student’s current addresses from the school’s alumni office and send the list to one of the authors. Twenty-two ABSEL members responded with a list of 602 names and addresses. An individualized cover letter and a questionnaire was sent to every name submitted. At the time of this analysis 78, questionnaires had been returned. There was 1 questionnaire which was not usable and 16 which were only partially completed. The partials did not complete the section regarding the rating of the various teaching methods. Thus, this analysis was based upon 62 completed questionnaires. (At the time of submitting this paper, 135 questionnaires had been returned.) THE DEVELOPMENT OF THE QUESTIONNAIRE The literature was searched to define the skills and attributes that "managers" need and the tasks they employ in plying their trade. A set of 41 tasks, skills and/or attributes was developed (Waters, 1980) (Livingstone, 1971) (Mintzberg, 1973). First, each respondent was asked to rate the importance of each skill or attribute to him or herself in terms of their current position. Exhibit 1 details the questionnaire’s instructions for the first section. EXHIBIT 1 Following is a list or 41 attributes that have proven to be critical in effective management. First, read the entire list. Second, select about 8 attributes which you consider to be the most important in your current position and circle the “I” beside each. Next, select about 8 more attributes which you consider to be slightly less important and circle a “2”. Continue selecting sets or about S attributes in descending order of importance until you have exhausted the list (the last set will have a rating or 5). if you have some sets with 9 and a few with 7, that is OK, but be sure to use all 5 scale values. The second section repeated the set of attributes and asked the respondent to evaluate the attributes on the basis of importance to their first position after being awarded their first college degree. The third section repeated the same set of attributes and ask the respondents to rate the quality of up to five educational experiences based upon where he or she had learned the listed skills. The educational experiences listed were 1) Undergraduate Program; 2) Graduate Program; 3) On the Job Training; 4) Professional Development or Continuing Education; and 5) Other (Specify). Exhibit 2 details the instructions for this part of the questionnaire. EXHIBIT 2 Below is the same list or attributes. This time we would like you to consider where you have learned or acquired these skills. We have listed S possible educational experiences. For each attribute, please rate the source where you acquired this skill with a ‘9” being the best possible source, and a “1” being the worst possible source. ir you have not been exposed to any one or the educational experiences listed, insert an “N” in the appropriate column(s). The fourth section repeated the attribute or skill list again. This lime, the respondents were asked 10 rate a set of teaching methodologies based upon the methods ability 10 teach the listed skills. Exhibit 3 provides the instructions provided for this part of the questionnaire. The balance of the questionnaire collected demographics on each subject. Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 66 EXHIBIT 3. Below is the same set of attributes once again. This time we would like you to consider the different teaching techniques that you have experienced in learning these sets of skills. These - teaching techniques include: Lectures. Case Study, Experiential Learning Exercises (Role Playing, etc.), Projects or Independent Study, and Business Simulations (Business Games(. For each attribute, please rate each teaching technique on its importance in your learning the skill. Please use a “9” for the very best teaching method down to a “1” for the worst teaching method. If you have e not experienced one of these methods, please insert an ‘ V in the appropriate column(s) THE SAMPLE BASE Because at the time of this analysis only a few of the coded questionnaires rated either Special Projects or internships, these teaching methodologies were not included in this analysis. Table 1 displays the distribution of completed questionnaires by the respondents last collegiate educations: experience. STANDARDIZING THE DATA ON THE TEACHING METHOD RATINGS When individuals fill out rating scales. some tend to be “yea” sayers and others are “nay” sayers. some individuals use only the upper end of the allowable responses, while others use only the lower end, and still others use the entire range. Since the measure of interest was the relative importance of each teaching method as it contributed to the learning of each attribute by each respondent, the data could be standardized within each subject without losing information. For each –respondent, the mean response along with its standard deviation across all teaching methods was found and a Z score (Mean 0.0 and the Standard Deviation = 1.0) for each response was calculated. These Z scores were then compared across subjects without concern about ‘yea’ and ‘nay’ sayers. The following analysis was done using the Z score data for those questions Pertaining to the ratings of teaching methods. THE RESULTS The Univariate Analysis The grand mean Z score for each attribute was calculated across all teaching methods as well as for each individual teaching method. An F lest was run so see it the distribution of responses for each skill or attribute was unique for each of :he leaching methods. The results were surprising. The distribution of ratings for virtually every skill or attribute was different for each teaching method. If one were to use the .05 level of significance, there was only one attribute (the ability to set goals whose distribution would not be considered to be different across the four reaching methods. The grand mean of the Z scores and the Z score mean for each teaching method along with the significance of the F test for each skill or attribute is shown in Table 2, in alphabetical order, the same order in which they were presented to the respondents. Table 2 also shows the significance of the F test of the differences between groups. The significance is shown rounded to the nearest one thousandths. In all but 8 of the skills, there is less than 5 chances in ten thousand that this difference across teaching methods is the result of chance. As described in Exhibit 1, the subjects were asked to rate the degree of the importance of each of the attributes to their current job or position. The ratings were based on a 5 point scale with 1 being labeled “Most Important” and 5 labeled “Least Important”. The subjects were asked to constrain their responses in a way that forced the use of all (lie values in approximately equal numbers. This provides the property of (almost) equal variance among the subjects of the ratings across the 41 attributes. The grand mean across all 41 attributes and 62 subjects was 3.01, the minimum attribute mean score was 1.96 (Make Decisions) and the maximum attribute mean score was 4.22 (Conduct Interviews). It seems ironic that while the attribute of "conducting interviews” was seen as the most important skill in the prescribed set to the respondents’ current jobs: the ability of any of the investigated teaching methods to teach this skill was considered to be very low, with a mean Z score across all teaching methods of -.34. Table 3 groups the 41 Skills into clusters based upon the Z score means across all the respondents who rated each particular teaching technique. Each cluster contains those skills with the highest Z score means for that particular teaching method. Note hat 9 skills had their maximum under teaching by the case method. There were 15 skills whose mean Z scores were at their maximum when the teaching method of experiential exercises was evaluated. Only 1 skill (listening reflectively) was at its maximum when evaluating the lecture method and 13 skills were at their maximum for teaching methods using simulations. The order of presentation in Table 3 is based upon the ranking of the Z score within each teaching method. The skill with the highest Z score under each teaching method is shown first. Those skills which were rated above the average level of importance are shown in bold type. Note those skills the respondents considered to be most important. The ability to conduct interviews. develop consensus, to supervise, to appraise performance, to enforce the rules and to speak in public were rated as the 6 most important skills and all 6 were best acquired through the same teaching method: experiential learning. The next most important skill, the ability to measure objectives, came from the teaching method of simulation. If one were to ask managers with much more experience than 3 to 5 years. the authors feel confident that the ratings of importance of the 41 skills would be quite different (Culbertson, 1980). Certainly the skill of planning, ranked 36th by the subjects in this study, would be more important for experienced managers. II is important for the reader to understand that this research confined its study to recent graduates and not experienced managers (Hayes, 1981). A discriminant analysis was performed to discover if the different teaching methods could be distinguished from each other on the basis of the ratings of each teaching method across all the attributes. Discriminant analysis is a statistical method in which group membership (a discrete variable) is the dependant variable and a linear combination of the independent variables is formed in a way that maximizes the probability of correct classification of the observations, For this analysis. the teaching method ratings for each of the attributes are the independent variables and the teaching methods are the dependent variables, This analysis was done using a step-wise procedure. The independent variables (the ratings) are not orthogonal to or independent of each other. Technically, the linear discriminant function requires the independent variables to have multivariate normal distributions. However, the discriminant technique is fairly robust even if the concision does not hold (Wahl & Kronmal, 1977). Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 67 Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 68 Multivariate Analysis Univariate tests of significant differences such as the F tests re. ported above arid the use of variable means provide basic information about groups of observations However, in multivariate analysis. the set of variables are considered simultaneously and not one at a time (Harris. 1975) Since the ratings were converted to Z scores, the variable distributions approached normality but they were still highly correlated with one another. Using all the variables, when intercorrelations exist, is a little like double counting. By employing a step-wise procedure. the variables are selected one at a time and a new variable is added only if the additional (orthogonal) information is sufficient to warrant its inclusion. Table 4 details the order in which the variables entered the discriminant analysis. The step-wise procedure employed in this study started with the variable that was best at discriminating among the four teaching methodologies (lectures, case methods, experiential exercises and simulations), based upon the ratings reported by the respondents. The first variable was the skill of analyzing problems. After the first variable was included, the analysis searched the remaining variables and found the one that explained most of the remaining variance. The second variable was the rating on the ability to forecast. This procedure was repeated for 19 steps, bringing in 19 variables. The 20th step was different. In this case, since all of the variables are correlated, the amount of explained variance accounted for by the rating on “Solve Problems Systematically” was no longer significant when all of the first 19 variables were considered simultaneously, and that variable was removed from the analysis. This entering and removing process continued for a total of 32 steps and. at the end, included the 24 variables listed in Table 4, and labeled “in” under the 2nd heading, “Included”. The Column labeled “F value’ is the result of an F test for the variable (attribute or skills, When the F value fell below 1.0. the variable was removed from the analysis. The column labeled “Mm D sq.” is a distance measure between the closest two group centroids. The greater this distance, the greater the ability to distinguish between the teaching techniques on the basis of the Z scores of the variables evaluated by tile subjects. This particular analysis was run in a way that maximized this distance function. Table 5 shows the set of variables that were not included in the final stage of the discriminant analysis. This does not mean that the ratings on these attributes or tasks are the same for all the teaching methods. It only indicates that the additional information, given the first 24 variables, is not significant in distinguishing between teaching methods. The included variables, taken as a whole, overlap the information contained in these remaining 17 variables. The “F to Enter” value is the value of an F Test, If this value was 1 the variable would have been included in the set above. One of the results of a discriminant analysis is a set of linear functions which are used for the classification of the observations or cases. Table 6 below provides the coefficients for each of the included variables in the analysis. One only need multiply these coefficients by the observed rating for the specified variable, sum these values across the variables, and add the constant. The result is a value of each function for the particular observations evaluated. Or: Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 69 Table 7 shows the results of applying these coefficients and classifying each set of ratings into a group on the basis of the 3 function values, Note that 01 the 62 observations that rated Lectures, 54 were correctly classified. Three were misclassified into the case study group, three were put into the experiential learning group and 2 were classified into the simulation category. Using the case study ratings, 49 out of 62 were correctly classified with most of the misclassification occurring when a case study rating was paced in the simulation group. In the experiential exercises ratings, 42 out of the 55 cases were correctly classified but in the simulation ratings only 34 out of 54 were correctly classified. The majority of misclassified observations in the simulations category were estimated to experiential exercises. A total of 179 ratings were correctly classified. If this were a random procedure, one would expect a correct classification of only 25 percent. As noted above, three functions were used to “discriminate" between the four teaching methods. It would have been possible to obtain less than three functions but no more. The maximum number of dimensions in which four (N) items (in this case the 4 teaching methods) can be placed is three (N- 1). From the original solution. Table 8 shows the explained variance of the solution (not the original data set) accounted for by each function or dimension. Keep in mind that these 3 functions are orthogonal. The first function, similar to a factor in factor analysis, explains over two thirds of the variance in classifying the teaching methods. As en factor analysis, it is possible to have a better understanding of the discriminating functions by a rotation of the axes in order to have the variables load heavier on one axis and less on the remaining ones. This process changes the amount of variance accounted for by each function and the variable loadings but will not change the classification results because the axes remain orthogonal. One result of rotation of the axes is that the meaning of each function may be more interpretable. Table 9 shows results of rotating the axes on the distribution of the explained variance, Note that the amount of variance explained by the first function went down from over 68 percent to just above 50 percent. The amount of variance explained by the second function went up from just over 20 percent to over 35 percent. The third remained relatively unchanged. Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 70 The next step is to determine the standardized coefficients for the rotated functions and group them in a way that shows which variables contribute most heavily to each function. This procedure should add some interpretability to the results since these coefficient values are directly comparable to one another and the larger the absolute values, the greater the variable contributions to the function. Thus, a large negative coefficient value contributes as much as a variable with a positive coefficient of the same magnitude. Table 10 shows these standardized coefficients for all 24 variables. The dashed line is used to separate the variables and cluster them into sets. Each set includes those variables which contribute the most to a corresponding function. The first function has Reflective Listening and lie ability to Enforce Rules weighted negatively while Make Decisions and Assessing a Situation Quickly are weighted positively. The second function has the ability to Adapt to New Tasks as its largest positive contributor with Developing Consensus and Prioritizing Tasks following close behind. Its important negative weighted variables are the abilities to Write Effectively and lo Analyze Data. The third function is weighted very heavily with the ability to Fore. cast with a negative value. The ability to Direct Others has a positive coefficient, but carries less than half of the weight of Forecasting.. The authors had hoped that the structure underlying these functions would become evident and easily identified. However, that is not the case and naming these functions or factors, as one would do if this were a marketing analysis. seems impossible. Therefore, the analysis will have to settle for functions 1, 2, and 3. CONCLUSIONS As business educators, we are concerned with teaching students both the internalization and the application of management skills. Since teaching management skills is considered more critical, often gathering information on the level of success in teaching these skills is often very limited. This is mainly due to a lack of accessibility to students after they graduate and start applying these skills in the work situation. The intent of this study was to take the first step n providing educators with information regarding a) an identification of critical management skills and their relevance to the students first jobs, b) the sources or programs where these skills are taught, c) the most effective teaching method in conveying any one of these skills. Each respondent was asked to report the importance of 41 managerial skills to his or her current position. They were also asked to rate various teaching techniques on how well each technique conveyed this set of predetermined skills. The analysis 01 the data (based upon 62 responses from 15 different university alumni) showed the following results: Experiential exercises were most effective in teaching skills of developing consensus, appraising performance and resolving conflict. Simulations best taught how to measure objectives, solve problems systematically and forecast. The case method was reported most successful in teaching how to conceptualizes put structure to unstructured problems and to think creatively. Lectures best taught reflective listening skills. Over 76 percent of grouped teaching method cases were correctly classified using multiple discriminant analysis. This compares to an expected value of 25 percent correct classifications if the data were based upon random responses. The results of the analysis clearly emphasized the effectiveness of utilizing multiple teaching techniques in teaching management skills. This finding alone could have major implications for educators who have predominantly employed a singe leaching method in conveying the art and science of management. Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 71 ENDNOTES and REFERENCES All the statistical analysis for this study was done using SPSSx Release 2.0, SPSS, Inc. 1986. Ackerman, Linda. The Transformational Manager. Facilitating the Flow State University Associates, 1985. p 242 Braadwell, Martin M., "Supervisory Training in the 80’s. Training and Development Journal, Vol 32:2, Feb., 1980 Brush, Donald H. "The Impact of Skill Learnability on the Effectiveness of Managerial Training and Development, Journal of Management, Vol 9:1 Spring/Summer, 1983 pp 27-39. Cohen , Effective Behavior in Organizations. 1984 Culbertson, K. and Thompson, M. "An Analysis of Supervisory Training Needs" Training and Development Journal. Vol 34:2 Feb., 1980, pp 58-62 Hayes, J. "Preparing Future Leaders’, Management Review May, 1981, pp 2-3 Harris, Richard J. A Primer of Multivariate Statistics, Academic Press Inc., 1975 Livingstone, J. S. "Myth of the well-educated manager’. Harvard Business Review, 1971 vol. 49, No 1 pp 79-89. Mintzberg, Henry The Nature of Managerial Work. Chapter 3. “Some Distinguishing Characteristics of Managerial Work Harper & Roe 1973, pp 28-53 Pascale, R. T. “Zen and the Art of Management”, Harvard Business Review, March 1978, p.156 Rocklin,C. Diagnosing the Training Situation: Matching Instructional Techniques With Learning Outcomes and Environments University Associates, 1987. Tough, Allen The Adult’s Learning Projects. A Fresh Approach to Theory and Practice in Adult Learning 2nd Edition, 1979 Wahl, P.W. and Wronmal, R. A. “Discriminant functions when covariances are unequal and sample sizes are moderate.” Biometrics, Vol. 33, 1977, pp 479-484. Waters. James A. Managerial Skill Development” Academy of Management Review 1980, Vol. 5, No 3 pp 449-453. Whelten, David A., Developing Management Skills. 1984 Zoffer, H. J. "Restructuring Management Education, Management Review, Vol 6. April 1981, pp 37-41 Table of Contents Volume 15, 1988 The Role of Experiential Knowledge and Human Information Processing in Decision Making A Semantic Differential Instrument to Evaluate Experiential Teaching Methods A Comparison of Two Approaches to Management Skill-Building in an Organizational Behavior Course: A Replication Integrating Simulations: A Model for Business Policy Success Capstone Renaissance = Simulation + Interaction + DSS A Hybrid Method of Executing a Management Simulation: Combining the Best of Mainframes and Microcomputers Providing an Experiential Dimension to Cost/Managerial Accounting Courses Utilization of Computerized Tax Research Services in the Tax Research Curriculum Using and Expert System Based Decision Aid in Accounting Information Systems Event-Extended Entity-Relationship Diagrams for Understanding Simulation Model Structure and Function Multiple Objectives in the Development of the Gordon Macro Game A Comparative Study of Strategic Performance Factors in Actual and Simulated Business Environments An Empirical Investigation of Integrated Spatial-Proximity MCDM-Behavioral Problem Solving Technology Group Decision models Computer Simulation of Human Interaction The Role of Experiential Learning and Simulation in Teaching Management Skills Expert Systems - The New Business Simulation Tool Integrating Prolog into and Undergraduate Logistics Course Simulating Material Requirements Planning on Lotus 1-2-3 Innovation in Management Education: The Impact of the AACSB Experiential Learning in the International Environment Educational Testing with the Microcomputer A Simulation of Investment Analysis, Portfolio Management and Reporting Using Lotus 1-2-3 The Use of an Expert System to Develop Strategic Scenarios Two Exercises for Teaching about Motivation Sex Roles and the Good Manager A Form and Process for Nonconfidential Peer Evaluations Simulation and the Recalcitrant Student Employee Rights-Student Rights: A Classroom Exercise Computer Simulated Competition: An Alternative to Team Play Management Simulation The Relationship of Locus of Control and Vividness of Imagination Measures to Simulation Performance Formal Planning, Simulation Team Performance, and Satisfaction: A Replication Experimental Analysis of Magnitude and Source of Students' Inequitable Classroom Perceptions in Three Reward Conditions Strategy Design, Process and Implementation in a Stable/Complex Environment: An Exploratory Study Matching a Strategy Simulation to the Business Policy Literature: A Black Box Approach to Simulation Development An Evolutionary Classroom Experiential and Computer Simulation Model of a Corporate Strategic Planning System Collective Bargaining in the City of Elson: A Public Sector Experience Should Students Play Games in Labor Relations? Applying Cognitive Educational Objectives to Business Management Cases Grading as a Teaching and Feedback Mechanism: Involving Students in the Grading Process Teaching Controversies: A New Approach to Computer-Assisted Instruction and Simulation Packages Simulating Demand in and Independent-Across-Firm Management Game Advertising Response in the Gold and Pray Algorithm: A Critical Assessment A Model for Pricing Decisions in First Period Marketing Simulation Games Jog Your Right Brain: An Exercise for the Classroom and for Research Six Thinking Hats: An Exercise to Combat Confusion and Develop Thinking Skills Communicating in Context: A Simulation for Learning Business Communication A Simulated Consulting Service for the Compete Marketing Simulation Game Action Exams in the Consumer Behavior Class Using Focus Groups to Teach Problem Definition in Basic Marketing Research The Use of Journals in Management Simulations: A Literature Review and an ABSEL Response An Initial Step Towards Developing and Using an Expert System with a Business Simulation Self-Managed Learning: An Experiential Course Design Using the QWL Paradigm A Review of Current Developments in Experiential Learning Bring the Real World into the Classroom Assessing Student Performance on a Business Simulation Experience Minimizing Startup Anxiety: Case Studies of Simulation Experiences A Tale of Two Shepards Or Using Simulation in a Class Without Walls Enhancing Business Simulations Through the Utilization of Experiential Activities Involving Local Community Executives Simulation with Integrated Spreadsheets: The Design and Development of a Conversational Marketing Concepts Decision Game Introducing INMART: An International Marketing Simulation Using a Computer-Based Business Plan Assistant in Conjunction with a Marketing Simulation Game Overview of the ABSEL Guide to Experiential Learning and Simulation Learning