EVALUATING THE EFFECTIVENESS OF ROLE PLAYING SIMULATION AND OTHER METHODS IN TEACHING MANAGERIAL SKILLS Developments In Business Simulation & Experiential Exercises, Volume 22, 1995 116 EVALUATING THE EFFECTIVENESS OF ROLE PLAYING SIMULATION AND OTHER METHODS IN TEACHING MANAGERIAL SKILLS James N. Trapp, Stephen A. Koontz, Derrell S. Peel, and Clement E. Ward Oklahoma State University ABSTRACT The effectiveness of lectures, case studies, laboratories, and role playing experiential simulation in teaching 41 key business management skills was analyzed based on survey results from a class of junior and senior agribusiness students. Role playing simulation was found to be clearly the most effective teaching method. In addition, role-playing simulation was found to be the most complementary teaching method to lectures. The managerial skills that role-playing simulation was rated best at teaching were found to more likely be skills that lectures were weakest at teaching. INTRODUCTION Traditional teaching evaluations of university classes do not ask students how effective the course was in teaching them skills they will use in their career. This is probably the case for several reasons. First, the bureaucracy of the university often favors using one standardized evaluation form for all classes. Thus, forms cannot be made career specific since they must span curriculums that prepare students for many diverse careers. Second, in many cases no functional definition of what skills are the most needed by a given career field exists. The objective of evaluating courses from the perspective of what relevant career skills are learned from the course seems to have merit, in spite of the problems just cited. This paper presents evaluations, not of specific courses, but of four teaching methods (lectures, case studies, laboratories and role playing simulation) for teaching business managerial skills. EVALUATION METHOD The study by Teach and Govahi published in the 1993 December issue of Simulation & Gaming provided the evaluation model used. Teach and Govahi surveyed 602 graduates from 36 business schools in order to prioritize a list of 41 managerial skills according to usefulness. Individuals surveyed were business school graduates with three to five years of employment experience. They were asked to divide the list of 41 skills into five groups, with seven to nine skills in each group, and to give a rating of one to the most critical group, followed by a two for the second most critical group and so on. The ratings were then aggregated to prioritize the 41 skills. A listing of skills according to the rating found is given in the left-hand column of Table 1. Teach and Govahi also asked the respondents to rate each of four teaching methods from one to nine with regard to which teaching method was most effective in teaching a given skill. The four teaching methods considered by Teach and Govahi were lectures, case studies, experiential exercises (internships and role playing simulators), and simulations (computerized simulations involving competition by individuals or teams against the computer or other individuals/teams simultaneously submitting information into the computerized simulation). Teach and Govahi found in general that simulations were judged by the respondents to have been the most effective teaching technique followed by experiential learning, case studies and lectures. However, they also found that each of the different techniques taught different skills more effectively. Lectures were most effective in teaching “reflective listening”. Case studies were the most effective in teaching how to “analyze problems”, “analyze data”, “gather pertinent information”, “write effectively”, etc. Case studies were rated the most effective in teaching 8 of the 41 skills considered. Experiential exercises were rated as being most effective for teaching how to “solve problems creatively”, “resolve conflict”, “conduct interviews”, “speak in public”, “lead”, “exert influence”, etc. Experiential exercises were rated as the most effective in teaching 12 of the 41 skills considered. Simulation was rated as being the most effective in teaching how to “make decisions”, “forecast”, “assess a situation quickly”, “adapt to a new task”, “solve problems systematically”, “develop people/teams”, “organize”, etc. Simulations were rated as the most effective in teaching 20 of the 41 skills. Teach and Govahi’s skills list and one through nine rating system were used to rate four agribusiness teaching methods used at Oklahoma State University (OSU) in the Agricultural Economics Department. Developments In Business Simulation & Experiential Exercises, Volume 22, 1995 117 Two fundamental differences exist between this study and that of Teach and Govahi’s. First, evaluations were conducted with junior and senior students. Second, because of the mix of teaching methods students in the Agricultural Economics Department and Oklahoma State University are exposed to, two of the teaching methods evaluated in this study were defined differently. The four teaching methods evaluated were lectures, case studies, laboratories, and role playing simulation. Laboratories are used in the OSU Agricultural Economics curriculum to teach computer skills, to work managerial economics and accounting problem sets, and to teach technical agricultural production science. Students in this survey had been exposed to only one role playing experiential learning class. That class was a one credit hour semester long class that used a fed cattle market simulator in an experiential learning format, i.e., students in the class role played as meat processing plant managers and cattle feedlot managers. Within the simulated environment, players bought or sold cattle to fulfill the objectives of their firm. All transactions and negotiations in the simulation occur face-to-face between individuals, rather than between individuals and a computer (see Koontz et al.). Several additional points should be noted regarding the students surveyed. Because students surveyed in this study had been exposed to only one role playing simulation class, the role playing simulation evaluations presented here are for a class as much as for a teaching technique in general. In addition to rating the effectiveness of four different teaching methods, the students surveyed were asked to rate the total curriculum with respect to its effectiveness in teaching each of 41 different managerial skills. Students were asked to rate the total curriculum’s effectiveness with a letter grade of “A” through “F” rather than using a one to nine scale. Finally, the students surveyed did not know that any of the managerial skills listed were deemed to be more critical than others. SURVEY RESULTS Role Playing Simulation was clearly rated as the most effective teaching method of the four, followed by case studies, laboratories and lectures (see Table 1). At the 5 percent level of statistical significance, no significant difference was found for the over-all average ratings given to case studies, laboratories and lectures. When a statistical significance level of 5 percent is considered, role playing simulation received significantly greater ratings than any of the other teaching technique for 1 7 of the 41 skills. The only skill for which a teaching method other that role playing simulation was rated as significantly more effective was the skill “write effectively”. Both case studies Developments In Business Simulation & Experiential Exercises, Volume 22, 1995 118 and lectures were rated as significantly better in teaching writing skills. In 24 out of the 41 skills at least one other teaching method was rated as equally effective to role playing simulation. Teach and Govahi state that, “When individuals fill out rating scales, some tend to be ‘yea’ sayers and others are ‘nay’ sayers. Thus some individuals use only the upper end of the allowable responses, some use only the lower end, and still others use the entire range.” Because of this, and because the measures of interest are primarily the relative effectiveness of different teaching methods, they argue (and we concur) that the data should be normalized before it is analyzed. To this end a mean score and standard deviation was found for each respondent. These means and standard deviations were then used to derive a normalized “z- score” for each response according to the following formula: The normalized ratings were then averaged to form Table 2. The z- scores reported in Table 2 essentially indicate how many standard deviations above or below the mean response the typical student rated the effectiveness of a given teaching method. Thus all scores in table 2 are directly comparable, including the total curriculum evaluation scores that were rated using a grading scale instead of a one through nine scale. In comparing Table 2 and Table 1, a few changes in ratings and significant differences can be seen, but by enlarge the results are very comparable. Indeed, it appears that normalization slightly strengthened the ratings for role playing simulation relative to other methods. In Table 2, role playing simulation has statistically significant higher ratings for 21 of the 41 cases versus only 1 7 of the 41 cases in Table 1. EVALUATION OF THE SURVEY RESULTS Role playing simulation seemed in general to have been given superior ratings by the students surveyed. However, questions remain as to whether one or more of the methods focuses more heavily on the skills rated most critical by Teach and Govahi’s study and/or whether certain teaching methods complement each other by teaching skills that others do not. Z-Score Ratings A basic sense of the focus of each teaching method can be gleaned by rating the skills for each teaching Developments In Business Simulation & Experiential Exercises, Volume 22, 1995 119 method according to their z-scores. Tables 3 through 7 do this for each of the four teaching methods considered and for the total curriculum. Column #1 reports the teaching effectiveness rank as determined by ordering the skills by z-scores listed in column #4. Column #2 reports the skill’s rank as determined by Teach and Govahi. most important. Lectures and case studies appear to be equally less effective. Table 8 summarizes how many of Teach and Govahi’s top ten rated skills are contained within the top quarter (top 10) and top half (top 20) of the ratings found here for each teaching method. Based on the counts reported in Table 8, laboratories and role-playing simulation seem to be most consistently focused towards effectively teaching the skills Teach and Govahi’s study rate as Correlations Between Skill Ratings and Teaching Effectiveness A more comprehensive look at the focus of each teaching technique can be derived by considering the correlation between Teach and Govahi’s skill ratings to the teaching effectiveness ratings for each teaching method, i.e., the correlations between columns #1 and #2 in Tables 3 through 7. Two forms of correlation will be calculated, simple linear correlation (r), and Spearman’s Rank-Difference Correlation Developments In Business Simulation & Experiential Exercises, Volume 22, 1995 120 (SRDC). SRDC is specifically designed to consider correlations between ranks. It is capable of ranging from -1 to + 1 and is calculated as follows: A third method of measuring the focus of each teaching method toward the most critical managerial skills is to determine the linear correlation between the skill ranks determined by Teach and Govahi and the z-scores found in this study, i.e., the linear correlation between columns #2 and #4 in Tables 3 through 7. These three forms of correlation are reported in Table 9. The correlation measures presented in Table 9 indicate that case studies and laboratories are focused most effectively on teaching the skills rated as most important by Teach and Govahi’s survey. The total curriculum seems to be well focused on teaching the skills rated as most critical. Figure 1 provides additional insight into the strengths and focus of each teaching method and the curriculum in general. This figure plots the ordinary least squares estimates of the functional relationship between z-scores for each teaching method and Teach and Govahi’s skill ranks, i.e., the function for which linear correlations are reported in column #3 of Table 9. All of the estimated relations have a negative and significant slope indicating that all of the approaches do relatively better at teaching the more Developments In Business Simulation & Experiential Exercises, Volume 22, 1995 121 important skills versus the less important skills as rated by Teach an Govahi. The clear superiority of role playing simulation in teaching nearly all skills, irrespective of their rating, is seen by the height of the role-playing simulation plot relative to other plots. The lecture method of teaching is shown to be the weakest teaching methods, but it is also shown to be the most consistent over the entire range of skills, i.e., its slope is the least of any of the functions. Complementarity and Competitiveness of Alternative Teaching Methods The strength of a total curriculum in effectively teaching key managerial skills, as well as a wide range of management skills, may rely upon using a portfolio of teaching methods. Teach and Govahi use discriminant analysis and conclude that lectures, case studies, experiential exercises and simulations teach different skills. More specifically their discriminant Developments In Business Simulation & Experiential Exercises, Volume 22, 1995 122 analysis allows them to conclude that their four sets of z-scores for the four teaching methods considered do not come from the same population. However, Teach and Govahi do not provide any analysis of which teaching methods complement each other best, i.e., which methods teach to the skills that others do not. The most pertinent question in this regard would seem to be “Which methods of teaching complement lectures the best?” This is because lectures are hypothesized to be the primary teaching method used in most classes, and therefore the method most in need of being complemented by other methods. Portfolio theory suggests a method for considering the question of teaching method complementarily. That method is to determine the correlations between z-scores for each of the teaching methods. Highly complementary teaching methods should have large negative correlations, while competitive methods will have large positive correlations. Table 10 presents the correlation coefficients between the z-scores of the four teaching methods studied here. No significant negative correlations were found at the 5% level of confidence. All correlation, except the correlation between role playing simulation and lectures, were found to be positive and significant. This implies the interaction between all teaching methods except lectures and role playing simulation are competitive. That is they tend to focus on the same skills. This is especially true for laboratories and case studies, which exhibit a correlation coefficient of 0.75. Thus, of the three alternative teaching methods to traditional lectures, role-playing simulation appears to be the only method that functions independently and perhaps in a slightly complementary manner to lectures in teaching. SUMMARY AND CONCLUSIONS This study used the survey and teaching method evaluation system of Teach and Govahi to analyze the effectiveness of four teaching methods used in Developments In Business Simulation & Experiential Exercises, Volume 22, 1995 123 the Oklahoma State University Agricultural Economics curriculum. Specifically, the effectiveness of lectures, case studies, laboratories, and role playing experiential simulation in teaching 41 key business management skills was analyzed. Role playing simulation was found to be clearly the most effective teaching method. In addition, role-playing simulation was found to be the most complementary teaching method to lectures, i.e., the managerial skills that role- playing simulation was rated best at teaching were found to more likely be skills that lectures were weakest in teaching. Complementarily with lectures is deemed to be an important feature of a teaching method since lectures undoubtedly will continue to comprise the bulk of university teaching efforts for some time to come, and because lectures are generally needed to teach the introductory material and basic theory necessary to conduct role playing simulations and other teaching approaches such as case studies and laboratories. The objective of this study was to evaluate teaching methods and courses from a perspective of their effectiveness in teaching relevant career skills. The conclusions of this study may be limited because the survey used was for only one set of students at one institution. However, the basic results seem statistically robust and it is believed they will endure the test of expanded samples. A second, and perhaps more significant contribution of this study is to have extended Teach and Govahi’s work into a procedure for individual programs and departments to evaluate their teaching methods, courses, and curriculum. It is believe that the approach used here can be helpful to all business and economics departments in learning more about their own teaching methods, courses, and curriculum. More specifically, it is believed that this approach to course and teaching evaluation can greatly enhance and extend what can be learned from more traditional teaching and course evaluations that do not pose questions about career skills learned. REFERENCES Koontz, S.R., Peel D.S., Trapp, J.N., and Ward C.E. (1993). The Packer-Feeder Game: A Commodity Market Simulator. Developments in Business Simulation & Experiential Exercises, Vol. 21, pp. 70-74. Teach, D.T. and Govahi, G. (1993). The Role of Classroom Techniques in Teaching Management Skills Simulation & Gaming, Vol. 24(4), pp. 429-445. Table of Contents Volume 22, 1995 Simulation Performance, Learning and Struggle Are Good Simulation Performers Consistently Good? Cognitive and Behavioral Consistency in a Computer-Based Marketing-Simulation-Game Environment: An Empirical Investigation of the Decision-Making Process Chalk & Cheese: Executive Short-Course vs. Academic Simulations Revisiting Personality Bias in Total Enterprise Simulations Are Good Strategies Consistently Good? Investigating the Use of a Computer Simulation as an Effective Pedagogical Tool for the Application of a Strategic Model The Problem of determining an Individualized Simulation's Validity as an Assessment Tool A Simulation Based Analysis of the Value of Information in the Hrebiniak Joyce Typology of Adaptation Relative to Porter's Generic Strategies The Impact of Sales and Income Growth on Profitability and Market Measures in Actual and Simulated Industries A Comparison of a Stand Alone Version of a Simulation with the Traditional Competitive Version Computer-Assisted Gaming of International Business Analyzing Simulations with Computer-Based Programs and Applying the Experience to a Real-World Business A Preliminary Investigation of the Use of a Bankruptcy Indicator in a Simulation Environment Graduates' Views on the Use of Computer Simulation Games Versus Cases as Pedagogical Tools An Analytical Advertising Model Approach to the Determination of Market Demand Dealing with the Complexity Paradox in Business Simulation Games A Prototyping Approach for Incorporating Large Data Bases into Media Planning Simulations: An Example Using Magazine Media Through the Looking Glass, Inc: Superior-Subordinate Personality Type and the Leniency effect Evaluating the Effectiveness of Role Playing Simulation and Other Methods in Teaching Managerial Skills Student and Teacher Perceptions of a Management Simulation Course Performance Evaluation: The Effect on the Propensity to Create Budgetary Slack Management Team Formation for Large Scale Simulations Comparative Static Analysis with the Complete PPA Package: A Strategic Market Planning Tool Consistency in Intent: Learning Objectives at the 1994 Intercollegiate Business Policy Competition A New Twist on an Old Game: The Business Strategy Game: A Global Industry Simulation 3ed Evaluation of Performance in Management Simulation: A Management Coefficients Model Building SimuWorlds: Strategic Management Games of the Future Bulls and Bears: A Stock Market Simulation A Systems Thinking Paradigm and Think Computer Simulation Model of Broadcast and Cable Television Industry Competition Jacket Factory The Sales Management Simulation The Marketing Management Simulation A Demonstration of Promodel Demonstrating A New, Cross-Functional Business Simulation: Vision+ A Cost Chain For The Business Strategy Game Simulation Special Session On Experiential Teaching Compensation Dilemmas: An Exercise In Ethical Decision-Making Organizational Storytelling: Telling Tales In The Business Classroom Evaluating Experiential Training: Case Study And Recommendations The Internship Portfolio: An Innovative Tool For Experiential Learning, Critical Thinking, And Communication An Ethnographic Analysis Of The Pedagogical Impact Of Cooperative Communicating Consumer Behavior: A Long-Term Integrated Exercise Using Personal Consumption Journals And Consumer Analysis Papers An Experiential Paradigm For Teaching Business Problem Solving Developing Leadership Skills The Spss® Student Assistant: The Integration of A Statistical Analysis Program Into A Marketing Research Textbook Negotiating With Your Students Using TQM Principles To Transform Accounting Systems Into An Experiential Exercise Enhancing The Effectiveness Of Outdoor-Based Experiential Training Using Virtual Reality Concepts Case Writing In A Developing Country: An Indonesian Example Experiential Learning Using Focus Groups How Real Should Experiential Pedagogy Be? A Viewpoint From Our Students Reengineering The Internship: A New Approach To Experiential Learning Utilizing The Cosmopolitan/Local And Marginal Man Constructs To Measure Students' Propensity For Creativity Developing Experiential Processes For Teaching Quantitative Techniques For Business Team Learning Roles: A Cooperative Learning Technique Creating the Ultimate Small Business Student Experience: Melding Score/Ace with SBI The Development of Trust in Work Teams: The Impact of Touch Some Outcomes of Experiential Learning: How the Cultural Dynamics of Different Countries are reflected in Workplace Norms & Values Incorporation of Job Analysis Results in Various Forms of Selection Interviews Chudesno, Inc.: An Evaluation of an Experiential Training and Development Simulation An Exercise for Exploring the Relationship between Jungian Psychological Types and Organizational Dynamics Partnership: A Radical Approach to Experiential Learning Partnership: A Nice Idea, But How Do I Get Started? Recognizing Discrimination at Work Using Critical Incident Skills Questions to Help Students Become More Successful at Job Interviewing The Video Project Introduction to Psychological Type Theory Come On Down Understanding Facilitation for Development and Continuous Learning: A Micro-Workshop Leadership and Empowerment: An Experiential Exercise in Decision Making Experiential Exercises and Pedagogy Track Workshop: Selecting a Manager for Maquiladora, Inc. Experiential Training In Multi-Cultural Corporate Settings The Role of Facilitation as an Aid to Complete Learning An Experiential Exercise to Illustrate Difference in Information Processing Behaviors and Styles How to Deliver Accessible Survey Results Age Diversity in the Workplace- Family Feud Style Nafta Standoff: A Cross-Cultural negotiating Role-Play Three Strikes and You're Out!: A Downsizing Experiential Exercise