A COMPARISON OF TWO APPROACHES TO MANAGEMENT SKILL-BUILDING IN AN ORGANIZATIONAL BEHAVIOR COURSE: A REPLICATION Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 11 A COMPARISON OF TWO APPROACHES TO MANAGEMENT SKILL-BUILDING IN AN ORGANIZATIONAL BEHAVIOR COURSE: A REPLICATION Glenn M. McEvoy, Utah State University ABSTRACT This study compared experiential exercise and behavior modeling approaches to teaching management skills using a sample of 69 undergraduate business students. The behavior modeling approach improved behavioral management skills more than the experiential exercise approach on one of two skills learning measures, providing a partial. replication of an earlier study by McEvoy and Cragun (1936-87) differences were found in student reactions to the two approaches. INTRODUCTION There is a growing recognition of the importance--some say imperative -of teaching interpersonal and behavioral skills to business students along with traditional cognitive skill enhancement. Several symposia at recent Academy of Management meetings have addressed this topic (Schoenfeldt, Powers, Whetten, & Albanese, 1987; Zoffer, Porter, Blood. & Steers, 1986). Lyman Porter an invited address at last years western Academy of Management meeting (Porter, 1927) reported the results of a survey in which he found that 68% of vice presidents and CEOs complained about the lack of emphasis on behavioral skill building in business schools. Four years earlier, Porter predicted that "teaching managerial competencies L5 quite likely to be one or major issues--perhaps major issue--facing business/ management schools ir. the last half of :he 1980s” (1983, p. 8). While Porter pleas for more attention to behavioral skill building nave not been totally ignored, there are to dare few published attempts at improving business student management skills. A recently published text is available for instructors with interest in skill building (Whetten & Cameron, 1984), but only; anecdotal reports exist of its effectiveness. The Outcome Measurement Project of the American Assembly of Collegiate Schools of Business (1984) has concerned itself with the measurement of management skills using assessment center technology, but has not as vet reported on the comparative effects teaching techniques designed to improve student skills. One exception :o this dearth of research is a recently published report by McEvoy and Cragun (1986-87). In a study of 182 undergraduate business students, they round that a skills training approach utilizing behavioral modeling training (BMT) was more effective than one utilizing experiential exercises (EE) an organizational behavior course when the criterion was either: a) a specially designed behavioral skills test involving role playing of management incidents or, b) a final exam measuring cognitive learning. Surprisingly they also found that students were generally less satisfied with the BMT approach to learning compared to the EE approach. This result is at variance with those from industrial applications of BMT where reports of satisfaction are quite high (Burke Day, 1986), The experimental and comparison groups two sections of the same course) hi McEvoy and Cragun's (1986-87) study did not differ on a cognitive pretest or on most demographic variables. Further, they established an acceptable level of interrater reliability for their behavioral skills test (r = 0.81), suggesting that certain key management skills (interpersonal communication, decision making, conflict resolution, delegation. performance feedback, employee motivation could be measured reliably in a role play setting. Despite the fact that the experimental and comparison groups used by McEvoy and Cragun (1986-87) did not differ on most demographic variables, it is still impossible to rule out alternative explanations for the results achieved without subject randomization. Technically, their study was a "quasi- experiment" using a nonequivalent control group design (Cook & Campbell. 1979). Further, this design falls under the general category of “between-subjects: designs because measures were taken once on each subject and comparisons were made between scores obtained b. different students. The importance of the McEvoy and Cragun (1986-87) study is that it begins the process of isolating effective means of building student behavioral management skills. The purpose of the present paper is to report a replication attempt of their stud; using a different sample, different methodology, and some additional measures. The research reported below rests the same hypotheses as McEvoy and Cragun (1986-87). Specifically, the research hypotheses are that: a) student behavioral skills are improved more using a BMT approach to learning compared to an EE approach, and b) students respond more favorably to EE learning than to BMT learning. METHOD This research uses a "within-subjects design. Such a design has the advantage that subjects “serve as their own controls” (Rosenthal & Rosnow, 1984), thus increasing the power of small sample designs and reducing concerns about the “nonequivalency” of the control group. Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 12 All 69 students in one section of an organizational behavior course were exposed to six management skill areas as part of the course content Three skills were taught using a behavior modeling training approach (Decker & Nathan, 1985) and three were taught using an experiential exercise approach (Kolb, Rubin, & McIntyre, 1971). Student groups developed presentations for both approaches. In BMT sessions, instructor-provided learning points were demonstrated in either a prepackaged video or through live modeling in front of the class All students then practiced the learning points in role play settings in smaller groups and got feedback from peers in terms of their ability to demonstrate the appropriate behaviors. In EE sessions, student groups drew exercises Out of resource books placed on library reserve and modified the exercises as needed for a particular topic. Selected exercises included such standards as “Lose on the Moon,’ ‘Win as Much as You Can, and so forth. Pretest, demographic, satisfaction, and other data were collected over the course of the quarter as with the original study. At the end of the course, peer evaluations were made and a behavioral skills test was given. Dependent Variables The behavioral skills test utilized as one measure of student behavioral learning was identical to the one reported by McEvoy and Cragun (1986-87). Briefly, students performed five-minute role-plays in response to stimulus scenarios depicting management situations. A faculty member who was not the course instructor was the other role player. The student’s performance was evaluated independently (i.e., without discussion) by both the faculty member and a graduate assistant using check list An example of the score sheet in one skill are is provided in Figure 1. Students engaged in two randomly; selected role-plays, one in a BMT-taught skill area and one in an EE-taught skill area. Neither later was aware of which skills were taught with which approach and were therefore blind to the experimental. treatment, Ten points were possible on each role play. scores from the two role plays were summed, as were the graduate assistant’s scores. The correlation between these two sets of scores was - 0.70 suggesting adequate interrater reliability Each students final score on the behavioral skills test was determined by averaging the faculty and assistant ratings. Peer evaluations have also been shown to be reliable and valid measures of managerial skills (Latham & Wexley, 1981; McEvoy & Cragun, 1986-87. Wexley & Baldwin, 1986). Therefore.. students who worked closely with each other in group projects during the course were asked to evaluate the behavioral skills of their peers at the end of the quarter. Ratings were gathered on each of the following dimensions: problem solving, conflict management, motivation. delegation, performance feedback, and an overall behavioral skills rating. Ratings were made on a five- point scale and on average each student was evaluated by 4.7 peers. Peer ratings from the EE-taught skill areas (problem solving, conflict) were averaged and compared to the peer ratings given in BMT-taught skin areas (motivation, delegation, performance feedback to provide a second assessment of the differential learning effects of the approaches. Finally, students’ reactions to BMT and EE learning activities were gathered by asking students to evaluate the day’s learning on a five-point scale at the end of each class period in which one of the skills areas was the focus of class. (Note that most classes were devoted to lecture rather than to skill-building exercises.) Ratings were averaged across all BMT classes for comparison with the EE average. Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 13 RESULTS Descriptive statistics for the variables in the study are reported in Table 1. It should be noted that the overall behavioral skills test score reported here (mean = 12.1) is slightly higher than the 11.0 average reported by McEvoy and Cragun (1986) suggesting more behavioral learning greater scoring leniency, or more skillful incoming students in the present study (or some combination of these). Comparison of demographic data, cognitive pretests, and cognitive final exam scores of this group and the McEvoy and Cragun group suggested that few differences in student samples were present. Therefore, the difference in behavioral skills test score is likely due to either more behavioral learning or rater leniency rather than to student differences. It can be seen in Table I that the scores on the BMT-taught skills areas were indeed higher than the EE-taught skills as hypothesized, but this difference was not statistically significant. Specifically a paired-samples t-test found 1.25, p 0.11 (one-tailed), suggesting, that the difference may have occurred by chance. Peer ratings for BMT-taught skills were slightly higher than the ratings for EE-taught skills. In this case, the hypothesis was supported because the difference was statistically significant (p < 0.05). Overall then, the results with both the skills test and the peer ratings provide mixed support for the hypothesis that BMT enhances management skill learning more than EE approaches. Results of student evaluations of class periods were opposite the direction hypothesized. However, as can be seen in Table 1, this difference was not statistically significant (p = 0.17). Intercorrelations of skills learning variables are presented in Table 2. Overall peer ratings were related to overall skills test scores (r - 0.30, 0.01) suggesting at least some overlap in these measures. That is. the skills exhibited ac the test site in two brief role plays were also observed by peers over the course of the quarter. Table 2 also shows fairly high intercorrelations between skills test scores for BMT- and EE-taught skills (r 0.64) and between peer ratings for BMT and EE taught skills (r 0.91), suggesting little differentiation in these areas. DISCUSSION This replication attempt provides limited support for McEvoy and Cragun’s (1986-87) original study. Only one of the two outcome variables found the predicted greater skills learning for the BMT approach compared to the EE approach. Further, there was no difference in student preferences for the two approaches. However, the direction of the differences was as predicted for the skills test dependent variable. This suggests the possibility that a lack of statistical power may have accounted for the failure to find a significant difference. This study used a sample of 69 compared to 182 in the original study by McEvoy and Cragun (1986-87). Post hoc power analysis revealed that for a sample of 69 and an expected medium effect size, the power of an alpha - 0.05 t-test is 0.82 (Rosenthal & Rosnow, 1984). Thus, power was adequate but not outstanding, with almost one chance in five of failing to find a true experimental effect if it were present. There were also differences between this study and the original one that may account for the disparity in outcomes. As McEvoy and Cragun described their BMT class section, students were quite heavily involved in behavior modeling even to the point of developing their own modelling displays (videotapes) for presentation in class. As noted earlier, in this study students relied on either prepackaged modeling displays or live modeling in front of class. Thus, the intensity of the BMT experience may not have been as great for the replication sample, and this may have transferred over into the behavioral test scores. However, since the test scores were on average higher for this sample as a whole than for the original sample, this possibility is suspect. Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 14 Interestingly. the average skills test score for the experimental group in McEvoy and Cragun (1986-87) was 12.26, very close to the overall skills test score in this sample (12.13). This suggests the possibility that exposure to only a few BMT-taught skills is sufficient to raise skills test scores the same amount as using a BMT-approach throughout the course for all skills areas. This study found no differences in student preferences for BMT versus EE approaches, a finding at odds with McEvoy and Cragun (1986-87). The difference here may be due to different measures. In the original study, anonymous course evaluation outcomes were compared and the EE class was more positive than the BMT class. In this study, a different measurement approach had to he used because all students were in the same class. The project evaluations that students turned in were done numerous times over the quarter and were not anonymous. Lack of anonymity may have created a different environment for raters, with differing results. Alternately, it could be that students react favorably to some BMT learning but not to an entire Quarter of it. In fact, anecdotal evidence from course evaluations reported by McEvoy and Cragun support this interpretation. Obviously, more research is needed before guidance can be provided to business schools and management instructors in terms of the most effective approaches for building student behavioral skills. Besides experiential exercises and behavior modeling training, numerous other possible instructional approaches exist: case study, lecture computer interactive video, independent study, role playing without modeling, and so forth. Each of these deserve attention in future studies. Further, the short-term nature of the present study is an obvious shortcoming. Future research should attempt to measure learning effects over longer time spans. Lastly, other measures of skills learning should be considered. Other raters could he used as well as other formats such as written management incidents or well- designed multiple-choice learning tests. REFERENCES American Assembly of Collegiate Schools of Business. (1984, December). Outcome measurement project of the Accreditation Research Committee (Phase II: An Interim Report). Burke, M.J. & Day, R.R. (1986). A cumulative study of the effectiveness of management training. Journal of Applied Psychology, 71, 232-245, Cook, T.D., & Campbell, D.T. (1979). Quasi- experimentation: Design & analysis issues for field settings, Boston: Houghton Mifflin. Decker, P.J., & Nathan, BR. (1985). Behavior modeling training. New York: Praeger. Kolb, D.A., Rubin, I.M. & McIntyre, J.M. (1971). Organizational Psychology: An experiential approach Englewood Cliffs, NJ: Prentice-Hall. Latham, G.P., & Wexley, K.N. (1981). Increasing productivity through performance appraisal. Reading, MA: Addison-Wesley. McEvoy, G.M., & Cragun, JR. (1986-87). Management skill-building in an organizational behavior course. The Organizational Behavior Teaching Review, 11, 60-73. Porter, LW. (L983). Teaching managerial competencies: An overview. Organizational Behavior Teaching Journal, 8 (2), 8-9. Porter, LW. (1987), Invited address at the 28th Annual Meeting of the Western Academy of Management. University City, CA, April 9-11. Rosenthal, R. & Rosnow, R.L. (1984). Essential of behavioral research, NY: McGraw-Hill. Schoenfeldt L.F. Powers, E.A., Whetten, D.A., & Albanese, R. (1987). Education for managerial competencies: Approaches, new developments, and issues. Symposium chaired by R. Albanese at the 47th Annual Meeting of the Academy of Management New Orleans, August 9-12. Wexley, K.N. & Baldwin, T.T. (1986). Posttraining strategies for facilitating positive transfer: An empirical exploration. Academy of Management Journal, 29, 503-520. Whetten, D.A. & Cameron, K.S. (1984). Developing Management skills. Glenview, IL: Scott, Foresman and Company. Zoffer, H.J., Porter, L.W., Blood, M.R., & Steers, R.M. (1986). Assessing learning outcomes and education for managerial competencies in business schools: Current issues and challenges. Symposium chaired by R. Albanese at the 46th Annual Meeting of the Academy of Management, Chicago, August 13-16. Table of Contents Volume 15, 1988 The Role of Experiential Knowledge and Human Information Processing in Decision Making A Semantic Differential Instrument to Evaluate Experiential Teaching Methods A Comparison of Two Approaches to Management Skill-Building in an Organizational Behavior Course: A Replication Integrating Simulations: A Model for Business Policy Success Capstone Renaissance = Simulation + Interaction + DSS A Hybrid Method of Executing a Management Simulation: Combining the Best of Mainframes and Microcomputers Providing an Experiential Dimension to Cost/Managerial Accounting Courses Utilization of Computerized Tax Research Services in the Tax Research Curriculum Using and Expert System Based Decision Aid in Accounting Information Systems Event-Extended Entity-Relationship Diagrams for Understanding Simulation Model Structure and Function Multiple Objectives in the Development of the Gordon Macro Game A Comparative Study of Strategic Performance Factors in Actual and Simulated Business Environments An Empirical Investigation of Integrated Spatial-Proximity MCDM-Behavioral Problem Solving Technology Group Decision models Computer Simulation of Human Interaction The Role of Experiential Learning and Simulation in Teaching Management Skills Expert Systems - The New Business Simulation Tool Integrating Prolog into and Undergraduate Logistics Course Simulating Material Requirements Planning on Lotus 1-2-3 Innovation in Management Education: The Impact of the AACSB Experiential Learning in the International Environment Educational Testing with the Microcomputer A Simulation of Investment Analysis, Portfolio Management and Reporting Using Lotus 1-2-3 The Use of an Expert System to Develop Strategic Scenarios Two Exercises for Teaching about Motivation Sex Roles and the Good Manager A Form and Process for Nonconfidential Peer Evaluations Simulation and the Recalcitrant Student Employee Rights-Student Rights: A Classroom Exercise Computer Simulated Competition: An Alternative to Team Play Management Simulation The Relationship of Locus of Control and Vividness of Imagination Measures to Simulation Performance Formal Planning, Simulation Team Performance, and Satisfaction: A Replication Experimental Analysis of Magnitude and Source of Students' Inequitable Classroom Perceptions in Three Reward Conditions Strategy Design, Process and Implementation in a Stable/Complex Environment: An Exploratory Study Matching a Strategy Simulation to the Business Policy Literature: A Black Box Approach to Simulation Development An Evolutionary Classroom Experiential and Computer Simulation Model of a Corporate Strategic Planning System Collective Bargaining in the City of Elson: A Public Sector Experience Should Students Play Games in Labor Relations? Applying Cognitive Educational Objectives to Business Management Cases Grading as a Teaching and Feedback Mechanism: Involving Students in the Grading Process Teaching Controversies: A New Approach to Computer-Assisted Instruction and Simulation Packages Simulating Demand in and Independent-Across-Firm Management Game Advertising Response in the Gold and Pray Algorithm: A Critical Assessment A Model for Pricing Decisions in First Period Marketing Simulation Games Jog Your Right Brain: An Exercise for the Classroom and for Research Six Thinking Hats: An Exercise to Combat Confusion and Develop Thinking Skills Communicating in Context: A Simulation for Learning Business Communication A Simulated Consulting Service for the Compete Marketing Simulation Game Action Exams in the Consumer Behavior Class Using Focus Groups to Teach Problem Definition in Basic Marketing Research The Use of Journals in Management Simulations: A Literature Review and an ABSEL Response An Initial Step Towards Developing and Using an Expert System with a Business Simulation Self-Managed Learning: An Experiential Course Design Using the QWL Paradigm A Review of Current Developments in Experiential Learning Bring the Real World into the Classroom Assessing Student Performance on a Business Simulation Experience Minimizing Startup Anxiety: Case Studies of Simulation Experiences A Tale of Two Shepards Or Using Simulation in a Class Without Walls Enhancing Business Simulations Through the Utilization of Experiential Activities Involving Local Community Executives Simulation with Integrated Spreadsheets: The Design and Development of a Conversational Marketing Concepts Decision Game Introducing INMART: An International Marketing Simulation Using a Computer-Based Business Plan Assistant in Conjunction with a Marketing Simulation Game Overview of the ABSEL Guide to Experiential Learning and Simulation Learning