A EVALUATION AND APPLICATION OF AN INSTRUMENT FOR MEASURING PEDAGOGICAL EFFECTIVENESS Developments in Business Simulation & Experiential Exercises, Volume 16, 1989 92 A EVALUATION AND APPLICATION OF AN INSTRUMENT FOR MEASURING PEDAGOGICAL EFFECTIVENESS Philip H. Anderson and Leigh Lawton College of St. Thomas St. Paul, Minnesota INTRODUCTION Research evaluating the learning achieved by students through the use of cases and general management simulations has been categorized into findings based on objective measures such as exams and written assignments and those based on student perceptions. Miles et al (1986) review of research regarding student perceptions of learning concluded that prior research yielded contradictory results. In an effort to overcome the methodological problems they identified, Miles et al (1986) offered suggestions for further research to resolve the contradictory findings on the superiority of cases versus simulations. One of those suggestions involved the use of a common measuring instrument to be able to more directly compare findings of various studies conducted. This paper has two objectives. One objective is to examine what the Miles et al (1986) instrument measures. As a part of their study, Miles et al (1986) presented an instrument they developed to be used for evaluating student perceptions of learning for simulations and cases. If their instrument is to be used in replications, its efficacy should be evaluated. To achieve this, the items in the Miles instrument were factor analyzed to determine the dimensions of perceived learning which occur through the use of management simulations and cases. The second objective is to use the dimensions identified by factor analysis to evaluate the perceived effectiveness of cases versus simulations; i.e., to compare student perceptions of cases and a simulation on these dimensions. RESEARCH METHODOLOGY The Subjects and the Course The subjects were either seniors or MBA students in a required business policy course at a medium-sized, private, midwest college. The data were collected from two undergraduate sections and three graduate sections taught by the same instructor over two semesters. The undergraduates were typical college seniors. The graduate students worked full- time, attending classes in the evening. Two-thirds of the subjects were male (64%), working on their MBA degree (67%). The grading criteria were the same for all sections. The cases and simulation received nearly equal weight and comprised 95% of the final course grade. The cases were the name for the two undergraduate sections and for the three graduate sections of the course. The Simulation The simulation used was Micromatic: A Management Simulation by Scott and Strickland (1985). It is moderately complex, requiring decisions in the areas of production, marketing, finance, and accounting. A total of 12 decision sets (i.e., simulated quarters of operation) were made. Students were evaluated both on their performance in the simulation and on a number of written assignments and examinations associated with the simulation. The simulation constituted 45% of the student’s final grade. The Cases The cases used In the course were selected from the Harvard Intercollegiate Case Clearinghouse. Nine (9) cases were used in each course section surveyed. The cases were typical of business policy course cases, designed to integrate the functions of business. Students were evaluated on classroom discussion of the cases and case analysis papers. Grades associated with the cases constituted 50% of the student’s final grade. The Questionnaire Since a major objective of this study was to evaluate the Miles et al (1986) instrument, the questionnaire they presented was used. The questionnaire was administered the last week of the semester after all graded assignments were completed. All students filled out the questionnaire, and all were usable. A copy of the questionnaire items is included in Appendix A. Students surveyed evaluated each Item with respect to the use of the simulation and cases. RESULTS Because each section was taught by the same instructor, the sections were treated as simultaneous replications. Factor analysis of the data collected shows that the Miles et al (1986) instrument taps three distinct dimensions. Two of these dimensions are directly related to the learning achieved through the use of cases and a business simulation. Table 1 shows these factors and the questionnaire items that made up each factor. The first factor was Skill in Working with a Most instructors use groups, or teams, when working with simulations. Groups are often used for case analysis as well. Groups have been found to be effective for helping students achieve the course goal of integrating the functions of a business enterprise. The students surveyed perceived cases and simulations as affecting their skill in working with a group. Developments in Business Simulation & Experiential Exercises, Volume 16, 1989 93 Understanding of Self was the second factor identified. Beyond the specifics of the course itself, students develop a more complete understanding of themselves. This factor includes issues of confidence and self assessment. A third dimension emerged from the factor analysis. This third dimension, Efficacy of the Pedagogy did not appear to be a dimension of learning. Rather it appears to be an evaluation of the efficacy of the simulation and cases as a learning tool. TABLE 1 DIMENSIONS OF LEARNING ACHIEVED THROUGH THE USE OF CASES AND A BUSINESS SIMULATION FACTOR 1 - SKILL IN WORKING WITH A GROUP a. Increase your effectiveness as a participant in group problem solving (Item 9). b. Motivate people who work with you (Item 15). c. Add to your ability to provide meaningful feedback to team members (Item 14). d. Learn to help people resolve conflicts (Item 16). e. Increase your ability to communicate clearly and effectively with your peers (Item 17). FACTOR 2 - UNDERSTANDING OF SELF a. Become more introspective about yourself (Item 18). b. Gain confidence in your ability to solve practical problems (Item 6). c. Clarify your career interests (Item 12). d. Increase your confidence in your ability to work independently (Item 8). e. Learn new behavior (Item 20). f. Experiment with new behavior (Item 19). FACTOR 3 - EFFICACY OF THE PEDAGOGY a. The simulation (or cases) helped me to better understand the basic principles of the course (Item 21). b. Using simulations (or cases) is a relatively inefficient way to study a subject (Item 28). c. The simulation (or cases) added a lot of realism to the class (Item 25). d. The simulation (or cases) took more time than it was worth (Item 23). e. The simulation (or cases) brought together material I learned in several other business courses (Item 24). In the process of examining the factor analysis, it became clear that scores on one item (Item 26) were inconsistent with snores on other, similar appearing items. After reflecting on the cause of the inconsistency, a decision was made to exclude Item 26 from the analysis. Interpretation of responses to this item is difficult, if not impossible. If a respondent agrees with the statement a pedagogy is “more entertaining than it is educational” does he/she mean that the pedagogy is quite educational and extremely entertaining or that the pedagogy has little entertainment value, but even less educational value? We know only the relationship between entertainment and education, but not the perceived value of the individual pedagogy. Before evaluating the cases versus the simulation on the dimensions identified by factor analysis, a comparison was made to test for differences between the graduate and undergraduate student perceptions of cases and the simulation. Table 2 shows that four of the six comparisons were significantly different at the .05 level or better. For both learning dimensions the direction of the difference indicates that undergraduate students think they learned more regarding the factor involved than did the graduate students, regardless of the pedagogy used. When evaluating the efficacy of the pedagogies, the graduate and undergraduate students had mixed views of cases and the simulation. The direction of the difference in ratings, while not significant, indicated undergraduate students saw the simulation as a more effective learning tool than did the graduate students. On the other hand, graduate students rated cases as significantly more effective than did undergraduate students. TABLE 2 GRADUATE VERSUS UNDERGRADUATE STUDENT PERCEPTION OF SIMULATION VERSUS CASES BY FACTOR Undergrad Grad Signif. Factor Mean Mean Level Working with a Group Simulation 2.69 2.92 NS Cases 2.77 4.00 .000 Understanding of Self Simulation 2.75 3.27 .001 Cases 3.16 3.65 .019 Efficacy of Pedagogy Simulation 1.88 1.93 NS Cases 2.26 2.02 .037 NOTES: 1. The lower the score, the more positive the perception. 2. NS r Non-significant Given the differences shown in Table 2, student perceptions of the effectiveness of the simulation and cases regarding each of the factors identified was analyzed for the undergraduate and graduate student populations separately. For each of the three factors, both graduate and undergraduate students rated the simulation more positively than the cases (See Table 3). Both saw a significant difference in the simulation’s ability to facilitate their understanding of themselves. Graduate students also saw simulations as significantly better for improving group skills. A significant difference between cases and the simulation was not found for the undergraduate students, but the direction of the means favors the simulation. It should be noted that while there are differences between graduate and undergraduate students, the differences are “in degree, NOT in kind.” They agree on which pedagogy they perceive as best. The disagreement is on how good each is. Developments in Business Simulation & Experiential Exercises, Volume 16, 1989 94 TABLE 3 STUDENT PERCEPTIONS OF SIMULATION VERSUS CASES BY TYPE OF STUDENT, BY FACTOR Undergraduate Graduate Sim. Case Signif. Sim. Case Signif. Factor Mean Mean Level Mean Mean Level Group 2.69 2.77 NS 2.92 4.00 .000 Self 2.75 3.16 .002 3.26 3.65 .001 Pedagogy 1.88 2.26 .007 1.93 2.02 NS NOTES: 1. The lower the score, the more positive the perception. 2. NS = Non-significant Since both student groups showed the same pattern of response for rating the pedagogies, the two groups (i.e., graduate and undergraduate student samples) were combined into one group and treated as a single group. Table shows a comparison of combined students’ perceptions of how well the simulation and cases facilitate learning of each of the factors identified. In every case, simulations were seen as significantly better than cases. It is worth noting that, for each factor, the level of significance was not marginal. All were significant at .008 or better. TABLE 4 SIMULATION VERSUS CASES BY FACTOR Simulation Case Signif. Factor Mean Mean Level Working with a Group 2.84 3.58 .000 Understanding of Self 3.09 3.48 .000 Efficacy of Pedagogy 1.91 2.10 .008 NOTE: The lower the score, the more positive the perception. Previous experience with a pedagogy was analyzed to determine its effect on perceptions. Because of the restricted range of prior experience with simulations, analysis methods differed between analyzing the effect of simulation experience and experience with cases. Table 5 shows that where students had previous experience with simulations, their perception of the value of the simulation tended to diminish. However, the only significant difference in changed perception was for learning a better understanding of self. TABLE 5 EFFECT OF PREVIOUS EXPOSURE TO CASES OR SIMULATIONS BY FACTOR Effect of Prior Effect of Prior Exposure to Simulations Exposure to Cases No Prior Prior Corr. Courses Courses Signif. with # of Signif. Factor Mean Mean Level Courses Level Group 2.79 3.02 NS .366 .000 Self 3.02 3.38 .046 .392 .000 Pedagogy 1.88 2.02 NS .187 .067 NOTES: 1. The lower the score, the more positive the perception. 2. NS = Non-significant The same relationship held for prior exposure to cases. The greater the number of previous case courses a student had, the less they perceived learning to occur regarding working with a group, understanding of self, or assessing course process. In each circumstance, this difference was significant. Students were also asked to indicate how well their assigned group functioned. Table 6 shows that for the simulation, the better the group functioned, the more students perceived they learned about how to work with a group and the better they evaluated the simulation’s effectiveness. While not significant, the data suggest that the better the group functioned, the less they learned about understanding themselves. TABLE 6 EFFECT OF HOW WELL GROUP FUNCTIONED ON CASES OR SIMULATIONS BY FACTOR Effect of Group Effect of Group On Simulations On Cases Corr. How Corr. How Well Group Signif. Well Group Signif. Factor Functioned Level Functioned Level Group .236 .014 .086 NS Self -.060 NS -.125 NS Pedagogy .210 .030 .022 NS NOTES: 1. The lower the score, the more positive the perception. 2. NS Non-significant Table 6 also shows that for eases, how well the group functioned had no significant effect on any of the three factors identified. However, while not significant, the direction of the relationships are the same as those for the simulation. DISCUSSION Factor analysis of the questionnaire items identified that three dimensions are measured by the Miles et al (1986) instrument; (1) skill in working wtth a group, (2) understanding of self, and (3) efficacy of the pedagogy. Due to interpretation problems caused by Item 26, it was omitted from the Developments in Business Simulation & Experiential Exercises, Volume 16, 1989 95 instrument. We suggest that this item be dropped from future replications. The students surveyed perceived a general superiority of the simulation over cases as a means to learn more about the factors identified. Regardless of their student status, graduate or undergraduate student, they saw the simulation more positively than the cases. Analysis also indicated that prior exposure to the pedagogy reduced perceived learning about oneself, for both cases and the simulation. Prior exposure to cases also lessened the perception of learning how to work with a group. The perception of how well a group functioned had no significant effect on any of the dimensions regarding cases, but did lessen the perception of learning how to work with a group regarding the simulation. Perhaps of equal interest to what was learned, is what was not learned, at least as far as factor analysis was concerned. There was an undercurrent of “business education" for all the questionnaire items. However, as soon as an item specifically mentioned business, the loading of that item for factor analysis became scattered across all factors. For example, Item 11 addresses learning about yourself as Students perceived the managerial dimension to take the learning beyond Just understanding of self to incorporate skill in working with a group. As a result, items which were business specific did not load on any single one of the factors identified. Nor was a factor on learning business skill; identified. Factor loadings that were specific to these skills were split among the general factors identified. In essence, the students perceived what was learned in a business course as broader than business skills. In this circumstance, the pedagogies involved were used to make personal applications of learning more so than business applications. This may suggest that these pedagogies have a broader reach than Just teaching business skills. They can be used to teach liberal arts "skills". In fact, if we try to teach students about business without any “personal’ application, they are less likely to be internalized by the students. APPENDIX A QUESTIONNAIRE ITEMS 1. Increase your ability to identify problems. 2. Integrate learning from functional areas: (Accounting, Finance, Marketing, etc.) 3. Gain a top management perspective on the operation of a complex organization. 4. Increase your competence for planning business operations. 5. Increase your ability to implement your ideas and plans. 6. Gain confidence in your ability to solve practical problems. 7. Gain new knowledge about operating a business. 8. Increase your confidence in your ability to work independently. 9. Increase your effectiveness as a participant in group problem solving. 10. Learn how to make decisions on the basis of incomplete information. 11. Learn something important about yourself as a manager. 12. Clarify your career interests. 13. Add to your understanding of how to seek and use information for problem solving. 14. Add to your ability to provide meaningful feedback to team members. 15. Motivate people who work with you. 16. Learn to help people resolve conflicts. 17. Increase your ability to communicate clearly and effectively with your peers. 18. Become more introspective about yourself. 19. Experiment with new behavior. 20. Learn new behavior. 21. The simulation (or cases) helped me to better understand the basic principles of the course. 22. Most of the students I know like the simulation (or cases). 23. The simulation (or cases) took more time than it was worth. 24. The simulation (or cases) brought together material I learned in several other business courses. 25. The simulation (or cases) added a lot of realism to the class. 26. Our simulation (or cases) was more entertaining than educational. 27. Performance in the simulation (or cases) is a good way for a student to tell how well s/he is learning the subject matter. 28. Using simulations (or cases) is a relatively inefficient way to study a subject. NOTE: The format for administering Questions 1 through 28 is shown in the Miles et al study (1986). BACKGROUND INFORMATION 1. Excluding this course, have you had any experience with a computer-based simulation in any of your college business courses? __No (Go to Question 2) __Yes . . . In how many courses (excluding this course) have you played a simulation? ______ 2. Excluding this course, have you had any experience with case-study analysts in any of your college business courses? __No (Go to Question 3) __Yes . . . In how many courses (excluding this course) have you done cases? _______ 3. Did you work in a group on the simulation used in my class? No (Go to Question 4) _Yes . . . All things considered, how well did your group function together on the simulation? (Circle appropriate number) Extremely Extremely Poorly Well 1 2 3 4 5 6 7 4. Did you work in a group on the used in my class? __No (Go to Question 5) __Yes . . . All things considered, how well did your group function together on the cases? (Circle appropriate number) Extremely Extremely Poorly Well 1 2 3 4 5 6 7 Developments in Business Simulation & Experiential Exercises, Volume 16, 1989 96 5. What is your academic status? __Undergraduate student Master’s student 6. What is your sex? Male Female 7. Do you have any lull-time work experience of three (3) or more years in any area? No (Go to question 8) Yes . . . In what area(s)? (Check all that apply.) Accounting Management Finance Marketing Social Services Engineering Health Care Other (Specify_) 8. What do you anticipate your career emphasis to be when you graduate? (Same options as in Question 7) REFERENCES [1] Anderson, P.H. and R.H. Woodhouse (1984) the Perceived Relationship Between Pedagogies and Attaining Objectives in the Business Policy Course," in D.M. Currie and J.W. Gentry (eds.) Proceedings of the Eleventh Annual Conference of the Association for Business Simulation and Experiential Learning. [2] Greenlaw, P.S. and F.P. Wyman (1973) The Teaching Effectiveness of Games in Collegiate Business Courses." Simulation and Games 4: 259-294. [3] Keys, B. (1976) ‘A Review of Learning Research in Business Gaming," in B. H. Sord (ed.) Proceedings of the Third Annual Conference of the Association for Business Simulation and Experiential Learning. [4] Miles, W.G. Jr., W.D. Biggs, and J.N. Schubert (1986) “Student Perceptions of Skill Acquisition Through Cases and a General Management Simulation: A Comparison. Simulation and Games 1: 7-24. [5] Wolfe, J. (1987) ‘The Teaching Effectiveness of Games in Collegiate Business Courses: A 1973-83 Update.’ Simulation & Games. Table of Contents Volume 16, 1989 Quality Control Circles (QC™s): Towards a Computerized Simulation The Canadian Hospital Executive Simulation System (CHESS) The Impact of Using Group Performance Evaluation as an Experiential Exercise The Impact of Leader and Team Member Characteristics Upon Simulation Performance: A Start-Up Study Planning for Career Success: Is Where you are Going Where you Really Want to Be? The Production Frontier: Modeling Production in the Computerized business Simulation A Study of the Need for Valid Business Game Algorithms Modeling the Human Component of Business Simulations A Stimulating Simulation in International Business Business Ethics, Experiential Exercises and Simulation Games Collective Bargaining Simulation: Adding Reality Through Point Scoring The Use of Experiential Teaching Techniques: Creativity vs. Conformity Visualization and Guided Imagery in the Organization Behavior Class: An Experiential Exploratory Approach Arranging an Agenda: An Activity on Running Better Meetings Harried Harry: An Experiential Capstone for Students of Organizational Behavior Coping with Stress: An Experiential Exercise Fairness in the Classroom: An Empirical Extension of the Notion of Organizational Justice A Study of the Relationship Between Student Final Exam Performance and Simulation Game Participation Competency Based Development: A Management Development Exercise Simulation Performance Revisited: The Fit Between Instructor Style and Learning Style An Evaluation and Application of an Instrument for Measuring Pedagogical Effectiveness A Knowledge Based System to Support Reasoning by Analogy for Business Simulation Gaming using Forecasting Accuracy as a Measure of Success in Business Simulations The Development of Algorithmic Functional Business Games Strategy Design, Process and Implementation in an Unstable/Complex Environment: A Second Exploratory Study Simulation Integration Contrasts Between MBAs and Undergraduates in the Capstone Policy Course An Investigation of the Real World Usefulness of a Strategy and Policy Course Using a Business Simulation Framework Duel (sic) Views of Internships, as Experiential Learning The Impact of Decision Support Systems on the Effectiveness of Small Group Decisions - An Exploratory Study An Investigation of the Relationship Between Formal Planning and Simulation Team Performance Under Changing Environmental conditions Sensitivity Analysis with the Complete IFPS/Personal Student Analysis Package: A marketing Decision Support System A New Approach to Teaching Salesmanship using Persona, Microskills, and a Sales Process A Rational Case for Synthetic Experience as a Prime Ingredient in the Marketing Curriculum SalesHire: A Microcomputer-Based Salesperson Selection Exercise TRANSECON: An Interactive Program for Learning Transportation Economics Hypercard as a Construction Tool for Short Instructional Exercises A Game to Introduce Accounting Information Systems Students to Certain Internal Control Concepts "Commitments" - A Demonstration Proposal An Analysis of Popular Games as Experiential Models for Corporate and Collegiate Management Education An Exploratory Study of the Effects of Strategic Emphasis in Management Games on Attitudes, Interest, and learning in the Business Policy Course Predicting Individual Decision Making Performance in a Business Simulation: An Empirical Study Strategic Planning And Organizational Performance In A Business Simulation: An Empirical Study, PAM (Planning Action Management) Simulation of a District Sales Territory Lifelong Learning and ABSEL: An Inquiry on Definitions and Relationships A Review of Salient Trends in Proceedings: A Fifteen year (1974 - 1988) Review of ABSEL Contributorship