AN INSTRUMENT FOR INVESTIGATING THE EFFECTIVENESS OF TEACHING METHODS IN THE BUSINESS POLICY AND STRATEGY FORMULATION COURSE Developments In Business Simulation & Experiential Exercises, Volume 20, 1993 11 AN INSTRUMENT FOR INVESTIGATING THE EFFECTIVENESS OF TEACHING METHODS IN THE BUSINESS POLICY AND STRATEGY FORMULATION COURSE William D. Biggs, Beaver College Wilford G. Miles, University of Hartford James D. Schubert, Alfred University ABSTRACT Since at least 1975 efforts hive been made to determine the perceived relative electiveness of cases and simulations in the business policy course. These efforts have been undercut by a lack of a common instrument to measure student perceptions. This paper reports on the development of such an instrument. The results of the study indicate that the instrument developed in this study provides researchers with scales that are conceptually clear, statistically satisfactory, and that measure variables ct importance to those who conduct research into the pedagogical effectiveness of business policy teaching methods. These scales measure student perceptions of how well cases and simulations teach Leadership and Group Interaction Skills, Business- management Skills, Personal Growth Skills, Mediation and Decision Making Skills, and Learning Effectiveness. INTRODUCTION Ever since the publication of the Gordon and Howell (1959) and Pierson (1959) reports must business schools have required red students to take a policy formulation and administration course, often called business policy, as the Capstone course in their business program. The broad purposes of this course are to provide students an opportunity to develop familiarity with the unique perspective or top management and to provide them an opportunity to develop some proficiency in analyzing top management-type problems. Until the early 1960 s the case method was unchallenged as the dominant teaching method in the business policy course. This dominance was challenged after the development. of the first practical business game by the American Management Association in 1956 (Meier, et. al. 1969. The challenge has continued as the number of general management and functional games available has grown dramatically, as shown in Table 1, all tables are grouped at the end of the paper) and this growth has been reflected in their use in business policy courses (Biggs, 1979, 1967; Faria, 1987; Keys, 1987. . Today both cases and business games are routinely used in the business policy course In the early years of their it development, much of the literature regarding business games was anecdotal This early research tended to suggest that students found business management games to be interesting and they appeared to learn from them. As their use spread, however, a number of writers (Greenlaw and Wyman, 1973; Miner and Miner, 1973; Neuhauser, 1976; Snow, 1976) raised serious questions concerning the effectiveness of business management games. This criticism generated a great deal of research into the educational effectiveness of business games and the relative effectiveness of business games and cases. These studies, which Wolfe and Roberts (1986) have referred to as internal validity studies, emphasized short- term results and typically compared business management games to other pedagogy, usually a combination of cases and lectures, either through the use of examinations or through the measurement of student perceptions about the effectiveness of games and cases. The more rigorous of those studies that used examinations are reviewed by Greenlaw and Wyman, 1973; Keys, 1976; and Wolfe, 1985 and the more rigorous of the perceptual studies are reviewed by Miles, Biggs and Shubert, 1986. While the results of these more rigorous studies generally supported the educational value of business games they did not resolve the questions about the effectiveness of business games and cases because the findings of some studies were inconclusive and the results across studies were often contradictory. The end result is that questions about the relative teaching effectiveness of cases and business games in the business policy course remains unsettled. There are several variables or conditions, which might account for the inconclusive nature of these studies. One such variable is rooted in the methodology used in the studies. In the Miles, Biggs, and Shubert (1986) review of the internal validity studies based on student perceptions (Anderson and Woodhouse, 1984; Roberts and Fields, 1975; Sampson and Sotirious, 1978; Sugges, 1982; Waggener, 1981; Wolfe and Byrne, 1976) it was found that no two studies used the same instrument. This, of course, makes it very difficult to interpret results across studies and reach generalizable conclusions about the relative effectiveness of business games and cases and other teaching methods. In addition, in only one study (Anderson and Woodhouse, 1984) was there any attempt to group the individual items into logical categories or scales. Thus, most studies merely asked a number of questions and indicated that students found one or another of the pedagogies to he superior on some questionnaire items. Developments In Business Simulation & Experiential Exercises, Volume 20, 1993 12 based on the existing literature the authors believe that research into the effectiveness of teaching methods used in business policy courses would be facilitated if a common measuring instrument were used. The primary focus of this article is on the development of an instrument which can be used to measure reliably student perceptions about the effectiveness of the pedagogies used in the policy formulation and administration course. Rather than merely relying on student responses to individual items as past studies have done, this study uses factor analysis to group items into categories. These categories were then analyzed to determine what learning objectives they were measuring. While the relative importance of educational goals of professors of business policy vary there appears to be a set of educational goals which are common to most. A review of business policy textbooks (Byars, 1991; Harvey, 1988; Higgins and Vincze 1989; Hofer, Murray, Charan, and Pitts 1984; Pearce and Robinson, 1992; Thompson and Strickland, 1992) provides the following set of goals: the development of a strategic orientation, an appreciation for top management perspectives, an emphasis on integration of functional departments and activities, the development of strong analytical and decision-making skills, and the ability to deal with varied situations and relationships. An appropriate instrument should address all of these topics. The need, then, is for an instrument with scales representing these key areas so that scale scores can be used to compare the effectiveness or alternative pedagogies or combinations of pedagogies. RESEARCH METHODOLOGY Data were collected from two sections of a business policy course at a small, private university in the eastern United States. The 64 students, all of whom were graduating seniors, were randomly assigned to one of two sections of the course. The two sections met concurrently. There were 32 students in each section and they were randomly assigned to teams that worked together on the simulation and on two case presentations. Students in each section were subject to identical course requirements and grading criteria. The sections met together for lectures and discussions of material on case analysis, strategy, control, decision making, and for an orientation to the business game. The sections met separately for case presentations. Each of the authors responsible for case grades in one of the sections however, both authors read and discussed all written case solutions from both sections in an attempt to ensure consistency of grading between sections. The Simulation The simulation used in this study was Tempomatic IV: A Management Simulation (Scott and Strickland, 1974) This is a moderately complex general management game that requires students to make decisions in all functional areas (Gomolka, Ward, and Parrot, 1982). As part of the course requirements each student team also developed written strategies and wrote and presented year-end reports. The year-end reports summarized the previous year’s performance and outlined operational and strategic changes planned for the following year. For any decisions, which would normally have to be approved by a board of directors, the students were required to obtain written approval from one of the instructors. Simulation performance, which made up 50% of the class grade, was based on 5 variables (sales in dollars, net income, return on sales, return on assets, stock price) and an evaluation of the strategic and year-end reports. The Cases The case portion of the course consisted of 15 cases chosen from those available through the Harvard Intercollegiate Case Clearinghouse. The cases covered both functional and strategic issues. If students were not responsible for presenting the case on a given day they were expected to he prepared to discuss the case. Performance on cases, including presentations and class discussion, made up 50% of the final grade. The Questionnaire During the final examination period each student completed two 28 item questionnaires. The questionnaires were identical except that one referred to cases and one referred to the business game. The questionnaires were constructed by combining questionnaires developed by Byrne (1979) and Chisholm, Krishnakumar, and Clay (1978, 1979). These questionnaires were chosen because they appeared to deal with educational goals that business policy text hook authors indicate are important in the business policy course, and contained questions which asked for evaluations of the teaching methods which were used in the class. At the time the students completed the questionnaires they knew that their grades had already been calculated and posted, but they had not yet seen them. All students from both sections completed the questionnaires and all questionnaires were usable. To be certain there was no ordering effect half of the students completed the case questionnaire first and half completed the business game questionnaire first. Statistical analysis revealed no ordering effect. Data Analysis An earlier study (the authors) presents the questionnaire and reports on the analysis of the individual items, therefore, those findings are not repeated here because the primary focus is on whether the items cluster into factors which represent pedagogical topics that conform to the set of educational goals previously outlined for the business policy course. To determine if Developments In Business Simulation & Experiential Exercises, Volume 20, 1993 13 common dimensions were present the data was factor analyzed using the SPSS "PAI" computer program (Nie, et. al, 1975). The analysis used varimax or orthogonal rotation. Factors were included if eigenvalues were equal to or greater than 1.0 and items were considered to load on a factor if their loadings were equal to or greater than .5. RESULTS In this section we present the results of the factor analysis. As is shown in Table 2 the principal components analysis extracted seven factors with eigenvalues greater than 1.0 and these factors accounted for 65 percent of the total variance. The results of the rotated factor matrix are presented in Table 3. This table shows the factors and their descriptors, and the items, which loaded, on each factor along with the factor loading and brief description of what the item covered. Only five of the 28 items (7, 13, 21, 25, and 28) failed to load on one of the seven factors These 5 items are included in Table 5 on the factor with the highest loading with parentheses around their loading and i tem number. One item, item 12, had factor loadings greater than .5 on than one factor. it is shown on both factors 7) although it appears to he more closely related to the items on factor 6. in addition, all variables except item 13 had commonalities greater than .5. These results approximate a simple structure solution. The first five factors are readily interpretable. Factor I is identifiable as a Leadership and Group Interaction Skills dimension by the exclusive loading correlation of leadership and group behavior questions such as “participation in group problem solving,” "motivating others, and “conflict resolution” on this factor. Factor 2 is defined by the cluster of questions tapping student perceptions of learning business Management Skills. These questions include “gaining top management perspective,” “planning business operations,” “implementing plans,” and “working in dependently.” Factor 3 appears to be a Personal Growth dimension characterized by “introspection,” “learning new behavior,” “and experimenting with new behavior. Integration and decision- making functions jointly identify Factor 4 as an Integration and Decision-Making dimension. The last three factors are evaluative dimensions. Factor 5 is clearly an Affect dimension with a liked -- didn't like or was worthwhile was not worthwhile polarity. Factor is primarily defined by a question evaluating Learning Feed back, although item 12, “clarify career interests" also loads on this factor as it does on Factor 7. Thus, Factor 6 provides an overall evaluation of the learning feedback provided by the pedagogy. Finally, Factor 7 also provides a summary evaluation of the pedagogy as defined by a question regarding the Entertainment versus the Educational Value of the pedagogy, although as we will indicate later there may be reason to discard this factor. Conceptually, Factors 6 and 7 are clearly distinct from the affective dimension of Factor 5, and from each other. The principal components analysis presented in Table 3 reveals a dimensional structure to these data that is conceptually clear and statistically satisfactory. Seven interpretable dimensions were extracted that account for a substantial portion of the total variance in student responses to the questionnaire. In addition, these factors appear to be related to learning criteria which are Important in the business policy course. Given these results the authors believed it was appropriate to compare the student’s perception of the cases and the simulation based on the derived scales. COMPARISON TO ANOTHER STUDY After the current study had been completed but before it was submitted for publication two other researchers (Anderson and Lawton, 1989) published a paper which was based on a study they conducted which was quite similar to the current study. In this section we describe their study and make comparisons of their results to those of the current study. The Anderson and Lawton Study The Anderson and Lawton study and the current study were both resulted from research need identified in a 1986 article by Miles, Biggs, and Shubert and many of the same research methods were used in both studies. These two studies sought to ascertain the validity of the same questionnaire by using factor analysis. There were a number of other similarities between the two studies. First, both used cases from the Harvard Intercollegiate Case Clearinghouse. Second, essentially the same simulation was used although the current study used the mainframe version (Scott and Strickland, 1974) while Anderson and Lawton used the microcomputer version (Scott and Strickland, 1985). Third, the overall grading weights were nearly the same. As noted earlier, in the current study the cases and simulation each accounted for 50% of the grade. Anderson and Lawton report that cases accounted for 50% of the grade, the simulation accounted for 45%. They do not specify the basis for the remaining 5%. There are some differences between the studies. First, the current study used only undergraduate students whereas Anderson and Lawton used three sections of graduate students and two sections of undergraduate students. They do not indicate how many students were involved in total nor do they indicate whether the cases and/or simulation were individual or group activities. Second, the current study involved two instructors (one for each section of the course) during a single semester whereas the Anderson and Lawton study involved one Developments In Business Simulation & Experiential Exercises, Volume 20, 1993 14 instructor over two semesters. Third, the number of cases required of the students differed between the two studies with fifteen being required in the current study versus nine in the Anderson and Lawton study. A Comparison of the Factor Analysis Results A comparison of the factor analytic results of these two studies shows very interesting similarities and equally interesting differences. It we examine the factor structure in the two studies, as presented in Table 4, we see that the first factor in both papers is identical and is interpreted in similar ways. In the current study this factor is interpreted to be a Leadership and Group Skills factor while Anderson and Lawton interpret these same items more narrowly as a Group Skills factor. Further, content analysis of the items shows that the items are similar and seem to he measuring common themes. The second factor in the current study includes 6 items that are primarily related to the development of Business Management Skills. There is no comparable factor in the Anderson and Lawton study. In fact, in the Anderson and Lawton study those items which most specifically relate to business skills wither did not load on any factor or as Anderson and Lawton stated, "...scattered across all factors." This was a surprising finding and is contrary to the findings in the current study which, as Table 2 shows, has these items clustering quite coherently. There is nothing in the Anderson and Lawton paper that helps to explain this unexpected result. Perhaps a more detailed examination of the differences in the cases, how the class was conducted, or differences in the behavior of the instructors would help to explain in these differences, but the available information suggests no explanation. The third factor in the present study and the second factor in the Anderson and Lawton study are not identical, but are very similar. They both identified as having a personal growth dimension and they both include items 18, 19, and 20. In addition, the Anderson and Lawton factor includes two items, and 8, that load on factor two, the Management skills factor, in the present study. Of the six items on factor two in the present study items 6 and 8 have the weakest conceptual connection with the development of business skills and the strongest conceptual connection with personal development. - Both scales represent a logical grouping of the items and constitute a coherent scale structure There is no factor in the current study which resembles the third factor in the Anderson and Lawton study. Factor number three] in the Anderson and Lawton study includes items number 21, 23, 24, 25, and 28 and was interpreted as representing pedagogical efficacy. The factor includes items, which relate to educational efficacy, items 21 and 24, and items that reflect the efficiency of the pedagogies, items 23 and 28, as well as an item that reflects the degree of realism that each method provided, item 25. In the current study Item 23 loaded on factor 5, the Affect factor, item 24 loaded on factor 4, the Integration and Decision Making factor, and items 21, 25, and 28 tailed to load on any factor. Anderson and Lawton found item 26 to be unclear and recommended that it be dropped from the instrument. The authors agree that this item is unclear and should be dropped. In fact, in the current study it loaded with only one other item (item 12) and that item was the only item that loaded on two factors and as previously mentioned it is conceptually more closely related to the other factor. There are a few other comments that can be made about the factor analytic aspects of the two studies. First, Anderson and Lawton did not report commonalities, percent variation explained by each item, or the cumulative percent of variation explained by all factors. This information might have made it possible to make a more precise determination of the relative value of each factor structure. Second, in the Anderson and Lawton study 11 of the 28 items failed to load on any factor, compared with only 5 items that failed to load in the present study. Nearly all of the difference in the number of items that failed to load on a factor, four of six, is explained by the failure of the business-related items to load in the Anderson and Lawton study. The failure of these items to load on a common factor is troubling. However, the fact that these items loaded coherently in the current study indicates that these items constitute a viable scale, CONCLUSION The purpose of this study was to develop an instrument that could be used to conduct research into student perceptions of the value of differing pedagogical methods in the business policy course. The results of this study, when viewed in conjunction with the Anderson and Lawton study, gives reason to believe that this instrument, or a modification of it, can serve this purpose. These two studies have provided several scales, which are coherent and are clearly related to the educational goals stated by those who teach the business policy course. Specifically, factors one to four in the current study and factors one to three in the Anderson and Lawton study are clearly related to these goals. Factors five through seven in the current study may also prove useful it information about student likes and dislikes or the perceived efficiency of feedback is needed. We encourage other researchers to use this instrument in order to help establish its usefulness REFERENCES References will be furnished upon request to the first author. Developments In Business Simulation & Experiential Exercises, Volume 20, 1993 15 Table of Contents Volume 20, 1993 Dominant Personality Types and Total Enterprise Simulation Performance Shelf Wars: A Grocery Channel Simulation Shared Cultural Perspectives: An Experiential Exercise Utilizing International Students to Globalize the Classroom An Instrument for Investigating the Effectiveness of Teaching Methods in the Business Policy and Strategy Formulation Course Providing Better Trained Graduates for Accounting Employers The Ambition Gradient Approach to Evaluation of Computer Simulation Game Team Performance Alphatec: A Negotiation Exercise with Logrolling and Bridging Potential Using the Ideafisher Idea Generation System as a Decision Support System in Marketing Strategy Courses A Dynamic Market Share Allocation Model For Computerized Business Simulations Multi-Cultural Adaptability Using Experiential Learning in a Graduate Course Development of Experiential Applications in HRM: Practicing What Preach and Preaching for Practice Linking Students and Business Leaders Through Portfolios Debriefing International Experiential Learning Exercises: Road Signs for Effectiveness Sales Manager: A Simulation Modeling Interactive Effects in Mathematical Functions for Business Simulations: A Critique of Goosen's Interpolation Reducing the Complexity of Interactive Variable Modeling in Business Simulations Through Interpolation Antecedent Biases of Experiential Learners: Trainee Occupation and Subgroup Diversity Pax in Terra Sancta: Simulating the Middle East Peace Negotiations A Multiple Regression Case In Experiential Learning Changes in Ethnocentric/Geocentric Orientation by Business Students after Exposure to a One Summer Course in International Marketing's A Systematic Approach to the Development and Evaluation of Experiential Exercises Entrepreneurs Evaluate Experiential Education A Linear Programming Approach to Open System Total Enterprise Simulations Reflecting Leader Behavior from the Looking Glass, Inc. Simulation Linking Cognitive Styles, Teaching Methods, Educational Objectives and Assessment: A Decision Tree Approach Restructuring Management Education in Post-Communist Countries: How Western Experts Can Help Managerial and Cultural Pre-Conditions for Superior Performance in a Global Setting: An Experimental Study with the Aid of Business Games Multiple Industries in Computerized Business Gaming Simulations Content or Process? - Content and Process! Some Observations and Reflections About Management Education in Central Europe Out-of-Class Experiences to Promote Volunteerism Enacting the Linguistic Consciousness of the Modern Managerial Mind: Post-Modernism and Experiential Learning Intergrating Experiential Exercises into the College Curriculum: The Case of Internationalizing the Business Curriculum Simulation Marketing Oversights Incorporating Advertising Creative Strategy into Computer-Based Business Simulations The Dynamics of a Partnership Between Business and Education Collaborative Education Done Globally Experiential Systems Analysis CADPLAN: A Simulation for Comparative Advertising A Doctoral Symposium: Preparing Students for Conference Behavior Comparing the Simulation with the Case Approach: Again! Total Quality Management: A Model for Continuous Quality Improvement The Quality Audit: An Experiential Exercise for Business Students Extending the Reach of Simulations: DECIDE Heads for the Inner City Lessons Learned from a Customized Management Development Simulation The Foreign Exchange Spot Trading Simulation Using Lotus 1-2-3 to complete a Triple Play in a Simulated Competition International Business Education: Is Enough Being Done? Matching of Student-Teacher Cognitive Style as a Factor in Student Success in an Introduction to Information Systems Course Breathing (More) Life into the Case Approach Lord of the Flies: A Live Case Approach to Leadership Cooperative Case Studies: Experiential Tools for Teaching Business Problem Solving Tools Strategy Simulations in Context: An Evaluation of Key Dimensions The Distribution Channel Game Evaluation of a Simulation Game as an Education Tool for Utility Professionals The Relationship Between Total Enterprise Simulation Performance and Learning Total Quality Management does not Happen by Magic, but it can be Taught Using a Pedagogical Methodology that Utilizes Magic Effectively Preparing Students for Careers in a Global Environment by Integrating Total Quality Management Thoughout the Business Curriculum An Empirical Investigation of Cognitive and Performance Consistency in a Marketing Simulation Game Environment Using MARSGAP with LAPTOP: (A Marketing Simulation Game Analysis Program) with LAPTOP: A Marketing Simulation Adapting TQM Implementation to Organizational Level An MBA Business Simulation: Executive Interaction Experiential Exercises and Pedagogy Track Workshop: Experiencing Cultural Diversity in the Classroom (and the Hotel Meeting Room) Closing the Gap between Corporate and National Culture The Dynamic Manufacturing Company The Use of Experiential Techniques in Corporate Training The State of Simulation Gaming in Easter European Countries- Principally Russia An Experiential Exercise in Cross-Cultural Training Valuing Differences: A Conceptual Framework Demonstration of an Experiential Exercise Effectively Using Experiential Learning to Impart TQM Concepts in a High Technology Environment The Older Worker Questionnaire: An Exercise Concerning Older Worker Stereotypes and Behaviors The Crime Fighting Task Force: An Exercise in Organizational Politics Welcome to the Party! An Expression of Vocational Preference Experiential Exercise for Imparting Cross-Cultural Appreciation Six Swift Simulations on Globalization Overview of BASF Delegate Program