A SEMANTIC DIFFERENTIAL INSTRUMENT TO EVALUATE EXPERIENTIAL TEACHING METHODS Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 5 A SEMANTIC DIFFERENTIAL INSTRUMENT TO EVALUATE EXPERIENTIAL TEACHING METHODS Gary Whitney University of San Diego San Diego, CA 92110 and William Soukup University of San Diego San Diego, CA 92110 ABSTRACT This paper discusses the applicability of semantic differential analysis as a means of gaining insight into the factors that students consider important in comparing various teaching methods. A survey was administered to a total of dive Sections of two undergraduate courses - a junior level management course and a senior level business policy course. The instrument asked students to react to each of three teaching methods (lecture, case discussion, and computerized business simulation) by indicating the appropriateness of each of 21 word pairs to each teaching method. The students perceived some significant differences among the three methods in terms of their predictability, complexity and motivational power. Demographic differences among the students were significantly correlated with some of the different perceptions of the teaching methods. These results strongly suggest that the semantic differential may offer substantial advantages over traditional course evaluation techniques for developing course designs that meet the specific needs of various student groups. INTRODUCTION In an earlier and simpler era, the design o[ business courses was largely a matter of deciding on course content and selecting a text. Alternative teaching techniques were few in number and tended to be course specific. Most courses were lectured. Business Policy courses and a few others relied heavily on the use of cases, and instructors o[ behavioral courses tended to use a limited variety of experiential deuces. The life of the business educator grows ever more complicated. The variety of available techniques has grown enormously, partly in response to the widespread availability of microcomputers, video recording, and teleconferencing devices. Apart from the obvious impact of technology on teaching techniques, instructors and educators have developed a vide range of approaches for presenting information and for increasing the students’ ability to internalize the information by direct involvement in course activities. In the use of computerized business simulations alone, there are a great variety of specific simulations each aimed at specific teaching objectives and/or student capabilities. In a recent survey, Edgar Williams 14J found 24 different computer simulations in use by 111 AACSB accredited colleges and universities in business policy courses alone. In addition to being able to access a large and growing catalog of simulation software, the instructor who elects to include a simulation in his/her course can “fine tune” the simulation to meet specific course objectives by making appropriate decisions regarding such matters as group size, the method used to form groups, the extent and manner in which the simulation will impact student grades, and the assignment of reports, presentations, or other activities that may be required. The trend is clearly toward a greater variety of simulation and related techniques. As Fritzsche [2] points out, many simulations that were originally designed to run on mainframes are now published for microcomputers and far more support software is now available for enhancing the basic simulation, There has been a concomitant increase in special and specific student needs, stemming away from a variety of causes, including: 1. The rapid growth of adult education and other “extension” programs. 2. A large influx of foreign-born students for whom English is a second language and “disadvantaged’ students with backgrounds substantially different from those of their more “traditional’ classmates. 3. A broadened view of the role of the instructor from that of simply providing and interpreting information to emphasizing skill development by the student. 4. Demands by the business community that graduates of business schools demonstrate better communication skills and more practical experience -- or at least familiarity with the problems and conditions faced by business professionals. In short, instructors are now equipped with a more formidable arsenal of teaching techniques than ever before, but at the same time face a more varied and complex student constituency than has been true in the past. Successfully matching teaching technique to student needs depends in part on the ability to accurately determine student perceptions of and reactions to, differing approaches used by the instructor. However, means for gathering information on student reactions that can be systematically and carefully analyzed seems to have lagged far behind developments in teaching techniques. Determining the amount and/or type of learning that has occurred as a result of simulations and other innovative experiential techniques has been far from easy, in part because of difficulties inherent in Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 6 research design and data analysis. (For a discussion of some of these problems, see Cooke III.) Closely related to the question of the effect on learning of different pedagogical techniques, although not identical to it, is the question of student reactions to various learning experiences. The almost universal use of student evaluations attests to the importance that is usually attributed to student evaluations of course design and instructor effectiveness. In addition to their use by college and university administrators, these student evaluations are commonly used as the basis for judging the effectiveness of specific innovative teaching methods. Unfortunately, many course evaluations instruments are not well designed for this purpose---partly because they were designed primarily for administrative con- trot, and partly because inadequate attention has been paid to the question of what is considered important BY STUDENTS. Even the so- called ‘menu’ evaluation systems assume that the instrument designer knows a priori the criteria that students apply in evaluating courses. As a result, attention has been focused on statistical analyses of student scoring of predetermined criteria instead of first determining the issues that the students themselves find important. This paper discusses the use of semantic differential as a means of determining the nature and extent of differences in student reactions to various teaching techniques. By focusing on student-perceived DIFFERENCES among teaching approaches, this technique can identify those dimensions along with the students find meaningful in distinguishing among course designs. The data for this paper were collected as part of an assessment of two undergraduate courses recently undertaken by the authors. These courses have been redesigned to incorporate computerized business simulations (a different simulation in each course,) In both courses, the simulations and related activities, including the submission of several reports by each student management team, consumed considerable class time and accounted for a substantial percentage of each students grade. The purpose of the assessment was to determine student reactions-- collectively and in each course separately--to the simulation as compared to other teaching methods (lecture and case discussion), METHOD The semantic differential questionnaire The two authors brainstormed to create twenty-one word pairs that we believed might capture the important perceptions of students regarding alternate teaching methods, especially experiential methods. Dimensions of particular interest were how realistic and applicable they found the experience. whether the longer time and continuity of experience was important, and how powerful they perceived the experience. The word pairs were randomly ordered on the questionnaire and individual pairs were randomly reversed to minimize response bias. The last page of the questionnaire asked for some demographic information. The instructions to students were; At the top of each of the following pages you will find a term that represents a teaching technique which you have experienced during college. below the technique are pairs of adjectives that could be used to describe that technique. Please put a check mark in. the position on the scale that best describes your reaction to that teaching technique. Work quickly. As soon as you finish one page, go on to the next. The form of the semantic differential was with the specific teaching method of interest at the top of a page followed by 21 word pairs as follows: Administration of Instrument The questionnaire was given to two sections of a senior level Business Policy course and three sections of a junior level Management course during the final exam period. The two courses were not an experimental design. They were targets of opportunity for testing the semantic differential questionnaire. The only requirement for selection was that both used all three methods of teaching. A comparison of the two classes is in Table RESULTS Valid responses were returned from 101 Management students and 65 Policy students. There were 53 percent male and 47 percent females respondents. Forty three percent had zero years of full time work, 30 percent had one or two, and 10 percent had three or four years. Factor- Analysis Factor analysis identifies “what tests or measures belong together--which ones measure virtually the same thing.... It helps the scientist to locate and identify unities or fundamental properties underlying tests and measurements (Kerlinger, 13], p 659). For example. if students respond with a 7 on “cold-hot’ and also responded with a 7 on icy- burning” and 7 on “snow-steam” they would be identified on the same factor and we could conclude these word pairs identified some fundamental property. If people did not perceive these items similarly they would not to show up on the same factor. Preliminary exploration showed that no more than three factors existed within the 21 word pairs. A principal- components factor analysis was performed Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 7 independently on each of the three semantic differentials (lectures, computer simulations, and cases) to discover which word pairs the students consistently found meaningful for describing the teaching techniques. As a general decision rule. word pairs which did not load above .6 on a factor were not considered any further. variance estimate between the two classes on each of the nine possible measures (motivation, complexity, and predictability for each of three teaching methods). Of the nine t tests, 4 were significant at the .05 level. (See table 3.) Therefore all further analysis was done separately for each class. The factors were consistent enough across the different teaching methods and were conceptually meaningful. The first seems to tap an affective response to the stimulation level or motivation of the teaching method, the second taps the perception of complexity while the third taps the predictability of the method. The first two factors appear to be useful measures of student perception across all three teaching methods. The third factor is not as clearly applicable with the random-logical’ word pair falling just under the .60 cutoff point. We decided to leave it in since it was of interest to simulations and, as we discovered, it did measure significant differences across methods. Each factor was combined into a single index by adding the scores and dividing by the number of word pairs in the factor thus creating an interval scale from one to seven. The simulations used in each class were organized, administered, and graded in approximately the same way so it is interesting to note the students’ perceptions of the twp simulations. Students independently but accurately perceived the different levels of complexity in the two simulations. The Boardroom (Management) requires only 8 decisions per quarter and has a moderately simple production function and demand function while Modern Business Decisions (Policy) requires 46 decisions and has a moderately complex production function and demand function The students also perceived the predictability as similar which is no surprise since, although both include some internal randomness, the primary source of uncertainty comes from competitor's actions which is equally unpredictable regardless of the specific computer simulation. Comparison of Management and Policy classes The data was collected from two different classes using different lectures, cases and simulations. Even though the questionnaire is worded in such a way as to solicit perceptions based on all college classes we reasoned that most students would respond to it from the perspective of the class in which they were sitting at the moment. Therefore the first step was to see if cases, lectures, and simulations were perceived differently by the students in each of the two classes. T tests were performed using a pooled In spite of the significant differences in level of complexity, students perceived no difference in the motivating power of the two simulations. If it is true that a simple simulation is just as motivating as a complex one, this is important information for simulation designers. However it could be that the less complex was motivating for students in the lower level (management) class whereas the more complex one was inst right for the higher level (policy) class. Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 8 Comparison of Teaching Methods The purpose of developing this semantic differential was to have a reliable tool to measure the perceived differences in teaching methods especially comparison of supplementary methods such as eases, simulations, experiential. exercises, etc. In particular cases and simulations are often seen as alternative methods toward the same objectives. Lectures are central to most classes and therefore are not generally seen as alternatives but are useful to include as a standard of comparison. Comparison of alternate teaching methods was done separately for each of the two classes, In the Management class, cases were perceived as significantly more motivating than lectures or the simulation. Cases and lectures were both perceived as more predictable than simulations while cases and simulations were both perceived as more complex than the lectures. In the policy class, simulations were perceived as less predictable than lectures while the cases were perceived as more complex than lectures. The last step was to do Pearson correlations with demographic variables. We informally hypothesized that cases and simulations would be more motivating for students with prior work experience. Of the demographic data collected, this is the only one which did not correlate significantly with our variables. Each student was asked to self-evaluate on a five point scale ranging from “very analytical.’ primarily analytical,’ ‘50- 50,’ “primarily intuitive,’ or “very intuitive” Every perception of every teaching method correlated with the analytical end of the scale with five of the nine significant at the .05 level (Table 6). DISCUSSION AND CONCLUSIONS Semantic differential appears to offer significant promise in the task of evaluating the effectiveness of different teaching methods for specific type of students. In particular, the semantic differential enjoys two specific advantages when compared to traditional course evaluations techniques: 1. While traditional instruments typically ask the evaluator to rate each item in terms of its effectiveness or lack thereof, the semantic differential approach can relatively easily explore multiple dimensions of student perceptions. That is, factors such as the motivating power, the complexity, and the ‘predictability” of various teaching methods can be examined separately. 2. Unlike most currently available course evaluation instruments, semantic differential analysis facilitates exploratory assessments to determine the factors that are considered important by the students themselves. This, in turn, enables faculty members to take student demographic variables into account when designing courses for students with different backgrounds, objectives, or competency levels. In the study reported here, undergraduate students in tjo different business courses (one junior level Management course, the other a senior level Policy course) differed significantly from each in their assessments of the motivational effects of lectures, case discussions, and simulations. At the same time, these classes agreed on the predictability of lectures as compared with simulations and the complexity of lectures as compared with case discussions. Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 9 Among the demographic variables considered in this research, work experience, contrary to the authors’ expectations, did not appear to affect student perceptions. The units of course work completed, and their prior exposure to simulations and to case-oriented courses were significant in some of their reactions to the three teaching methods, Of greatest significance among the demographic variables for these students was their “analytical-intuitive: orientation. with students who described themselves as analytically oriented responding more positively than their intuitive peers to both lectures and case discussions but no differently to simulations. The data collected in these surveys were intended primarily to validate and demonstrate the usefulness of the semantic differential in two applications: 1. The course designer who has available two or more experiential techniques (or major variations on a single technique) can use semantic differential analysis to determine students’ reactions to the candidate techniques across a wide range of specific criteria. since the responses are not limited to ‘good’ or “bad,” the survey instrument can probe for evaluation criteria that match more closely the instructor’s objectives for the course. As shown in this paper, factor analysis can then be applied to determine which of the criteria are regarded as significant BY THE STUDENTS. 2. The semantic differential technique can also be very valuable in determining differences in reactions from different student populations (in terms of demographic such as age, prior work experience, career objectives, etc.) a particular experiential teaching method may be viewed very differently by dissimilar groups of students, as was illustrated by the analytic-intuitive dichotomy reported above. APPENDIX We were primarily interested in developing an instrument for measuring the perceptions of students to simulations. Other researchers may be interested in comparing one simulation to another and may only be concerned with the word pairs that factored out on perceptions of simulations. This table shows the results of the factor analysis from simulations only. It includes all respondents from the two classes. Only word pairs loading greater than .6 are shown. Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 10 REFERENCES 1. Cook, E. F. ‘The Dilemma in Evaluating Classroom Innovations’ Developments in Business Simulation and Experiential. Exercises. 13, 110-114. 2. Fritzsche. D. J. ‘The Impact of Microcomputers on Business Educational Simulations” Simulation & Games Vol. 18, No. 2, 1)6-191. 3. Kerlinger, F. N. Foundations of Behavioral Research. New York Holt, Rinehart, and Winston. 1973 4. Williams, E. L. Jr. “Business Simulation in the Policy Course: A Survey of American Assembly of Collegiate Schools of Business’ Developments in Business Simulation and Experiential Exercises. 14, 235-238. Table of Contents Volume 15, 1988 The Role of Experiential Knowledge and Human Information Processing in Decision Making A Semantic Differential Instrument to Evaluate Experiential Teaching Methods A Comparison of Two Approaches to Management Skill-Building in an Organizational Behavior Course: A Replication Integrating Simulations: A Model for Business Policy Success Capstone Renaissance = Simulation + Interaction + DSS A Hybrid Method of Executing a Management Simulation: Combining the Best of Mainframes and Microcomputers Providing an Experiential Dimension to Cost/Managerial Accounting Courses Utilization of Computerized Tax Research Services in the Tax Research Curriculum Using and Expert System Based Decision Aid in Accounting Information Systems Event-Extended Entity-Relationship Diagrams for Understanding Simulation Model Structure and Function Multiple Objectives in the Development of the Gordon Macro Game A Comparative Study of Strategic Performance Factors in Actual and Simulated Business Environments An Empirical Investigation of Integrated Spatial-Proximity MCDM-Behavioral Problem Solving Technology Group Decision models Computer Simulation of Human Interaction The Role of Experiential Learning and Simulation in Teaching Management Skills Expert Systems - The New Business Simulation Tool Integrating Prolog into and Undergraduate Logistics Course Simulating Material Requirements Planning on Lotus 1-2-3 Innovation in Management Education: The Impact of the AACSB Experiential Learning in the International Environment Educational Testing with the Microcomputer A Simulation of Investment Analysis, Portfolio Management and Reporting Using Lotus 1-2-3 The Use of an Expert System to Develop Strategic Scenarios Two Exercises for Teaching about Motivation Sex Roles and the Good Manager A Form and Process for Nonconfidential Peer Evaluations Simulation and the Recalcitrant Student Employee Rights-Student Rights: A Classroom Exercise Computer Simulated Competition: An Alternative to Team Play Management Simulation The Relationship of Locus of Control and Vividness of Imagination Measures to Simulation Performance Formal Planning, Simulation Team Performance, and Satisfaction: A Replication Experimental Analysis of Magnitude and Source of Students' Inequitable Classroom Perceptions in Three Reward Conditions Strategy Design, Process and Implementation in a Stable/Complex Environment: An Exploratory Study Matching a Strategy Simulation to the Business Policy Literature: A Black Box Approach to Simulation Development An Evolutionary Classroom Experiential and Computer Simulation Model of a Corporate Strategic Planning System Collective Bargaining in the City of Elson: A Public Sector Experience Should Students Play Games in Labor Relations? Applying Cognitive Educational Objectives to Business Management Cases Grading as a Teaching and Feedback Mechanism: Involving Students in the Grading Process Teaching Controversies: A New Approach to Computer-Assisted Instruction and Simulation Packages Simulating Demand in and Independent-Across-Firm Management Game Advertising Response in the Gold and Pray Algorithm: A Critical Assessment A Model for Pricing Decisions in First Period Marketing Simulation Games Jog Your Right Brain: An Exercise for the Classroom and for Research Six Thinking Hats: An Exercise to Combat Confusion and Develop Thinking Skills Communicating in Context: A Simulation for Learning Business Communication A Simulated Consulting Service for the Compete Marketing Simulation Game Action Exams in the Consumer Behavior Class Using Focus Groups to Teach Problem Definition in Basic Marketing Research The Use of Journals in Management Simulations: A Literature Review and an ABSEL Response An Initial Step Towards Developing and Using an Expert System with a Business Simulation Self-Managed Learning: An Experiential Course Design Using the QWL Paradigm A Review of Current Developments in Experiential Learning Bring the Real World into the Classroom Assessing Student Performance on a Business Simulation Experience Minimizing Startup Anxiety: Case Studies of Simulation Experiences A Tale of Two Shepards Or Using Simulation in a Class Without Walls Enhancing Business Simulations Through the Utilization of Experiential Activities Involving Local Community Executives Simulation with Integrated Spreadsheets: The Design and Development of a Conversational Marketing Concepts Decision Game Introducing INMART: An International Marketing Simulation Using a Computer-Based Business Plan Assistant in Conjunction with a Marketing Simulation Game Overview of the ABSEL Guide to Experiential Learning and Simulation Learning