EXPERIENTIAL EXERCISES OR COMPUTER SIMULATIONS? HOMOGENEOUS, HETEROGENEOUS, SUPPLEMENTARY, COMPLEMENTARY OR NONE OF THE ABOVE? Developments in Business Simulation and Experiential Learning, Volume 25, 1998 EXPERIENTIAL EXERCISES OR COMPUTER SIMULATIONS? HOMOGENEOUS, HETEROGENEOUS, SUPPLEMENTARY, COMPLEMENTARY OR NONE OF THE ABOVE? Gayle Baugh, The University of West Florida E. Brian Peach, The University of West Florida Walter J. Wheatley, The University of West Florida ABSTRACT In an earlier time and place the educators limited choice of pedagogical tools consisted primarily of a text, a piece of chalk, and a slateboard. Today, the pedagogical choices are practically limitless. Computer ancillaries, interactive exercises, data prone cases, audio/visual accessories, and unlimited access to the world wide web are but a few selections from today’s pedagogical menu. Staying current with the various methodologies with which to educate consumes almost as much of the educators time and effort as staying current in his or her respective fields of teaching expertise. It is obvious that no one paper is going to study the relationships, aspects and efficacies of all of the pedagogical tools currently available. Thus, the authors of this paper chose to study the relationship of two commonly used pedagogies with which they have had the most experience: experiential exercises and computer driven business policy simulations. The paper’s findings reflect the opinions of the most important variable in the education equation - the student which the authors trust will be of benefit to other educators in their choice of pedagogical tools. INTRODUCTION The rise of experience-based learning theory has significantly influenced business school pedagogy, leading to a wide variety of experiential learning devices. Experiential learning requires learners to actively participate in personally meaningful activity allowing practical application of theory and knowledge (Gentry, 1990). Participants should feel a personal sense of commitment to the success of the exercise (Walter & Marks, 1981). In a review of the literature, Bowen (1987) concluded learning is enhanced by a) an optimal level of emotional arousal, b) an environment perceived as safe, and c) sufficient time for the participant to process the learning. Experiential learning that is epigrammatic in nature, lasts a relatively short time, and is not computer based is generally considered as experiential exercises (Keys & Wolfe, 1990). Although technically a subset of experiential learning, business simulations are typically considered separately. Simulations use simplified environments with a sufficient illusion of reality to invoke attitudes and responses comparable to real world situations. Simulations may be computer based or non-computer based, and may be functional in that they concentrate on a single part of a business (Biggs, 1987) or total enterprise in that they require the strategic integration of decision variables from all major components of an organization (Keys, 1987). Due to the complexity and duration of many computer based simulations, it is often difficult to accurately determine what participants are responding to and how they are learning. In addition, the actual value of learning from experiential exercises and simulations has been challenged. Wheatley, Hornaday & Hunt (1988) concluded that the literature indicates students enjoyed business games, and that simulations were effective for teaching complex concepts but were otherwise irrelevant to student classroom and career success. In a significant review of 60 studies, Keys and Wolfe (1990) concluded there is a “general yet 236 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 problematic educational efficacy” (p. 311) from business simulations, and that they are “generally effective and. possess internal validity” (p 316). Within ABSEL, the debate on learning from simulations continues unabated, and Anderson & Lawton (1997c) identified four “camp? whose research streams have produced conflicting conclusions. To address this problem, they (Anderson & Lawton, 1997b) proposed a series of suggestions to guide future research on simulation efficacy. The rising popularity of simulations has given rise to a number of studies to determine if simulations were superior to other forms of teaching. Studies by Estes & Smith (1979), Kaufman (1976), McKenney (1962, 1963), Raia (1966), and Wolfe & Guth (1975) found simulations superior to the case method on a variety of performance measures. Evaluating experiential learning has a checkered history and has many problems (Anderson & Lawton, 1997c; Burns, Gentry & Wolfe, 1990). This study accepts that there is sufficient research evidence to conclude that experiential learning does have some validity, and instead investigates the question as to whether simulations are superior to other experiential approaches. Specifically, this study addresses the relationship of students’ perceptions of a highly comprehensive computer-based simulation versus highly dynamic experiential exercises. METHODS Participants Participants in this study were 66 students enrolled in a computer simulation driven Business Policy course who had just recently completed a Principles of Management course that emphasized experiential exercises. All participants were junior or senior level students. The classes were reflective of the general student body at the university. Participants were a little older than traditional college age (university average age was 29), and almost all had some full- or part- time work experience. The class was about equally divided with respect to gender. All students were business majors, and most were full-time students holding part-time jobs concurrently. Procedure The Team Teaching Methodology Diagnostic Questionnaires (TTMDQ) were distributed to the students in class by the instructor of that class. students were provided class time to respond. Participation was voluntary, but no one refused to participate. Instrument The TFMDQ consists of a set of items constructed to assess perceptions of student learning via experiential methods as compared to computer simulations. The items were derived in part from previous research comparing various computer- based simulations (Biggs, Miles, & Schubert, 1990), and previous research studying ways to enhance student team effectiveness (Wheatley & Armstrong, 1997). All of the questions are included in Table 1 along with their respective descriptive statistics. RESULTS AND DISCUSSION Descriptive Statistics Basac descriptive statistics on all of the items were first generated (see Table 1). The values of the descriptive statistics are relatively consistent even though many of the questions were written to yield reverse responses in order to prevent response bias. The mean response to the items as a set was 4.30 with an average standard deviation for the items as a set of 1.53. The range of mean responses was 2.07 and the range of item standard deviations was 0.39, respectively. 237 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 In reviewing the descriptive statistics, only a few questions yielded mean responses which were significantly different than the responses to the questionnaire as a whole. Responses indicated that students enjoyed the computer-based simulation more than they did the experiential exercises [Item 4; mean = 4.77]. Previous research has also indicated a preference for computer simulation may be due to the fact that the technology in developing experiential exercises is not keeping pace with the computer-based simulation technology. A second possible explanation is the perception of the students. Lacking the “scientific” aura of a computer, they may feel that experiential exercises are nothing more than “games.” The exercises may not offer for them the same sense of closure that they receive when participating in a computer-based simulation, which has a clear beginning and end. In other words, they paid their tuition to learn something about managing businesses and not to engage in activities such as “playing with Legos,” which does not provide the same sense of doing something truly business-related. Perhaps students have not yet learned to respect the difficulty presented by the types of interpersonal issues that corporate executives face. Another finding of note to all educators was that students, as a whole, no longer prefer working by themselves than working in teams [Item 19; mean = 5.28]. Many educators have heard students complain they could have earned a much better grade if they had worked alone. Their tolerance for the group dynamics that must be dealt with while working in a team seems to have increased somewhat over the years. Perhaps as a result of this greater tolerance, teams appear to be pulling together mere than in the past [Item 18; mean = 3.97 and Item 20; mean = 4.82]. Somehow, educators have found a way to convince students that working effectively in teams is important in tenns of career success. A final area of interest is that students reported preferring activity-based learning [Item 17; mean = 3.26] as compared to “standard” lecture courses. This comes as somewhat of a surprise finding because one would suspect that the many years learning in a rote environment would engender some resistance to change. Particularly given how fast the education times, they are a changing (Dylan, Bob, 1965). However, this is good news for those of us that realize the efficacy of new forms of pedagogy’ Exploratory Factor Analysis Next, the questions which compared experiential learning to computer simulation were factor analyzed to determine if there was an underlying dimensionality to the items. The items which specifically compared perceptions of learning using experiential-based and computer-based methods (16 of the 20 items) were subjected to a principle factors analysis with varimax rotation. The number of factors extracted was based on two rules. First, the eagenvalue for that factor had to be greater than 1. Second, a Screen test was used. These criteria resulted in two distinct factors. A question was considered to load on a factor if the item loading was greater than .4 and the difference between the absolute values of loadings on the two factors was greater than .3. Table 2 shows the questions, factor loadings, and eigenvalues. The first factor included six items, and was composed of items that queried the respondent about experiential learning methods as compared to computer-based simulation. Reliability analysis of these six items resulted in a coefficient alpha of .88. The second factor included five items, and was composed of items which queried the respondent about computer-based simulations as compared to experientially-based courses. Reliability analysis of the five items resulted in a coefficient alpha of .74. Five of the 16 items included in the factor analysis showed complex loadings and were not included in further analyses. 238 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 Before scale scores were computed, all of the questions comprising Factor 1 were reverse scored, so that high scores on both Factor 1 and Factor 2 indicated a preference for experiential learning methods over computer-based simulation. Scale scores for both Factor 1 and Factor 2 were then computed using unit weighting for all questions comprising the factor. A paired 1-test was run between the scale scores. The results of this analysis indicated significant differences between the scale scores [t(65) = 4.85, p < .001]. This difference was found despite the fact that all of the items loading on Factor 1 were recoded, so that high scores on EACH scale represented a preference for experiential learning methods over computer-based simulation. The paired t-test provides results on within-person, not between-person, differences. Thus, it is the SAME respondents who indicate a preference for both computer simulation and computer simulation. The results of the factor analysis combined with the results of the paired 1-test led to the disappointing conclusion that students reported a preference for whatever teaching method was identified first in the item. That is, both factors may well indicate nothing more than method variance. The items that identified experiential teaching methods first resulted in an apparent preference for that method of teaching, whereas the items that identified computer-based simulation first indicated an apparent preference for that teaching method. And indeed, those preferences co-existed within the same individual, depending on which methodology was identified first in the item! Thus, as educators, we must be wary of apparent results. While the classic scale development procedures were not carefully followed in this study, very few researchers are willing to spend time on careful scale development. Yet if we are not willing to do so, we may find that we have accumulated results that are not easily interpretable, as in this study. It is doubtful that the existence of method factors is confined to this study. As a result, we must be careful about prescribing pedagogical approaches based on single studies, or even on a cumulation of studies which use differing measurement methods. Although the rewards for careful studies of measurement are not immediately forthcoming, it is important to spend time on measurement issues before proceeding to test substantive questions. Too few among us are willing to make that investment of time. CONCLUSION The major purpose of this paper was to investigate the relations, as perceived by our students, between two popular and widely utilized pedagogies -experiential exercises and computer- based simulation. While the study does support some substantive conclusions, one of the major findings was that our students are susceptible to response bias. To the. extent that student responses are used to measure the effectiveness of pedagogy, we must be very careful in the construction of measures and in interpretation of results. To the extent that student responses are used to measure the effectiveness of instructors, we would be well advised to take the results of this study to heart! REFERENCES ON REQUEST 239 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 240 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 241 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 242 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 243 Table of Contents Volume 25, 1998 Marketing Goes to the Movies Bringing Experiential Learning to a Principles of Marketing Course Investment Analysis Application Using In-house Spreadsheet Models SugarCoated Statistics: An Exercise for the First Day of Class Improving Undergraduate Student Involvement in Management Science and Business Writing Courses Using the Seven Principles in Action Establishment and Funding for Interuniversity / Multidisciplinary Student experiences The Prospects of Creative Teaching: A Discussion with Patricia Sanders The Simulation and Classroom Assessment Techniques Developments of Management Skill Assessment Games as Instruments of Assessment: A Framework for Evaluation The Role of Artificial Intelligence in Business Curricula Threshold Solo Competitor: A Management Simulation (V1.0) a Windows-Based. Play Alone, Total Enterprise Simulation and Assessment Instrument Toward An Understanding of One's self-concept Total Enterprise Simulations and the Internet: Improving Student Perceptions and Simplifying Administrative Workloads The Expatriate an Assignment Orientation Game An Expatriate's Nightmare: An Experiential Exercise in Coping with Overseas Assignments Analyzing Experiential Exercise: Using the Scientific Method for Problem Solving The Second Component to Experiential Learning: A Look Back at How ABSEL has handled the Conceptual and Operational Definitions of Learning Predictive Models of Learning: Participant Satisfaction of Experiential Exercises in Business Education Accelerating Moral Development through Use of Experiential Ethical Dilemmas Ethical Dilemmas to use with Business Simulations to Teach Ethics The Class Approach in Behavioral Simulation in a Business Policy/Strategic Management course: A Progression toward Greater Realism An Exploration of the Emergence of Process Prototypes in a Management Course Utilizing a Total Enterprise Simulation How Organizations Are Improving Their Performance Utilizing Electronic Commerce: Examples From The Internet The Market Access Planning System (Maps): A Computer-Based Decision Support System For Facilitating Experiential Learning In International Business An Excel Workbook For Student Planning And Interface With A Simulation Game Design Of Multi-Media Based Pedagogy For Leadership Training Panel Discussion On Using The Internet For Courses Valuing And Enhancing Teaching: Sharing Tips Via The Web The Buddy Project: A Semester Long Project Aimed At Developing An Appreciation For Diversity Enhancing The Excitement And Learning Retention In The Classroom: The Power Of Magic The Supervised Management Internship: A Job Or Learning Experience Team Ware™ An Online Moderated Class Discussion Facility And Beyond A Neophyte Distance Educator's Experience Learning Management By Practicing Management: A Report Of Significant Student Service In 1997 Integration Of Academic And Service Learning: Students' Perceptions About Its Effects And Outcomes The Value Of Incorporating A Service Learning Component Into Course Content: A Presentation And Roundtable Discussion Business Games Teach: Thoughts on the Sources of Conflicting Conclusions on their Effectiveness Antecedents Of Learning In The Simulation: A Replication Using Student Journals To Enhance Learning From Simulations Technological Change And Intertemporal Movements In Consumer Preferences In The Design Of Computerized Business Simulations With Market Segmentation Integrating The Marketing Curriculum Using Collaborative Learning Teaching Time Management In A Sales Program: The Application Of A Computer Simulation Game Adapting Interactive Computer Simulations For Content Based Esl Instruction Multimedia And Student Expectations Synthesizing Data For Media Simulations Composing A Team Health Promoting Behaviors-A Decision Making Exercise Does it really Work? An Application of the Group Interaction Framework Administering the MIT Beer Game: Lessons Learned A Paperless Economy? Instructing Students on the Aspects of Successful Electronic Commerce Maximizing Learning Gains in Simulations: Lessons from the Training Literature Observing General Ability in a Total Enterprise Gaming Simulation FReach Teach: A Computer-Based System for Teaching Advertising Media Planning An Integrated Business Instruction System An Experiential Exercise you can Tinker With Experiential Exercises or Computer Simulations? Cash Flow Statements: Are They Important in Business Simulations? Holistic Cognitive Strategy in a Computer-Based Marketing Simulation Game: An Investigation of Attitudes Towards the Decision-Making Process Barnga: A Game on Cultural Clashes The Many Faces of Culture: Understanding Country and Corporate Culture Students' View of the Use of Business Gaming in Hong Kong Assessing General Management Interest What is the Future of Business Gaming? Starting a Small Music Trivia Business Exercise and Other Innovative Icebreakers An Integrated Approach to Behavioral Skill Development Career Focus: A Student and Business Learning Experience The Use of Concept Mapping in Teaching Strategic Management An Experiential Approach to Developing Mission Statements Business Games in Brazil-Learning or Satisfaction A Simulation within a Simulation: Job Layoff's and Emotional Reactions