PREDICTIVE MODELS OF LEARNIN: PARTICIPANT SATISFACTION OF EXPERIENTIAL EXERCISES IN BUSINESS EDUCATION Developments in Business Simulation and Experiential Learning, Volume 25, 1998 PREDICTIVE MODELS OF LEARNING: PARTICIPANT SATISFACTION OF EXPERIENTIAL EXERCISES IN BUSINESS EDUCATION Gregory H. Patton, University of Southern California Daniel Cochece Davis, University of Southern California Gita Govahi, University of Southern California ABSTRACT This study explores the use and evaluation of experiential exercises in the business classroom. It briefly examines the research foundations of business gaming, highlighting the emphasis on management simulations and the need for explorations of experiential exercises. This study builds on existing research by comparing two competing models of “learning enhancement” across three avenues of participant learning: mega lectures, lab sessions, and experiential exercises. Results of regression analyses on the competing models surprisingly contradict previous theorizing about experiential learning’s role in management. Specifically, findings indicate that experiential exercises may be misplaced in learning theories and should not be considered as pedagogically interchangeable with simulations. The interaction of instructors on participant satisfaction and monotonic trends among means of the three learning categories are explored. The emerging predictive model for experiential exercises provides a key stepping stone toward a consumer satisfaction orientation allowing a better transition between the business classroom and corporate settings. INTRODUCTION This study explores the use and evaluation of experiential exercises in the business classroom, with the hope that authoritative claims regarding the general efficacy of functional and skill-based experiential exercise programs will soon be forwarded, based on research programs utilizing contemporary research designs. The present study forwards this process. By briefly exploring the research foundations emerging in the investigation of business gaming, this review highlights the continuing emphasis on management simulations and investigates the role of experiential exercises in business classrooms while identifying the lack of authoritative general claims as to the efficacy of these exercises. Expanding on experiential exercise research, the present study introduces an examination of the effects of a series of exercises on participants’ learning of basic organizational behavior concepts. Additional steps are taken toward creating an experiential training model, identifying the need for differentiation of simulation and exercise research, and providing an examination of the role of exercises within the core curriculum of business schools. RESEARCH FOCUS IN SIMULATION & EXPERIENTIAL EXERCISE LITERATURE Targeted research efforts that test constellations of variables found within general management simulations have increasingly eclipsed questions of overall effectiveness within recent research reported in ABSEL publications. While these efforts greatly improve the statistical rigor of studies, they also de-emphasize the most basic questions of overall efficacy of simulation and experiential exercises. These questions remain unanswered. Writing as part of the field’s silver anniversary, Klabbers (1994) notes that “A review of three decades of business games 69 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 does not reveal how effective they are” (p. 509). Similarly, Anderson and Lawton (1997a) argue that, despite the extensive literature and reviews that contain truly “awe-inspiring” bibliographies, “it remains difficult, if not impossible, to support objectively even the most fundamental claims for the efficacy of games as a teaching pedagogy” (p. 68). Three decades of investigation have produced much insight, while simultaneously producing increasingly complex, unanswered, questions. Key to past research efforts was a focus on general management games that originally defined the field (Greenlaw & Wyman, 1973). These general games (e.g., total/full enterprise or top management simulations) continue to demand the primary attention of research and development (Keys & Wolfe, 1990; Wolfe 1985). In efforts to develop greater understanding of the interacting variables influencing learning outcomes in these simulations, researchers have increasingly focused on factors relating to the nature of individual simulations, participant’ abilities, and administrative elements (see Bredemeier & Greenblat, 1981; Keys & Wolfe, 1990; Wolfe 1985). Reviewing the management game movement and the current state of the field, Keys and Wolfe (1990) chose to “concentrate on the total enterprise management gaming literature, because this is historically where most of the research and development has taken place. . .“ (p. 309). Currently, this trend continues in programs of research examining different variables within specific simulations, such as investigative research on “MICROMATIC” (e.g., Gosen & Washbush, 1997). Omitted from this research agenda is the overall effectiveness of specific, skill-based, experiential exercises, or programs of exercises, in teaching participants foundational business education information. Despite different approaches, current research efforts share a dominant preference toward the study of complex simulations at the expense of research into far less complex experiential exercises. These experiential exercises are unique in that they are often highly focused and are likely to develop a particular skill rather than focusing on the applications of a constellation of skills. The differentiation, however, of total enterprise gaming and experiential exercises is more than a concern over the number and type of learning variables. The Methodological Debate Issues of appropriate measurement techniques in assessing exercise or simulation effectiveness have also been a significant focus within the field. The challenges of researching participant learning, and the appropriateness of methodologies, have led to lively discussions at ABSEL Conferences (see Anderson & Lawton, 1997b for the latest forum), a solidification of distinct world views and programs of research, and the emergence of two trends. In the first trend, researchers have generally disfavored an overall approach to the assessment of participant learning, deferring instead to the determination of which individual skills were developed by participants (see Anderson & Lawton, 1997a; Hemmasi & Graf, 1991). This necessary step, nonetheless, increased fragmentation across research efforts, making general claims regarding overall effectiveness more difficult. Further compounding these difficulties is the nature of the individual skills serving as possible learning outcomes. Anderson and Lawton (1997a) suggest that these outcomes are aimed at higher levels of Bloom’s taxonomy, and that, since few of the objectives are cognitive or affective, “It will require an enormous amount of ingenuity to develop appropriate objective measures for most of these learning objectives; and for most of these objectives, we haven’t even begun to develop objective instruments” (p. 73). 70 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 The second trend within business gaming calls for more sophisticated research techniques and new approaches to designing research (Anderson & Lawton, 1997a; Keys & Wolfe, 1990). In answer to this call, researchers have increasingly questioned the use of self-reports in evaluating perceptions and attitudes, instead preferring actual assessment of knowledge gains (see Anderson & Lawton, 1997a; Whiteley & Faria, 1989). The attack on self-reporting of perceptions and attitudes is not unique to business gaming. This methodological debate parallels ongoing dialogues in the broader domain of training and development. Jones (1990) suggested 26 separate “faults” or “limitations” with this type of instrumentation. Most of these concerns relate to poor instrumentation design, and call for better instrumentality, rather than an abandonment of the process. Given appropriate design, the major structural impediments are primarily linked to concerns over the application, or transfer, of the training and rater characteristics (e.g., their qualifications, judgment, history, and mood). However, the general faults with self-reported data are not fatal, especially within the context of other business gaming research efforts. Inquiry into the transfer, or application, of training through assessment of knowledge gains, often by exploring specific variables, is a well developed area of research within the simulation and experiential learning field. The concerns based on rater characteristics in assessing learning from experiential activities, and particularly simulations within college settings, is also a developing area of research in business gaming. Many of these participant/rater issues are described by Wolfe (1985) under “player and group characteristics” and by Bredemeier and Greenblat (1981) in their discussion of variables. Drawing on these previous inquiries into the transfer of training and participant/rater characteristics, the use of self reported data on experiential exercises can offer valuable insight without the shortcomings attributed to the more generalized use of the method. The efforts to further improve perception and attitude based measurements of experiential activities, and to answer issues of training efficacy, ultimately turns to the creation of better instruments and modeling. Focus on Experiential Exercises Experiential exercises enjoy wide spread usage in traditional business classrooms, and current trends within higher education hold the potential for a promising future. Increasingly, classes at large colleges and universities are moving towards very large, “mega” lecture sections for introductory business classes supported by class discussion laboratories. This trend, primarily driven by economic concerns, offers a new forum and potential opportunity for providing students with many separate experiential exercises serving as a second approach in developing the student’s knowledge of core concepts. The movement toward laboratories has received little attention in the field of business simulations and experiential learning. An exception to this is Keys and Wolfe (1990) who undertook an examination of trends influencing management gaming development. They noted that “Management games will play a more significant role in management development and assessment efforts in business schools” and that “Laboratories designed around simulations and games are becoming the accepted teaching environment for many of these schools” (pp. 324-325). Despite the trend within business schools, research on management games has not responded to the unique attributes and experiential potential of business laboratories. Rather than a forced choice of pedagogical methods, laboratories utilizing experiential exercises offer an additive dimension. Given classroom changes, new research questions will develop relating to: a) the interaction of instructors in the dis- 71 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 cussion labs conducting activities and professors providing lectures, b) the interaction effect of experiential exercises with content lectures and regular discussion sessions, and c) the relative merits of different proportionality of the three methods in teaching information. PURPOSE The purpose of the present study is to explore and expand present understanding of the unique ways in which experiential exercises contribute to overall participant learning. Albertson (1995) sought predictor items in creating a model of “drivers” to explain overall effectiveness of experiential training. He found “learning enhancement” to be the primary component of overall participant satisfaction when engaging in experiential exercises. Building on this work, the present study targets Albertson’s “learning enhancement” component for analysis by comparing two competing models, each potentially capable of predicting major portions of variance in students’ overall evaluation of their educational experience. The first model is rooted in the student’s perceived “utility” of the educational experience (e.g., did I learn how to apply this information?). The second model is grounded on the student’s perceived “mental understanding” of the educational experience (e.g., did this information improve my understanding of the material?). Expanding beyond the scope of Albertson (1995), this study also seeks to explore the relationships between three methods or categories of participant learning (mega lectures, lab sessions, and experiential exercises) to better understand their potential roles in business education. The focus is not on isolating and building support for a single best method, but rather to better understand the interactional elements when these complementary methods are used in a business education program. METHODOLOGY A survey at the end of the semester was utilized to understand students’ perceived learning of organizational behavior concepts (a copy of the survey is available on request). Participants included 317 upper-division business students enrolled in a required introductory organizational behavior class taught by the management department at a large private west coast university. The four unit management course requires enrollment in one of two weekly lecture sections taught by the department and one of twelve weekly lab/discussion sections taught by graduate student teaching assistants. These educational experiences are supplemented by experiential activities at the business school’s experiential learning center conducted by staff facilitators. During the sixteen week semester, students are provided sixteen lectures, twelve discussion sessions, and engage in four experiential activities. Surveys were administered at the end of the Spring 1997 semester, in place of regular experiential learning center evaluation forms. The survey, developed by the researchers, sought quantitative feedback on fifteen questions divided among three categories (lecture, lab, and experiential learning sessions). Each category contained five standard questions, measured on seven-point scales, relating to students’ abilities in using and understanding course material. Data analysis was completed using SPSS for Windows (version 6.13). RESULTS The Alpha reliability level for the 15 item survey instrument was excellent (.94; N 317). There is a monotonic trend (e.g., Braver & Sheets, 1993) among the means of overall satisfaction within each category (e.g., experiential learning = 6.14; mega lecture = 5.54; lab = 5.42). Additionally, the skewness patterns across the three categories varied, with 72 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 students often evaluating their experiences at the highest evaluation level possible (7.00) resulting in a data set “leaning” or skewed toward that end of the scale. Skewness was most pronounced in the experiential learning category (lab -.90; mega = -1.12; experiential learning = -1.96) indicating a consistency in student’s evaluation of their experience with this educational category. The combination of the professor providing mega lectures, instructors in the discussion labs, and individuals facilitating experiential exercises, collectively contributes to participants’ overall satisfaction and learning of organizational behavior concepts and practices. Additionally, instructors of discussion labs specifically contribute to participants’ overall satisfaction levels with mega lectures and experiential learning sessions. Facilitators of experiential exercises significantly contribute to overall evaluations of lab sessions, but not mega lecture sessions. Stepwise multiple regression analyses facilitated testing of the two competing business education satisfaction models. Survey instrument items supporting a “mental understanding” model and a “utility” model functioned as predictor variable clusters in the analyses. Nearly equal amounts of participant variance were accounted for across the two models (“mental understanding”: r2 = 82%; “utility”: r2 = “79%). Moving from a macro satisfaction orientation to examine specific categories of educational method, a second series of regression analyses was used to determine micro satisfaction level predictors. For each of the three methods of participant learning (mega lecture, lab, experiential learning) the “mental understanding” and “utility” items served as predictor variables in a stepwise multiple regression for the category’s satisfaction evaluation. In each of the three categories, the predictor variables accounting for the greatest variances of overall satisfaction were consistently related to “mental understanding” rather than “utility” (mega lecture, r2 = 66.4; discussion lab, r2 = 72.7; experiential learning, r2 = 68.6). DISCUSSION Although the present study is of an exploratory nature, some results appear solid enough to strongly suggest a representativeness of business students’ experience on a more general level. Additionally, the data set displays consistent trends that, while not statistically tested for significance, still provide useful information to researchers. At the macro level, the two competing models surprisingly contradict previous theorizing about the role of experiential learning in management education. Specifically, Hsu (1989) forwarded a four phase learning process model. The first two phases (“retention” and “organizing”) correspond to cognitive learning elements while phase three and four (“experiencing” and “firming”) focus on skill building. Hsu logically suggests that the home of gaming and simulation lies in the third phase (“experiencing”) involving skill-building. The present research indicates that experiential activities actually span Hsu’s second and third phases in the learning process model, as experiential activities and complex behavioral simulations are equally at home in the second phase (“organizing of knowledge”). Hsu describes this phase as when “students take initiatives to digest the information received by verbalizing, discussing, and applying the principles and concepts to a situation, they begin to organize the information into knowledge and represent it in long-term memory in one of the following forms: semantic nets, frames, scripts, rule-based presentations, or logic-based representations” (p. 418). The phase two classification is especially true when the experiential exercises are conducted in introductory management courses, assisting in the learning of core concepts. Given the 73 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 ability of experiential activities to span the realm of cognitive learning and skill-building, it may follow that models predicting major portions of the variance in student overall satisfaction for these types of exercises are more involved, representing more learning phases and comprising a more balanced approach between cognitive learning and skill building than overall management games. A benefit of the appearance that experiential exercises scans two learning phases, is that the assessment of an experiential activity based on its cognitive learning component may be more appropriate and beneficial than a similar approach to overall management games. Two different counter-intuitive results were also observed. At the micro level, “mental understanding” variables not only predicted satisfaction levels in the mega lecture category as logic would dictate, but also predicted satisfaction levels for the experiential learning category where one would expect "utility” variables to dominate and in the discussion lab category where it was unclear whether “mental understanding” or “utility” would prevail. Additionally, at the macro level, one would expect the dominance of “mental understanding” variables at the micro level to translate into a macro level dominance as well. Yet, the “mental understanding” and “utility” models appear to explain relatively similar amounts of variance. Of additional interest are the results stemming from the interaction of instructors on participant satisfaction and the monotonic trend across means of the three learning categories. While the present study was limited in its ability to examine interactions across all categories by the existence of a single mega lecture professor, the results still point to the existence of important results. For example, the data suggests that the individual lab instructors have an impact on their student’s overall satisfaction of the mega lecture section taught by another faculty member. Meanwhile, satisfaction means for the learning categories and the monotonic trend appear to-indicate that students are favorable toward the use of mega lectures in conjunction with discussion labs and experiential exercises. These experiential exercises may play important roles in this trilogy. Monotonic trends have proven useful in other business settings (e.g., Merenda & Jacob, 1987). When explored in greater detail, these trends may provide prima facia evidence that discussion labs in the field of business, and more generally in universities, may provide improved educational experiences when several experiential activities are incorporated into the curriculum. Additionally, this predictive model is a key stepping stone toward a consumer satisfaction orientation that allows a better transition between business classrooms and corporate settings. This new orientation will allow practitioners greater access to pedagogical efforts in the field. IMPLICATIONS FOR FUTURE RESEARCH Future studies should investigate several distinct areas to provide a more systemic understanding of business students’ educational experience with experiential exercises. Initially, further examination and understanding of the interaction and predictive ability of “mental understanding” and “utility” models is needed. Further examination is especially necessary within the three learning categories (mega lecture, discussion lab, experiential exercise) to understand reasons for the similarity in the predictor variable for participant satisfaction and when differences may exist. Closely related to this is the need to better understand the placement of experiential exercises within learning process models and the differentiation in placement between simulations and experiential exercises that appears necessary. Perhaps related to this is the 74 Developments in Business Simulation and Experiential Learning, Volume 25, 1998 fact that the data suggest the presence of colinearity (shared variance) among the predictor variables. Future studies should develop orthogonal variables (variables that have little to no correlation with each other) accounting for unique variance, better distinguishing themselves as independent predictors of the criterion variable (i.e., overall satisfaction). Next, the monotonic trend, noted across categories, suggests higher satisfaction levels with experiential exercises than other categories of education experience. Future research should test this type of data for group differences to ascertain whether these differences are statistically significant. Closely aligned to this is the need for future investigation of the skewness pattern found across educational categories. Further analysis of this area may also prove fruitful in the inquiry of group differences across educational categories. REFERENCES Albertson, D. S. (1995). Evaluating experiential training: Case study and recommendations. Developments in Business Simulation and Experiential Learning, 22, Anderson, P. H., & Lawton, L. (1997a). Demonstrating the learning effectiveness of simulations: Where we are and where we need to go. Developments in Business Simulation and Experiential Learning, 24, 68-73. Anderson, P. H., & Lawton, L. (1997b). Simulations and learning: Can we prove a relationship? Developments in Business Simulation and Experiential Learning, 24, 109- 110. Braver, S. L., & Sheets, V. L. (1993). Monotonic hypotheses in multiple group designs: A Monte Carlo study. Psychological Bulletin, 113(2), 3 79-395. Bredemeier, M. E., & Greenblat, C. S. (1981). The educational effectiveness of simulation games: A synthesis of findings. Simulation & Games, 12(3), 307-322. Greenlaw, P. 5., & Wyman, F. P. (1973). The teaching effectiveness of games in collegiate business courses. Simulation & Games, 4(3), 259-294. Gosen, J., & Washbush, J. (1997). Antecedents of learning in simulations. Developments in Business Simulation and Experiential Learning, 24, 60-67. Hemmasi, M., & Graf, L. A. (1991). Educational effectiveness in business simulation gaming: A comparative study of student and practitioner perspectives. Developments in Business Simulation & Experiential Exercises. 18, 53-56. Hsu, E (1989). Role-event gaming simulation in management education. Simulation & Games, 20(4), 409-438. Jones, J. E. (1990, December). Don’t smile about smile sheets. Training & Development Journal, 44(12), 19+. Keys, B., & Wolfe, J. (1990). The role of management games and simulation in education and research. Journal of Management, 16(2), 307-336. Klabbers, J. H. G. (1994). The 25th anniversary of ISAGA: The orchestration of organized complexity. Simulation & Gaming, 25(4), 502-5 13. Merenda, P. F., & Jacob, S. (1987). Validity of self-concept measures for selection of sales personnel. Psychological Reports, 60(2), 508. Whiteley, T. R., & Faria, A. J. (1989). A study of the relationship between student final exam performance and simulation game participation. Developments in Business Simulation and Experiential Learning, 16, 78-82. Wolfe, J. (1985). The teaching effectiveness of games in collegiate business courses. Simulation & Games, 16(3), 25 1-288. 75 Table of Contents Volume 25, 1998 Marketing Goes to the Movies Bringing Experiential Learning to a Principles of Marketing Course Investment Analysis Application Using In-house Spreadsheet Models SugarCoated Statistics: An Exercise for the First Day of Class Improving Undergraduate Student Involvement in Management Science and Business Writing Courses Using the Seven Principles in Action Establishment and Funding for Interuniversity / Multidisciplinary Student experiences The Prospects of Creative Teaching: A Discussion with Patricia Sanders The Simulation and Classroom Assessment Techniques Developments of Management Skill Assessment Games as Instruments of Assessment: A Framework for Evaluation The Role of Artificial Intelligence in Business Curricula Threshold Solo Competitor: A Management Simulation (V1.0) a Windows-Based. Play Alone, Total Enterprise Simulation and Assessment Instrument Toward An Understanding of One's self-concept Total Enterprise Simulations and the Internet: Improving Student Perceptions and Simplifying Administrative Workloads The Expatriate an Assignment Orientation Game An Expatriate's Nightmare: An Experiential Exercise in Coping with Overseas Assignments Analyzing Experiential Exercise: Using the Scientific Method for Problem Solving The Second Component to Experiential Learning: A Look Back at How ABSEL has handled the Conceptual and Operational Definitions of Learning Predictive Models of Learning: Participant Satisfaction of Experiential Exercises in Business Education Accelerating Moral Development through Use of Experiential Ethical Dilemmas Ethical Dilemmas to use with Business Simulations to Teach Ethics The Class Approach in Behavioral Simulation in a Business Policy/Strategic Management course: A Progression toward Greater Realism An Exploration of the Emergence of Process Prototypes in a Management Course Utilizing a Total Enterprise Simulation How Organizations Are Improving Their Performance Utilizing Electronic Commerce: Examples From The Internet The Market Access Planning System (Maps): A Computer-Based Decision Support System For Facilitating Experiential Learning In International Business An Excel Workbook For Student Planning And Interface With A Simulation Game Design Of Multi-Media Based Pedagogy For Leadership Training Panel Discussion On Using The Internet For Courses Valuing And Enhancing Teaching: Sharing Tips Via The Web The Buddy Project: A Semester Long Project Aimed At Developing An Appreciation For Diversity Enhancing The Excitement And Learning Retention In The Classroom: The Power Of Magic The Supervised Management Internship: A Job Or Learning Experience Team Ware™ An Online Moderated Class Discussion Facility And Beyond A Neophyte Distance Educator's Experience Learning Management By Practicing Management: A Report Of Significant Student Service In 1997 Integration Of Academic And Service Learning: Students' Perceptions About Its Effects And Outcomes The Value Of Incorporating A Service Learning Component Into Course Content: A Presentation And Roundtable Discussion Business Games Teach: Thoughts on the Sources of Conflicting Conclusions on their Effectiveness Antecedents Of Learning In The Simulation: A Replication Using Student Journals To Enhance Learning From Simulations Technological Change And Intertemporal Movements In Consumer Preferences In The Design Of Computerized Business Simulations With Market Segmentation Integrating The Marketing Curriculum Using Collaborative Learning Teaching Time Management In A Sales Program: The Application Of A Computer Simulation Game Adapting Interactive Computer Simulations For Content Based Esl Instruction Multimedia And Student Expectations Synthesizing Data For Media Simulations Composing A Team Health Promoting Behaviors-A Decision Making Exercise Does it really Work? An Application of the Group Interaction Framework Administering the MIT Beer Game: Lessons Learned A Paperless Economy? Instructing Students on the Aspects of Successful Electronic Commerce Maximizing Learning Gains in Simulations: Lessons from the Training Literature Observing General Ability in a Total Enterprise Gaming Simulation FReach Teach: A Computer-Based System for Teaching Advertising Media Planning An Integrated Business Instruction System An Experiential Exercise you can Tinker With Experiential Exercises or Computer Simulations? Cash Flow Statements: Are They Important in Business Simulations? Holistic Cognitive Strategy in a Computer-Based Marketing Simulation Game: An Investigation of Attitudes Towards the Decision-Making Process Barnga: A Game on Cultural Clashes The Many Faces of Culture: Understanding Country and Corporate Culture Students' View of the Use of Business Gaming in Hong Kong Assessing General Management Interest What is the Future of Business Gaming? Starting a Small Music Trivia Business Exercise and Other Innovative Icebreakers An Integrated Approach to Behavioral Skill Development Career Focus: A Student and Business Learning Experience The Use of Concept Mapping in Teaching Strategic Management An Experiential Approach to Developing Mission Statements Business Games in Brazil-Learning or Satisfaction A Simulation within a Simulation: Job Layoff's and Emotional Reactions