Experiential Learning - What Do We Know? A Meta-Analysis of 40 Years of Research Page 279 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 EXPERIENTIAL LEARNING - WHAT DO WE KNOW? A META-ANALYSIS OF 40 YEARS OF RESEARCH Gerald (Jerry) F. Burch Tarleton State University gburch@tarleton.edu John H. Batchelor University of West Florida jbatchelor1@uwf.edu Nathan A. Heller Tarleton State University heller@tarleton.edu Joanna Shaw Tarleton State University jshaw@tarleton.edu Walter Kendall Tarleton State University kendall@tarleton.edu Beverly Turner Tarleton State University bturner@tarleton.edu ABSTRACT Experiential exercises have been used for decades under the assumption that more active forms of curriculum delivery result in better learning outcomes. This study evaluates every article published in the ABSEL Proceedings over the past 40 years to identify the true level of learning as evaluated by both objective measures and all student perceptions. The results show strong support for continued use of experiential exercises and also the need to continue to conduct empirical analysis grounded in sound measures and using control groups. INTRODUCTION There is once again growing interest surrounding experiential learning. This could be partially based on the new AACSB 2013 Business Accreditation Standards (AACSB, 2013) which states that curricula should facilitate and encourage active student engagement where “students engage in experiential and active learning to improve skills and the application of knowledge.” Included in this statement is the assumption that experiential/active learning leads to learning, and perhaps even more substantive learning than more passive forms of instruction. The purpose of this article is to provide a quantitative review of the four decades of research on experiential learning to determine what we know and what is still left to learn about the learning outcomes achieved by experiential learning. We examine every article in the ABSEL Proceedings from 1974 to 2013 to examine the empirical evidence that has accumulated by ABSEL members. Our primary focus is to determine if experiential exercises lead to increased “learning” outcomes above that which is obtained in less active forms of instruction. We accomplish this task using meta-analytic practices that allow us to accumulate the empirical findings in other studies to calculate a “true” or Page 280 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 “corrected” level of increased learning. Our results are then used to provide practical implications for the use of experiential exercises to increase learning and to make recommendations for future research. THE FIRST FORTY YEARS ABSEL has enjoyed a rich, forty year history of sharing theoretical, empirical, and practical research results amongst members in a unique ABSEL “style” (Hoover, 1979). Periodically literature reviews have addressed the learning outcomes associated with simulations (Greenlaw & Wyman, 1973; Keys & Wolfe, 1990; Klein & Fleck, 1990; Malik & Howard, 1996; Wolfe, 1985) and have led to the general consensus that simulations lead to increased learning of concepts over traditional teaching techniques. One problem is that most of the evidence surrounding the benefits of experiential experiences has been performed as an “Act of Faith” since the instructor “just knows that learning has taken place” (Gentry, Commuri, Burns, & Dickinson, 1998 p. 64). What is missing from the literature is a rigorous evaluation of the empirical evidence that exists concerning the benefits of a more active form of teaching. This comment is supported by the pleas from researchers to improve experimental designs to provide more credible comparisons of pedagogies (Butler, Markulis, & Strang, 1985; Gentry, et al., 1998; Gosenpud, 1990; Wolfe, 1976; Wolfe, 1981). The purpose of this paper is to determine if there is enough accumulated empirical evidence to finally determine if including experiential exercises in business courses results in increased learning or development of skills. One problem associated with answering this research question is how to determine if learning has occurred. To facilitate this discussion we adopt the definition of experiential learning from the whole person perspective where “experiential learning exists when a personally responsible, participant cognitively, affectively, and behaviorally processes knowledge, skills, and/or attitudes in a learning situation characterized by a high level of active involvement” (Hoover & Whitehead, 1975, p. 25). Experiential exercises are therefore defined as any activity that requires the student to be an active part in their education. From this perspective it can be seen that some experiential exercises (e.g. simulations) require more student engagement than other forms of experiential exercises (case studies). This creates a continuum of experiential exercises that includes case studies, role plays, simulations, internships, and others. Adopting this idea allows us to include more empirical research and at the same time adds to the complexity of how to measure learning. One means of assessing learning is to evaluate the extent to which the student mastered the learning objectives devised by the instructor. At first glance this appears to be a benign statement since it makes “little sense to attempt to measure learning if what is intended to be learned has not been clearly defined.” (Gentry, et al., 1998, p.63). However, the need to specify learning objectives still creates difficulties in assessing learning (Anderson & Lawton, 1997). Our research focused on those ABSEL studies where learning objectives were stated and where treatment and control groups were used to enhance the study design (Campbell & Stanley, 1963). In addition to this form of learning we also believe that assessing student learning based on the student’s ability to meet the instructor’s learning objectives fails to acknowledge the “whole person” perspective (Gentry, et al., 1998) since the student may “learn” something that the instructor never intended (Gosenpud, 1996). This type of learning is difficult for the instructor to anticipate and may only be available by asking the student to evaluate their TABLE 1 SUMMARY OF EXPERIENTIAL LEARNING OUTCOMES AND EFFECTS RESULTS Page 281 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 level of learning. It is therefore imperative to evaluate those studies where the student perception of learning is evaluated. METHODOLOGY Two reviewers independently scanned all articles, total 2,429, published in Developments in Business Simulation & Experiential Learning (ABSEL Proceedings) from 1974- 2013. Full text copies were obtained for all 311 empirical papers. These papers were reviewed to identify empirical papers that addressed learning outcomes based on experiential learning. This search resulted in 42 studies that addressed individual learning outcomes based on evaluating learning objectives through tests or evaluations. From this group 26 studies were rejected since they did not provide both a control group and a treatment group that had received a form of experiential learning. This resulted in 16 usable studies (1,048 individual respondents) that contained sufficient information to be included in the primary meta- analysis investigating actual student learning outcomes. A separate group of studies were identified that evaluated the student’s perception of learning. The initial list of perception studies included 37 articles. Further examination of these articles identified only four articles with eight studies (2,672 individual respondents) that contained the student perception of the experiential learning experience and an evaluation of traditional lecture effectiveness. These meta-analyses evaluated all obtained mean differences between experiential learning and control groups. In papers consisting of multiple time periods, the last time period only was coded. Some papers presented multiple studies on various research topics. For these papers, only the studies relevant to our primary research questions were coded. RESULTS PRIMARY ANALYSIS OF EXPERIENTIAL LEARNING OUTCOMES Our analysis computed the fixed and random effects of experiential learning outcomes using Comprehensive Meta Analysis (CMA).The results in Table 1 show a point estimate for the standard difference in means for experiential learning and the control group of .86 (p<.001) and .75 (p<.01) for the fixed and random effects models respectively. Because individual studies varied in how experiential learning was measured and the possibility of moderating variables exists, which may result in additional heterogeneity, the results of the random effects model should be evaluated as more appropriate in this setting than those of the fixed effects model. With that said, both models show a significant positive improvement in the means of the experiential learning groups over the control groups. Further, an evaluation of the forest plots for both the fixed and random models shows the lower limits do not cross zero, indicating support for the positive effect of experiential learning on participants. Finally a one study removed analysis was performed on the data to see if any one study exerted undue influence on the overall results. The results of this analysis showed a range of individual study point estimates from .77 to 1.00 for the fixed model and .61 to .86 for the random model. These estimated indicate that no one study exerted undue influence on our results. STUDENT PERCEPTIONS OF LEARNING The student perceptions of learning analysis also computed the fixed and random effects of student perceptions of learning using Comprehensive Meta TABLE 2 SUMMARY OF STUDENT PERCEPTION STUDIES AND EFFECTS Page 282 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 Analysis (CMA). The results in Table 2 show a point estimate for the standard difference in means for student perceptions of experiential learning and the control group of .70 (p<.001) and 1.05 (p<.001) for the fixed and random models respectively. Similar to the primary analysis, the studies investigating student perceptions of learning varied in how experiential learning is measured and the possibility of moderating variables exists, which may result in additional heterogeneity, thus again, the results of the random effects model should be evaluated as more appropriate in this setting than those of the fixed effects model. Both models show a significant positive improvement in the means of the experiential learning groups. Further, an evaluation of the forest plots for both the fixed and random models shows the lower limits do not cross zero, indicating support for the positive effect of experiential learning on student perceptions of learning. Finally, as with the primary analysis, a one study removed analysis was performed on the student perception of learning data to see if any one study exerted undue influence on the overall results. The results of this analysis showed a range of individual study point estimates from .45 to .87 for the fixed model and .87 to 1.18 for the random model. These estimated indicate that no one study exerted undue influence on our results. DISCUSSION The results of this study show that the use of experiential learning activities has a positive, significant effect on student learning and the student’s perception of learning. Many have hypothesized this result, while others simply performed the exercises on an “act of faith” (Gentry, et al., 1998, p. 64). This information provides the necessary empirical evidence that is needed to encourage others to include more active forms of curriculum delivery. A second important finding in this study is that the level of demonstrated learning (according to tests and evaluations) is very similar to the student’s perception of learning. In both studies the standard difference in means was .86 and .70 for fixed effects and .75 and 1.05 for random effects for demonstrated learning and perceptions, respectively. Students therefore acknowledge a similar increase in learning as demonstrated by the learning objectives when instructors use experiential learning techniques in the class room. One weakness in this study was the paucity of studies that compare student perception of learning in traditional classrooms to learning in classes with experiential exercises/opportunities. Even though this study analyzed every article published in the ABSEL Proceedings for 40 years, only four studies evaluated this relationship. There were several studies that measured student perceptions towards experiential learning that had to be rejected since they did not provide a frame of reference (control) for their perceptions. Future research should continue to work towards developing studies that provide a high level of control for both perception and objective studies. CONCLUSIONS This study sheds considerable light on the true relationship between experiential learning exercises and student learning outcomes. Although this lack of knowledge did not stop many ABSEL members, and others, from using experiential learning exercises in their classrooms, it is reassuring to know that the numbers support the practice. As we continue to move forward in the development of the “perfect” learning environment we must shift our focus to finding the right mix of experiential and traditional teaching methods. Students need conditions where they can explore new knowledge in a safe environment, time to reflect on their learning, and guidance to integrate both. Learning takes time and experience. As the demand to decrease the time and cost associated with obtaining a higher education degree it will become more important to add experience where ever possible. We still have much to learn about this process, but it nice to know that we are on the right path. REFERENCES Anderson, P.H. & Lawton, L. (1997). Demonstrating the learning effectiveness of simulations: Where we are and where we need to go. Development in Business Simulation and Experiential Learning, 24, 68-73. (Reprinted from Bernie Keys Library (11th ed.)) Butler, R.J., Markulis, P.M., & Strang, D. (1985). Learning theory and research design: How has ABSEL fared? Development in Business Simulation and Experiential Learning, 12. (Reprinted from Bernie Keys Library (11th ed.)) Campbell, D.T., & Stanley, J.C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally & Company. Gentry, J.W., Commuri, S., Burns, A.C., & Dickinson, J.R. (1998). The second component to experiential learning: A look back at how ABSEL has handled conceptual and operational definitions of learning. Development in Business Simulation and Experiential Learning, 25. (Reprinted from Bernie Keys Library (11th ed.)) Gosenpud, J. (1990). Evaluation of experiential learning. Guide to business gaming and experiential learning. 301-329. (Reprinted from Bernie Keys Library (11th ed.)) Greenlaw, P.S., & Wyman, F.P. (1973). The teaching effectiveness of games in collegiate business courses. Simulation & Games, 4, 259-294. Page 283 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 Hoover, J.D. (1979). “Foreword” In S. Certo and D. Brenenstuhl (Eds.), Development in Business Simulation and Experiential Learning, 6. (Reprinted from Bernie Keys Library (11th ed.)) Hoover, J.D., & Whitehead, C.J. (1975) An experiential- cognitive methodology in the first course in management: some preliminary results. Simulation Games and Experiential Learning in Action, Volume 2. . (Reprinted from Bernie Keys Library (11th ed.)) Keys, B., & Wolfe, J. (1990). The role of management games and simulations in education and research. Journal of Management, 16, 307-336. Klein, R.D., & Fleck, R.A. (1990). International business simulation/gaming: An assessment and review. Simulation and Games, 21, 147-165. Malik, S.D., & Howard, B.J. (1996). How do we know where we’re going if we don’t know where we have been: A review of business simulation research. Development in Business Simulation and Experiential Learning, 25, 49-53. (Reprinted from Bernie Keys Library (11th ed.)) Wolfe, J. (1976). Comments on the perception, identification, and measurement of learning in simulation games. In Bernard Sord (Ed.) Computer Simulation and Learning Theory. 3. (Reprinted from Bernie Keys Library (11th ed.)) Wolfe, J. (1981). Research on the learning effectiveness of business simulation games – A review of the state of the science. Development in Business Simulation and Experiential Learning, 25. (Reprinted from Bernie Keys Library (11th ed.)) Wolfe, J. (1985). The teaching effectiveness of games in collegiate business courses: Simulation & Games, 16, 251-288