Evaluating Experience-Based Learning Activities: Working Through the Morass Page 99 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 ABSTRACT The Association to Advance Collegiate Schools of Business (AACSB) recognized in the 2002 report titled “Management Education at Risk” that experiential education keeps students connected to rapidly changing business models. In the years since the report was issued, a number of colleges and universities have made experience-based learning (EBL) a part of their curriculum. This paper examines the manner in which a large Middle Atlantic university’s business faculty implemented a strategic initiative to make EBL central to the curriculum. Specifically, it investigates the spectrum of activities recognized by these faculty members as EBL in orientation and the discipline specific nature of evaluation utilized for each such EBL activity with the goal of developing a framework for identifying best practices for experience-based learning assessment. Preliminary findings from this qualitative analysis indicate a need for closer attention to alignment of desired course outcomes with EBL activities and their assessment. INTRODUCTION This paper began as a conversation between colleagues as to how to best evaluate experience-based learning (EBL) activities in courses offered at a large Middle Atlantic research university. As data collection began with the goal of categorizing best practices in evaluation and assessment of curriculum based EBL activities, it became apparent that a major research project covering many facets of EBL as it is practiced in the classroom was unfolding. Accordingly, this paper is the first in a series of analyses designed to understand the nature of EBL as it is operationalized in the classroom and how these activities are assessed and to ultimately create a set of developmental teaching techniques and best practices for faculty. It has been over a decade and a half since the 2002 Association to Advance Collegiate Schools of Business (AACSB) report, Management Education at Risk, was issued. The report noted that “preparation for the rapid pace of business cannot be obtained from textbooks and cases, many of which are outdated before they are published” (AACSB, 2002). In addition, the report called for “outward-facing curricula and experiential education [that] can create the critical intersection between classroom and business learning that keeps faculty and students connected to rapidly changing business models” (AACSB, 2002). The report triggered a school-wide initiative to create an experience-based learning focus for the business curriculum in the undergraduate, and to a lesser extent, graduate programs at a major Atlantic Coast Conference (ACC) institution. Championed by the school’s dean, training programs were offered to inform faculty members of EBL options and funding was made available to develop and expand EBL activities in each academic major. A current university-wide initiative to ascertain best teaching practices with an accompanying goal of enhancing teaching techniques from delivery of curriculum to assessment of learning underlays the aforementioned colleague conversation on EBL evaluation. As this project began in earnest, a series of questions quickly created the basis for this multi-faceted research project: What is the current state of EBL in business school classrooms? What kinds of activities are used? Is the nature of EBL activities dependent on the discipline? How are the activities evaluated? How do the activities align with course objectives and student learning outcomes? What changes are beginning to occur in the nature of EBL course activities and the manner in which they are evaluated as many colleges and universities now move courses to online platforms? How can faculty best be trained to develop, implement and evaluate EBL activities that will achieve discipline, course, and college objectives and also achieve identified learning outcomes for students? This first paper will center on creating a framework to utilize in discerning the discipline specific nature of EBL activities utilized in business classrooms as well as the nature of assessment for each of those activities. The next phase of this research will look at utilizing the framework in examining the same issues in other schools of business. Subsequent papers will cover the remaining questions with the end goal to be a delineation of best practices for evaluating EBL activities. Evaluating Experience-Based Learning Activities: Working Through the Morass Debbie Good University of Pittsburgh debgood@katz.pitt.edu Elise Boyas University of Pittsburgh eaboyas@katz.pitt.edu Paul Klein University of Pittsburgh pmklein@katz.pitt.edu mailto:debgood@katz.pitt.edu mailto:eaboyas@katz.pitt.edu mailto:pmklein@katz.pitt.edu Page 100 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 THEORETICAL FRAMEWORK Experience-Based Learning Hoover’s 1974 paper on the conceptualization and definition of experiential learning opens with Carl Rogers’ view that experiential learning “… has a quality of personal involvement — the whole-person in both his feeling and cognitive aspects being in the learning event.” Hoover then broadens that definition by noting “Experiential learning exists when a personally responsible participant (s) cognitively, emotionally, and behaviorally processes knowledge, skills and/or attitudes in a learning situation characterized by a high level of active involvement.” (35) A decade later David Kolb (1984) introduced what many consider to be the best known definition of experience-based learning “the process whereby knowledge is created by the transformation of experience.”(41) He suggested that the model is characterized by three key elements: • Involvement of the whole person – intellect, feelings, and senses in learning; • Recognition and active use of relevant life and learning experience; and • Continued reflection. These three elements lead to six underlying propositions of experience-based learning: 1. Learning is best conceived as a process, not in terms of outcomes. 2. All learning is re-learning. 3. Learning requires the resolution of conflicts between dialectically opposed modes of adaptation to the world. 4. Learning is a holistic process of adaptation. 5. Learning results from the synergistic transaction between the person and the environment; and 6. Learning is the process of creating knowledge. (4) Therefore, the educational process must be focused on engaging students where it best enhances their learning since learning is not just a cognitive process but a thinking, feeling, perceiving and behaving one. Kolb’s resulting Experiential Learning Theory Model (1984) features two dimensions. The first focuses on grasping experience through concrete experience and abstract conceptualization. The second dimension, transforming experience, looks at reflective observation and active experimentation. Andresen, Boud, & Cohen (2000) adopt a similar perspective suggesting that for learning to be experiential, three necessary conditions must exist: 1. The whole person must be involved. 2. The learner’s relevant life and learning experiences must be recognized and actively used. 3. There must be continued reflection on earlier experiences in order to add and transform them into deeper understanding. Once experiential learning activities are developed and implemented in a course, their efficacy in achieving identified learning goals must be assessed. The following section of the paper expands the notion of appropriate assessment of student learning within the context of EBL curriculum. Assessment Assessment of student learning has been studied for decades (Palomba & Banta, 1999; Huba & Freed, 2000 and Walvoord, 2004) with four core principles emerging: 1. Assessment must be defined in terms of improving student learning. 2. Must be the purview of faculty. 3. Must be implemented through course assignments. 4. Must be a collaborative venture. As assessment tools are developed, these four principles suggest a necessary focus upon and alignment of course outcomes, context and content, the prior knowledge and experience of students, and instructional strategies and resources (McConnell & Doolittle, 2014). Culver and Van Dyke’s assessment cycle model (2009) begins with the identification and articulation of student learning outcomes followed by gathering and analyzing information with regard to the achievement of those outcomes. Finally, the information gathered is used to improve student learning. Clearly, EBL activities are a critical component of business school curriculum, and student achievement of learning objectives in those EBL activities should be assessed. So how exactly are experience-based learning activities currently operationalized in curriculum within the context of student learning outcomes? Further, what are the tools that are used to gather information on the achievement of student learning outcomes aligned with EBL activities? Page 101 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 RESEARCH BACKGROUND In the fall of 2006, the undergraduate and graduate business programs at a major Middle Atlantic research institution began to focus on incorporating experience-based learning into their programs as a stated initiative in an AACSB reaccreditation report. At that time, an EBL committee was established with financial support for faculty area chairs to attend seminars, conferences and other events that would help define, educate and further develop EBL in the curriculum. Over the course of that academic year, a series of workshops were held for all school faculty members to introduce the concept of EBL to them and provide them with specific examples of such activities. In addition, school-based grants were offered to facilitate development of EBL activities for the classroom, knowledge gathering on the topic, or demonstrations of successful implementation of experience-based activities. Throughout the ensuing years, experience-based learning has become a cornerstone of the curricula at both the undergraduate and graduate levels and has become a key artifact and value of the culture of the school. EBL is a core proposition in the strategic mission of the school as reflected in its Spring 2018 AACSB Reaccreditation Report where a goal of merging knowledge and practice to create exceptional experience-based learning outcomes for students was offered. During students’ time in both the graduate or undergraduate programs, the role of experience-based learning is maximized both as a pedagogical and developmental tool. In short, EBL is part of the very fabric of the college and is the backbone of the curriculum. For academic year 2018-19 a new initiative has been developed to further enhance teaching effectiveness in the business programs. Although this is a research institution, good and exceptional teaching has regularly been recognized and valued with many members of the faculty receiving university and national recognitions and awards for outstanding teaching in their specific disciplines or more broad-based areas. Regardless of the accolades received, there is a culture of continuous improvement and thus a teaching effectiveness committee has been established to support this program. One facet of the committee’s plan is to enhance EBL activities. This multi-faceted study is part of the initiative of the teaching effectiveness committee’s mission. RESEARCH DESIGN Though a number of definitions of experience-based learning exist and some authors have attempted to discern types of assessment methods used with specific methods (Gosenpud & Washburn, 1996; O'Keefe & Chadraba, 2013) few have looked at the content, structure and application of evaluation methods utilized with those EBL activities as they assist in determining grades for students enrolled in courses with embedded EBL activities. This study seeks to provide some preliminary findings in this area. Faculty members in the graduate and undergraduate divisions at a large Middle Atlantic research university were interviewed about their use of EBL in their classes. All faculty members are required annually to self-report the percentage of EBL activity in each course in comparison to the course as a whole. After self-reporting that percentage, faculty members are required to list and describe the EBL activity or activities in the class. Identified faculty members in each discipline are also being interviewed by the researchers. This intake interview opens with the question: What kind of evaluated EBL activities do you have in your course which are aligned with the course learning objectives? Participating faculty members then list and describe EBL activities that are part of their course and are evaluated for a student’s grade. The resulting findings will be categorized by type of EBL activity, discipline of faculty member and class title. Size of class, online/face to face nature of class, undergraduate/graduate designation and time schedule for class were also obtained. Following the listing of the EBL activities, faculty members are asked: How are these EBL activities evaluated? Any evaluative tools like rubrics are requested. In addition, faculty members are questioned as to whether their rubrics or other evaluative forms and formats were revealed to students prior to the actual performance and assessment of the activity. Finally, faculty members are asked whether students have commented on the evaluation tools in emails, in person or in teacher evaluations and are asked to characterize that commentary or supply it if possible. Once rubrics and other evaluative forms are obtained, each individual assessment method is subjected to a qualitative analysis. Evaluative tools will be categorized as to type (rubric, log, etc.), whether they assess content, delivery, process or other in the activity. In addition, the nature of the scaling used for evaluation is noted. This same information and material will also be gathered from ABSEL faculty members at other universities throughout the world and will be assessed as a point of potential comparison to findings obtained for the target university in future stages of this project. PRELIMINARY FINDINGS Faculty self-report information on the types of experiential-based learning activities utilized in a large research oriented Middle Atlantic business school was examined. Across the disciplines faculty members identified the following activities as EBL activities: 1. Written case (purchased from sources such as Harvard Business Publishing) 2. Written case (developed by instructor) Page 102 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 3. Written case (developed by student) 4. Live case (purchased) 5. Live case (delivered on video) 6. Live case (delivered in person) 7. Field trip 8. Field trip in the context of a study abroad program/course 9. Speaker 10. Company/organization-based exercise 11. Simulation (purchased) 12. Game 13. Consulting projects with real companies 14. Developed projects based on real companies 15. Internships 16. Competitions Individual faculty members interviewed to date for further detail on the nature of assessment tools used in evaluating student performance on these EBL activities suggest that they are using rubrics, journals or logs, problem examples and process sequences. The following table presents examples from three disciplines in the focus business programs and will serve as the framework for understanding choice of EBL activity and the accompanying evaluation. The table’s design is based on the ADDIE training model originally developed by Florida State University for the U.S. Army (Blanchard & Thacker, 2013). ADDIE suggests that the objectives of a course inform the activities utilized within the course with the final step, evaluation, examining the level of achievement of the initial goals. In short, the table offers a framework for examining the types of EBL activities used in light of course objectives and the tools used to discern attainment of those objectives. PRELIMINARY OBSERVATIONS Instructor course learning outcomes are usually very clearly noted and denote subject-specific orientations as well as more general career oriented goals. However, the alignment between the EBL activities and those outcomes are not as clear as they might be in all cases. Conversations sometimes led the researchers to discern that specific EBL activities were chosen that might not be as aligned with outcomes as others might be. An anecdotal case in point: An instructor of an elective class in the graduate programs reported that the choice of EBL activity, a client project, was implemented in the online class even though the fit with the overall course objectives was not as “tight” as it might be. The core course objectives of the MBA “survey” class focus on acquainting the student with all functional areas of the discipline. The EBL project however only focused on three of five core areas. Though the activity was not as comprehensive in its orientation as it might have been, the instructor, wanting to maintain a relationship with a corporate partner of the school, decided to implement the project. Research by McConnell & Doolittle (2014) emphasizes the importance of alignment at each step in the assessment cycle process. As the sample size increases, this topic will be monitored for impact on overall student outcomes. Even the small sample size thus far has indicated that the meaning of EBL activities continues to expand with regard to the manner in which the concept is operationalized. Previous work on the nature of EBL activities has tended to use broad categorizations for EBL activities. However, the refinements of each has offered new avenues of addressing student outcomes and warrants a full review and understanding of each. Now that a current baseline has been established as to how EBL activities are delineated and how they are assessed, more specific research in identified areas can be undertaken. For instance, these very preliminary findings have led to a currently ongoing experiment related to discerning the impact on student outcomes of EBL activities and their assessment. A common curriculum for a core required class in the undergraduate college of business including a universal EBL client project is offered to 300 students each semester by four instructors. Common rubrics are used for evaluating the project’s deliverables, a written report and presentation for the client. The nature of and method for delivering feedback will be manipulated and compared to a control group to understand impacts on student learning. Data is currently being collected and will also be collected in the future to gain temporal and comparison effects. As indicated earlier, one of the intentions behind this preliminary benchmarking study was to develop specific teaching effectiveness workshops. One possible workshop arising from these findings, intended for junior faculty, is simply a look at the types of activities used by those in the same discipline and how those used in other disciplines vary. Understanding the nature of these activities and the reasons each are implemented in the classroom allows each discipline to gain a better understanding of their counterparts’ areas, invites interaction in teaching and research and enhances the collegial culture of the school. Other workshops cutting across disciplines and levels in the faculty structure will also be developed based on the current findings and the results of phase two of this research project. Although no clear conclusions can be drawn from the limited data collected to date, what has been gathered indicates that the EBL initiative has been widely instituted and expanded at this university. Individual instructors are offering exciting projects and exercises to their students which expose and prepare those students for challenges and situations they will face in the workplace. Systematic focus on individual details of those projects and clarification of tools used to evaluate each will continue to enhance student preparation and development. Page 103 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 T A B L E 1 D IS C IP L IN E S P E C IF IC E B L A C T IV IT Y A N D A C C O M P A N Y IN G A S S E S S M E N T T O O L D is ci p li n e C o u rs e L ea rn in g O b je ct iv es E B L A ct iv it y S tu d en t A ct iv it y S u m m a ti v e o r F o rm a ti v e T o o l D im en si o n s H u m an R es o u rc es S ta ff in g (U n d er g ra d u a te M aj o r R e- q u ir ed ) (1 ) In tr o d u ce s ta ff in g c o n ce p ts a n d p h il o so - p h ie s. (2 ) P re p ar e fo r H R c ar ee r b y d ev el o p in g sk il l se t in t h e fu n ct io n i n cl u d in g i d en ti fi ca - ti o n , an al y si s an d a p p li ca ti o n . (3 ) Im p ro v e in te rp er so n al a n d l ea d er sh ip sk il ls . C o n su lt in g P ro je ct W ri tt en R ep o rt O ra l G ro u p P re se n ta ti o n S u m m at iv e R u b ri c: A ss es s ex te n t o f co v er a g e o f sp ec if ic d is ci - p li n e b as ed e le m en ts o n a 0 t o 5 s ca le A ss es s st ra te g ic a li g n m en t o f d is ci p li n e el em en ts Q u al it y o f w ri tt en e x p re s- si o n Q u al it y o f o ra l p re se n ta - ti o n D at ab as e E x er ci se A n al y si s o f d at a- b as e fo r tr en d s R u b ri c: A cc u ra cy V ia b le r at io n al e fo r ap - p ro ac h A cc o u n ti n g A cc o u n ti n g R es ea rc h a n d W ri ti n g ( M S A cc o u n ti n g E le ct iv e ) (1 ) F a m il ia ri z e st u d en ts w it h p ri m ar y so u rc es f o r fi n an ci al a cc o u n ti n g r es ea rc h . (2 ) Im p ro v e c ri ti ca l th in k in g s k il ls . (3 ) Im p ro v e w ri tt en p ro fe ss io n al c o m m u n i- ca ti o n s k il ls . C as es o n C u rr en t Is su es ( fo r ex am p le , re q u ir in g s tu d en ts to l is te n t o c u rr en t d el ib er a- ti o n s o f th e F A S B r e: t h e n at u re o f b it co in a n d c ry p to - cu rr en c y a n d h o w i t sh o u ld b e ac co u n te d f o r o n a c o rp o - ra ti o n ’s f in an ci al s ta te m en ts O ri g in al r es ea rc h W ri te w ri tt en r e- p o rt C o n tr ib u te t o c la ss d is cu ss io n o n t o p - ic C an b e b o th d ep en d in g o n th e n at u re o f th e ca se E x te n t o f co v er ag e o f ea ch le ar n in g o b je ct iv e o n a 5 p o in t sc al e fr o m e x ce ed s re q u ir em en ts t o b el o w ex p ec ta ti o n s Q u al it y o f w ri tt en e x p re s- si o n A cc o u n ti n g M B A F in an - ci al A cc o u n t- in g ( C o re C o u rs e) H el p s tu d en ts u n d er st an d t h e b as ic s tr u ct u re an d s u b st an ce o f a fi rm ’s f in an ci al s ta te - m en ts i n o rd er t o (1 ) E n ab le s tu d en ts t o u se t h e fi n an ci al i n - fo rm at io n f o r d ec is io n m ak in g . (2 ). T o u n d er st an d w h at i n fo rm at io n i s (a n d w h at i s n o t) i n cl u d ed i n t h e fi n an ci al s ta te - m en ts , an d h o w a n d w h en e co n o m ic e v en ts af fe ct t h e fi n an ci al s ta te m en ts . (3 ) U n d er st an d t h e im p ac t o n t h e fi n an ci al st at em en ts o f m o re a d v an ce d t o p ic s an d m o re c o m p le x t ra n sa ct io n s. (4 ) A cq u ir e an u n d er st an d in g o f th e in fe r- en ce s th at c an b e m ad e fr o m f in an ci al i n fo r- m at io n a b o u t th e p as t o p er at io n s, p re se n t p o si ti o n , an d f u tu re p ro sp ec ts o f a fi rm . R es ea rc h P re se n ta ti o n P o w er p o in t p re se n ta ti o n , 3 -4 p er so n g ro u p m ad e to t h e in - st ru ct o r an d a sm al l au d ie n ce ; d es ig n ed t o r ep li - ca te a m ee ti n g i n a co rp o ra te s et ti n g ra th er t h an a c la ss - ro o m p re se n ta ti o n P ee r re v ie w B o th Q u al it y a n d c la ri ty o f sl id es T ea m w o rk C la ri ty o f id en ti fi ca ti o n o f ac co u n ti n g i ss u e C la ri ty o f d es cr ip ti o n o f im p ac t o n f in an ci al s ta te - m en ts Im p ac t o n f in an ci al r at io s In cl u si o n o f to p ic e x te n - si o n s th at m ak es s en se H u m an R es o u rc es E m p lo y m en t & L ab o r R e- la ti o n s (U n d er g ra d u a te M aj o r R e- q u ir ed ) (1 )T o e n su re s tu d en ts u n d er st an d t h e el e- m en ts o f a d iv er se o rg an iz at io n . (2 )T o e n su re s tu d en ts u ti li ze c o u rs e co n - ce p ts i n b u il d in g a d iv er se o rg a n iz at io n . (3 )T o e n su re s tu d en ts r ec o g n iz e th e o b st a- cl es a n d b en ef it s to c re at in g i n cl u si v e o r- g an iz at io n s. P ar tn er w it h a H u m an R e- so u rc es D ep ar tm en t to u n - d er st an d t h e d ev el o p m en t an d i m p le m en ta ti o n o f a d iv er si ty i n it ia ti v e. C la ss P re se n ta ti o n S u m m at iv e C o n te n t o f fi n d in g s D el iv er y o f fi n d in g s P re se n ta ti o n o f m aj o r co m p o n en ts o f d iv er si ty p ro g ra m . Page 104 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 Andresen, L., Boud, D., & Cohen, R. (2000). Experience-based learning. Understanding adult education and training, 2, 225-239. Association to Advance Collegiate Schools of Business. (2002). Management education at risk: Report of the management education task force to the AACSB International Board of Directors. Tampa, FL: AACSB International. Blanchard, P.N., & Thacker, J.W. (2013) Effective training: Systems, strategies, and practices. Saddle River, N.J.: Pearson. Culver, S.M., & Van Dyke, R. (2009). Developing a receptive and faculty-focused environment for assessment. In C. Schreiner (Ed.) Handbook of research on assessment technologies, methods and applications in higher education (pp. 337-347). Hershey, PA: Information Science Reference. Gosenpud, J., & Washbush, J. (1996, March). Correlates of learning in simulations. In Developments in Business Simulation and Experiential Learning: Proceedings of the annual ABSEL conference (Vol. 23). Hoover, J. D. (1974, March). Experiential learning: Conceptualization and definition. In Developments in Business Simulation and Experiential Learning: Proceedings of the annual ABSEL conference (Vol. 1). Huba, M.E. & Freed, J.E. (2000). Learner centered assessment on college campuses: Shifting the focus from teaching to learning. Boston, MA: Allyn & Bacon. Kolb, D.A. (1984). Experiential learning: Experience as the source of learning and development. Englewood Cliffs, N.J.: Prentice Hall. McConnell, K.D. & Doolittle, P.E. (2014). Classroom level assessment: Aligning pedagogical practices to enhance student learning. In C. Secolsky & D.B. Denison (Eds.) Handbook on measurement, assessment and evaluation in higher education. Abingdon, United Kingdom: Routledge Taylor & Francis. Palomba, C. & Banta, T. (1999). Assessment essentials: Planning, implementing and improving assessment in higher education. San Francisco: Jossey-Bass. O'Keefe, R. D., & Chadraba, P. C. (2013). Suggestions for the preparation, presentation and evaluation of case study reports in strategically-allied global business education programs. International Journal of the Academic Business World, 7(2). Walvoord, B.E. (2004). Assessment clear and simple: A practical guide for institutions, departments and general education. San Francisco: Jossey-Bass. REFERENCES