Effectiveness of Student co-created Flip: A comparative Study in a Marketing Principles Class Page 86 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 ABSTRACT Experiential learning has gained tremendous importance in recent times both from a pedagogical perspective as well as accreditation perspective. Flipped classroom is one such experiential approach which has many proponents but not enough evidence to support its adoption by educators. Theoretically derived from the notions of Consumer as co-creator (Vargo and Lusch, 2004) and the Co-Creative Educational Model (Geddes et. al., 2017), this paper describes the student co-created flip classroom approach. The paper also presents comparative effectiveness of the flipped classroom with a traditional classroom in terms of student outcomes in an introductory marketing class. INTRODUCTION For the things we have to learn before we can do them, we learn by doing them -Aristotle The above quote nicely summarizes why experience in the context of learning is important. Experience provides an opportunity for the learner to get involved in the learning process. Kolb (1984) defines experiential learning as a “holistic integrative perspective on learning that combines experience, cognition and behavior”. Kolb & Kolb (2005) also describes learning as a cycle with four distinct stages Thinking (Abstract Conceptualization), Feeling (Concrete Experience), Reflecting (Observation) and Acting (Active Experimentation). Whereas traditional pedagogical approaches do a good job of fostering thinking and reflecting components of learning, their effectiveness on the experiencing and acting components are quite limited thereby affecting effective learning process. Experiential learning, on the other hand has the ability to bridge the gap in traditional approaches by fostering experiencing and acting components of the learning cycle. Experiential learning is also of importance to business educators as a critical component of their institutional accreditation. AACSB, the premier accrediting organization for Business schools, expects “students [to] engage in experiential and active learning to improve skills and the application of knowledge” as part of their accreditation requirement (AACSB, 2013). The face of education is changing with the recent advances in technology. These technological advances have significantly brought down the cost of information availability through sources like Youtube® Videos, Khan Academy, MOOCs etc. This low cost information availability has also resulted in an ideological change in the education system that calls for letting learners learn at their own pace. A flipped classroom provides a conducive environment for incorporating these changing trends and still maintain a more traditional approach to education. The key objectives of this study are twofold – (i) To describe a flipped class approach where students take an active role as co-creators in designing and implementing the flip activity and (ii) to present comparative evaluation of the impact of a flipped class approach to a more traditional lecture based approach on student outcomes. The remainder of the paper will be organized as follows. First a review of literature will be performed to provide an overview of the flipped classroom approach and justification for the students as co-creators approach in the flip classroom. The next section will describe how the student centric flip was implemented in the classroom. Then details of the methods for comparative assessment will be presented along with the findings of the analysis. The paper will finally end with the discussion of the results and conclusions. REVIEW OF LITERATURE The Flipped Classroom Flipped Classroom is one of many experiential approaches available to educators in business and other programs whereby the principles of experiential learning can be implemented in the context of learning. According to Harris et. al. (2016), a Flipped Classroom is defined as a “learning environment that provides students with a variety of means to study basic knowledge content as part of the pre-class-meeting homework, so teachers can use class time more effectively for hands-on activities to practice, apply and demonstrate mastery of the content learned from the pre-class requirements” (p.326). Thus from this definition, it is clear that flipped classroom aims to address all the four stages of learning as defined by Kolb. According to Lage, Platt and Treglia (2000), a key justification for flipping the classroom is the apparent mismatch between the teaching style of the teacher and the learning style of the student. This mismatch may result in students being less interested and thus less involved in the overall learning process. One way institutions could address this problem is to ensure that this mismatch between teaching and learning styles is minimal which is harder to implement. By adopting the flipped classroom approach, lower level learning components (lectures, assigned readings etc.) are completed outside the class at the student’s own pace and prepare Effectiveness of Student co-created Flip: A comparative Study in a Marketing Principles Class Raghu Kurthakoti Arcadia University, kurthakotir@arcadia.edu mailto:kurthakotir@arcadia.edu Page 87 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 them to be on par with each other by the time of class meeting. This similar level of understanding among all learners irrespective of their speed of learning will enable the class time to be more fruitfully devoted to developing mastery of the content. A flipped classroom, by its nature provides opportunities for lower level learning (Remember, Understand) to happen outside the classroom whereas classroom time can be effectively used to foster higher level learning (Apply, Analyze, Evaluate and Create). The Student Co-Created Flip Vargo and Lusch (2004) propose a dominant logic for marketing where service provision is envisioned as the primary tool of exchange. They propose that value is not determined by the producer but is perceived and determined by the consumer based on how they decide to act on the offerings of the producer. They also propose the notion that the consumer is always a co-producer in the exchange process. This is because the consumer derives value from the producer’s offering by acting upon it and is therefore critical to the value generation process. If one considers a traditional learning environment in this context, the instructor takes on the role of the producer and the student takes on the role of consumer. The value of the course offering is defined by the instructor and a series of unidirectional activities (from producer to consumer) are then initiated to deliver this value (e.g. lectures). However, the value of the learning process is often defined by the consumer on what they can take away from the course (evidenced through teaching evaluations where students are often asked to respond questions that sound similar to – how valuable was this course to you). Such consumer defined value will often act as the basis for the assessment of the instructor and the course outcomes. Thus it becomes important that the consumer (student) be involved in defining and create value in the exchange (learning) process In 2015, Geddes et al., apply this notion of service dominant logic to the domain of education and learning. By applying the service dominant logic, they classify educational strategies into two types – Informational strategy and Transformational Strategy. The informational strategy resonates more with the traditional classroom where high quality knowledge is imparted from the instructor to the student who is a passive receiver of this knowledge. The transformational strategy, on the other hand, focuses on inspiring (and thereby transforming) students to be active learners under the guidance of the instructor. The notion of the flipped classroom is related to the notion of transformational education strategy where students become active learners in pursuit of mastery under the guidance of the instructor in a classroom. Taking this notion of service dominant logic in educational environment further, Geddes et al. (2017) suggest that all education is necessarily co-creative and the extent of experiential learning experience is dependent on the degree of co-creative effort invested in each of the stages of experiential learning (Geddes et al., 2017, p61). Flipped classrooms provide a higher degree of co-creation than a traditional didactic classroom. However, the flow of information is still primarily coming from the instructor to the students albeit with more opportunities to experience and act than in a traditional classroom. The instructor is primarily involved with lesson planning to design and execute the various activities that will be done during class time to reinforce the knowledge acquired outside the classroom through online lectures. Thus, although on one hand, a Flipped classroom is more experiential in nature, on the other hand it still limits itself to be an instructional strategy and does not represent a truly transformational strategy. As seen from the discussion on the service dominant logic and its application to educational settings, value is defined by the consumer (student), but an experiential approach like flipped classroom still relies on actions originating from the instructor to deliver value. A truly transformational education strategy rooted in the service dominant logic should therefore actively involve the students in creating the flipped classroom, not just as mere participants. This involvement in the co-creation process is likely to enhance their motivation (as an outcome of transformational education strategy) to seek and gather information under the guidance of the instructor thereby co-creating value in the learning process. With this motivation and rationale, a principles of marketing class at a small private liberal arts university in the Northeastern part of the U.S. was taught in a student co-created flipped classroom setting. The next section of the paper briefly describes the details of how the student co-created flipped classroom was operationalized. METHODOLOGY The marketing principles class in question typically enrolls 20-25 students in each section. Even before the implementation of the flip methodology to the class, the class was provided opportunities for experiential learning through simulations and case analysis. By adopting the flipped approach, the extent of experiential component of the course was significantly enhanced. To enhance student clarity about the nature and expectations of a flipped classroom, the course was divided into 8 modules as follows – Module 1 – Introduction to Marketing and Marketing Strategy; Module 2 – Marketing Enviromnent; Module 3 – Consumer Behavior; Module 4 – Segmentation; Module 5 – Products; Module 6 – Distribution; Module 7 – Pricing; Module 8 – Promotions. Most of the students enrolled in the class were unaware of the notion of the flipped classroom and had mostly participated in traditional lecture based classes with some experiential components (projects, cases, simulations etc.). Therefore, to ease the students into the flipped classroom the first module was conducted in a traditional lecture based format. The second module onward were flipped. However, to minimize the anxiety of students and provide clarity on what it means to flip a class, Module 2 (Marketing Environment) was demonstrated as a flipped class by the instructor. A set of activities related to the topic was designed by the instructor and the activities were implemented in the classroom. To further enhance the clarity of expectations from the students, the mechanism of flipping and the expected outcomes of the flipped activity was provided in detail to the students through a form (See Appendix 1 for a completed form for Marketing Environment) The class was divided into six groups (self-selected groups) and each group was randomly assigned one of the remaining six Page 88 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 modules to lead with a flipped activity. The expectation was that students, in close consultation with the instructor, will research and create a suitable activity for the class to engage in to achieve a predetermined set of outcomes. Each group was expected to complete items 4-6 of the form similar to Appendix 1 (with specific outcomes for their module) and provide clear details of what the class will be expected to do, how the activity will be debriefed and finally what learning will be achieved. (See Appendix 2 for a sample form completed by students for a different module). Item 7 on the form provided the rubric for student grading and thus gave a guidance to students on what aspects to focus on during the creation of the flip activity. Since students were actively involved in the process of researching and creating an activity to be implemented in the classroom under the guidance of the instructor, the entire process fits the notion of co-created learning environment. Extensive meetings were held between the instructor and the team leading the flip to (a) fill conceptual gaps in understanding of the material (b) refine and develop a focused activity that would be creative and engaging for the participants and (c) ensure adequate and focused debriefing to achieve identified outcomes. Typically, the team and the instructor had three face to face meetings lasting about 20 minutes each and a constant collaborative work online, developing the form and creating the activity through a shared GoogleDocs®. DATA AND ANALYSIS In Fall 2018, two sections of the principles of marketing class was taught by the author. One of the two sections had an enrollment of 21 and the other section had an enrollment of 10. Given the nature of the course design and the past experiences of the author with small classes, it was deemed that using a flip in the smaller class was likely to be less valuable due to the limited number of students available for active participation. Therefore, the author made a decision to teach the smaller class in a traditional lecture format and the larger class in the flipped format. All grade components of the course (tests, assignments, papers, simulation etc.) remained the same across both sections. The only difference was in the pedagogical approach employed in each class. Thus this situation provided an opportunity to directly compare and contrast the effectiveness of the pedagogical approach on student outcomes. TABLE 1 STUDENT SCORES ON COURSE COMPONENTS ACROSS PEDAGOGIES Course Component Max Score Possible Flip Class Lecture Class Mean Difference % Difference M SD M SD Flip - Lecture A B C D E F G H Quiz 40 34.25 4.66 31.10 7.74 3.15 7.88 Test 60 43.03 5.03 46.55 7.01 -3.53 -5.88 Assignment 80 64.73 7.03 62.45 5.61 2.27 2.84 Team Paper 30 24.65 2.60 24.50 3.21 0.15 0.50 Individual Paper * 40 29.70 5.06 25.30 8.62 4.40 11.00 Group Simulation 90 77.80 5.13 80.20 5.81 -2.40 -2.67 Individual Simulation 90 75.75 7.67 77.19 5.30 -1.44 -1.61 Cases ** 20 16.15 1.31 13.68 3.08 2.48 12.38 Total 500 411.25 27.32 397.82 35.78 13.43 2.69 * p < 0.10 ** p < 0.05 Description of the Course Components • Quiz – Students were given a multiple choice quiz solely based on facts presented in the textbook to ensure that they read the material before class. • Tests – Tests were application based multiple choice questions related to concepts of the class covered in the presentations • Assignments – Assignments were intended to comprehend and apply the concepts of marketing to personal situations and draw inferences related to marketing • Team Paper was a comprehensive case where students were expected to develop a marketing plan. Students did this as pairs • Individual Paper was a compare and contrast paper where students were expected to critically analyze the marketing strategies of a firm/brand in two countries. • Group Simulation – Students’ performance on a simple marketing simulation based on combination of marketing, financial met- rics • Individual Simulation – Simulation performance weighted by students’ individual contribution based on peer evaluations • Cases – Students had to analyze two cases that dealt with issues related to marketing environment • Total – Total score of students (out of 500 points) that included all the above components and other miscellaneous components like Class Participation, Simulation Quiz, Presentations etc. Page 89 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 For the purpose of this comparative analysis, all course deliverables (tests, assignments, case analysis, simulation performance, term papers etc.) were compared between the two sections. In addition, a survey was administered to both sections seeking their feedback on the pedagogical approach as well as information about their preferred learning style. These survey responses were also compared across the two pedagogical approaches. During initial analysis of data, it was found that one of the students in the flip class demonstrated outlier behavior due to issues beyond control during the course. So the data pertaining to this student was eliminated from the analysis. The final data set was comprised of 20 students in the flip class and 10 students in the lecture class. RESULTS AND DISCUSSION As discussed in the Methodology section, different types of analysis were planned to compare the effectiveness of the pedagogical approach on student learning outcomes. This included quantitative assessment of grades of students across course deliverables, a survey to gather student feedback and perception about the pedagogical approach, and qualitative analysis of the students’ written assignments using content analysis. Given the timing of the course, results of only the quantitative assessment of student grades on course deliverables will be presented and discussed in the paper. Qualitative and student perception analysis will be presented at the Conference as the data is still being analyzed. Table 1 presents the average scores of the students in various course deliverables across both pedagogical formats. As can be seen from Table 1, the Flip Class demonstrated a higher score on most components of the course. Of these, two components that primarily dealt with articulation and insightful analysis of marketing situation viz., Individual paper and Individual Cases showed a significantly higher mean score for the flipped class compared to the lecture class. Literature suggests that instructors use case studies and analytical reports to help assess higher order learning outcomes in students (Ben-Zvi, 2010; Anderson & Schiano, 2014) This indicates that the flipped pedagogy was a contributing factor in achieving these higher order learning outcomes for the students. Further, when one analyzes columns G and H from Table 1, we see that for these two components the flipped class resulted in over 10% improvement in students’ grade compared to the lecture only mode. Although differences for other course components were statistically non-significant (possibly due to smaller sample sizes), it is nevertheless important to analyze and discuss the directionality of the results obtained. Average scores on Quizzes, and Assignments were more for the flipped class than the lecture class. This could be possibly due to the fact that not having formal lectures, but requiring students to lead a module put the ownership on the students. This ownership of the course content possibly brought in more responsibility for the students by being pro-active in their preparation leading to higher scores on the quizzes. For the lecture mode, this was not an expectation and the lack of ownership is likely to have made students less enthusiastic to prepare prior to class as they knew that the materials would be covered in detail during lectures. Similarly, assignments required students to analyze marketing concepts and relate them to personal experiences (e.g. Shopping behaviors, segmentation and position strategies etc.). The responsibility owing to content ownership and gaining practice through creation of flip activities might have resulted in a higher score for the flip class. Additionally, given that lectures were provided on the Learning Management System (LMS), class time was sometimes utilized for students to work together on certain complex assignments with strategic input from the instructor for clarification purpose. Due to lack of time for such activities in the lecture class, those students did not have that opportunity and had to rely on assignment instructions and very generic help from the instructor. Although the total score was not statistically different between the two classes and the percentage difference was only about 2.7, it has to be noted that this difference actually suggests a difference of one letter grade for the class. This means, on an average the students in the flipped class secured a final grade that was one letter grade better than the average for the lecture class. Thus, it appears to be a very significant outcome of the flip pedagogy for the students. Although the tests were application based, they were still multiple choice tests and were based on the concepts discussed in the lectures. For the flipped class, since the lectures were posted online, the level of student comprehension was not observable by the instructor. Also these lectures were static and there was little scope for adaptation and mid-course corrections based on student feedback. Additionally, the student centric flip was able to achieve depth in only certain concepts for each course module and not all concepts were dealt with in equal detail. Thus it is possible that some concepts were not comprehended fully by the students resulting in a weaker score on certain sections of the test. For the lecture class, on the other hand, the instructor could modify delivery mid-course depending on student body language and so this might have helped students comprehend the concepts better and do well on the tests. Also tests were a small portion of the overall course grade (about 12%) and this low weight of tests might have made some students complacent to attempt and do better on the test. However, it is possible that a flipped class with occasional brief lectures (to cover concepts not covered in the flip activity) might be able to supplement the online lectures and thus help students score better on the tests. It is also interesting to see that the lecture class performed (directionally) better on simulation compared to the flip class. The directionality of the results is counter intuitive and is further supported by statistical non-significance. However, it is possible that with absolutely no opportunity for experiential learning in the lecture class until the introduction of simulation, the introduction of an experiential exercise might have triggered higher involvement resulting in better performance on the simulation. The flipped class, on the other hand, had numerous opportunities to engage in experiential learning and a computer based simulation might not have excited them as much as the flip activity (which gave them total ownership and creative freedom to design experiences for the class) making it less engaging for the students. LIMITATIONS AND CONCLUSION As with any study, this study faces certain limitations. First, the sample sizes in the study are extremely limited and as such Page 90 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 these results can be treated as a pilot study. This implies further large scale sample based studies need to be conducted to clearly document the specific advantages of the flipped class approach. Second, as with any between-subjects design, this study assumes that students in both classes are identical in all other aspects, but there was no attempt made to gather additional data in the survey from the students (partly because of IRB restrictions due to relationship between participants and researcher and partly to minimize workload for students. The course by itself was very intensive and the additional burden on students to gather information was seen as counterproductive to achieve desirable sample sizes). Future research should conduct a more rigorous controlled study to isolate the true effects. Third, although the grading of the qualitative components (papers, assignments etc.) were done by the same instructor using the same rubric for both classes, some variability in grading might have been introduced due to the fact that grading was relative to performance in each class independent of the other. This might have made in the scores reported in Table 1 less reliable than desired. A more objective assessment, for example, by using a content analysis software might be able to mitigate this variability and provide a clearer picture. Finally, there was no attempt made to analyze the relationship between students’ performance on the tests and the topics/concepts covered in the flipped class. Future studies might want to address this aspect specifically to see if flipped class demonstrates higher scores on specific concepts covered through flipped modules. Such studies might help instructors in creating flipped activities for specific concepts to enhance student understanding and helping them perform better on assessments. Finally, due to lack of time for proper analysis, students’ qualitative data pertaining to their perception and feedback on the pedagogy could not be analyzed. This information will be presented to the audience during the ABSEL meeting in San Diego. In conclusion, this study provides an overview of a student centric flipped classroom as an experiential pedagogy. Based on Vargo & Lusch’s (2004) notion that the consumer is a co-creator in the consumption process, and the co-creative education model proposed by Geddes et. al. (2017). Initial results of a comparative assessment of student performance and outcomes of such a flipped class with a more traditional lecture based class in a marketing principles course indicated that the flipped class does provide some significant advantages to students enabling them to perform better in certain specific course components. AACSB Accreditation Standards pertaining to Academic and Professional Engagement (2013). Accessed from http://www.aacsb.edu/en/accreditation/standards/2013- business/academic-and-professional-engagement- standards/ Anderson, E., & Schiano, B. (2014). Teaching with cases: A practical guide. Harvard Business Review Press. Ben-Zvi, T. (2010). The efficacy of business simulation games in creating Decision Support Systems: An experimental investigation. Decision Support Systems, 49(1), 61-69. Geddes, B. C., Cannon, H. M., Cannon, J. N., & Feinstein, A. H. (2015, February). Developing educational strategies for experiential learning: An application of service dominant logic from marketing. In Developments in Business Simulation and Experiential Learning: Proceedings of the Annual ABSEL conference (Vol. 42). Geddes, B. C., Cannon, H. M., & Cannon, J. N. (2017). Conceptualizing Co-Creative Strategies in Experiential Education: Individual versus Group Approaches. In Developments in Business Simulation and Experiential Learning: Proceedings of the Annual ABSEL conference (Vol. 44, No. 1). Harris, B. F., Harris, J., Reed, L., & Zelihic, M. M. (2016). Flipped Classroom: Another Tool for Your Pedagogy Tool Box. Developments in Business Simulation and Experiential Learning, 43(1). Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. Englewood Cliffs, NJ: Prentice Hall. Kolb, A. Y., & Kolb, D. A. (2005). Learning styles and learning spaces: Enhancing experiential learning in higher education. Academy of Management Learning & Education, 4(2), 193–212. Lage, M. J., Platt, G. J., & Treglia, M. (2000). Inverting the classroom: A gateway to creating an inclusive learning environment. The Journal of Economic Education, 31 (1), 30-43. Vargo, S. L., & Lusch, R. F. (2004). Evolving to a new dominant logic for marketing. Journal of marketing, 68 (1), 1-17. REFERENCES http://www.aacsb.edu/en/accreditation/standards/2013-business/academic-and-professional-engagement-standards/ http://www.aacsb.edu/en/accreditation/standards/2013-business/academic-and-professional-engagement-standards/ http://www.aacsb.edu/en/accreditation/standards/2013-business/academic-and-professional-engagement-standards/