Australasian Journal of Educational Technology, 2020, 36(3). 163 Introducing the flip: A mixed method approach to gauge student and staff perceptions on the introduction of flipped pedagogy in pre-clinical medical education Margaret Simmons, Deb Colville, Shane Bullock, Julie Willems, Michelle Machadoa, Adelle McArdle, Marianne Tare, Jayden Kelly, Mohammad Ali Taher, Sallyann Middleton, Marion Shuttleworth, David Reser Monash University aWarwick University, United Kingdom Flipped learning has become a popular blended learning approach in higher education and is now being adopted in medical schools across Australia and internationally. There are a number of principal educational justifications for the introduction of this approach, primarily, that it fosters deeper student learning through active engagement in the classroom. As a pedagogical intervention however, what do the various stakeholders think about its introduction? This paper explores reactions to implementation of a flipped learning approach to pre-clinical medical education in a regional Victorian medical course, via a mixed method approach. A range of quantitative and qualitative data was collected concerning the implementation, including a student survey, student focus groups, a staff survey for both academic and professional staff members involved in the implementation of the approach, and an independent student-driven social media questionnaire conducted in the second year post implementation survey. These data provide critical feedback for refinement of the flipped learning approach, including more robust student and faculty development and support during implementation of this pedagogy. Taken together, our results provide a unique perspective of the introduction of the flipped approach through different stakeholder lenses, and over time. Implications for practice or policy: • Innovative learning and teaching approaches should meet the needs of the modern student. • Tertiary education can be improved and efficiency increased by adopting a flipped classroom approach. • Change is often challenging; capturing the opinion of students and staff may enhance adoption and acceptance of new teaching and learning methods. • It is important to be reflexive in adopting new approaches with regular appraisal and evaluations in parallel allowing time for stabilisation and consolidation. Keywords: flipped learning, medical education, student experiences, evaluation, change management, academic development Introduction Traditional face-to-face teaching methods in medical education are being enhanced through, and sometimes replaced by, technology. As Williams (2016) notes, the cost and efficiency of technology for medical education now represents a paradigm shift for teaching and learning. This paradigm shift has given rise to “substantial recent expansion and innovation” (Prideaux, 2009, p. 100) in the pedagogy for teaching and learning in higher education, creating “disruptive innovations” (Mehta, Hull, Young, & Stoller, 2013, p. 1419). However, disruptive innovations may create complex problems (Schell & Mazure, 2015), notwithstanding staff unfamiliarity or discomfort with some of the ways they promise to disrupt teaching and learning (Adams Becker et al., 2018). Wang, Han, and Yang (2015) note that the integration of technology-mediated learning with campus-based learning has made learning increasingly complex, and that this complexity “lies not only in the emergence of new elements in teaching and learning, but also in the changes brought about by the interaction between these new elements” (p. 381). The growth of digital and blended learning methods in medical education is a case in point. Blended learning is a large umbrella term (Torrisi-Steele, 2011), of which flipped learning Australasian Journal of Educational Technology, 2020, 36(3). 164 (FL) is but one component (McLaughlin et al., 2014). Therefore, the introduction of any new pedagogical approach in higher education must account for a broad range of influences and contributing factors at play. Complexity approaches in education embrace, rather than attempt to simplify, the intricacies of educational decision making (Fraser & Greenhalgh, 2001), and the subsequent application of innovation. Here we report a mixed methods evaluation (Denzin, 1978) of the adoption of a FL approach to pre-clinical medical education in a regional Victorian medical education site, providing a multiple stakeholder perspective. Through a range of quantitative and qualitative data sources, including a student survey, student focus groups, a staff survey for both academic and professional staff members involved in the implementation of the FL approach, and an independent student-driven social media questionnaire conducted in the second year post implementation survey. Flipped learning FL is a pedagogical approach in which direct instruction in the delivery of content moves from the group learning space to the individual learning space, so that the resulting group learning space becomes the site of active inquiry and strategic application exercises (Honeycutt & Garrett, 2013). As such, the group space is transformed into a “dynamic, interactive learning environment where the educator guides students as they apply concepts and engage creatively in the subject matter” (Flipped Learning Network, 2017, n.p.). FL is most often associated with a mixed-mode or blended learning approach. The blend might be in terms of modalities (online and face-to-face) or related to time (asynchronous and synchronous). This association with blended learning is evident from definitions such as Strayer’s (2012, p. 171): “a specific type of blended learning design that uses technology to move lectures outside the classroom and uses learning activities to move practice with concepts inside the classroom”. Indeed, O’Flaherty and Phillips (2015) note, blending learning, and FL in particular, is becoming increasingly popular in higher education internationally. Specifically, Monash University has committed to the best use of a blended learning approach, combining digital technology with in person teaching and learning techniques (Monash University: Better Teaching, Better Learning 2016). Under the university’s remit, there is an imperative that students become active participants in their education. However, it is to be noted that FL approaches can also be constructed in fully online environments (Willems, Young, Cardilini, & Teychenne, 2016). In the blended format, in contrast to the traditional lecture format, technology-mediated, online resources are utilised for pre-class and post-class activities, and campus-based, face-to-face learning opportunities are used during the in-class sessions. The pre-class environment involves delivery of the core curriculum content through a digital environment, allowing engagement in the content to be driven by the individual student’s own abilities, interests (Prober & Kahn, 2013), and at their own pace and opportunity within a bounded period. Students subsequently participate actively during the in-class session (as opposed to passive consumption), completing applied activities in the presence of, and in cooperation with, the subject matter expert, to enhance the learning that was initiated with the pre-class content. The face-to-face learning environment allows students the opportunity to apply, consolidate, and build upon their pre-class learning by encouraging them to work collaboratively with colleagues on problems, tests and other interactive activities, as well as benefitting from the opportunity to ask questions of the expert lecturer. FL fosters students’ involvement in active inquiry and strategic application, thereby contributing to higher order thinking (Honeycutt & Garrett, 2013). It facilitates and enhances retention of relevant knowledge, allowing in-depth learning driven by the individual student’s own ability and interest (Prober & Kahn, 2013). Finally, student learning is reinforced through engagement with the post-class activities (Abeysekera & Dawson, 2015), before the cycle repeats with new content. There is increasing advocacy of the FL approach for medical education (Prober & Kahn, 2013; Williams, 2016) and medical workplace training (Tan, Brainard, & Larkin, 2015), despite reports of mixed or unsuccessful implementation (McLaughlin, 2018). However, these reports of less positive outcomes are largely offset by multiple reports of the successful adoption of FL in tertiary education, using an array of outcome measures (Johnson et al., 2016; Strayer, 2012). These measures include: greater student active involvement (McCallum, Schultz, Sellke, & Spartz, 2015); enhancements in effective peer collaboration (Strayer, 2012); better understanding of the curriculum content (Vaughan, 2014); increased student ownership of their learning (Enfield, 2013); and increased interaction between instructors and students (Lage, Platt, & Treglia, 2000). These findings are generally evaluated by changes in student results after https://www.intranet.monash/learningandteaching/media/documents/btbl-master-docs/btbl-program-september-2016.pdf https://www.intranet.monash/learningandteaching/media/documents/btbl-master-docs/btbl-program-september-2016.pdf Australasian Journal of Educational Technology, 2020, 36(3). 165 implementation (Ferreri & O’Connor, 2013). Alternatively, instructors have been approached for their perspectives on introducing FL into an existing course (Karabulut-Ilgu, Jaramillo Cherrez, & Hassall, 2018). Recommendations have been the need for release time for course development; having meaningful interaction between online and in-class tasks; and clearly explaining the rationale for the FL course structure to students (e.g., how the pedagogical decisions were made, and how these will impact the type and quality of their learning experience). In the context of the current article, the term course refers to the overarching structured combination of approved units, generally leading to an award such as an undergraduate degree. Introduction of any new pedagogy, such as a flipped approach to the traditional lecture format, can be met with resistance from all stakeholders. As Rotellar and Cain (2016, p. 6) note, the intervention: “may be initially met with resistance from students and faculty members alike, but the opportunities to improve teaching and learning practices should not be ignored”. Rotellar and Cain argue that “students deserve to be taught in a way that best prepares them to practice within a highly demanding work environment” (p. 6), with the understanding that the principles behind the flipped pedagogy are designed to best meet that need. Our purpose with this article is to assess the degree to which FL meets those student needs, across as many indicators as feasible for our cohort, given the constraints of course delivery and scheduling. Our research questions included: 1. What benefits were encountered from the perspective of students and staff with the introduction of flipped learning to a pre-clinical medical course? 2. What challenges were encountered from the perspective of students and staff with the introduction of flipped learning to a pre-clinical medical course? 3. How can the potential of flipped learning be maximised from the perspective of students and staff? Research context Monash University offers a 4-year graduate entry medical course, first offered in 2008. Students enrolled in this program complete Year A, the first (pre-clinical) year of study, at Monash Rural Health (Churchill) facilities in Gippsland, a rural community to the east of Melbourne, Victoria. Year A is aligned with Years 1 and 2 of the direct entry medical course, which is delivered at the university’s Clayton and Malaysia campuses. The subsequent 3 (clinical) years of the program are merged with the direct entry pathway. Students have the opportunity to complete the clinical years (Years 3B, 4C, and 5D) in metropolitan Melbourne, or in a rural location across the broad footprint of Monash School of Rural Health in Victoria (south eastern, north eastern, or north western). In 2015, FL was introduced for the Year A cohort, comprised of 88 students who had completed undergraduate training in Australia or through an equivalent international degree program. The FL model adopted at this site followed the usual 3-part formula: pre-class, in-class, and post-class activities. As part of the pre-class component, online lectorials covering key curriculum content were made available to the students for individual viewing prior to class. While it must be acknowledged that some educationalists use the term lectorial to refer to a hybrid lecture/tutorial which is delivered face-to-face (McCullough & Munro, 2016, p. 2), for our purposes, a lectorial refers to a mini online lecture to be viewed before class. The aim was that these videos were initially (and ideally) to be of up to 10 minutes in total duration, to maximise student concentration. This length of video was in accordance with Zainuddin and Attaran (2015), who reported that students prefer shorter videos with 15 minutes being the optimal. In reality, and in the first iteration, the lectorials typically ranged from 15 to 30 minutes in duration. Following the pre-class delivery of curriculum content, the in-class component mainly comprised face-to- face classroom activities and quizzes, facilitated by qualified instructors. Post-class activities involved formative quizzes or problem-solving exercises to further reinforce and consolidate learning. This FL cycle was repeated again for the next major topic over the duration of the 15-week semester, with each topic typically comprising 1 to 3 weeks of instruction. Research methodology The study aimed to evaluate the student perceptions about the learning experiences of the FL pedagogy, and to elicit the views of both academic and professional staff who had been involved with the Australasian Journal of Educational Technology, 2020, 36(3). 166 implementation of this approach in the first year of delivery (2015). This research comprised a mixed method approach, spanning 2 years, and utilised numerous data sources yielding both qualitative and quantitative data. Mixed method approaches to research have three principal purposes. These include convergent validation (triangulation), the development of analytic richness, and provision of illustration (Fielding, 2012). Participants included Year A students, academics, and professional staff. Staff and student surveys and student focus groups made up the multiple data sources used for the major evaluative review. An external medical educator was recruited by Monash Rural Health (Churchill) to undertake the bulk of the data collection and analysis. The project was approved by the Monash University Human Research Ethics Committee (MUHREC), number CF15/978 - 2015000448. An independent student-driven social media questionnaire conducted in the second year post implementation survey was also included as the final piece of research reported here for triangulation purposes. The data collected was purely quantitative data. The inclusion of this de-identified student data was approved by the Monash University Human Research Ethics Committee (MUHREC), number 19170 (25/03/2019). Transcripts of the qualitative data were jointly analysed thematically and using constant comparison to investigate attitudes, outcomes and the limitations of the introduction of flipped learning from both student and staff perspectives. Using the process of constant comparison, each item is checked or compared with the rest of the data in an inclusive and coherent way in order to establish analytical categories (Pope, Ziebland, & Mays, 2000). Constant comparison follows a process of familiarisation with the data; identification of all the key issues, concepts, and themes by which the data can be examined and referenced; indexing; charting and mapping; and interpretation (Pope et al., 2000). The various data collection approaches – student and staff surveys, student focus groups, student Facebook questionnaire – are briefly outlined below. Major review of the introduction of flipped pedagogy (student survey) The entire cohort of 88 students was invited to participate in an online survey enquiring into their perceptions of each course component, and any suggested improvements to the course design. Students were asked to consider the following broad questions: What are the students’ experiences of converting to the FL model for a first-year graduate-entry medical program?; What is the overall level of medical students’ satisfaction with their learning experiences during Year A?; and Based on students’ perceptions about their learning experiences, what are suggested areas for improvement in areas such as: curriculum, course structure, assessment and course delivery? Students were surveyed in both first and second semesters, and both quantitative and qualitative responses to the survey questions were self-recorded anonymously using Qualtrics© online survey software. The survey had both fixed answer and open-ended response options. In Semester 1, 39 students (n = 88; 44%) participated in the online survey, while 36 students (n = 87; 41%) completed the survey in Semester 2. The number of students who participated in both semester surveys was not monitored. The Qualtrics software did not allow for determining how many students participated in both the Semester 1 and 2 anonymous surveys, but there was the possibility of some overlap in the participant groups. Major review of the introduction of flipped pedagogy (student focus groups) In Semester 2, students were invited to participate in a focus group to clarify and extend the data collected through the online surveys (collected in both Semester 1 and 2). In order to remove any potential power imbalances and adopt an impartial recruitment strategy for the student focus groups, professional staff members (non-academics who had no influence over student outcomes) recruited students. Invitations to participate in the focus groups were distributed via the student group via university supported student group email, along with notification via posters in common areas, noticeboards, and learning spaces located on campus. The initiating questions for the focus groups explored student experiences with the FL design. Two academics reviewed the focus group interview schedule of questions, and their feedback was incorporated by the evaluator. Australasian Journal of Educational Technology, 2020, 36(3). 167 Seven students were recruited into two focus groups: four students were allocated to the first focus group and three in the second. Focus groups were held on site at the Churchill campus to encourage attendance. Duration ranged from 60 to 90 minutes. Each group of students self-selected to participate. In the focus groups, discussion was guided, monitored, and recorded by two external facilitators to preserve the anonymity of the qualitative data collection. Major review of the introduction of flipped pedagogy (staff survey) At the conclusion of the academic year, staff were also surveyed. All staff who had been involved in the introduction of the FL approach were invited to participate. The key evaluation questions for staff included: What flipped classroom benefits and challenges were encountered in 2015 from the perspective of students and staff?; and How can the potential of the flipped classroom model be maximised from the perspective of staff and students? Several open-ended questions were also included to allow staff to express their views without the limitation of a forced choice methodology. The survey yielded both quantitative and qualitative data. Of 24 staff members, 18 (75%) completed the survey. Seven of the staff were full time or fractional academics (39%). Seven were sessional academics (39%). Four were professional staff (22%). Independent student-driven social media questionnaire In 2016, two students from the 2015 Year A cohort independently asked their peers from the 2015 and 2016 cohorts to evaluate the FL approach. Their questionnaire was created on Google forms and delivered via the cohort’s Facebook group, independent of the academic staff. This is an example of peer learning. De- identified findings were shared with the academic staff involved in developing and delivering the flipped classroom, for the purposes of ongoing improvement. There were 14 fixed-answer questions in the student survey, which examined the general perceptions of the flipped pedagogy from a peer perspective. Participants were asked what they found beneficial, and to describe their feelings about the pedagogy. Twenty-eight participants responded to the Facebook survey, including 3 who had commenced graduate entry in 2015, and 25 who commenced in 2016. Results The results speak to the benefits and challenges of the introduction of FL to Year A of a pre-clinical medical course from the perspective of students and staff. They also point to ways in which the potential of FL could be maximised from the perspective of students and staff. The discussion of results is broken down into several aspects: considerations of how to better prepare staff and students prior to implementing a new FL model; the benefits and challenges experienced following the introduction of FL; perceptions of acceptance over time; and overall attitudes. Better preparation of both staff and students prior to implementation of the FL model Issues around better preparation for the implementation of the flipped classroom arose for both students and staff. Two staff requested better support and preparation for the introduction of FL, including provision of training and additional information about types of delivery and teaching strategies. Some staff wanted additional time to participate in professional development, such as conferences, courses, discussions, seminars, or to have access to resources. Only one staff respondent stated they did not need further professional development. Two staff members requested formal flipped classroom training covering delivery methods and strategies, including “[w]orkshops on different modes and strategies for the delivery of content tailored for flipped class room teaching”. One staff member noted the need for higher-level oversight of this process, commenting on the “[n]eed [for] guidance and a structured pathway from the School on professional development opportunities”. Time was also a common denominator in staff responses, with staff commenting that they felt time- pressured. One staff member responded that they would like to have “[t]ime to take advantage of the development opportunities offered by the university. Training in various IT modalities. Sufficient time to be able to attend medical education conferences and courses”. Similarly, another staff member noted “[t]he problem is that there is not any time to do that without compromising output. (But perhaps not being able to spend time doing this, also compromises quality of output!)”. This notion of time was not simply for professional development leading up to the change, but also time required for the staff as a whole unit to Australasian Journal of Educational Technology, 2020, 36(3). 168 better prepare for the implementation of a major curriculum change. One staff participant wrote that they would like “[t]ime for discussion and learning as to how best to achieve our aims”. Time also was a theme in staff comments around the consideration of structure in transforming a course previously taught in traditional lecture format into a flipped design, including time for the preparation of student resources, such as the preparation and recording of the lectorials. One staff member noted that “[i]t takes a lot of time to prepare learning resources of a suitable type and standard”. Similarly, another said that“[p]reparing a class in this manner is time-consuming and given that there was little lead-in time, it can be hard for staff to get materials organised…”, expressing their hope that “some material can be rolled over” for future years. Time constraints to develop learning resources also related to the creation of materials in more specialised areas of focus, for example: My topic area just doesn't have the large wealth of pre-prepared study material that some areas have. These resources will have to be developed up over time, a task that was difficult this year because of the need to get the main components of the curriculum sorted out first. Such observations relate directly to issues around change management with the introduction of any new approach. Some students, for example, believed that course changes and benefits should have been communicated to them earlier. As one focus group participant explained “at the beginning of the semester, it (FL) was not sold to us in a way that made it seem inviting and good for us”. Staff also considered the need for students to be better prepared for the flipped leaning model: It would help for students to be "trained-up" a bit early on in how to go about this. Some more resources on how to ensure that the students engage in the pre-class activity would be helpful, otherwise there may be potential to be covering material online prior to class and then covering the same material in class. Staff also struggled in implementing the new pedagogical approach. Additional factors that posed a challenge for staff included time to seek assistance with the uploading of resources within the learning management system from staff in the central IT support group. In relation to seeking help, one staff member wrote: Sometimes there was a delay at e-Solutions [the Monash IT service], and other times if you gave them material to load on a Friday, it didn't get loaded until Monday. This was extremely frustrating. If we could do it ourselves, it would reduce the angst at all levels. Another related issue raised by both staff and students, was the necessity for there to be consistency across all units. From the students’ perspectives, several referred to frustrations with how the method was inconsistently implemented across topics and subjects. For example, a response in the focus groups was “I do not believe that the flipped classroom model has been effective, particularly as all lecturers have not adopted the approach across subject areas”. Expanding on this point, another student noted that “[s]ome tutors provide little or no resources, which makes self-directed learning extremely difficult, as we spend ridiculous amounts of time trolling through textbooks or online resources”. Similarly, a third student reflected that: It has some benefits but has not been implemented well at all. Many classes lack resources and the lectures, which are required for our learning. [Necessary resources] are not uploaded on time and this causes stress and lack of learning. This issue extended to the variability in delivery across instructors, with one student commenting: Either implement a standard model of the flipped classroom or abandon it. Variations lecturer to lecturer leads [sic] to inequality both in the delivery and what the student believes is expected of them. For example, late posting of any material not only interferes with the preparation for that class but the preparation for the surrounding classes and the students’ planning thereof. Australasian Journal of Educational Technology, 2020, 36(3). 169 Perceived benefits and challenges upon introduction of FL The flipped pedagogy involved a blended delivery format in contrast to the traditional face-to-face delivery of the lecture and 71% of students in the Facebook survey felt that the “flipped classroom provides a greater opportunity for conversation and interaction with the teaching staff’. Some students also felt that it helped to consolidate their learning, stating: The flipped classroom model has really helped my learning this year. Being able to learn and study the material in my own time before classes meant I was able to check and solidify the information during class. Staff also felt FL to be a useful way to consolidate student learning, commenting that the “[b]enefits (if done well) is[sic] good background information available to the students at their leisure - and then consolidation of that learning.” However, one staff member added a caution that while they considered the flipped approach as promoting active student learning, that the in-class activities needed to be carefully considered. They commented that FL: Encourages active participation and a higher level of understanding when done well. If the contact time activities are not carefully planned, then students can become disengaged and bored and view it as a waste of time. If the learning resources are not appropriate for this mode of delivery or are not provided in a timely manner, then that undermines the process. Reflecting on the change of pedagogical approach in the student-initiated Facebook survey, 68% felt that the flipped pedagogy did not make “learning easier than the traditional lecture and tutorial method”. However, 64% did feel that “listening to recorded lectures [lectorials] is better than going to an actual lecture theatre”. Students struggled with the timeliness of resource availability for the pre-class work. A key issue for students was not having timely access for their pre-class preparation to the online resources, including the amount of content that was uploaded prior to a tutorial. One student explained: There's [sic] been a lot of issues with [staff] not putting up content [to] have prepped, until 5 o'clock the night before. And then they put 3 hours of work up. So it completely defeats the purpose. In the student survey, one question focussed on the notion of feeling lost with respect to what was required from them. Students were asked to complete the following sentence: At times I felt lost during the flipped classroom sessions; because … The survey contained six rating options for students to select from to rate their degree of feeling lost. Of these, no respondents (0%) selected the never felt lost option, meaning that all respondents all felt lost at various times and to varying degrees. Students in the focus group felt that the teachers themselves were vital to the success of the flipped model, and in ensuring a positive learning experience for students. This was also reflected in the Facebook survey, where two questions with set choice multiple choice answers gathered responses on this matter. When reflecting on the statement, “What I took from flipped classroom depended on the quality and the approach of teacher”, 89% felt that this statement was true. When responding to the item, “At times I felt lost during the flipped classroom sessions because …” 89% of participating students selected “the teacher was not good”. Key considerations were student agency, self-directed learning, and taking responsibility for their own learning. Some students found it difficult to adjust to the expectation that they come to class prepared. In response to the item, “I believe flipped classroom, compared to traditional methods, forces students to take responsibility for their learning”, 82% agreed with the statement. For those respondents who did not agree, delivery of content was the main focal point. As one student in the focus groups noted, they wanted: More delivery of content. Self-directed study should be a supplement to learning its foundation. Numerous studies indicate that online learning is not as effective as face to face, inductive, teaching methods. Australasian Journal of Educational Technology, 2020, 36(3). 170 For some students, a more instructivist, directed pedagogy was preferred. Conversely, another student felt that the new pedagogical approach supported their learning, stating: “I think the Flipped Classroom model is successful for me as a relatively self-motivated learner.” The staff perspective was slightly different. One staff member stated: Some [students] were uncertain at first as they were used to the old ways. They soon accepted that the Flipped Classroom approach worked and enhanced self-direction and facilitated investigation of any special interest that a student might be keen to explore. In relation to the provision of learning resources for students, the majority of staff in the staff survey described learning resources as satisfactory, reasonable, or adequate. One exemplar response was: “I think we provide plenty of adequate resources to students. I haven't heard that students were unhappy with the quality - maybe the quantity (as in too much??).” Another staff member stated that the program was “[v]ery well resourced, perhaps not always fully utilised [by the students]”. However, from the student perspective in the survey, 61% felt that resources such as the class hand-outs were not good. One staff member acknowledged that the resources were reasonable and that: The resources available to students will improve as the staff have time to quality add to their program, rather than simply grind out a large amount of content in the Flipped Classroom model all at once. This staff member mentioned “[f]urther suggestions [on] different types of ways of presenting learning resources would be really helpful”. The student-led Facebook survey provided some further granular insight on the implementation of the flipped innovation in terms of completion of the pre-class activities. When asked if they completed the pre- class activities, 18% said always, 64% stated often, and a further 18% replied seldom. Interestingly, however, no participant said that they never completed the pre-class activities. Further to this, one staff member identified the lack of pre-class preparation by some students as a problem because it increased the amount of in-class time that needed to be devoted to covering content, impacting on the in-class work. One staff member wrote in the survey: “I underestimated the amount of in class time that was actually needed (particularly because students did not keep up with the pre-class preparation).” When asked what materials they used to prepare for the in-class sessions in the Facebook survey, participants provided a number of responses: 100% responded they listened to the lectorials; 54% looked up other online resources; and 32% indicated that they read the relevant textbook chapters. A disparity was apparent between the pre-class material, including the lectorials, and the content of the in-class sessions. Some students expressed concerns about the lack of integration between information in lectorials compared with the content covered in-class and the quality and information provided in some online resources such as videos (lectorials). Students indicated that they were not well prepared for tutorials and were particularly concerned when new content was added in the tutorials. One student commented: “And then you go to the tutorial after watching the online lectures and it's completely different content [to the lectorials].” Another student stated: “You do your online lecture but when you come to the tutorials, completely different ... not topic but [a] completely different idea.” Changes in perception of FL over time For the staff, there was a perception that the student body as a whole initially rejected the pedagogical innovation, but that this changed from initial rejection to gradual acceptance. One staff member wrote: “They seem more settled in Semester 2. In Semester 1, there was a lot of concern from students about the change in teaching methods.” Another staff member noted in the survey: “Early in the year some students risked being left behind as they waited for lectures to set the scene. As the year progressed this ceased to be of concern.” An academic staff member reported: “As the students got used to this model they seemed to feel much more comfortable with the flipped classroom and appear to appreciate the ability to undertake self-directed learning.” Staff also reflected that an issue with regards to acceptance might be in the name: Australasian Journal of Educational Technology, 2020, 36(3). 171 It enables students to learn actively, which benefits them in the end, but they don't necessarily see it that way when they first start. Part of the problem is with the name ‘Flipped Classroom’. It gives the students the impression that they are doing all the work and the lecturers are not doing any. Another name would help with the ‘stigma’ around this mode of delivery. Quantitative results from the student survey relating to student satisfaction over the two semesters were consistent with the staff perception of changes over time. There were more affirmative student responses in Semester 2, with increases in the completely satisfied (from 3 to 10%) and moderately satisfied (from 25 to 55%) alternatives, and a reduction in the limited satisfaction (38 to 22%) and not satisfied (34 to 13%) responses, compared to the Semester 1 survey (Figure 1). Interestingly, the greatest shift in perception was in the moderately satisfied category. The changes in perception of the student group may be aligned with the staff view that there was increased acceptance of the flipped classroom. Clearly some students reported benefits associated with the flipped learning approach. However, for other students the change may well be related to an acceptance of the reality that this is the way the course is taught. Figure 1. Student satisfaction with FL In the student-led survey, feedback was slightly different, possibly reflecting that these were graduate-entry students who had received the bulk of their undergraduate education in the traditional lecture mode. Thirty- nine percent of respondents indicated “I was not used [to] this method of teaching, but I felt better as the year passed”, while 25% selected the option, “I was not used [to] this method of teaching and it never got better across the year”. Overall perception For staff, successful implementation of FL required a greater need for student support and familiarisation, with more judicious use of terminology also identified as a potential contributing factor. One staff member reflected It might need more scaffolding if students are unfamiliar with the model and I think that it can be used judiciously with some classes lending themselves to the model, some being delivered by lectures. That way, both types of learners are catered for. I think we should use the term "multi-mode" delivery rather than "Flipped Classroom" as the latter implies that all the learning is done with that model, which is not necessarily the case. Students were split fairly evenly in their responses to their overall perception of flipped learning. In the independent student-led survey, responses to the subjective assessment item, “My overall perception of the flipped classroom has been positive or negative”, were 46% negative and 54% positive. Representative of the negative response, one student noted: “I think that the course is good. It's just there's been issues ... not satisfied limited satisfaction moderately satisfied completely satisfied semester 1 semester 2 30 30 60 60 0 Australasian Journal of Educational Technology, 2020, 36(3). 172 because of the introduction of this flipped classroom model, it's not working the way it's supposed to.” In contrast, a student who viewed the method positively stated: I can sit at home and Google it. I think my point is, the flip classroom model works. It has definitely worked when it's embraced by those [staff] who are running it. The staff who go along and provide us [with] great resources, make sure there's lectorials online and then additional reading if we want it and then come to class to answer questions and go through questions together in small groups and then work through any problems with that. That totally works for me. I really think that's fantastic. Discussion From all data sources (survey, focus groups, and the student-led social media survey), it is clear that successful transition to an FL approach requires extensive stakeholder training and preparation time, and careful management of student perceptions to ensure that they understand the approach is based on sound pedagogy as opposed to an attempt to save on cost or staff time. This correlates with concerns that while it may be a positive notion for students to take responsibility for their learning, as is anticipated in adopting a FL approach, there is often little consideration for the impact that this may have on students themselves. Students may perceive that their workload is additional rather than distributed, after years embedded in traditional learning environments (Rotellar & Cain, 2016), or may struggle with the radical re-organisation of their approaches to learning and their study (Miles, Lee, Foggett, & Nair, 2017). The same applies to staff used to traditional academic approaches and the perceived primacy of face-to-face teaching. From the students’ perspective, improvement of the learning experience in the pre-clinical course requires: (1) consistent implementation of the flipped model across a course; (2) timely access to quality online learning resources, including the timely uploading of lectorials and other resources; (3) identification of specific course content which is better taught using some traditional instruction methods; (4) clear and consistent communication of pre-class and in-class expectations, and the estimated time required; and (5) alignment between content covered in pre-class activities and in-class activities. From the staff perspective, the effective use of flipped pedagogy requires increased, rather than reduced, staff time and commitment, aligning with literature (cf. Rotellar & Cain, 2016). Staff data in this research indicate that FL implementation requires: (1) professional development training for all academic staff (including sessional staff); (2) identification of topic areas that are most appropriate for FL; (3) frequent assessment of student learning (e.g. self-assessment quizzes) to ensure that students are adequately prepared for the face-to-face tutorial activity; and (4) the provision of additional time, support and professional resources for preparation of consistently high quality learning opportunities. The latter provision incorporates a clear emphasis on selection and training around learning management systems and IT-related issues, in order to minimise lag and down time and enhance the in-class experience. Research limitations and future research directions There were several limitations in this research. First, the research cohort was a rural-based, graduate-entry cohort for pre-clinical medical education, which constrained the studies to a relatively small sample of specialist students, which may affect the generalisability of the findings. Students who participated in the online surveys were not tagged to monitor how many responded to both semester surveys, therefore student participation in both surveys was a limitation. Second, the staff and student survey and the student focus group data span only a single academic year, whereas the student-initiated Facebook survey included students from the initial and subsequent years of the FL rollout. The normal process of experience and maturation of the students in this cohort may account for some of the enhancement of satisfaction with the content delivery method. Future research directions are numerous. These include the tracking of this cohort, and other subsequent cohorts, as they progress into later years of the medical education program for their reflections on the pedagogy over time. As a longitudinal study, this would include the repetition of the study in second and subsequent years. Additionally, it would be useful to compare and contrast the original cohort with subsequent graduate intakes and their perceptions of the flipped model. Third, it would include considerations of the influence of any sell of the FL pedagogical approach from one cohort to the next. Australasian Journal of Educational Technology, 2020, 36(3). 173 Indeed, anecdotally, staff have felt that the need to sell the approach to the student cohort has reduced over the years because they are now experiencing FL approaches in other undergraduate courses across the university, which helps to reassure incoming students that the new pedagogy is to their benefit. Finally, it would be interesting to run a comparison between the official staff-initiated student survey and focus group evaluation and student-led, parallel surveys, in order to assess potential differences with the different data collection techniques. Conclusion Universities can be seen as responding to, and engaging with, the needs of the modern student, particularly where that student is learning to become part of an increasingly complex health care system (Betihavas, Bridgman, Kornhaber, & Cross, 2016). For institutions, FL is viewed as one means towards achieving this. However, this response may also be driven by economic arguments surrounding cost-effectiveness and staff efficiency (O’Flaherty & Phillips, 2015). As Gunyou (2015) stresses, the FL modality may be particularly cost-effective when compared with the high costs of the traditional large bricks and mortar higher- educational structures. This greater benefit takes the form of value to students as measured by course fees, the value of the learning approach and student satisfaction (Nicklen et al., 2016), and considerations about quality improvement (Rotellar & Cain, 2016). The data presented in this paper provide critical feedback for refinement of the FL approach. Key facets of this feedback include the identification of a more robust student and faculty development in the lead up to implementation, plus ongoing support during implementation of this pedagogy. Taken together, our results provide a unique perspective of the introduction of the flipped approach through different stakeholder lenses, and over time. The implications for practice and policy from this research are numerous. It is clear that successful transition to an FL approach is critically dependent on preparation of stakeholders, provision of adequate resources to staff and students, and consistent, evidence-based, and truthful explanation of the rationale for the change. Each phase of this transition must also be accompanied by transparent, reliable monitoring of effectiveness, with support for continuous improvement during and after implementation. The research also reflected the efficacy of the use of peer-led social media feedback as evidenced through the Facebook survey. This approach highlights the use of social media as the site of a member-driven formal learning in higher education, enabling co-learning and technology-infused interactions (Willems & Bateman, 2013, p. 111). In summary, the introduction of alternative pedagogies, such as FL, require careful consideration, preparation, and change-management strategy from the outset. They require evaluation of implementation which needs to consider both staff and student experiences. Finally, to fulfil their educational mission, conversion of medical education programs to the FL modality requires continued refinements in response to student and staff reflections. References Abeysekera & Dawson, P. (2015). Motivation and cognitive load in the flipped classroom: Definition, rationale and a call for research. Higher Education Research and Development, 34(1), 1-14. Adams Becker, S., Brown, M., Dahlstrom, E., Davis, A., DePaul, K., Diaz, V., & Pomerantz, J. (2018). NMC Horizon Report: 2018 Higher Education Edition. Louisville, CO: Educause. Betihavas, V., Bridgman, H., Kornhaber, R., & Cross, M. (2016). The evidence for ‘flipping out’: A systematic review of the flipped classroom in nursing education. Nurse Education Today, 38, 15-21. https://doi.org/10.1016/j.nedt.2015.12.010 Denzin, N. K. (1978). The research act. (2nd ed.). New York, NY: McGraw Hill. Enfield, J. (2013). Looking at the impact of the flipped classroom model of instruction on undergraduate multimedia students at CSUN. TechTrends, 57(6), 14–27. https://doi.org/10.1007/s11528-013-0698-1 Ferreri, S. F., & O’Connor, S. K. (2013). Redesign of a large lecture course into a small-group learning course. American Journal of Pharmacology Education. 77(1), Article 13. Fielding, N. G. (2012). Triangulation and mixed methods designs: Data integration with new research technologies. Journal of Mixed Methods Research, 6(2), 124-136. https://doi.org/10.1016/j.nedt.2015.12.010 https://doi.org/10.1007/s11528-013-0698-1 Australasian Journal of Educational Technology, 2020, 36(3). 174 Flipped Learning Network (2017). Flip learning. Retrieved from http://flippedlearning.org/definition-of- flipped-learning/ Fraser, S., & Greenhalgh, T. (2001). Coping with complexity: Educating for capability. British Medical Journal, 323(7316), 799–803. Gunyou, J. (2015). I flipped my classroom: One teacher’s quest to remain relevant. Journal of Physician Assistant Education, 21(1), 13-24. Retrieved from http://www.naspaa.org/JPAEMessenger/Article/VOL21-1/03_Gunyou.pdf Honeycutt, B., & Garrett, J. (2013). The flipped approach to a learner-centered class [White Paper]. Maddison, WI: Magna. Retrieved from https://www.magnapubs.com/publications/the-flipped- approach-to-a-learner-centered-class-3098-1.html Johnson, L., Adams Becker, S., Cummins, M., Estrada, V., Freeman, A., & Hall, C. (2016). NMC Horizon Report: 2016. Higher Education Edition. Austin, TX: The New Media Consortium. Karabulut-Ilgu, A., Jaramillo Cherrez, N., & Hassall, L. (2018). Flipping to engage students: Instructor perspectives on flipping large enrolment courses. Australasian Journal of Educational Technology, 34(4), 123-137. https://doi.org/10.14742/ajet.4036 Lage, M. J., Platt, G. J., & Treglia, M. (2000). Inverting the classroom: A gateway to creating an inclusive learning environment. The Journal of Economic Education, 31(1), 30-43. https://doi.org/10.2307/1183338 McCallum, S., Schultz, J., Sellke, K., & Spartz, J. (2015). An examination of the flipped classroom approach on college student academic involvement. International Journal of Teaching and Learning in Higher Education, 27(1), 42–55. Retrieved from http://www.isetl.org/ijtlhe/pdf/IJTLHE1880.pdf McCullough, K, & Munro, N. (2016). Finance students’ experiences of lecture-based active learning tasks. Innovative Education & Teaching International, 55(1), 1-9. https://doi.org/10.1080/14703297.2016.1189843 McLaughlin, J. E. (2018). Flipped classrooms, by design. Medical Education 2018, 52(9), 882-893. McLaughlin, J. E., Roth, M. T., Glatt, D. M., Gharkholonarehe, N., Davidson, C. A., Griffin, L. M., Esserman, D. A., & Mumper, R. J. (2014). The flipped classroom: A course redesign to foster learning and engagement in a health professions school. Academic Medicine, 89(2): 236-43. https:/doi.org/10.1097/ACM.0000000000000086 Mehta, N., Hull, N., Young, J., & Stoller, J. (2013). Just imagine: New paradigms for medical education. Academic Medicine, 88(10), 1418-1423. Miles, C. A., Lee, A. C., Foggett, K. A., & Nair, B. (2017). Reinventing medical teaching and learning for the 21st century: Blended and flipped strategies. Archives of Medicine and Health Sciences, 5(1), 97- 102. https://doi.org/10.4103/amhs.amhs_53_17 Nicklen, P., Rivers, G., Ooi, C., Ilic, D., Reeves, S., Walsh, K., & Moloney S. (2016). An approach for calculating student-centred value in education - A link between quality, efficiency, and the learning experience. PLOS ONE. 10(1371),1-11. https://doi.org/10.1371/journal.pone.0162941 O'Flaherty, J., & Phillips, C. (2015). The use of flipped classrooms in higher education: A scoping review. Internet and Higher Education, 25(1), 85-95. Pope, C., Ziebland, S., & Mays, N. (2000). Qualitative research in health care. Analysing qualitative data. British Medical Journal, 320(7227), 114-116. https://doi.org/10.1136/bmj.320.7227.114 Prideaux, D. (2009). Medical education in Australia: Much has changed but what remains? Medical Teacher, 31(2), 96-100. https://doi.org/10.1080/01421590802509157 Prober, C. G., & Kahn, S. (2013). Medical education reimagined: A call to action. Academic Medicine, 88(10), 1407-1410. Rotellar, C., & Cain, J. (2016). Research, perspectives, and recommendations on implementing the flipped classroom. American Journal of Pharmacology Education, 80(2), Article 34. https://doi.org/10.5688/ajpe80234 Schell, J., & Mazur, E. (2015). Flipping the chemistry classroom with peer instruction. In J. Garcia- Martinez & E. Serrano Torregrosa (Eds.), Chemistry education: Best practices, opportunities and trends (pp. 319-344). Weinheim: Wiley-VCH Verlag GmbH & Co. Strayer, J. F. (2012). How learning in an inverted classroom influences cooperation, innovation and task orientation. Learning Environment Research, 15(2), 171-193. Tan, E., Brainard, A., & Larkin, G. L. (2015). Acceptability of the flipped classroom approach for in- house teaching in emergency medicine. Emergency Medicine Australasia, 27(5), 453-459. Torrisi-Steele, G. (2011). This thing called blended learning — A definition and planning approach. In K. Krause, M. Buckridge, C. Grimmer, & S. Purbrick-Illek (Eds.) Research and development in higher http://flippedlearning.org/definition-of-flipped-learning/ http://flippedlearning.org/definition-of-flipped-learning/ http://www.naspaa.org/JPAEMessenger/Article/VOL21-1/03_Gunyou.pdf https://www.magnapubs.com/publications/the-flipped-approach-to-a-learner-centered-class-3098-1.html https://www.magnapubs.com/publications/the-flipped-approach-to-a-learner-centered-class-3098-1.html https://doi.org/10.14742/ajet.4036 https://doi.org/10.1080/14703297.2016.1189843 https://www.ncbi.nlm.nih.gov/pubmed/?term=McLaughlin%20JE%5BAuthor%5D&cauthor=true&cauthor_uid=24270916 https://www.ncbi.nlm.nih.gov/pubmed/?term=Roth%20MT%5BAuthor%5D&cauthor=true&cauthor_uid=24270916 https://www.ncbi.nlm.nih.gov/pubmed/?term=Glatt%20DM%5BAuthor%5D&cauthor=true&cauthor_uid=24270916 https://www.ncbi.nlm.nih.gov/pubmed/?term=Gharkholonarehe%20N%5BAuthor%5D&cauthor=true&cauthor_uid=24270916 https://www.ncbi.nlm.nih.gov/pubmed/?term=Davidson%20CA%5BAuthor%5D&cauthor=true&cauthor_uid=24270916 https://www.ncbi.nlm.nih.gov/pubmed/?term=Griffin%20LM%5BAuthor%5D&cauthor=true&cauthor_uid=24270916 https://www.ncbi.nlm.nih.gov/pubmed/?term=Griffin%20LM%5BAuthor%5D&cauthor=true&cauthor_uid=24270916 https://www.ncbi.nlm.nih.gov/pubmed/?term=Esserman%20DA%5BAuthor%5D&cauthor=true&cauthor_uid=24270916 https://www.ncbi.nlm.nih.gov/pubmed/?term=Mumper%20RJ%5BAuthor%5D&cauthor=true&cauthor_uid=24270916 https://doi.org/10.1097/ACM.0000000000000086 https://doi.org/10.1080/01421590802509157 Australasian Journal of Educational Technology, 2020, 36(3). 175 education: Reshaping higher education (pp. 360-371). Retrieved from http://www.herdsa.org.au/system/files/HERDSA_2011_Torrisi-Steele.PDF Vaughan, M. (2014). Flipping the learning: An investigation into the use of the flipped classroom model in an introductory teaching course. Education Research and Perspectives, 41(1), 25–41. Wang, Y., Han, X., & Yang, J. (2015). Revisiting the blended learning literature: Using a complex adaptive systems framework. Educational Technology & Society, 18(2), 380–393. Willems, J., & Bateman, D. (2013). Facing up to it: blending formal and informal learning opportunities in higher education contexts. In G. Trentin, & M. Repetto (Eds.) Using network and mobile technology to bridge formal and informal learning (pp. 93-118). Cambridge: Woodhead Publishing Limited. Willems, J., Young, K., Cardilini, A & Teychenne, S. (2016). WIL-fully flipping online: A novel pedagogical approach in STEM. In S. Barker, S. Dawson, A. Pardo, & C. Colvin (Eds.), Show me the learning. Proceedings ASCILITE 2016, Adelaide, 656-665. Retrieved from http://2016conference.ascilite.org/wp-content/uploads/ascilite2016_willems_full.pdf Williams, D. E. (2016). The future of medical education: Flipping the classroom and education technology. Ochsner Journal, 16(1), 14-15. Zainuddin, Z., & Attaran, M. (2015). Malaysian students’ perceptions of flipped classroom: A case study. Innovative Education Teaching International, 53(6), 660-670. https://doi.org/10.1080/14703297.2015.1102079 Corresponding author: Margaret Simmons, Margaret.Simmons@monash.edu Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY-NC- ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY- NC-ND 4.0. Please cite as: Simmons, M., Colville, D., Bullock, S., Willems, J., Machado, M., McArdle, A., … Reser, D. (2020). Introducing the flip: A mixed method approach to gauge student and staff perceptions on the introduction of flipped pedagogy in pre-clinical medical education. Australasian Journal of Educational Technology, 36(3), 163-175. https://doi.org/10.14742/ajet.5600 http://www.herdsa.org.au/system/files/HERDSA_2011_Torrisi-Steele.PDF http://www.deakin.edu.au/research/admin/pubs/reports/database/dynamic/output/person/person.php?person_code=willejua&cat_code= http://2016conference.ascilite.org/wp-content/uploads/ascilite2016_willems_full.pdf https://doi.org/10.1080/14703297.2015.1102079 mailto:Margaret.Simmons@monash.edu https://creativecommons.org/licenses/by-nc-nd/4.0/ https://creativecommons.org/licenses/by-nc-nd/4.0/ https://doi.org/10.14742/ajet.5600 Introduction Flipped learning Research context Research methodology Major review of the introduction of flipped pedagogy (student survey) Major review of the introduction of flipped pedagogy (student focus groups) Major review of the introduction of flipped pedagogy (staff survey) Independent student-driven social media questionnaire Results Better preparation of both staff and students prior to implementation of the FL model Perceived benefits and challenges upon introduction of FL Changes in perception of FL over time Figure 1. Student satisfaction with FL Overall perception Discussion Research limitations and future research directions Conclusion References McLaughlin, J. E., Roth, M. T., Glatt, D. M., Gharkholonarehe, N., Davidson, C. A., Griffin, L. M., Esserman, D. A., & Mumper, R. J. (2014). The flipped classroom: A course redesign to foster learning and engagement in a health professions school. Aca...