Australasian Journal of Educational Technology, 2021, 37(3). 187 Preparation and synchronous participation improve student performance in a blended learning experience Charlotte Emily Jane Clark, Ger Post University of Melbourne Blended learning can create flexibility for students, more efficiently utilise infrastructure, and can provide high-quality learning at scale. We investigated perceived value and learning gains associated with asynchronous eLearning and synchronous face-to-face (f2f) components of a blended learning experience. We hypothesised that individual student preference for eLearning and f2f learning would be variable, but that participation in f2f classes would enhance student learning. Using a design-based research approach, we have evaluated two iterations of a blended learning experience, combining qualitative survey data and quantitative attendance data and student grades. Students overwhelmingly valued active learning, both within eLearning materials and f2f classes. Final marks positively correlated with the number of f2f classes students attended. Analysis of a subset of intended learning outcomes (ILOs) showed that students who accessed eLearning independently and students who attended f2f classes performed equally-well in ILO-related assessment tasks, however, students were more likely to choose an assessment task directly-related to a class they attended. In addition, completion of required eLearning prior to f2f class attendance significantly enhanced student performance in related assessment tasks. We suggest that f2f attendance as part of blended learning is beneficial, however students can obtain selected ILOs from engaging eLearning materials. Implications for practice or policy: • Instructors will gain insight into aspects of blended active learning that students value. • We present evidence that supports the benefits to students of completion of pre- eLearning prior to participation in synchronous f2f classes. Keywords: blended learning, eLearning, active learning, student attendance, constructive alignment, design-based research Introduction This study explored the value (perceived and actual) of attendance at synchronous face-to-face (f2f) active learning classes (either on-campus or via video conferencing) and completion of asynchronous eLearning tasks as part of a blended learning experience. Students are increasingly requesting more flexible study options, including the ability to engage with learning online due to inability to attend f2f classes (Brown et al., 2018; Norton & Cakitaki, 2016). Many major universities, including The University of Melbourne, are adopting teaching and learning strategies that require their educators to provide fully online and blended learning opportunities as part of the FlexAP project (The University of Melbourne, https://about.unimelb.edu.au/teaching-and-learning/innovation-initiatives/pedagogy-and-curriculum- innovation/flexap-project). This strategy has been expedited through enforced COVID-19 pandemic restrictions. Here, we utilised a design-based research approach to evaluate iterative changes made to a blended second year undergraduate cell biology course. We refer to these iterations as “Iteration 1” (2019) and “Iteration 2” (2020). Early definitions of blended learning refer to a blend of asynchronous text-based online material and synchronous f2f learning (Garrison & Kanuka, 2004), but as technology has evolved, so too has the definition (Sharpe et al., 2006). Further descriptions of blended learning introduce the concept of strong and weak blends, depending on the amount of eLearning and also discuss the variable media and activity blends available (Littlejohn & Pegler, 2007). The Joint information systems committee (Jisc) define blended learning as “a combination of face-to-face learning and dynamic digital activities and content that facilitate anytime/anyplace learning” (Hibberson et al., 2020) (https://www.jisc.ac.uk/guides/creating- blended-learning-content). We define blended learning as requiring two key components: (1) a f2f component that must occur synchronously; and (2) eLearning that can be accessed asynchronously. Our initial definition of the synchronous f2f component was of an on-campus experience however, due to https://about.unimelb.edu.au/teaching-and-learning/innovation-initiatives/pedagogy-and-curriculum-innovation/flexap-project https://about.unimelb.edu.au/teaching-and-learning/innovation-initiatives/pedagogy-and-curriculum-innovation/flexap-project https://www.jisc.ac.uk/guides/creating-blended-learning-content https://www.jisc.ac.uk/guides/creating-blended-learning-content 188 COVID-19 restrictions we extended our analysis to compare synchronous on-campus classes (Iteration 1) and synchronous online video conference classes (Iteration 2). It is of course possible for students to form study groups and access eLearning materials synchronously and in groups, but here we assume that the majority of our students access eLearning materials independently and asynchronously. The Jisc definition of blended learning refers to dynamic digital activities. We believe that the dynamic nature of these digital activities could be interpreted in at least two different ways: (1) the digital activities are variable within and between learning sessions and between subsequent iterations of the same learning session with different cohorts; and (2) the digital activities themselves are interactive and require elements of active student participation. In designing our digital activities, we included a range of different activities, chosen to best support attainment of intended learning outcomes (ILOs), and to encourage student learning in different ways. Many of the activities themselves were also interactive rather than being static and predominantly didactic in nature. In blended learning Iteration 1, attendance at the f2f classes was not compulsory and many students were unable to or chose not to attend the f2f classes. In blended learning Iteration 2, an attendance hurdle was initially enforced, but this was relaxed during the semester. We understood from personal communication with students that many students preferred online learning while others prefer f2f on-campus learning. This led us to question the value of these synchronous f2f classes. Specifically, we were interested in investigating how students value the synchronous and asynchronous components of the course and how does each component support student learning. If students do not see value in synchronous classes and these do not further support their learning, we should consider whether a completely online asynchronous option should be provided. Conversely, if these classes significantly enhance student learning, regardless of whether students see value in them, this would support retention of an attendance hurdle. Active learning takes a constructivist approach to learning whereby students learn through active participation rather than passive transmission (Freeman et al., 2014; Waldrop, 2015). A key aim of active learning is to increase deep, transformative learning that can change learners perception of the world and develop new representations of knowledge (Biggs & Tang, 2007; Entwistle & Ramsden, 1983; Marton & Saljo, 1976; Prosser & Trigwell, 1999). We therefore define active learning as that in which students do not passively absorb information, but actively develop their understanding and practice application of knowledge and skill through interactive learning activities, discussion with their peers and teaching staff and ultimately learn through a process of discovery. Examples of active learning include: group problem- solving; completing worksheet activities; participation in tutorials; answering clicker questions or in-class polls; participating in peer instruction; and participating in workshops (Freeman et al., 2014; Matsushita, 2017). There is little disagreement that active learning is beneficial and should be included where appropriate (Chickering & Gamson, 1987; Freeman et al., 2014; Waldrop, 2015). Active learning is often combined with flipped classroom (EDUCAUSE, 2012) where pre-class materials must be studied by students at their own pace (asynchronously) before attending a synchronous active learning class in which students engage in discussions with teachers and peers. Although research on flipped classrooms is still in a nascent stage (DeLozier & Rhodes, 2017) it has been argued that the method is particularly beneficial to students whose performance in traditional educational environments is impaired (Du et al., 2014). A key element of active learning that we took advantage of in both our online and f2f components was interactive knowledge checks, as flipped classrooms have been shown to increase attainment of learning outcomes when quizzes are included in their design (van Alten et al., 2019) and several studies have shown that practice tests improve learning (Butler & Roediger, 2007; Cranney et al., 2009; Vojdanoska et al., 2009). As it has been reported that practice testing with feedback consistently outperforms practice testing alone and protects against perseverance errors, we ensured that all knowledge checks provided students with formative feedback (Dunlosky et al., 2013). We therefore acknowledge the significant potential to maximise student learning through the use of both blended and active learning strategies. Based on our definition of blended learning, both asynchronous eLearning and synchronous learning components can be inherently active and as such our blended learning could be described as active blended learning. Here we describe two iterations of an intervention in which the use of active learning strategies was increased in both asynchronous eLearning and synchronous f2f components of a blended course and evaluate these changes in an attempt to differentiate between the value provided by the asynchronous active online learning opportunities and the synchronous f2f active learning opportunities. 189 A design-based research approach Design-based research is a relatively new but well-established methodology which combines “empirical educational research with the theory-driven design of learning environments” (The Design-Based Research Collective, 2003, p. 5). More recently, design thinking (Elliott & Lodge, 2017) has emerged as a key theme in Visions for Australian tertiary education (James et al., 2017). A framework has been developed that places educational design research along a continuum of the design process: from analysis and exploration of a pedagogical issue and context; through design and construction of an intervention; and finally to evaluation of and reflection on this intervention and implications to the broader context (Kopcha et al., 2015; McKenney & Reeves, 2018). A comprehensive review of undergraduate student experience of blended eLearning has highlighted the requirement to “use blended learning as a driver for transformative course redesign” (Sharpe et al., 2006, p. 4). These recommendations encompass a design-based research approach, reminiscent of that which we have utilised. Here, we report on the design and delivery of the first iteration of an intervention required to address changes in infrastructure. Based on evaluation of and reflection on this first iteration, we report on a second iteration of this intervention which also incorporates further changes based on necessary COVID-19 restrictions. We designed and implemented changes to a second-year cell biology course, with approximately 120 students, at a large Australian university. In Iteration 1, the course was taught through lectures and computer-aided learning (CAL) classes, in Iteration 2, the course was taught through eLearning materials that replaced lectures, and workshops which were an evolution of the CAL classes. Due to a change in learning management system (LMS), existing CAL eLearning materials were required to be redeveloped in a new platform. In addition, infrastructure changes enabled the f2f CAL classes to be relocated from a 1:1 (student:computer) computer lab to a brand new, purpose-built collaborative learning space. The CAL eLearning materials were originally designed to be accessed by students individually in the 1:1 CAL lab. The building change and changing student demographics, specifically increased student bring-your-own- device behaviour has decreased the necessity to provide timetabled teaching in computer labs and a global shift toward collaborative active learning has identified the opportunity for these eLearning materials and the associated f2f classes to be redesigned. Attendance at CALs was not compulsory and it has been observed that attendance rates have progressively declined over the past years. We wanted to make optimal use of the new collaborative learning space and create more engaging eLearning materials. We hoped to encourage student attendance by providing a valuable f2f learning experience but also to provide an active, solely online learning experience for students unable to attend the f2f classes. This proved to be beneficial for us in Iteration 2 of this intervention, when on-campus f2f attendance was not possible due to COVID- 19 restrictions. In an attempt to minimise the impact on teaching staff and to mitigate student expectations we implemented a soft transition, whereby more minor changes were introduced in Iteration 1 with an aim to iteratively make further developments over subsequent iterations. In Iteration 1 there were eight CALs and changes were progressively introduced across the semester such that early CALs relied less on completion of pre- eLearning and involved less group-work and later CALs relied more heavily on completion of pre- eLearning and involved significant in-class group-work. In Iteration 2, the course was modularised, with each week of the 12-week semester comprising a module. In each module, lectures were fully replaced with eLearning materials and group-work was encouraged in weekly workshops across the whole semester. We initially examined student opinions of the eight CAL classes in Iteration 1 and examined the correlation between attendance at CAL classes and final scores. We then conducted a more focused analysis on CAL7 to compare the value of the asynchronous active eLearning materials and participation in the synchronous active learning tasks in terms of perceived learning and demonstration of intended learning outcomes (ILOs) in constructively aligned summative assessment tasks. We then conducted a similar evaluation of Iteration 2, examining student opinions of the 12 modules and correlation between completion of pre- eLearning prior to or after attendance at workshops. Qualitative data analysed were anonymous student evaluations of the course. Data were obtained from online and paper-based questionnaires, transcribed, and imported into NVivo for coding. Data were progressively coded to identify key themes and then calculate coverage of relevant comments by the key themes. Specific quantitative data analysed were student LMS access dates, f2f class attendance records, mid-semester test scores, final exam question item analysis, and final subject scores. These data were collated in Excel and statistically analysed in GraphPad Prism 8.1.2 190 (2019). Only data from students who provided consent for their data to be included in this study were analysed. Mid-semester test data were excluded for two students who received zero for this assessment due to academic misconduct. This research study was approved by the University of Melbourne School of Biomedical Sciences Human Ethics Advisory Group (Ethics IDs 1953765.1 and 1955758.1). Results To gain an overall impression of the benefit of attendance at CAL classes (Iteration 1) and workshops (Iteration 2) we conducted an analysis of student final marks and correlated these to the number of CAL classes or workshops they attended. In both iterations of the intervention we saw a modest positive relationship with students who attend more CAL classes or workshops scoring higher final marks (Figure 1). Figure 1. Attendance at f2f classes Individual final marks (y axis, %) are plotted against the total number of CALs (A, Iteration 1) or workshops (B, Iteration 2) the students attended (x axis). Linear regression demonstrates that the slope of each of the lines of best fit (A slope = 2.5, R2 = 0.1292 and B slope = 1.6, R2 = 0.0055) are statistically significantly positive (A p = 0.0002, B p = 0.0304) demonstrating a relationship between attendance and final mark. To gain an understanding of which CAL classes students found most beneficial, we analysed specific data from an end-of-semester subject evaluation. Students were asked: “Which CAL module assisted your learning the most and why?” Some students mentioned more than one CAL, but out of the eight CALs for this course, CAL7, was most frequently mentioned (n = 26 of 104 respondents). The reasons students provided were subjected to a coded analysis and five major themes emerged: content; integration; interactivity; engagement; and teamwork (Table 1). When asked: “Which CAL module assisted your learning the least and why?” only 5 students mentioned CAL7. The comparable block of learning in Iteration 2 of this subject was module 9. The eLearning materials associated with this module were based on those developed as pre-eLearning materials for CAL7. The workshop for module 7 was modified slightly from the active learning tasks students completed in CAL7. In the end-of-semester subject evaluation for iteration 2 students were asked: “Which module assisted your learning the most and why?” Many students mentioned more than one module, but of the 12 modules, module 9 was most frequently mentioned (n = 35 of 60 respondents). The reasons students provided were coded based on the five themes identified from evaluation of Iteration 1. Four of the five themes were evident in feedback from Iteration 2: content; integration; interactivity; and engagement (Table 1). When asked “Which module assisted your learning the least and why?” only 7 students mentioned module 9. Together these data suggest that overall, CAL7 in Iteration 1 and module 9 in Iteration 2 were well designed and delivered and contributed positively to students’ self-reported learning. These data also demonstrate that students valued the content of the block of learning similarly in both iterations, students valued the integration of concepts and skills in Iteration 2 more highly than in Iteration 1, and students valued opportunities to participate in teamwork in Iteration 1 but not in Iteration 2. 191 Table 1 Aspects of CAL7 and module 9 that students self-reported helped their learning the most Theme Details and student quotes Number of mentions/ percentage text coverage CAL7 (Iteration 1) Module 9 (Iteration 2) Content Information, topics, concise, thorough and detailed, pre-CAL, related to lecture material, able to be self- taught, layout, clear explanations, diagrams, extensive and informative, assessment, intended learning outcomes. “The CALs were a good source for me to review and revise all the content which I learn in lectures.” “Module 9 […] despite likely being the topic with the most information, it felt rather streamlined making it much easier to learn and recall.” 16 (42.4%) 20 (38.4%) Integration Consolidation, relationship to prior knowledge, bigger picture. “The CALs were great, almost like forced revision, really consolidated learning.” “Module 9 assisted my learning the most because it related to many of the other modules and presented some really key concepts that allowed me to make sense of a lot of the other content I had learned previously.” 8 (27.0%) 16 (43.4%) Interactivity Interactive, application, questions, workshop dynamic, examples, real life. “The CALs resulted in my deepest learning in general. The interactive format and in-depth content explanations worked very well.” “These topics really help us understand in depth issues that are often talked about in real life but are not explained scientifically.” 8 (16.8%) 2 (14.3%) Engagement Fun, interesting, engaging. “The CALs were the most beneficial as they were particularly engaging and fun to go through.” “Module 9 has been very interesting to me.” 4 (8.0%) 9 (4.0%) Teamwork Team-based learning, incentive to participate, talking with peers, group-work. “[L]earning it together with my friend clarified my understanding and stimulated my learning more.” 4 (5.8%) Nil Specific Iteration 1 analysis Given that students reported that they felt that CAL7 helped their learning the most out of any of the CALs, we were interested to explore this in more detail and so conducted an ILO-based evaluation of constructively aligned assessment tasks. We analysed student scores in a mid-semester test directly related to CAL7 and scores in a final exam question that assessed concepts and skills covered in CAL7. We were interested in evaluating the performance of students who attended the f2f class and participated synchronously (attend) and students who accessed the eLearning materials independently and asynchronously but did not attend the f2f class (async). We also examined the performance of students who 192 did not attend the f2f class or access the eLearning materials (no access, as determined by LMS analytics). Students who attended CAL7 and students who accessed the CAL7 eLearning materials asynchronously achieved a significantly higher CAL7-related mid-semester test score compared to students who did not attend the class and who did not access the eLearning materials (Figure 2). Interestingly, we saw no significant difference in CAL7-related mid-semester test scores between students who attended the f2f class and those who accessed the eLearning material asynchronously and independently. We also conducted a similar analysis of student scores in a final exam question directly related to CAL7 ILOs. Students who attended or asynchronously accessed the CAL7 eLearning material performed slightly (although narrowly missing statistically significantly) better than students who did not access the CAL7 eLearning materials (Figure 2). These data suggest that the summatively assessed ILOs can be obtained through interaction with the eLearning materials and that there is no added benefit to students in attending the f2f class, in terms of directly-related ILO attainment. Figure 2. Student comparison for CAL7 From mid-semester test scores, students who attended CAL7 (attend, median = 32/40) and students who accessed CAL7 asynchronously (async, median = 30/40) performed better (p < 0.0001) than students who did not access the CAL7 eLearning materials (no access, median = 20/40). There was no statistically significant difference in performance between the attend and async groups. From the exam question scores, students who attended CAL7 (attend, median = 19/25) and students who accessed CAL7 asynchronously (async, median = 20/25) performed slightly, though narrowly not statistically significantly, better in a final exam question directly related to CAL7 content and skills than students who did not access the CAL7 eLearning materials (no access, median = 15/25, p = 0.0507). Statistical significance was determined by Kruskal-Wallis and Dunn’s multiple comparison tests. While we didn’t see a difference in median CAL7-related final exam question scores between CAL7- attendees, CAL7-asynchronous participants and students who did not access CAL7 we noticed that there were fewer students in the async and no access groups who answered the CAL7-related exam question compared to the attend group. We therefore performed a subsequent analysis of this observation. The final exam was divided into three parts: Part A comprised 20 x 1-mark multiple-choice questions; Part B comprised 4 x 10-mark short answer questions; and Part C comprised 2 x 25-mark long answer questions. Students were required to answer all questions in Parts A and B, but only two of 4 questions in Part C. Given that we saw no significant difference in scores for the CAL7-related Part C exam question we further asked whether attendance at CAL7 impacted the tendency of students to select the CAL7-related question. Forty-eight students attended CAL7 and of these 30 (62.5%) chose to answer the CAL7-related exam question. Forty-two students accessed CAL7 asynchronously and of these 21 (50%) chose to answer the CAL7-related exam question. Finally, 22 students did not attend CAL7 or access CAL7 eLearning materials and of these, 6 students (27%) chose to answer the CAL7-related question. Given the available choices, there was a 50% expected chance that any student would answer any given question. Thus, there was a significantly higher percentage of students who attended CAL7 and chose to answer the CAL7-related exam question (p = 0.0104, two-tailed binomial test). The number of students who chose the CAL7-related 193 exam question and accessed CAL7 asynchronously fitted with the expected distribution of 50% and unsurprisingly, students who did not access CAL7 eLearning materials were less likely to choose the related exam question (p = 0.0059, two-tailed binomial test). These data suggest that students who attended the CAL7 active learning class and were therefore able to discuss the material with their peers and tutors may have been more confident in selecting an exam question related to this material. Specific Iteration 2 analysis As described above, based on our analysis of Iteration 1 and the impacts of COVID-19-related restrictions, we made changes to the structure of the course in Iteration 2. As such, our analysis of Iteration 2 differs slightly for that of Iteration 1. It is important to note however, that the ILOs remained the same in both iterations of this course. For comparison to Iteration 1 CAL7-related data, we conducted an ILO-based evaluation of constructively aligned assessment tasks related to Module 9 in Iteration 2. Attendance at workshop 9 was encouraged through addition of an attendance hurdle (students had to attend at least 9 out of 12 workshops, and indeed 97/104 students attended workshop 9). As such, it was not appropriate to compare student performance based on whether they attended the workshops. Instead we grouped students according to whether they asynchronously completed the required eLearning before attending the workshop (before), after attending the workshop (after), or did not access or complete the eLearning materials but did attend the workshop (incomplete). Only a few students did not attend workshop 9 (n = 7) and the majority of these did not provide consent for their data to be included in this study so we do not present data for students who did not attend workshop 9. Students who completed the relevant eLearning materials before attending workshop 9 performed significantly better than students who did not complete the eLearning materials, in both a related mid-semester test and in the relevant final exam question (Figure 3). Importantly, students who completed the eLearning before attending the workshop also performed better than students who did not complete the eLearning until after the workshop in a related mid-semester test. This trend was present in scores for the final exam, however this difference was not significantly different. These data demonstrate a significant benefit to students who completed pre-eLearning prior to attendance at the workshop. Figure 3. Student comparison for Module 9 From the mid-semester test assessing ILOs related to Module 9 content, students who completed the Module 9 eLearning materials asynchronously before attending the workshop (before, median = 36.5/45) performed better than students who did not complete the eLearning materials until after attending the workshop (after, median = 32.5/45, p = 0.0276) and performed better than students who did not access or complete the Module 9 eLearning materials at all but who did attend the workshop (incomplete, median = 27.5/45, p = 0.0005). From the final exam scores for a question directly related to Module 9, students who completed the Module 9 eLearning materials asynchronously before attending the workshop (before, median = 23.5/30) performed better than students who did not access or complete the Module 9 eLearning materials but who did attend the workshop (incomplete, median = 14.25/30, p = 0.0003). Students who attended the workshop and completed the eLearning materials asynchronously after the workshop did not perform statistically differently from the students in the other two groups (after, median = 20.75/30, p = 0.3197). Statistical significance was determined by Kruskal-Wallis and Dunn’s multiple comparison tests. 194 Discussion Using a design-based research approach, we designed, delivered, and evaluated two iterations of blended learning with interactive asynchronous pre-eLearning and synchronous f2f active learning classes. Our qualitative evaluations suggest that students valued the interactive learning activities both in the eLearning material and the active learning activities in f2f classes. Aspects of the blended learning that students self- reported as valuable in both iterations were content, integration, interactivity, and engagement. Students also reported on the value of teamwork in Iteration 1. Student final marks positively correlated with the number of f2f classes students attended. The class that most students identified as best supporting their learning was CAL7 in Iteration 1 and Workshop 9 in Iteration 2. Both delivered the same set of ILOs. We undertook further analysis of these classes, utilising pre-class eLearning content delivery followed by a collaborative problem solving-based active learning class. In Iteration 1, our data show that students who attended the f2f class and students who accessed the eLearning materials independently and asynchronously performed equally-well on assessment of CAL7-related ILOs in a mid-semester test. In contrast, students who did not access the eLearning material performed worse on average. We also found an effect of f2f class attendance on students’ choice to answer a specific exam question, in that students who attended the CAL7 f2f class were more inclined to answer an exam question directly related to CAL7. In Iteration 2, we found that students who completed the eLearning materials prior to attending the workshop performed better in a mid-semester test and final exam question compared to students who did not access or complete the eLearning materials but did attend the workshop. We also saw a lower median score in these assessment tasks in the group of students who completed the related eLearning materials after attending the workshop (but before submitting the assessment tasks). This was significantly different for the mid-semester test but not the final exam question. We suggest that there is a clear benefit to students who complete the eLearning materials prior to participation in an active learning class related to this material. Collectively, these data suggest that while the content can be taught and learned via the eLearning materials, students may gain more confidence in their knowledge and skill by participation in the f2f active learning components of the class. Comparison of the qualitative data provided by students related to Iterations 1 and 2 revealed some similarities and some notable differences. Table 1 described these data in terms of the number of mentions and the percent of text coverage. Unsurprisingly, the content itself was mentioned as the most valuable component of the blended learning in both iterations. In Iteration 1, integration was important to many students, but the value of this aspect was significantly increased in Iteration 2. A considerable change in the structure of teaching and learning was introduced between Iterations 1 and 2, notably the modularisation of the course in Iteration 2. A significant emphasis was put into creating links between modules and scaffolding learning to build on prior modules. These data suggest that this has been performed particularly successfully. The f2f classes in Iteration 1 were held on-campus, while the majority of the f2f classes in Iteration 2 were held via video conferencing. We observed significantly less intra-group interactivity in Iteration 2 as a result of the online teaching format and unsurprisingly this was reflected in fewer student mentions of the value of interactivity. Interestingly, many students commented on the interactivity within the eLearning materials for Iteration 1, but despite the fact that the eLearning materials for Iteration 2 were similar, no students commented on this aspect of the educational design for Iteration 2. Perhaps surprisingly, many students mentioned the level of engagement of Module 9 learning activities in Iteration 2, many students described this module as “fun”. We were pleased that in spite of significant hardship and challenges being experienced by both students and teaching staff at this time, we were still able to deliver a fun and engaging online learning experience for many students. Teamwork was mentioned as a valuable component of the blended learning in Iteration 1 but not at all for Iteration 2. We believe this was due to a general reluctance of many of our students to participate in group work during synchronous online video conferences. It has been reported that the shift to online education due to the COVID-19 pandemic accentuated factors that challenge students’ learning, including issues related to technology access, information communication competencies, language proficiencies, and absence of traditional classroom socialisation (Adnan & Anwar, 2020; Naffi et al., 2020). We believe that especially the collaborative online aspects of the course, which were new to both teachers and students, may have been challenged by these factors. Hopefully, this will be mitigated by allowing students who prefer on-campus learning to attend on- campus f2f classes in Iteration 3. We also plan to address this issue for students who will still be learning online by allowing students to form groups based on preferences for group study format, for example, working collaboratively primarily via video conferencing versus primarily working via chat or messaging 195 apps. We will also trial the use of a more user-friendly collaborative document platform. There was already significant momentum towards increasing online learning opportunities to facilitate flexibility for students in terms of what, when, and where they can study. This has been dramatically accelerated by COVID-19. We must continue to assess whether it is appropriate to offer this subject solely online or if it should continue to be offered as a blended subject with either compulsory or non-compulsory attendance at f2f sessions. Others have shown a robust positive relationship between class attendance and student scores (Devadoss & Foltz, 1996; Louis et al., 2016; Oldfield et al., 2017) and our data support this observation in the context of Australian education, which has been the subject of few studies of this kind (Louis et al., 2016). At an individual class-level, despite students self-reporting that CAL7 helped their learning the most, we did not observe any significant learning gains in students who attended the f2f class versus students who accessed the eLearning materials asynchronously. This demonstrates that we achieved our aim of creating engaging eLearning materials that benefit students equally. However, given that we showed that the average final score is higher depending on how many f2f classes students attend, this could suggest that there is a collective benefit to students in attending a series of f2f classes. Reasons for this may include development of confidence in working as part of a team over the semester, as well as increased subject-specific mastery due to more content engagement. In light of the data from Iteration 1 we imposed an attendance hurdle in Iteration 2, which was informally relaxed due to COVID-19, however we still saw significantly higher attendance overall in Iteration 2. We therefore shifted our analysis to investigate the value of attendance based on the level of preparedness of students. At face value, the groupings of students for quantitative analysis in Iterations 1 and 2 seemed slightly different, but we believed they were able to be compared. In Iteration 1, the students who attended CAL7 had completed the required pre-eLearning and therefore this group was directly comparable to the group of students in Iteration 2 who completed the pre-eLearning before attending Workshop 9. In Iteration 1, the students who did not attend CAL7 but completed the eLearning asynchronously were comparable to the group of students who attended Workshop 9 prior to accessing or completing the required eLearning. We believe that these students would have found it difficult to participate fully in the group tasks, due to their lack of preparation, and as such, likely did not gain any additional collaborative learning benefits. In Iteration 1, the students who did not access the eLearning materials or attend the f2f class were comparable to the group of students in Iteration 2 who attended the f2f class but did not access or complete the eLearning at all before completing the assessment tasks related to this material. In Iteration 1, we saw a statistically significant difference between the mid-semester test marks of students who attended CAL7 or accessed the CAL7 eLearning materials independently and asynchronously compared to students who did not access the CAL7 eLearning material. Data from a CAL7-related exam question showed a similar trend but the differences were not significantly different. One reason for this could be the variable group sizes as a result of skewed student selection of this exam question based on whether students attended the CAL class or did not access the eLearning materials at all. This hypothesis was supported by the results for Iteration 2, which demonstrated comparable data for both assessment tasks, when sample sizes were less skewed. An opposing hypothesis, was that Iteration 1 resulted in short-term but not long-term benefits of the learning intervention, as it has previously been shown that team-based learning may induce short-term learning gains which do not persist in the long-term (Emke et al., 2016). The CAL7-related mid-semester test occurred earlier than the final exam, so the learning gains we observed in mid-semester test scores may have reflected a short-term increase in student learning that was not retained over a longer timeframe. Pedagogically, it has been suggested that offering question choices in assessment tasks may reduce the reliability of student marks due to variability in question difficulty (Bloxham & Boyd, 2007). Indeed, we observed variable average marks for each of the four questions in Part C of the final exam for Iteration 1 (data not shown). Based on these data and as a result of changed final exam format due to COVID-19- related restrictions, we did not include question choices in the final exam for Iteration 2. In addition, as already mentioned we did not have sufficient data from students who did not attend Workshop 9, so analysis of mapped exam question performance against attendance was not possible. Our quantitative analysis of Iteration 2 demonstrated that completion of eLearning materials prior to attendance at a workshop and participating in collaborative group problem-solving activities resulted in higher performance in related assessment tasks compared to attending without having completed the pre- 196 eLearning. This was shown clearly in scores for the mid-semester test. Data for the related final exam question did not demonstrate a statistically significant difference between these groups, despite a lower median score in the latter group. The number of students in this subject and the number of students who provided consent for their data to be included in our analyses limited the power obtained from these analyses. We believe that the data demonstrated a trend towards a significant difference in the Workshop 9-related final exam question score in these groups but that this difference was not statistically significant due to sample size. An additional observation was the clearly expanded distribution of exam question scores in the group of students who completed the eLearning after the workshop. This may have reflected different performance of students based on many other factors, including engagement with other relevant eLearning materials, attendance at other related workshops, and prior or concurrent learning from other relevant courses. Forced online facilitation of these classes due to COVID-19 restrictions actually provided us with a unique opportunity to evaluate whether we could replicate the benefits of an on-campus f2f experience with an online f2f experience. We believe that our qualitative data demonstrates that an on-campus experience is more beneficial to students, particularly in facilitation of interactivity during active learning classes and development of teamwork skills. In future, as attendance restrictions allow, we will offer a synchronous on-campus active learning experience to all students. Additionally, we will further refine our online offering to try to increase the value of this learning environment for those students unable to attend on-campus classes, allowing students the option of performing group work semi-synchronously, in that the group itself must work synchronously but that all groups do not need to be working synchronously. One potential confounding aspect of our study was that the assessment tasks analysed were completed by students individually, yet we related this to both their completion of asynchronous individual eLearning and their attendance at a synchronous collaborative active learning class. Future investigations of these and similar data will also include analysis of student performance in group assessment tasks. To aid analysis of these activities we will draw on aspects of the developmental assessment of twenty-first century skills framework which has been designed to effectively assess collaborative problem solving skills, in addition to other twenty-first century skills (Care & Kim, 2018). We hypothesise that students who complete pre- eLearning and participate regularly and actively in synchronous group work will perform much better in assessment tasks that directly assess and are more highly dependent on these skills. In Iteration 1, student performance was similar after the f2f class and after accessing the eLearning independently and asynchronously, we hypothesised that this may have been impacted by the fact that these assessment tasks asked students to remember, understand, and apply knowledge, while collaborative problem solving may have been effective in fostering learning in higher levels of Bloom’s Taxonomy (Allen et al., 2013; Anderson et al., 2001). In Iteration 2, we designed assessment tasks that required more application and analysis and our data suggest that differences were evident between more prepared (before) and less prepared (after) students. Additionally, in Iteration 2, as intended and as demonstrated by student feedback, integration of content and skills was a significant aspect of the blended learning that assisted student learning. However, our analysis focussed on a more granular aspect of the subject: one module and the associated assessment tasks. It may therefore be regarded as overly simplistic to analyse the value of one eLearning module and one workshop for those ILOs. A more holistic analysis may reveal further insight. For example, a more detailed evaluation of engagement with eLearning materials, participation in multiple, linked workshop activities and a more comprehensive assessment analysis could potentially reveal further insights. Our findings support the further development and evaluation of interactive and active blended learning. One specific aspect we will continue to focus on and develop is assessment reform and shifting the focus from individual to group assessment and including assessment of a range of lower to higher order cognition and twenty-first century skills. Although our results showed support for student attainment of ILOs via a fully online educational experience, it was also clear that on-campus f2f interactions can significantly enhance the student experience, which is key at the undergraduate level as students develop their self- regulation skills. References 197 Adnan, M., & Anwar, K. (2020). Online learning amid the COVID-19 pandemic: Students’ perspectives. Journal of Pedagogical Sociology and Psychology, 2(1), 45–51. https://doi.org/10.33902/JPSP. 2020261309 Allen, R. E., Copeland, J., Franks, A. S., Karimi, R., McCollum, M., Riese, D. J., & Lin, A. Y. F. (2013). Team-based learning in US colleges and schools of pharmacy. American Journal of Pharmaceutical Education, 77(6), 115. https://doi.org/10.5688/ajpe776115 Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., Raths, J., Wittrock, M. C. (Eds.) (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Pearson. Biggs, J., & Tang, C. (2007). Teaching for quality learning at university: What the student does (3rd ed.). Open University Press: McGraw-Hill Education. Bloxham, S., & Boyd, P. (2007). Developing effective assessment in higher education: A practical guide. Open University Press: McGraw-Hill Education. Brown, C., Davis, N., Sotardi, V., & Vidal, W. (2018). Towards understanding of student engagement in blended learning: A conceptualization of learning without borders. In M. Campbell, J. Willems, C. Adachi, D. Blake, I. Doherty, S. Krishnan, S. Macfarlane, L. Ngo, M. O'Donnell, S. Palmer, L. Riddell, I. Story, H. Suri, & J. Tai (Eds.), Open oceans: Learning without borders. Proceedings of ASCILITE (pp. 318–323). https://2018conference.ascilite.org/conference-proceedings/ Butler, A. C., & Roediger, H. L. (2007). Testing improves long-term retention in a simulated classroom setting. European Journal of Cognitive Psychology, 19(4–5), 514–527. https://doi.org/10.1080/09541440701326097 Care, E., & Kim, H. (2018). Assessment of twenty-first century skills: The issue of authenticity. In E. Care, P. Griffin, & M. Wilson (Eds.), Assessment and teaching of 21st century skills (pp. 21–39). Springer. Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. Washington Centre News. Cranney, J., Ahn, M., McKinnon, R., Morris, S., & Watts, K. (2009). The testing effect, collaborative learning, and retrieval-induced facilitation in a classroom setting. European Journal of Cognitive Psychology, 21(6), 919–940. https://doi.org/10.1080/09541440802413505 DeLozier, S. J., & Rhodes, M. G. (2017). Flipped classrooms: A review of key ideas and recommendations for practice. Educational Psychology Review, 29(1), 141–151. https://doi.org/10.1007/s10648-015-9356-9 Devadoss, S., & Foltz, J. (1996). Evaluation of factors influencing student class attendance and performance. American Journal of Agricultural Economics, 78(3), 499–507. Du, S.-C., Fu, Z.-T., & Wang, Y. (2014). The flipped classroom-advantages and challenges. Proceedings of the International Conference on Economic Management and Trade Cooperation (pp. 17–20). Atlantis Press. Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, Supplement, 14(1), 4–58. https://doi.org/10.1177/1529100612453266 EDUCAUSE (2012). 7 Things you should know about flipped classrooms. https://library.educause.edu/resources/2012/2/7-things-you-should-know-about-flipped-classrooms Elliott, K., & Lodge, J. M. (2017). Engaging university teachers in design thinking. In R. James, S. French, & P. Kelly (Eds.), Visions for Australian tertiary education (pp. 55–66). Melbourne Centre for the Study of Higher Education. Emke, A. R., Butler, A. C., & Larsen, D. P. (2016). Effects of team-based learning on short-term and long-term retention of factual knowledge. Medical Teacher, 38(3), 306–311. https://doi.org/10.3109/0142159X.2015.1034663 Entwistle, N., & Ramsden, P. (1983). Understanding student learning. Routledge, Taylor & Francis Group. Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410-8415. https://doi.org/10.1073/pnas.1319030111 Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher education. The Internet and Higher Education, 7(2), 95–105. https://doi.org/10.1016/j.iheduc.2004.02.001 https://doi.org/10.5688/ajpe776115 https://2018conference.ascilite.org/conference-proceedings/ https://doi.org/10.1080/09541440701326097 https://doi.org/10.1080/09541440802413505 https://doi.org/10.1007/s10648-015-9356-9 https://doi.org/10.1177/1529100612453266 https://library.educause.edu/resources/2012/2/7-things-you-should-know-about-flipped-classrooms https://doi.org/10.3109/0142159X.2015.1034663 https://doi.org/10.1073/pnas.1319030111 https://doi.org/10.1016/j.iheduc.2004.02.001 198 Hibberson, S., Barrett, E., & Mote, K. (2020). Developing blended learning approaches. https://www.jisc.ac.uk/guides/creating-blended-learning-content James, R., French, S., & Kelly, P. (2017). Visions for Australian tertiary education. https://melbourne- cshe.unimelb.edu.au/__data/assets/pdf_file/0006/2263137/MCSHE-Visions-for-Aust-Ter-Ed- web2.pdf Kopcha, T. J., Schmidt, M. M., & McKenney, S. (2015). Editorial 31(5): Special issue on educational design research (EDR) in post-secondary learning environments. Australasian Journal of Educational Technology, 31(5). https://doi.org/10.14742/ajet.2903 Littlejohn, A., & Pegler, C. (2007). Preparing for blended e-learning. Routeledge. Louis, W. R., Bastian, B., McKimmie, B., & Lee, A. J. (2016). Teaching psychology in Australia: Does class attendance matter for performance? Australian Journal of Psychology, 68(1), 47–51. https://doi.org/10.1111/ajpy.12088 Marton, F., & Saljo, R. (1976). On qualitative differences in learning: I - Outcome and process. British Journal of Educational Psychology, 46(1), 4–11. Matsushita, K. (2017). Deep active learning: Toward greater depth in university education. Springer. https://doi.org/10.1007/978-981-10-5660-4_10 McKenney, S., & Reeves, T. C. (2018). Conducting educational design research. Routeledge. Naffi, N., Davidson, A.-L., Patino, A., Beatty, B., Gbetoglo, E., & Duponsel, N. (2020). Online learning during COVID-19: 8 ways universities can improve equity and access. The Conversation. https://theconversation.com/online-learning-during-covid-19-8-ways-universities-can-improve-equity- and-access-145286 Norton, A., & Cakitaki, B. (2016). Mapping Australian higher education 2016. Grattan Institute. http://grattan.edu.au/report/mapping-australian-higher-education-2016/ Oldfield, J., Rodwell, J., Curry, L., & Marks, G. (2017). Psychological and demographic predictors of undergraduate non-attendance at university lectures and seminars. Journal of Further and Higher Education, 42(4), 509–523. https://doi.org/10.1080/0309877X.2017.1301404 Prosser, M., & Trigwell, K. (1998). Understanding learning and teaching: The experience in higher education. Open University Press. Sharpe, R., Benfield, G., Roberts, G., & Francis, R. (2006). The undergraduate experience of blended e- learning: A review of UK literature and practice. The Higher Education Academy. https://www.advance-he.ac.uk/knowledge-hub/undergraduate-experience-blended-e-learning The Design-Based Research Collective (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5–8. https://doi.org/10.3102/0013189X032001005 The University of Melbourne (n.d.). Teaching and learning, innovation initiatives, pedagogy and curriculum innovation. FlexAP project. https://about.unimelb.edu.au/teaching-and- learning/innovation-initiatives/pedagogy-and-curriculum-innovation/flexap-project van Alten, D. C. D., Phielix, C., Janssen, J., & Kester, L. (2019). Effects of flipping the classroom on learning outcomes and satisfaction: A meta-analysis. Educational Research Review, 28. https://doi.org/10.1016/j.edurev.2019.05.003 Vojdanoska, M., Cranney, J., & Newell, B. R. (2009). The testing effect: The role of feedback and collaboration in a tertiary classroom setting. Applied Cognitive Psychology, 24(8), 1183–1195. https://doi.org/10.1002/acp.1630 Waldrop, M. (2015). The science of teaching science. Nature, 523, 272–274. https://doi.org/10.1038/523272a Corresponding author: Charlotte Clark, charlotte.clark@unimelb.edu.au Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY-NC- ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY- NC-ND 4.0. https://www.jisc.ac.uk/guides/creating-blended-learning-content https://melbourne-cshe.unimelb.edu.au/__data/assets/pdf_file/0006/2263137/MCSHE-Visions-for-Aust-Ter-Ed-web2.pdf https://melbourne-cshe.unimelb.edu.au/__data/assets/pdf_file/0006/2263137/MCSHE-Visions-for-Aust-Ter-Ed-web2.pdf https://melbourne-cshe.unimelb.edu.au/__data/assets/pdf_file/0006/2263137/MCSHE-Visions-for-Aust-Ter-Ed-web2.pdf https://doi.org/10.14742/ajet.2903 https://doi.org/10.1111/ajpy.12088 https://doi.org/10.1007/978-981-10-5660-4_10 https://theconversation.com/online-learning-during-covid-19-8-ways-universities-can-improve-equity-and-access-145286 https://theconversation.com/online-learning-during-covid-19-8-ways-universities-can-improve-equity-and-access-145286 http://grattan.edu.au/report/mapping-australian-higher-education-2016/ https://doi.org/10.1080/0309877X.2017.1301404 https://www.advance-he.ac.uk/knowledge-hub/undergraduate-experience-blended-e-learning https://doi.org/10.3102/0013189X032001005 https://about.unimelb.edu.au/teaching-and-learning/innovation-initiatives/pedagogy-and-curriculum-innovation/flexap-project https://about.unimelb.edu.au/teaching-and-learning/innovation-initiatives/pedagogy-and-curriculum-innovation/flexap-project https://doi.org/10.1016/j.edurev.2019.05.003 https://doi.org/10.1002/acp.1630 https://doi.org/10.1038/523272a https://creativecommons.org/licenses/by-nc-nd/4.0/ https://creativecommons.org/licenses/by-nc-nd/4.0/ 199 Please cite as: Clark, C. E. J., & Post, G. (2021). Preparation and synchronous participation improve student performance in a blended learning experience. Australasian Journal of Educational Technology, 37(3), 187-199. https://doi.org/10.14742/ajet.6811 https://doi.org/10.14742/ajet.6811