Australasian Journal of Educational Technology, 2020, 36(4). 1 Conceptualising a visual representation model for MOOC- based blended learning designs Laia Albó and Davinia Hernández-Leo ICT Departament, Universitat Pompeu Fabra, Barcelona This article reports on a study about how massive open online course (MOOC)-based blended learning designs can be visually represented to facilitate their comprehension and sharing. We carried out an iterative co-creation process with different stakeholders to conceptualise a visual learning design representation model within the context of blending MOOCs with face-to-face courses. The data analysed was derived from questionnaires and the generated representations. Results indicate that the representation enabled educators to easily visualise the overall structure of the learning designs and the relationships between the different design elements, providing a context for fostering reflection and decision-making during the planning of MOOC-based blended learning designs. Implications for practice or policy: • Course designers can use the proposed visual representation model for planning their MOOC-based blended learning designs and as a reflection and decision-making support tool during the design process. • Educators may benefit from sharing their MOOC-based designs representations and access the designs of others as they could learn from the successful and unsuccessful educational practices within their communities. Keywords: visual representation, hybrid MOOCs, blended learning, learning design, co- creation Introduction There is a growing body of research that describes the significant impact that massive open online courses (MOOCs) have had on universities in terms of introducing changes to their teaching and learning (Andone, Mihaescu, Ternauciuc, & Vasiu, 2015). Specifically, the use of MOOCs as part of face-to-face (f2f) regular university courses has emerged as a new form of blended learning (BL). MOOC-based BL designs have spread taking different forms and combinations (Albó & Hernández-Leo, 2017; Delgado-Kloos, Muñoz- Merino, Alario-Hoyos, Ayres, & Fernández-Panadero, 2015; Pérez-Sanagustín, Hilliger, Alario-Hoyos, Delgado Kloos, & Rayyan, 2017). But there is still abundant room for further progress in the sharing of these types of BL practices in order to explore their full potential as well as offer more quality learning opportunities to learners. Whereas a number of studies have found that BL has a positive impact on teaching and learning effectiveness (Garrison & Kanuka, 2004; López-Pérez, Pérez-López, & Rodríguez-Ariza, 2011; Means, Murphy, & Baki, 2013; Moskal, Dziuban, & Hartman, 2013), others have highlighted that the “studies of effectiveness lack consistency in what constitutes BL environments, and what outcomes are being compared” (Siemens, Gašević, & Dawson, 2015, p. 74). Thus, there is limited evidence on which pedagogy or technology influences learning outcomes in BL scenarios (Arbaugh, 2014; Littlejohn & Pegler, 2007). Furthermore, and by extension, very little is known about the best ways of designing effective MOOC- based BL (Bralić & Divjak, 2018). Research in this area largely appears in the form of case studies composed of long text descriptions, which often provide design recommendations based on the lessons learned. Yet they sometimes omit details about how the blended schema has been articulated (e.g., structure of the course, activity descriptions, technology used, or pedagogy applied). This hinders practitioners who want to understand the reported case, learn from it, or even replicate it – encumbering the effective sharing of the BL practices as well as the evaluation and comparison of the outcomes. Therefore, some studies have reported the need to provide support to practitioners who are willing to implement BL and help them to face the challenges which arise during the process (Albó, Hernández-Leo & Oliver, 2016; Albó & Hernández-Leo, 2017; Moskal et al., 2013; Porter, Graham, Spring, & Welch, 2014). Australasian Journal of Educational Technology, 2020, 36(4). 2 In this respect, the field of learning design (LD) might shed some light as one of its main aims is to provide support on “how to represent teaching practice in an appropriate form to enable teachers to share ideas about innovative online pedagogy and think about the process of design” (Agostinho, Bennett, Lockyer, & Harper, 2011, p. 97). The use of a systematic way of representing BL designs would facilitate their comprehension and would enable sharing and comparison between the outcomes of the different blended LDs to study their effectiveness in a more accurate way and to lead to an improvement of these types of practices. Thus, the main aim of this study was to investigate how MOOC-based BL designs (as an example of complex BL) can be represented visually to facilitate their comprehension and sharing. The article focuses on the following general research questions: • G-RQ1: What are the necessary LD elements to visually represent a MOOC-based BL design? • G-RQ2: How can these LD elements be articulated in a visual representation to facilitate comprehension of the whole design? To address these questions, we carried out an iterative co-creation process with different stakeholders to conceptualise a visual BL design representation model for educators within the context of mixing MOOCs with f2f courses. The iterative process was composed of three cycles: (a) creation and evaluation of a first model proposal, (b) creation and evaluation of the second model proposal, and (c) the formulation of the final model based on the evaluation results. Visual representation of MOOC-based BL designs Studies over the past two decades have provided important information on how to represent LDs (Persico et al., 2013). Although most of the frameworks used by practitioners for describing teaching practices are text-based (Conole & Wills, 2013; Goodyear, 2004), some researchers have highlighted the potential of visual approaches for representing LDs (Agostinho, 2011). Conole and Wills (2013) identified three benefits of visualisation: it can support teacher’s design thinking; it helps make the design explicit and sharable to others; and it provides a way of representing and articulating the design process. These are part of the reasons why the representation we sought in this study should be mostly visual. However, visual representations also present some challenges regarding the level of abstraction, which we also considered during the conceptualisation of our model. In some cases, graphical solutions can be too abstract for easy interpretation by educators and the time needed to use this type of representation can make their use not worth it – a similar abstraction challenge was well described in the case of pattern language by Winters and Mor (2009). Thus, it is necessary to find a balance between the capability of the graphical elements of the visual representation to be able to represent the educational practices and level of abstraction. Most representations combine visual elements with text-based support. However, very few consider and highlight the BL concept as a challenge to be represented. In the meantime, debate continues on providing a definition of BL. Heinze and Procter (2004, p. 11) defined BL as follows: Blended Learning is learning that is facilitated by the effective combination of different modes of delivery, models of teaching, and styles of learning, and founded on transparent communication amongst all parties involved with a course. Littlejohn and Pegler (2007) presented the concept of blended e-learning by defining it as a hybrid model that enables the coexistence of conventional f2f teaching methods and newer e-learning activities and resources in a single course. Nonetheless, they argued that the term can often refer to the proportion of e- learning (BL), the mix of media (media blend), or the way in which activities are used together (activity blend). They further added that there is also another way of looking at blending: as a combination of on- campus and off-campus activity – arguing that “the level of student experience with e-learning as well as their location relative to campus can be important factors in determining how, where and when to blend e- learning with conventional teaching” (p. 43). Other researchers have claimed that, by definition, education has always been blended, and they have highlighted time and synchronicity as the primary elements of learning environments (Norberg, Dziuban, & Moskal, 2011). Turning to the key elements of an LD, most representations draw upon common model elements such as activities (or tasks), people or actors (students and tutors), and resources. Thus, LD representations aim to Australasian Journal of Educational Technology, 2020, 36(4). 3 define how these elements are related to each other, for example, by defining how the activities are sequenced and which actors and resources are involved in them. To achieve this objective, Littlejohn and Pegler (2007) argued that sequencing requires a timeline. Likewise, more authors have highlighted the importance of the time element in designing for learning, stating that a learning sequence is essentially time-based and that it demands a plan (Dalziel, 2003, 2015; Laurillard et al., 2013; Pozzi, Ceregini, & Persico, 2016). Reinforcing the above arguments, a study by Lai, Portolese, and Jacobson (2017) demonstrated that activity sequencing can have a meaningful impact on deep learning and transfer. Accordingly, we think that our model should have time (e.g., a timeline) as one of its main elements (according to which activities and resources are sequenced). Regarding the resources, it is important to emphasise that the particular representation of the use of MOOC resources was not considered at all when conceptualising the representations listed above (even if in some cases they considered technology-enhanced learning (TEL) interventions). We argue that the complexity of planning and representing BL which incorporates MOOCs is underexplored. Whereas problem-based LDs are considered by Littlejohn and Pegler (2007) as complex blended because of their non-linear sequencing of activities, planning MOOC-based blended courses also presents some challenges regarding time representation. MOOCs are bound to specific time offerings – they usually have registration periods and run during set times with corresponding starting and end dates (there are cases when MOOCs remain open-ended or present other time configurations), often defined by the MOOC platform or by the owner of the course. Thus, these time constraints add complexity to the planning of MOOC-based blended courses, especially when a professor wants to use an external MOOC and needs to combine it with the schedule of their f2f course. In this context, time becomes an even more important factor as a key part of the design process as well as for planning the blended elements in advance. Thus, we argue that MOOC-based BL designs can be considered as complex BL, which need to be carefully planned, resulting in positive institutional transformation when this (together with receiving proper support) occurs (Moskal et al., 2013; Porter et al., 2014). Therefore, a visual representation of MOOC-based BL designs is deemed necessary as it could help advance the planning and sharing of these types of educational practices by educators. Methodology Participants and sample This study aimed at defining the main structure and elements for the visual model in collaboration with the practitioners and experts. Thus, we carried out two workshops in different contexts scheduled according to opportunity: a teaching innovation conference and a research project event. Participants had different profiles depending on the workshop; 39 people attended (Table 1). Table 1 Research context descriptions and participant profiles Context Participants Workshop Dates Description Profile Number Workshop with practitioners May 2016 Local teaching innovation conference MOOC-related staff & university professors 24 Workshop with experts June 2017 Members of a research project event Expert researchers on TEL – also with teaching experience 15 Total: 39 Of the 24 practitioners who attended the first workshop, 13 were staff responsible for supporting the development of MOOCs in their universities; 8 were university teachers; one participant was a university student; one was a researcher; and another was a university staff member with no direct responsibility around MOOCs. All participants were from 10 Catalan universities (six public and four private) and attended the workshop voluntarily when they registered for the conference. Despite the sampling being accidental, the group’s main characteristics are shared with those of the population of interest of the current study: people connected and experienced with MOOCs who were interested in learning how to use them in BL approaches on campus (Albó & Hernández-Leo, 2016). Australasian Journal of Educational Technology, 2020, 36(4). 4 In the case of the workshop with experts, participants were 15 researchers from four universities (three of them from different regions of Spain), most of whom had expertise in BL and/or LD (Figure 1). Moreover, 11 participants had experience in teaching at undergraduate and/or graduate levels, whereas two of them also had experience at high school level and another two in adult education. Only one participant indicated they had experience in primary school and another one in vocational training. Regarding the years of teaching experience, three of them had 1 year of experience, four participants had between 6 and 9 years of experience, and most of them (seven) had more than 10 years of teaching experience. Figure 1. Participants’ expertise level (second workshop – with experts) Despite the model aiming to serve teachers as designers of (MOOC-based) BL scenarios, other roles such as administration, MOOC production, or teaching support may also be involved in this process. This is why all these stakeholders, as well as experts, were involved in the co-design process of the model. The data collection and analysis followed ethical considerations avoiding harm to participants, respecting confidentiality, and ensuring that their participation was voluntary. At the beginning of the two workshops, we explained the context of the study and sought informed consent from the participants. In the case of the first workshop (where participants were expressing their own BL cases), we asked them explicit permission for sharing the generated artefacts. Therefore, only those designs for which they gave permission have been published in the final instance (anonymised) and used in this study (17 LDs). Procedure and instrumentation As described above, the iterative co-creation process for conceptualising the visual representation model was composed of three cycles: (a) creation and evaluation of a first model proposal, (b) creation and evaluation of the second model proposal, (c) the formulation of the final model based on the evaluation results. During the conceptualisation phase of the visualisation model, we used co-creation (Albó & Hernández-Leo, 2018) and carried out two participatory design workshops using paper prototyping (Muller & Druin, 2012; Novick, 2000) activities (resulting in potential artefacts to be analysed). Participatory designers act as facilitators, empowering users in making their own decisions (Clement, 1994). As literature has reported, including end users as active participants in the technology design process (using participatory design techniques) can be an effective means for improving technology designs (in our case the design of a visual representation) as well as a valuable resource in design-based research frameworks (Bonsignore et al., 2013; Muller & Druin, 2012). In our study, participants shared and discussed in groups and completed a final individual questionnaire for sharing their reflections with us (Figure 2). First of all, we developed a first visual representation model proposal based on the existing literature about BL, which was used by the participants during the first workshop. Then, at the end of the first workshop, we redesigned the first model based on the workshop results (also with the support of the literature) leading to a second model which was used in the second workshop. Ultimately, the results of the second workshop enabled us to elaborate a final model proposal (see the red boxes in Figure 2). In order to evaluate the two intermediate models, we formulated three evaluation questions, as follows: • E-RQ1. To what extent is it possible to represent a MOOC-based blended LD using the model? • E-RQ2. Which are the main strengths and weaknesses of the model? • E-RQ3. To what extent is the visual model helpful for understanding others’ LDs? Australasian Journal of Educational Technology, 2020, 36(4). 5 The above evaluation questions enabled us to identify the design elements that could contribute to clarifying and facilitating comprehension of the design, and in the process, to discard the ones which could confuse its understanding (answering G-RQ1: What are the necessary LD elements to visually represent MOOC- based BL design?). The evaluation questions were also aimed at validating the articulation potential of these elements within the intermediate models to propose a final visual representation model (answering G-RQ2: How can these LD elements be articulated in a visual representation to facilitate comprehension of the whole design?). Figure 2. Conceptualisation phase procedure and instrumentation The aim of the first 2-hour workshop was to challenge each participant to design a BL course using MOOCs. The workflow of the activity was a five-step process described in detail by Albó and Hernández- Leo (2016) and involved the use of printed materials. Specifically, a step of the workshop was asking participants to describe their own blended cases and represent them using the first visual representation model. The main objective of the second workshop, on the other hand, was to explore with the participants how to visually represent blended LDs and how these visualisations can facilitate the understanding of others’ LDs. In this event, participants did not work with their own cases. Instead, we prepared description sheets of two cases (using MOOCs mixed with f2f courses) extracted from the literature (see the case descriptions in Appendix 1). Moreover, the main material used was a paper template with the second visual model. The template (Figure 3) included the timeline as the main element of the visual representation and an empty space for placing paper activity cards below it. These cards were to be drawn and filled in with stickers by the participants using the legend provided. Figure 3. Material for the second workshop: printed visual LD model template in A3 paper, activity cards, and legend for completing the activity cards with stickers Australasian Journal of Educational Technology, 2020, 36(4). 6 The second workshop lasted an hour, with the 15 participants working in groups of two or three people using a template per workgroup. The two printed case descriptions were divided equally between the existing groups, in such a way that half of the groups worked with one of the examples and the other half with the other one. Once each group had a case description sheet, they went through the following steps: (1) In the first step, participants had to read the case provided and represented it using the template sheet – placing the activities and the resources described in the case, using the timeline, and filling in the activity cards (and placing them on the sheet) (Figure 3). (2) After completing the LD template for their case (Figure 4, left), participants had to exchange the completed templates between work groups and interpret the template produced by another group, by only looking at the visual representation and without knowing the LD case description of the template received as the exchange occurred between groups with different cases. (3) Finally, each group could check if they had understood the template received by looking at the corresponding case description. Lastly, participants were asked to complete a questionnaire providing their insights about the process. Figure 4. Design template completed during the second workshop Data collection, analysis, and ethics In both workshops, the artefacts generated by the participants served as research data for the study. In the case of the first workshop (with practitioners), we collected the design templates, which included the context and description of the participants’ cases as well as their LD representations using the first model. Similarly, the design templates of the second workshop (with experts), which used the second model, were collected for their analysis. Furthermore, at the end of both workshops, we requested participants to complete an online questionnaire to get their insights about the design process using the models (Figure 2). In the questionnaire of the first workshop, we used an open-ended question to ask whether participants had found difficulties representing their cases using the model provided and to seek for suggestions for improving the model. Australasian Journal of Educational Technology, 2020, 36(4). 7 The questionnaire of the second workshop had six questions as follows: (1) What difficulties did you find during the design process? (related to E-RQ1) (2) Indicate your level of agreement with: “The visual representation helped me to understand the LD explained in the sheet” (related to E-RQ1) (3) What do you think are the main strengths of the visual representation? (related to E-RQ2) (4) What do you think are the main weaknesses of the visual representation? (tackling E-RQ2) (5) Do you think it provides a useful summary of an LD? (tackling E-RQ1/RQ3) (6) What suggestions would you recommend to improve the visual representation? (related to E- RQ2/RQ3) The first question aimed to identify the main difficulties of using the provided model for its later improvement. The second question served to evaluate the usefulness of model for others. Finally, the last four questions – extracted from Agostinho (2011, p. 966) – addressed the usefulness of the representation. All questions were open-ended except the second one, which was a 5-point Likert scale of agreement. The resulting qualitative data from the questionnaires were coded with inductive thematic analysis driven by the research questions of each phase and were cross-referenced to justify interpretations. The main topics were then categorised in order of dominance and triangulated with the artefacts, resulting in a more in- depth analysis for corroborating the overall consistency of the findings. Evaluation of the first visual representation model The first model is based on a timeline composed of two layers: online and f2f learning (Figure 5). The granularity of the representation is variable, and it is defined between the start and end dates of the module, course, or activity, depending on what the teacher is willing to describe. Within the timeline, the teacher can mark as completely dark the parts of the timeline where the activity is mandatory or use stripes to indicate the parts of the timeline where the activities are optional for the students. In both cases, marking the corresponding layer is needed depending on whether the activity is online or f2f. In the case of this study, the online layer refers to online work which happens in a MOOC or MOOCs since participants of the workshop would report MOOC-based BL cases. By contrast, the f2f learning layer refers to the brick and mortar classes at the university. Figure 5. The first version of the model The use of the timeline as one of the main elements of the visual representation is justified based on the importance of the time element in designing for learning; as some authors state, a learning sequence is essentially time-based and it demands a plan (Dalziel, 2003, 2015; Laurillard et al., 2013; Pozzi et al., 2016). Furthermore, the two layers distinguishing between online and f2f refer to the definition of BL provided by Littlejohn and Pegler (2007). The importance of setting the obligatory nature of the activities is supported by the insights by Kirkwood and Price (2005) (cited by Littlejohn & Pegler, 2007), who stated that where the use of new media is optional or incidental, students will typically not value the material presented. Thus, the blended elements could be undervalued, which would affect the overall design effectiveness evaluation. Therefore, we argue that it is an important factor to consider and visualise in the representations of BL designs to ensure an accurate comparison of the final outcomes. The first evaluation aimed to provide an understanding of the potentialities and weaknesses of the first model, and used the evaluation questions E-RQ1 (To what extent is it possible to represent a MOOC-based blended LD using the model?) and E-RQ2 (Which are the main strengths and weaknesses of the model?). Australasian Journal of Educational Technology, 2020, 36(4). 8 Results and discussion All participants were able to represent their cases using the visual model provided with no major challenges (answering E-RQ1). The questionnaire collected only three minor suggestions from the participants for improving the LD template without making direct reference to the visual representation model. Turning now to the specific information gathered from the LD templates, the most striking result to emerge from the data is that the cases reported present a certain variability regarding how the MOOCs were integrated into the f2f course. The complete design descriptions and their corresponding educational contexts can be seen in Appendix 2 (Table A2-1). Interestingly, looking only at the text information provided by the participants in the LD template, it has been challenging to group them into similar cases, as from the description and context provided in some designs we did not find enough similarities to be able to put them in a specific group. Surprisingly, the visual representations compiled facilitate the classification of the LDs according to known pedagogical models much faster and easier (Figure 6), allowing for the recognition of the existing variability in the designs in a totally visual way. Looking at the visualisations generated by the participants made it much more effective to classify the designs than only reading the description and context provided (which in some cases were confusing). Figure 6. Artefacts of representations from the first workshop (visually classified according to their pedagogical model) Australasian Journal of Educational Technology, 2020, 36(4). 9 The four models in Figure 6 have been extracted from the experiences of other researchers with MOOC- based BL designs (Delgado-Kloos et al., 2015). One of the main strengths of the visual representation model (answering E-RQ2) is that it enables us to understand how the f2f teaching and the online work using the MOOC are related in terms of time, thus facilitating the identification of the models in a visual way. Lastly, the classification in Figure 6 has been done based only on the visual representations; we then checked the corresponding LD text descriptions to corroborate the results. In three designs out of 17 (IDs 1, 4, and 12 in Figure 6), the visual representations were somehow confusing, and it was necessary to consult the LD descriptions provided as a complement to the visual representations to finally decide in which model they would be classified. An implication of this is the possibility that anyone can relate an LD to an example of a specific model without the need to be an expert (identifying strength of the model, E- RQ2), facilitating the search for similar designs in a visual, easy and effective way, and avoiding long descriptions. Regarding the weaknesses of the visual LD representation, some limitations of the model have been detected (addressing E-RQ2). Firstly, it is challenging to visualise cases where the blended part differs from combining brick and mortar courses with online teaching from the MOOC; for example, an online course from a distance university which uses MOOCs, where the f2f contact is left for non-formal hours – although this case can be considered as MOOC-based BL, it is not possible to represent it using the first model as it has only an online layer and it would be necessary to have more than one (i.e., one for the online course and another one for the MOOC). This is the case of representation ID 2 (Figure 6), where the participant indicated, in the annotations next to the visualisation, that she used the f2f layer to represent the online formal teaching part of the course. Secondly, it is important to highlight that although the simplicity of the model makes it easy to use and understand (i.e, a strength), at the same time it is a limitation in the sense of providing detail and depth in the information regarding the represented LD. In addition, the model stays in the surface layer of the MOOC-based blended courses as it does not allow for more depth in the representation of online activities which may occur in the f2f sessions, but which use other online sources different from the MOOC (e.g., virtual learning environment–related activities, online educational applications). In summary, the results show that it is possible to use the first visualisation template representing MOOC- based LDs (answering E-RQ1). The main strengths of the first model include providing a visual, easy, and effective way of representing the blended LDs, which facilitates their understanding and classification according to existing blended models. However, it has some limitations related to its flexibility and the lack of detail in descriptions or representations of combinations of different types of blended elements beyond the f2f sessions with the MOOC (answering E-RQ2). Evaluation of the second visual representation model In the second model, three main changes were made from the first model. Firstly, instead of having online and f2f learning layers, it has in-class and outside-of-class activities (Littlejohn & Pegler, 2007). Secondly, below these two main layers, there are the resources layers (Figure 7, layers in grey), which are extendable and can be defined by the designer depending on his/her needs. Lastly, below the activity and resources layers, there is an open space to place detailed descriptions of the activities (using the corresponding template, Figure 7). These modifications have been introduced to overcome the limitations of the previous model in relation to the representation of different types of BL which occur in the same course (online activities or resources beyond the MOOCs) and the lack of detailed information about the design (and its activities and resources) that the first model provides. By contrast, in accordance with the positive results from the first model, in the second model, it is still possible to indicate whether the blended activities will be mandatory or optional by placing time indications. Australasian Journal of Educational Technology, 2020, 36(4). 10 Figure 7. A second version of the model Furthermore, in line with providing more detailed designs, we have conceptualised an activity template (Figure 8) which can be used to describe in more detail the activities placed on the timeline and can be situated on the open space of the bottom part of the model (Figure 7). For each activity, a name or ID and several tasks which are composed (a., b., c., etc.) should be indicated. For each task, the designer can provide four descriptors: the teacher’s role (no teacher, supervisor, lecturer, or facilitator); the type of students work (individual or collaborative); the type of task (read/watch/listen, discuss, investigate, collaborate, produce, practice); and the grading mode (not graded or graded). Although the types of tasks have been extracted directly from the literature (Laurillard, 2012), we have defined the other descriptors according to the definitions and views from the specific literature about BL. Figure 8. Activity template (card) with its corresponding legend The second evaluation aimed to provide an understanding of the potentialities and weaknesses of the second model answering the three evaluation questions E-RQ1, E-RQ2, and E-RQ3. Results and discussion Whereas Figure 9 shows the timelines and resources layers of the LD templates filled out by the six participants’ teams, Figure 10 presents the activity cards. The two parts of the templates collected have been separated into two figures to facilitate their analysis and comparison (see an example of a complete LD template in Figure 4. Australasian Journal of Educational Technology, 2020, 36(4). 11 Figure 9. Completed timelines and resources layers extracted from the design templates The first step for filling out the visual model template was to separate the timeline into weeks depending on the case descriptions. From Figure 9, we can see that all teams representing Course #1 separated the timeline into 6 weeks correctly. Moreover, they placed the activities in each corresponding week depending on whether they were in-class or out-of-class exactly how the case was described in the provided paper sheets (see case descriptions in the Appendix 1). By contrast, not all the teams representing Course #2 used the timeline in the same way. Interestingly, Teams #4 and #5 separated the timeline into 5 weeks proportionally, leaving little space for placing the in-class activities during the first week. On the other hand, Team #6 separated the timeline in a non-proportional way, using as much space as possible to describe the activities of the first week and indicating the remaining 4 weeks in a shorter space. This finding may suggest the need for removing the limitation of the paper prototype model (where participants have a static space for representing their LDs) and offering the possibility of a web-based visual model (enabling zooming in and zooming out of the timeline and facilitating the allocation of different activities, especially when they are numerous). Regarding the use of the resources layers, three teams used extra layers beyond the learning management system (LMS) and MOOC ones. Closer inspection of the representations shows that all the participants who indicated resources aligned them according to the activities in which they were used, as well as indicating whether they were mandatory or optional. Overall, these results indicate that all the teams were able to use the timeline as well as the resources layers correctly. In general, the visual representation was used in the expected manner. The results show that it is possible to represent a MOOC- based LD using the model (answering E-RQ1) and that the limitation of the first model – regarding the representation of several online resources in the same blended course – has been solved. In addition, the results obtained from the representation of the activities (Figure 10) indicate that all the teams were also able to use the activity cards codes and fill out the different task descriptor details. However, some variations between the interpretation of the activities by the teams could be identified. Moreover, we can observe that in three teams (#2, 5, and 6), allocating the type of the task was challenging for the participants because they found that one task could have multiple values for the same descriptor. To face this challenge, they decided to cut pieces of the task circles to create composite tasks, for example, Team #2 (A3: a, c, d; A6: a, b; and A7: a). Australasian Journal of Educational Technology, 2020, 36(4). 12 Figure 10. Completed card activities extracted from the design templates A possible explanation for these results might be that the participants may have had difficulties in understanding and interpreting the provided case descriptions during the limited time of the workshop. Another explanation could be that the codes for describing the tasks were not clear enough. Lastly, from the use of the activities, we can see that two participants placed the activity cards in the order in which they appear in the timeline (Teams #2 and #6), whereas the other participants put their activity cards in numerical order (Teams #1, #3, #4, #5). This result suggests confusion on how to use the open space for placing the activity cards. The most surprising aspect of the data from the questionnaire is that 63% of the reported difficulties were related to the LD case descriptions provided (see the complete analysis, the corresponding categories, and participants’ excerpts in Appendix 2, Table A2-2). Of 14 participants, six found the explanations of the cases not clear enough. Three of them highlighted difficulties regarding the allocation of the task descriptors in the activities due to the unclear descriptions of the cases, while the other three had trouble allocating the activities in the timeline for the same reason. On the one hand, this result is encouraging as it is not directly related to any limitation of the visual model. But on the other hand, it indicates that the examples of cases provided to the participants need to be improved for the next workshops. Adding a last note to this, if the participants had been describing their own LDs, they would have had more flexibility in separating the tasks and the problem would have been minimised. Otherwise, 32% of the difficulties were related to the activity task descriptors, specifically to three of them: the teacher’s role, the type of student work, and the type of task. Regarding the challenges in selecting the appropriate type of task, other researchers who tested the Learning Designer tool (Laurillard, Kennedy, Charlton, Wild, & Dimakopoulos, 2018), which also uses Laurillard’s (2012) activities taxonomy, reached similar findings, stating that the activities may correspond to more than one existing type (Prieto et al., 2013). In the case of the other descriptors, sometimes they were not clear enough to be correctly used or some items for describing certain cases were missing (e.g., difficulties in differentiating between the supervisor and facilitator roles or deciding whether a lecture is a collaborative or individual task). These results corroborate the findings from the artefact analysis, which pointed out that the use of the timeline and the resources layers did not present difficulties, but the cases and the activity cards were a source of doubt for the participants. Regarding the model, further work is required for rethinking and improving the task descriptors and for developing guidelines for using the open space to place the activities. Australasian Journal of Educational Technology, 2020, 36(4). 13 In the case of the second question of the questionnaire (related to E-RQ3), half of 14 participants who answered the question strongly agreed or agreed (4 and 3 respectively) that the visual model was helpful in understanding the LD of others. However, three participants were neutral, while three disagreed and one strongly disagreed with the statement. These findings may be somewhat limited by the difficulties participants experienced in understanding and representing their cases due to the unclear descriptions and the difficulties they had using the activity descriptors (adding a certain bias to the results). More research is therefore needed here, using an improved version of the representation model. Regarding the main strengths of the model (answering E-RQ2), we identified six categories according to the frequencies obtained (see the complete analysis, the corresponding categories, and participants’ excerpts in Appendix 2, Table A2-3). Most participants (7 out of 14) recognised the fact that it provides an overall idea of the whole LD in a simple view as the main strength of the model. Four highlighted the potential of the model in facilitating organisation (planning), reflecting about the design, and in making decisions about LD. The last strengths detected were its ease of interpretation (2); its potential for communicating the work to others (1): the time representation and separation of in-class and out-class activities (1); and the visual representation of the activities (1). In contrast, the main weaknesses of the model (answering E-RQ2) identified by the participants were the allocation of the task types (4), in line with the results discussed above; the limitation of the model for representing complex designs (2); the lack of provision of details (2); the time needed to learn the model and its codes (2); the need for alignment of the activity descriptions with the timeline and the resources (1); and that the use of the timeline is open to errors by the users (1) (the complete analysis, the corresponding categories, and the participants’ excerpts in Appendix 2, Table A2-4). From these results, we can state that although the aim of adding the descriptions of the activities in this second model was to eliminate the limitation of the first model in terms of providing more details in the design, this aspect still needs to be improved. A web-based model (as already suggested) could overcome the limitations of the current model (e.g., offering automatic error feedback and control, especially regarding the use of the timeline, which is now subject to errors as it is a paper prototype). Furthermore, a web-based model could visually align the various elements, such as the timeline, resources and activities, automatically. Interestingly, a common view amongst participants was that the visual representation provides a useful summary of an LD (Appendix 2, Table A2-5), a result aligned with the main strength of the model reported above. All the responses to the fifth question (Do you think it provides a useful summary of an LD?) were positive except for one occasion (to which the answer was “partially”), although some of the participants added some positives to their answers: the model is especially useful for complex designs, that it could be refined, or that teachers would need to be trained for using these kinds of representations. In the final part of the questionnaire, respondents were asked to answer the question of how the visual representation model could be improved. The answers were classified into six categories: (1) Providing more flexibility for representing the tasks. (2) Improving the visualisation with significant icons, colour representation and metaphors. (3) Providing an online version of the model. (4) Facilitating the alignment between the activities placed on the timeline and the activity templates at the bottom of the model. (5) Providing the content of the activities within the visual representation. (6) Rethinking the teachers’ roles provided. See the complete analysis, the corresponding categories and participants’ excerpts in Appendix 2, Table A2-6. Final model proposal Figure 11 presents the final proposal of the design layers of the visual model. Answering G-RQ1 (What are the necessary LD elements to visually represent a MOOC-based BL design?), the main element of the model is a timeline which is composed of two activity layers and an extensible number of resource medium layers. Due to the results obtained in the two evaluations presented in the study, we have decided to use the in-class/out-of-class representation; as we have argued in the previous sections, it allows for the Australasian Journal of Educational Technology, 2020, 36(4). 14 representation of different online type of resources (beyond the MOOC) which can be used in both places (in and out of class). Moreover, in the final proposal, we can distinguish between the resources and the resource medium layers (which indicate the medium through which the resource is provided to the student). For example, a book (resource) would be placed in a physical resource medium layer (as would other physical resources such as paper sheets, laboratory material), whereas a MOOC medium layer could contain a video, an online test, or a web-text resource among others. Regarding G-RQ2 (How can these LD elements be articulated in a visual representation to facilitate comprehension of the whole design?), activities are sequenced on time and situated on the place where they occur (in class or out of class). Resources are aligned with the activities where they are used and placed in the corresponding resource medium layer, informing how they are available (e.g., physical resource, online resource – virtual learning environment, web, cloud). Figure 11. Final proposal: visual representation model for BL designs The results of the last evaluation have led us to the following decisions regarding modifications to some of the task descriptors, as follows: • Teacher’s presence: teacher available f2f, online or not present. This has been simplified based on the categories used in the Learning Designer tool (Laurillard et al., 2018) and adapted adding the category of teacher available online. • Type of student work: individual, in groups or the whole class. We have added a new category for being able to represent, for example, the case when the whole class is attending a lecture – the same approach as Dillenbourg (2015) in the orchestration graphs. • Type of task: remembering, understanding, applying, analysing, evaluating and creating. This has changed from Laurillard et al.’s (2018) categories to Bloom’s taxonomy cognitive levels (Krathwohl, 2002). This allows for the indication of each task in terms of its associated cognitive process levels, avoiding confusion when there is a possibility of representing a task with more than one level. • Grading mode: graded task, not graded, or task for auto-evaluation. We have added a category auto-evaluation, but further research is necessary to evaluate these categories and address the different possibilities of assessment. From this study’s findings, we argue that with this final model, the designer can consider the impact on space (e.g., the physical location of the students), the impact of time (synchronous or asynchronous) and the level of interaction in a visual way – three concerns which Littlejohn and Pegler (2007) identified as necessary for making design decisions and making LD choices. Conclusion and future work The results of the study presented in this research article show the main strengths and weaknesses of the visual representations proposed and are useful for analysing whether the elements provided by the model (the timeline with the activity layers, the resources layers, and the activity descriptors) are valid for visually Australasian Journal of Educational Technology, 2020, 36(4). 15 representing and designing MOOC-based BL courses. The outputs of the workshop have helped to improve the different versions of the model to come up with a final model proposal which allows for a visual representation of blended courses. These results show that the representation enables educators to easily visualise the overall structure of the LDs and the relationships between the different design elements, providing a context for fostering reflection and decision-making during the planning of MOOC-based BL designs. Future work includes the implementation of the model into an LD authoring tool, where an actionable version of the model will be developed, and a computer-supported authoring experience based on the model will be evaluated. The aim of the study was conceptualising a model for visually representing MOOC-based BL designs. Yet it would also be interesting to go further and test whether the model would be able to represent other complex blended cases beyond the use of MOOCs and beyond the higher education context. Moreover, further research might explore whether the representation model can also be useful for students, as it has the potential for showing what teachers but also what students are expected to do – which may help them visualise the total workload at a glance and better plan their courses and be more effective in their learning process. Thus, it can contribute to one of the most important challenges that BL courses need to face: facilitating the necessary alignment of expectations between instructors and students (McGee & Reis, 2012). Moreover, future work needs to explore whether and how the model can express synchronicity in a more visual way (Littlejohn & Pegler, 2007; Norberg et al., 2011) and address the requirement of providing a context to the visualisation beyond the fact of representing the activities (e.g., providing details about the topic of the course, the number of students, the educational level). As Goodyear (2004, p. 342) stated, “context is important in helping constrain and communicate the nature of both problem and solution”, avoiding over-generalisation. Besides, it would be interesting to investigate how to connect the learning outcomes with the visual representation (Bralić & Divjak, 2018). More broadly, research is needed to determine the impact of the model in the planning process as well as whether it can serve to communicate LD ideas to others, promoting their sharing among teachers’ communities. Acknowledgements The authors would like to thank all the participants of the workshops. This work has been partially funded by “la Caixa Foundation” (CoT project, 100010434) and FEDER, the National Research Agency of the Spanish Ministry of Science, Innovations and Universities MDM-2015-0502, TIN2014-53199-C3-3-R, TIN2017-85179-C3-3-R. Davinia Hernández-Leo (one of the authors) is a Serra Húnter Fellow. References Agostinho, S. (2011). The use of a visual learning design representation to support the design process of teaching in higher education. Australasian Journal of Educational Technology, 27(6), 961–978. https://doi.org/10.14742/ajet.923 Agostinho, S., Bennett, S., Lockyer, L., & Harper, B. (2011). The future of learning design. Learning, Media & Technology, 36(2), 97–99. https://doi.org/10.1080/17439884.2011.553619 Albó, L., & Hernández-Leo, D. (2016). Blended learning with MOOCs: Towards supporting the learning design process. In G. Ubachs & L. Konings (Eds.), Proceedings of the Online, Open and Flexible Higher Education Conference 2016 (pp. 578–588). Maastricht, The Netherlands: European Association of Distance Teaching Universities. Retrieved from https://repositori.upf.edu/handle/10230/27478 Albó, L., & Hernández-Leo, D. (2017, May). Breaking the walls of a campus summer course for high school students with two MOOCs. Paper presented at the HybridEd Workshop: Innovations in blended learning with MOOCs, Leganés, Spain. Retrieved from http://hdl.handle.net/10230/32157 Albó, L., & Hernández-Leo, D. (2018). Co-creation process and challenges in the conceptualization and development of the edCrumble learning design tool. In A. Piotrkowicz, R. Dent-Spargo, S. Dennerlein, I. Koren, P. Antoniou, P. Bailey, T. Treasure-Jones, I. Fronza, & C. Pahl (Eds.), Joint Proceedings of the CC-TEL 2018 and TACKLE 2018 Workshops. Aachen, Germany: CEUR-WS. Retrieved from http://ceur-ws.org/Vol-2190/ Albó, L., Hernández-Leo, D., & Oliver, M. (2016). Blended MOOCs: University teachers’ perspective. In C. Delgado Kloos, P. J. Muñoz-Merino, R. M. Crespo-García, & C. Alario-Hoyos (Eds.), Trends in Digital Education: Selected papers from EC-TEL 2015 Workshops CHANGEE, WAPLA, and https://doi.org/10.14742/ajet.923 https://doi.org/10.1080/17439884.2011.553619 https://repositori.upf.edu/handle/10230/27478 http://hdl.handle.net/10230/32157 http://ceur-ws.org/Vol-2190/ Australasian Journal of Educational Technology, 2020, 36(4). 16 HybridEd (vol. 1599, pp. 11–15). Aachen, Germany: CEUR-WS. Retrieved from http://ceur- ws.org/Vol-1599/ Andone, D., Mihaescu, V., Ternauciuc, A., & Vasiu, R. (2015, May). Integrating MOOCs in traditional higher education. Paper presented at the Third European MOOCs Stakeholder Summit, Mons, Belgium. Retrieved from https://www.researchgate.net/publication/289991756_Integrating_MOOCs_in_Traditional_Higher_E ducation Arbaugh, J. B. (2014). What might online delivery teach us about blended management education? Prior perspectives and future directions. Journal of Management Education, 38(6), 784–817. https://doi.org/10.1177/1052562914534244 Bonsignore, E., Ahn, J., Clegg, T., Guha, M. L., Yip, J. C., Druin, A., & Hourcade, J. P. (2013). Embedding participatory design into designs for learning: An untapped interdisciplinary resource? In N. Rummel, M. Kapur, M. J. Nathan, & S. Puntambekar (Eds.), Proceedings of the 10th International Conference on Computer-Supported Collaborative Learning (vol. 1, pp. 549–556). International Society of the Learning Sciences. https://doi.org/10.13140/2.1.3961.7920 Bralić, A., & Divjak, B. (2018). Integrating MOOCs in traditionally taught courses: Achieving learning outcomes with blended learning. International Journal of Educational Technology in Higher Education, 15(1). https://doi.org/10.1186/s41239-017-0085-7 Clement, A. (1994). Computing at work: empowering action by “low-level” users. Communications of the ACM, 37(1), 52–63. https://doi.org/10.1145/175222.175226 Conole, G., & Wills, S. (2013). Representing learning designs - making design explicit and shareable. Educational Media International, 50(1), 24–38. https://doi.org/10.1080/09523987.2013.777184 Dalziel, J. (2003). Implementing learning design: The Learning Activity Management System (LAMS). In G. Crisp, D. Thiele, I. Scholten, S. Barker, & J. Baron (Eds.), Interact, Integrate, Impact: Proceedings of the 20th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (pp. 593–596). Retrieved from http://www.ascilite.org/conferences/adelaide03/program/conf_prog_index.htm Dalziel, J. (2015). Learning design: Conceptualizing a framework for teaching and learning online. New York, NY: Routledge. Delgado-Kloos, C., Muñoz-Merino, P. J., Alario-Hoyos, C., Ayres, I. E., & Fernández-Panadero, C. (2015). Mixing and blending MOOC technologies with face-to-face pedagogies. In Proceedings of the 2015 IEEE Global Engineering Education Conference (EDUCON) (pp. 967–971). IEEE. https://doi.org/10.1109/EDUCON.2015.7096090 Dillenbourg, P. (2015). Orchestration graphs: modeling scalable education. Lausanne, Switzerland: EPFL Press. Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher education. Internet and Higher Education, 7, 95–105. https://doi.org/10.1016/j.iheduc.2004.02.001 Goodyear, P. (2004). Patterns, pattern languages and educational design. In R. Atkinson, C. McBeath, D. Jonas-Dwyer, & R. Phillips(Eds.), Beyond the Comfort Zone: Proceedings of the 21st ASCILITE Conference (pp. 339–347). Perth, Western Australia: ASCILITE. Retrieved from http://www.ascilite.org/conferences/perth04/procs/pdf/goodyear.pdf Heinze, A., & Procter, C. (2004). Reflections on the use of blended learning. In Proceedings of Education in a Changing Environment (pp. 13–14). Salford, United Kingdom: University of Salford. Retrieved from http://www.ece.salford.ac.uk/proceedings/papers/ah_04.rtf Krathwohl, D. R. (2002). A revision of Bloom’s taxonomy: An overview. Theory into Practice, 41(4), 212–218. https://doi.org/10.1207/s15430421tip4104_2 Lai, P. K., Portolese, A., & Jacobson, M. J. (2017). Does sequence matter? Productive failure and designing online authentic learning for process engineering. British Journal of Educational Technology, 48(6), 1217–1227. https://doi.org/10.1111/bjet.12492 Laurillard, D. (2012). Teaching as a design science: Building pedagogical patterns for learning and technology. New York, NY: Routledge. Laurillard, D., Charlton, P., Craft, B., Dimakopoulos, D., Ljubojevic, D., Magoulas, G., … Whittlestone, K. (2013). A constructionist learning environment for teachers to model learning designs. Journal of Computer Assisted Learning, 29(1), 15–30. https://doi.org/10.1111/j.1365-2729.2011.00458.x Laurillard, D., Kennedy, E., Charlton, P., Wild, J., & Dimakopoulos, D. (2018). Using technology to develop teachers as designers of TEL: Evaluating the learning designer. British Journal of Educational Technology, 49(6), 1044–1058. https://doi.org/10.1111/bjet.12697 http://ceur-ws.org/Vol-1599/ http://ceur-ws.org/Vol-1599/ https://www.researchgate.net/publication/289991756_Integrating_MOOCs_in_Traditional_Higher_Education https://www.researchgate.net/publication/289991756_Integrating_MOOCs_in_Traditional_Higher_Education https://doi.org/10.1177/1052562914534244 https://doi.org/10.13140/2.1.3961.7920 https://doi.org/10.1186/s41239-017-0085-7 https://doi.org/10.1145/175222.175226 https://doi.org/10.1080/09523987.2013.777184 http://www.ascilite.org/conferences/adelaide03/program/conf_prog_index.htm https://doi.org/10.1109/EDUCON.2015.7096090 https://doi.org/10.1016/j.iheduc.2004.02.001 http://www.ascilite.org/conferences/perth04/procs/pdf/goodyear.pdf http://www.ece.salford.ac.uk/proceedings/papers/ah_04.rtf https://doi.org/10.1207/s15430421tip4104_2 https://doi.org/10.1111/bjet.12492 https://doi.org/10.1111/j.1365-2729.2011.00458.x https://doi.org/10.1111/bjet.12697 Australasian Journal of Educational Technology, 2020, 36(4). 17 Littlejohn, A., & Pegler, C. (2007). Preparing for blended e-Learning. London, United Kingdom: Routledge. https://doi.org/10.4324/9780203961322 López-Pérez, M. V., Pérez-López, M. C., & Rodríguez-Ariza, L. (2011). Blended learning in higher education: Students’ perceptions and their relation to outcomes. Computers & Education, 56(3), 818– 826. https://doi.org/10.1016/j.compedu.2010.10.023 McGee, P., & Reis, a. (2012). Blended course design: A synthesis of best practices. Journal of Asynchronous Learning Networks, 16(4), 7–22. https://doi.org/10.24059/olj.v16i4.239 Means, B., Murphy, R., & Baki, M. (2013). The effectiveness of online and blended learning: A Meta- analysis of the empirical literature. Teachers College Record, 115, 1–47. Retrieved from http://www.tcrecord.org/Content.asp?ContentId=16882 Moskal, P., Dziuban, C., & Hartman, J. (2013). Blended learning: A dangerous idea? Internet and Higher Education, 18, 15–23. https://doi.org/10.1016/j.iheduc.2012.12.001 Muller, M. J., & Druin, A. (2012). Participatory design: The third space in HCI. In J. A. Jacko (Ed.), The human-computer interaction handbook: Fundamentals, evolving technologies and emerging applications (3rd ed., pp. 1125–1154). Boca Raton: CRC Press. https://doi.org/10.1201/b11963 Norberg, A., Dziuban, C. D., & Moskal, P. D. (2011). A time-based blended learning model. On the Horizon, 19(3), 207–216. https://doi.org/10.1108/10748121111163913 Novick, D. G. (2000). Testing documentation with “low-tech” simulation. In Proceedings of the 18th Annual Conference on Computer Documentation (pp. 55–68). IEEE. https://doi.org/10.1109/IPCC.2000.887261 Pérez-Sanagustín, M., Hilliger, I., Alario-Hoyos, C., Delgado Kloos, C., & Rayyan, S. (2017). H-MOOC framework: Reusing MOOCs for hybrid education. Journal of Computing in Higher Education, 29(1), 47–64. https://doi.org/10.1007/s12528-017-9133-5 Persico, D., Pozzi, F., Anastopoulou, S., Conole, G., Craft, B., Dimitriadis, Y., … Walmsley, H. (2013). Learning design Rashomon i: Supporting the design of one lesson through different approaches. Research in Learning Technology, 21(SUPPL.1). https://doi.org/10.3402/rlt.v21i0.20224 Porter, W. W., Graham, C. R., Spring, K. A., & Welch, K. R. (2014). Blended learning in higher education: Institutional adoption and implementation. Computers & Education, 75, 185–195. https://doi.org/10.1016/j.compedu.2014.02.011 Pozzi, F., Ceregini, A., & Persico, D. (2016). Designing networked learning with 4 Ts. In S. Crammer, N. B. Dohn, M. de Laat, T. Ryberg, & J. A. Sime (Eds.), Looking Back – Moving Forward: Proceedings of the 10th International Conference on Networked Learning 2016 (pp. 210–217), Lancaster, United Kingdom: Lancaster University. Retrieved from https://eprints.lancs.ac.uk/id/eprint/84840/ Prieto, L. P., Dimitriadis, Y., Craft, B., Derntl, M., Émin, V., Katsamani, M., … Villasclaras, E. (2013). Learning design Rashomon II: Exploring one lesson through multiple tools. Research in Learning Technology, 21(SUPPL.1), 1–20. https://doi.org/10.3402/rlt.v21i0.20057 Siemens, G., Gašević, D., & Dawson, S. (2015). Preparing for the digital university: A review of the history and current state of distance, blended, and online learning. Athabasca, Canada: Athabasca University. Winters, N., & Mor, Y. (2009). Dealing with abstraction: Case study generalisation as a method for eliciting design patterns. Computers in Human Behavior, 25(5), 1079–1088. https://doi.org/10.1016/j.chb.2009.01.007 Corresponding author: Laia Albó, laia.albo@upf.edu Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY-NC-ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY-NC-ND 4.0. Please cite as: Albó, L., & Hernández-Leo, D. (2020). Conceptualising a visual representation model for MOOC-based blended learning designs. Australasian Journal of Educational Technology, 36(4), 1–26. https://doi.org/10.14742/ajet.5178 https://doi.org/10.4324/9780203961322 https://doi.org/10.1016/j.compedu.2010.10.023 https://doi.org/10.24059/olj.v16i4.239 http://www.tcrecord.org/Content.asp?ContentId=16882 https://doi.org/10.1016/j.iheduc.2012.12.001 https://doi.org/10.1201/b11963 https://doi.org/10.1108/10748121111163913 https://doi.org/10.1109/IPCC.2000.887261 https://doi.org/10.1007/s12528-017-9133-5 https://doi.org/10.3402/rlt.v21i0.20224 https://doi.org/10.1016/j.compedu.2014.02.011 https://doi.org/10.3402/rlt.v21i0.20057 https://doi.org/10.1016/j.chb.2009.01.007 mailto:laia.albo@upf.edu https://creativecommons.org/licenses/by-nc-nd/4.0/ https://doi.org/10.14742/ajet.5178 Australasian Journal of Educational Technology, 2020, 36(4). 18 Appendix 1 Descriptions of the course cases Course #1 Context: This course implemented a blended model of learning by merging content from a MOOC with in-class, team-based instruction as part of a required undergraduate circuit theory course. Course duration: 6 weeks course MOOC dates: available during all the 6 weeks. In-Class Activities A1. Special class session: the 1st f2f session a. The first class session (number 1) was a presentation of the course by the teacher. A2. Final exam: the last f2f session a. The last class session (number 12) was the final exam (part of course grade) A3. Regular class sessions: 75 minutes, twice a week (the first and the last week just once a week, 10 in total) a. Mental ramp-up period (10 minutes): Professor asks questions about students’ activity to gauge students’ understanding while grad students collect weekly "student online activities survey" and summarize results for the professor to be discussed on that day’s class session. b. In-class mini-review lecture (20 minutes): Based on the grad students’ survey analysis of online topics marked as “difficult to understand” or “hard” by students that week, the professor reviews the more difficult concepts in class. If no topics emerge as difficult, the professor solves a sample problem that embodies the most important concept of that week’s topic. In addition, a summary of the online lectures is distributed to all students twice a week. c. Group quiz (20 minutes): Students work on a group quiz as a team of three. Professor leads and answers questions on strategies for how to solve different types of problems. The group quiz is collected and graded as part of overall course grade. The last 5 min, professor reveals some of the best strategies to solve problem and the solution is distributed among students in class. d. Individual quiz (20 minutes): The individual quiz is given to each student to gauge their understanding of subject material. The quiz is collected and graded as a part of students’ final course grade. In the last 5 minutes, the best strategy to solve this type of the problem is discussed (students and professor) with the solution distributed to the students in class. e. Preview for next class session (5 minutes): Preview by professor of next class’ material. A4. Optional class sessions: 60 minutes, once a week (not including last week, 5 in total) a. An optional, Friday, one-hour, f2f walk-in session (i.e., optional recitation office hour) held weekly by the professor. Outside-of-Class Activities A5. Mandatory work in the MOOC: 30 minutes, twice a week (begins just after the first special f2f session and ends before the last f2f regular session) a. Watch edX the topical mini-lecture video of up to 10 minutes and answer embedded questions online. b. Read assigned sections of the edX online textbook. (15min) c. Watch edX videos of MIT faculty arguing with each other in presenting and modelling competing alternative solutions to a single problem. (5min) A6. Mandatory work in the MOOC: 60 minutes, once a week a. Solve edX problem sets and submit answers online for automated grading by edX. (30min) b. Complete edX online lab experiments and submit answers online for automated grading by edX. (30min) Australasian Journal of Educational Technology, 2020, 36(4). 19 A7. Mandatory work: after each class session, 10 minutes a. Finally, after each class session, students were given (or could download) an assessment handout for the next class session that asked each student to evaluate their understanding or level of difficulty (i.e., “easy”, “elementary”, “intermediate”, “hard”, “advanced”) for each of the edX topics to be covered in the next class session. If a student rated a topic as “hard” or “advanced”, he or she was required to briefly explain what was difficult or confusing. Students were required to complete this survey before coming to each class and give it to the graduate assistants at the beginning of each class session. The two graduate assistants compiled the results of these surveys during the first ten minutes of each class so that the professor could focus on the most difficult topic areas during the F2F mini- review lecture. Assessment The student course grades are based on: ● Online activities (Assignments and labs) (20%) ● Team quizzes (15%) ● Individual quizzes (15%) ● Final comprehensive exam (50%) Case based on: Ghadiri, K., Qayoumi, M. H., Junn, E., Hsu, P., & Sujitparapitaya, S. (2013). The transformative potential of blended learning using MIT edX’s 6.002 x online MOOC content combined with student team-based learning in class. Retrieved from https://www.edx.org/sites/default/files/upload/ed-tech- paper.pdf https://www.edx.org/sites/default/files/upload/ed-tech-paper.pdf https://www.edx.org/sites/default/files/upload/ed-tech-paper.pdf Australasian Journal of Educational Technology, 2020, 36(4). 20 Course #2 Context: This course presents a case study of using two external MOOCs in a face-to-face (f2f) summer course for high school students. The course used an LMS (Moodle) to articulate the course content. For supporting the theory part, 19 videos downloaded from two different MOOCs (both from Coursera MOOC platform) were embedded in the LMS. Six videos were from MOOC #1 “Videojuegos: de qué hablamos” from the Universitat Autònoma de Barcelona, whereas 13 videos were from MOOC #2 “¡A programar! Una introducción a la programación” from the Universidad ORT Uruguay and the University of Edinburgh. Both MOOCs were running parallel to the f2f course and finishing several weeks later. One of the added values of the course was offering students continued learning through the MOOCs after the f2f course finished. Course duration: 5 days of mandatory f2f course plus 4 optional weeks following the two MOOCs online. MOOC #1 dates: Starting at the same time of the f2f course and ending 4 weeks later. MOOC #2 dates: Starting at the same time of the f2f course and ending 4 weeks later. In-Class Activities A1. First day: f2f class session (9:30 - 14:00) a. Course Welcome and introduction (60 minutes): Teachers introduced the course content and did a Scratch presentation/demo to the students (using the Scratch online platform). b. Getting to know activity (30 minutes): Students had to create a Scratch account (individually in the Scratch online platform) and design a short animation with Scratch presenting themselves. Teachers were around the class helping with the students’ doubts. c. Introduction to Coursera MOOCs (30 min): Teachers introduced the Coursera platform and the two MOOCs to the students (using Coursera online platform). d. Introduction to computer games developing (30 min): The theory concepts of the course related to video games were explained showing the 6 videos from MOOC #1 (usually an extract of it) in class to all the students (downloaded from Coursera, hosted in a private YouTube channel and embedded in the LMS). Teachers played the videos and showed only the selected parts (stopping when was necessary) as well as commenting on some key parts of the videos. e. Introduction to programming (90 min): First part of the concepts related to programming were explained showing 7 videos from MOOC #2 (usually an extract of it) in class to all the students (downloaded from Coursera, hosted in a private YouTube channel and embedded in the LMS). Teachers played the videos and showed only the selected parts (stopping when was necessary) as well as commenting on some key parts of the videos. f. Exercises between videos of tasks d and e (10 min): After each video of the tasks “d” and “e”, students were asked to solve a multiple-choice online question in the LMS related to the concept in order to check if they had understood it. The teacher clarified the concepts when the results of the question were bad (he/she was checking the results just after the students were submitting them). A2. Second day: f2f class session (9:30 - 14:00) a. Introduction to programming (part 2) (90 min): Second part of the concepts related to programming were explained showing 6 videos from MOOC #2 (usually an extract of it) in class to all the students (downloaded from Coursera, hosted in a private YouTube channel and embedded in the LMS). Teachers played the videos and showed only the selected parts (stopping when was necessary) as well as commenting on some key parts of the videos. b. Invited talk 1 (30 min): An expert about computer games developing came to the class to do a short lecture. c. Collaborative activity about Scratch (30min): d. Individual Scratch practice (90 min): Students practised programming using Scratch (in the online platform) individually, following several computer game samples that the teacher uploaded in the LMS of the course. Teachers were around for attending to students’ questions. Australasian Journal of Educational Technology, 2020, 36(4). 21 A3. Third day: f2f class session (9:30 - 14:00) a. Makey-Makey presentation (60 minutes): A teacher introduced the Makey-Makey gadget to the students by a demo with examples. b. Individual Scratch practice (90 min): Students practised programming using Scratch (in the online platform) individually, following several computer game samples that the teacher uploaded in the LMS of the course. Teachers were around for attending to students’ questions. c. Presentation of the Sensors in Scratch (30 min): Teachers made a demonstration of using Scratch sensors (using the online Scratch platform). d. Collaborative work preparing the final project (90 minutes): Students were grouped in teams of 3 or 4 people. Students had to work collaboratively within their teams in order to develop their own computer game with Scratch (using the online Scratch platform). A4. Fourth day: f2f class session (9:30 - 14:00) a. Invited talk 2 (30 min): An expert on 3D graphics came to the class to do a short lecture and a demo of how to design a computer game 3D character. b. Collaborative work preparing the final project (240 minutes): Students had to continue working collaboratively within their teams in order to develop their own computer game with Scratch (using the online Scratch platform). A5. Fifth day: f2f class session (9:30 - 14:00) a. Presentations of the final projects (240 min): Each team of students presented their developed computer game in front of the other students in class. The teacher asked several questions (regarding the development process) after each presentation. b. Farewell (30 minutes): Teachers gave course certificate to the students and invited them to keep learning about programming and computer games through the two MOOCs in the Coursera platform. Outside-of-Class Activities A6. Optional work in the MOOC #2: between the first and the second f2f sessions a. Students were invited to watch four optional videos from MOOC #2. b. And try to develop (individually) two optional games (appearing in the “a” task optional videos) in the online Scratch platform. A7. Optional work in the MOOCs: once the f2f course finished a. Students were invited to follow the 2 MOOCs during the 4 weeks after the f2f course. Assessment There was no formal assessment as it was a summer course. Students had to deliver a final game developed in teams and present it the last day of the course. Case based on: Albó, L., & Hernández-Leo, D. (2017, May). Breaking the walls of a campus summer course for high school students with two MOOCs. Paper presented at the HybridEd Workshop: Innovations in blended learning with MOOCs, Leganés, Spain. Retrieved from http://hdl.handle.net/10230/32157 http://hdl.handle.net/10230/32157 Australasian Journal of Educational Technology, 2020, 36(4). 22 Appendix 2 Table A2-1 BL design descriptions by 17 participants ID Context description How is online teaching (MOOC) combined with f2f education? 1 Introduction to quantitative methods and programming for students of Economics and Administration and Business Management degrees. The MOOC is used to propose exercises and/or complementary material to the students, while in the classroom they develop the theoretical aspects and solve doubts. 2 Technical subject of data analysis. Administration and business management degree in an online university. MOOC: Basic statistical concepts and techniques. Virtual classroom space: Complex activities, application of real cases, resolution of doubts, deepening ... 3 A MOOC on strength training, in a core subject of sports training of the degree in Physical Activity and Sports Science (CAFE). The MOOC is a part of the f2f subject to vary the teaching methodology and try to make the theoretical part of the subject more attractive, given the high number of students in the classroom and not very high interest in the theoretical class. 4 Training course for trainers. Objective: to promote online innovations for the f2f teaching. Within the section of activities there is a previous knowledge base that is delivered in a session of MOOCs available (preparation of online activities, use of teaching videos). This section of activities is the basis for design by teachers: to develop their own activities and redesign their f2f courses. 5 Subject on proteins (biotechnology). The MOOC is used to teach the basic and theoretical part of the subject and to be able to do the practical part directly in f2f format. 6 Resources and activities in Moodle for teachers. Collection of reference resources (videos) with practical examples. f2f sessions combining MOOC content to expand applicability. 7 Subject with many 1st grade student groups (12 ECTS credits). In the classroom, more generic contents are delivered. On the online platform (MOOC), more specific contents are given on the various topics discussed to promote the collective creation of knowledge among the various class groups – allow students with similar interests to get in touch with peers from their own and other universities. 8 Digital video editing course. The MOOC would be used as an introductory part of the course to explain the basic concepts of the digital edition of video to ensure that participants have the same level of knowledge and that they can arrive prepared to the f2f teaching to address and delve into the practical part. 9 Subject of a university course. Knowledge-levelling MOOC course before starting a subject (for example, finances) in the case of classes with students that come from different disciplines. The MOOC is a complement to f2f classes. What has been seen in f2f classes is deepened. 10 Master's degree. Optional subject: teacher training in law. Take advantage of contents and experience in the classroom to work with the flipped classroom. 11 Primary school teaching degree. Optional subject of the ICT degree. The MOOC was proposed as a voluntary complement to expand knowledge, as a space for debate, as a supplement. Australasian Journal of Educational Technology, 2020, 36(4). 23 12 Fundamentals of taxation. Compulsory subject of the degree in Administration and Business Management (ADE). Complementary to existing material. Application of the concept of tax. 13 Training course to teach in English at the university. The MOOC provides explanatory videos on the material of the course. The f2f part of the course enables teaching staff to attend the specific needs of each individual teacher attending the course. It enables students (who are teachers) to have quality personal attention with the trainer and colleagues. 14 A course on the characteristics of Catalan literature of the XVIII century. The MOOC would be compulsory for the students, who would have to do it before the f2f classes began. The course would give students the theoretical basis on the context of Catalan literature of the XVIII century so that the classroom can be more practical and focused on the subject itself. During the course there will be optional content in the MOOC. 15 Postgraduate course in Emotional Education and Welfare. Blended. Produce some modules in a MOOC format to serve as an introduction. MOOC is not combined much with the f2f component. The MOOC will be the introduction of the postgraduate course. 16 Within a subject of introduction to ethics. Introduce a part of the content that is more practical. A part of the syllabus that aims to be more interactive (bioethics, examples, and problems) is delivered to the students in the form of a MOOC that will collect practical cases presented in videos. 17 Master in Neuroscience and Nanotechnology. Compulsory assignment of 5 ECTS credits, “Characterization and manipulation of the nanoscale”. Each topic of the MOOC is used as introductory material of the same topic of the f2f course. The questionnaires implemented in the MOOC are worked on in class and the subject is broadened. The new questions or extension material could be incorporated into the MOOC to enrich it with different levels of difficulty. Australasian Journal of Educational Technology, 2020, 36(4). 24 Table A2-2 Difficulties found during the design process: Questionnaire responses and frequencies from 14 participants Meta- category # Category Freq. Selected excerpts LD example sheets 1 Examples descriptions 6 “The precision of the descriptions was not enough in some cases.” “Some explanations were unclear (for example, activity A4).” 2 Allocation of task types 3 “In some cases, it is unclear which type of activity (investigation, production, discussion) should we choose since it's rather a mix” 3 Activities allocation in the timeline 3 “The main difficulty faced was at organizing the activities in timeline and deciding how many times a specific type of activity should be repeated in other weeks or not.” “(…) sometimes was difficult to assume how to divide the space according to the number of activities which we had” Activity tasks descriptors 4 Student’s work (individual vs. collaborative) 2 “In class sessions with Q&A, whether the activity can be classified as collaborative or not.” “In a lecture (listening), is this collaborative or individual? (…)” 5 Teachers’ roles not clear 2 “It was also difficult to differentiate between the role of teacher as supervisor and facilitator.” 6 Task types not clear 1 “The taxonomy for learning tasks and for teacher roles was not clear. I was specifically missing a task category of reproducing knowledge.” 7 Collaborative task over- described 1 “Sometimes the triangles for collaborative activity were merging with the orange colour of the collaborative activity.” Model elements alignment 8 Alignment between activities and timeline 1 “It was not clear how we should arrange the activity cards in the open space that was available in the page. It was necessary to align very well the activities in time among them, and then align them with the resources” Australasian Journal of Educational Technology, 2020, 36(4). 25 Table A2-3 Strengths of the visual representation: Questionnaire responses and frequencies from 14 participants # Categories Freq. Selected excerpts 1 Overall idea of the whole LD in a simple view (capacity of synthesis) 7 “You can see the whole course design at a glance.” “High level overview of complexity level of tasks.” “Rapid identification of what is going on in the process.” “you can obtain, in a simple view, things that can be useful to have account for a teacher (i.e., how is the course structured, what are the main role of the teacher, what kind of the activities are more extended in the course” 2 Good for organising, reflecting and taking LD decisions 4 “It allows you in a glance to decide the type of activities you wish to introduce in your course and balance them well during your design.” “It gives a good template to structure the course not only being aware of the main components of the course, but also type of activities, when there is grading, along with the temporal sequence of all components.” “You can (…) understand what resources are needed, and better organise the time inside and outside the class.” 3 Easy to interpret 2 “Easy to interpret.” 4 Good for communicating ideas to others 1 “I think it is good way to organize your own work and communicate it with others.” 5 Time representation and separation of in- class/out-class activities 1 “Time representation, good separation of in-class/out-class activities.” 6 Visual representation of the activities 1 “I think the visual representation is very representative with the teacher´s roles and the tasks for make with the different colours (produce/collaborative...)” Table A2-4 Weaknesses of the visual representation: Questionnaire responses and frequencies from 14 participants # Categories Freq. Selected excerpts 1 Allocation of task type 4 “The taxonomy of the colours, as it is very difficult to divide an activity in "pure" read or produce or practice. Maybe inserting a percentage would be useful.” 2 Limited for not complex designs 2 “It seems it cannot be used to represent more complex designs (e.g. alternative paths depending on the results of a given activity).” 3 Lack of details provision 2 “Some details of the design might be missing: number and size of groups; flow of generated artefacts; assessment methods” 4 Learning curve 2 “I think that teachers should be trained in order to interpret this kind of visual helps. There is a need to know well the coding system” 5 Activities not aligned with the timeline 1 “Details of the activities are not organized along with time, but in the other panel.” 6 Timeline open to errors by users 1 “Timeline of course is too open to errors by users.” Australasian Journal of Educational Technology, 2020, 36(4). 26 Table A2-5 Questionnaire responses and frequencies about whether the representation provides a useful summary of a learning design (14 participants) Do you think it provides a useful summary of a learning design? Freq. “Yes.” 6 “Yes, for complex designs.” 1 “Yes, at the coarse grain. At the fine grain, not very much.” 1 “Yes, it does. I would not say summary. It is just a concise representation of learning design. Summary may miss some components, but the provided visual representation does not miss anything. It just put them all together in a more concise way.” 1 “I think that it includes all main elements of a design in a comprehensive way. Some elements cannot be ‘decrypted’ very easily, at a glance.” 1 “Yes, although it can be refined.” 1 “Yes, exactly the word summary could describe what I did during this design process. A summary could help to see how dependencies between the activities and what students and teacher are expected to do.” 1 “Yes, but as I mentioned before I think that teachers need to be trained in this kind of representations.” 1 “Partially.” 1 Table A2-6 Recommendations for improving the visual representation: Questionnaire responses and frequencies from 14 participants # Categories Freq.* Selected excerpts from the research questionnaire 1 Provide flexibility for representing the tasks 4 “... the system might be flexible enough to modify some aspects of the coding allowing teachers to include new ones.” “... some activities may have an individual part and a collaborative part, and this is not easy to be depicted with the current visual representation.” 2 Improve the visualization with significant icons, colour representation and metaphors 4 “Better colour representation.” “Maybe use more friendly icons.” 3 Provide an online approach for the visualization 3 “... a zoom in/zoom out approach in which the detail of the activities can be seen in place according to time would be more comprehensible.” “In a textual format maybe is difficult to add more things but in a digital format you can probably click on the activity and then select the activity task with the different colours etc.” 4 Facilitate the alignment between the activities and the timeline 2 “The cards with the description of the activities are collected in the big canvas, but maybe they should be explicitly connected with the activities in the timeline…” 5 Provide the content of the activities within the visual representation 1 “...giving a summary with combination of visualizations and the content of the activities will help others to see the overall design and what is described in the activities.” 6 Rethink the teachers’ roles types 1 “Some of the codes for the teaching roles were not completely clear.” * The answer of one of the participants was classified in two categories. Introduction Visual representation of MOOC-based BL designs Methodology Participants and sample Procedure and instrumentation Data collection, analysis, and ethics Evaluation of the first visual representation model Results and discussion Evaluation of the second visual representation model Results and discussion Final model proposal Conclusion and future work Acknowledgements References Appendix 1 Descriptions of the course cases Course #2 Appendix 2