Australasian Journal of Educational Technology, 2022, 38(6). 53 Putting TPACK into action in learning design: The case of PeerLAND Kyparisia A. Papanikolaou School of Pedagogical and Technological Education, Greece Katerina Makri University of Thessaly, Greece Ioannis Sofos National Technical University, Greece Maria G. Tzelepi, Eleni Zalavra University of West Attica, Greece Although previous research highlights the complementary relationship of learning design with TPACK, this is not the case for TPACK informing the development of digital learning design tools. In this paper, we present PeerLAND (Peer Evaluation of LeArNingDesigns). This learning design tool interweaves design and peer evaluation in an integrated process based on TPACK, promoting teachers' roles as designers and reviewers. It adopts a modular design approach to support teachers as designers explicitly represent their design ideas starting from pedadogical content knowledge and gradually cultivating all the TPACK knowledge domains. The learning design process ends with peer evaluation where teachers use TPACK-based criteria to provide constructive feedback to peers. We report on a study conducted in a teacher education context to evaluate PeerLAND. Specifically, we investigate: (i) how student teachers' knowledge develops through the learning design process supported by PeerLAND, and (ii) how they value peer evaluation through PeerLAND. Our findings suggest that putting TPACK into action through PeerLAND developed student teachers’ knowledge in every TPACK domain, except for content knowledge. Furthermore, peer evaluation is considered advantageous to student teachers for getting timely constructive feedback and refining their designs, and several ideas for improving the peer evaluation mechanism are proposed. Implications for practice or policy: • PeerLAND is an online tool supporting the development and peer evaluation of technology-enhanced learning designs allowing teachers to work together and switch roles between designers and reviewers. • The learning design process in PeerLAND is a ready to use, step by step process for training teachers in technology-enhanced learning design. It provides a replicable blueprint for organising curricula. Keywords: learning design, technology pedagogy content knowledge, peer evaluation, teacher education Introduction As digital technologies rapidly infiltrate educational structures, the educational technology and teacher professional development research areas work with a high priority on understanding what knowledge is required for teachers to incorporate technology into their teaching practice appropriately, and how to develop and measure this knowledge (Harris et al., 2017; Nguyen et al., 2022; Saubern et al., 2020). In these contexts, the Technology Pedagogy Content Knowledge (TPACK) framework is a well-known, suggesting that knowledge of technology, pedagogy, and content, along with their intersections, is required by teachers to teach effectively with technology (Mishra, 2019; Mishra & Koehler, 2006). TPACK-focused research significantly impacts teacher education, resulting in various proposals for TPACK development that consider factors such as teachers’ background (pre-service, in-service), the discipline, and the context (Mishra, 2019; Rosenberg & Koehler, 2015). Also, a key aspect considered is the open-ended and rapid rate of technology advancements (Koehler et al., 2014). TPACK measurement is usually approached as a Australasian Journal of Educational Technology, 2022, 38(6). 54 concrete process involving several approaches organised around self-reports or artefacts evaluation (Wang et al., 2018). A quite challenging approach in TPACK measurement through artefacts evaluation is by involving peers. Considering learning designs as artefacts, the area of learning design which views teaching as a design science (Laurillard, 2018) may be valuable for organising peer evaluation aligned with the design process, with an aim to promote TPACK development and measurement simultaneously. To this end, digital learning design tools have been acknowledged for helping teachers improve how they incorporate technology and innovative pedagogy in their practice (Asensio-Pérez et al., 2017). It is worth investigating how they may support this idea as well. Indeed, despite the variety of learning design tools developed, the need to further teachers’ capabilities as technology enhanced learning designers remains an open issue (Nguyen & Bower, 2018). A challenging research goal is how learning design tools supporting the development of learning designs of several granularity levels (single lesson, course, curriculum) may also support developing and measuring teachers’ knowledge. Although learning design practices seem to take up peer evaluation as an inherent element of the underlying need for learning design evaluation (Bjælde et al., 2019; Sagy & Kali, 2014), they: (1) employ general-purpose tools to support ideas and feedback exchange, (2) promote a free-form discussion as evaluation process, and (3) organise the evaluation process in various phases of the learning design process. learning design evaluation has yet to be operationalised to a level accessible to teachers to align with and feed the design process. To this end, we think it is worth further investigating how the TPACK framework's theoretical underpinnings may inform the development of digital learning design tools. Currently, learning design tools neither support developing and evaluating learning designs based on the TPACK framework, nor approach the design and evaluation process holistically toward reforming learning designs. This paper addresses this research gap by putting TPACK into action through an learning design tool. First the theoretical background is elaborated. We then present PeerLAND, an learning design tool which interweaves design and peer evaluation in an integrated process based on TPACK, promoting the roles of teachers as designers and reviewers. In Papanikolaou, Gouli et al. (2016), initial evaluation results of PeerLAND were provided based on students’ perceptions as designers and reviewers. To collect evidence about the impact on students’ knowledge and get feedback about the peer evaluation process employed, we report on a study conducted in a teacher education context. We elaborate the methodology following a quantitative design to investigate how student teachers' knowledge develops through the learning design and peer evaluation processes supported by PeerLAND, and a qualitative design to determine how student teachers value integrating peer evaluation in the learning design process through PeerLAND. Finally, we discuss the findings and conclude with implications for both the learning design and TPACK research areas. Theoretical background Pathways to cultivate TPACK Regarding TPACK development, several approaches have been proposed concerning the pathway to cultivate specific teacher knowledge types (Koehler et al., 2014; Nguyen et al., 2022). Mishra et al’s (2009) proposed taxonomy of TPACK-based activity types organised content-specific activities in categories associated with relevant technologies aiming at helping teachers develop TPACK. This proposal has been considered a pathway to move from pedagogical-content-knowledge (PCK) to technological-knowledge (TK) and technological-pedagogical-knowledge (TPK), and finally to TPACK. Such a pathway looks appropriate for in-service teachers with a certain level of PCK when starting a teacher education program on technology integration. For pre-service teachers with minimal TPACK knowledge levels, pathways starting from TPK and evolving to TPACK are also adopted. An interesting proposal in this direction was technology mapping (Angeli & Valanides, 2013). This proposal emphasised mapping, or connecting, technological tool affordances to transform content representations and/or support student-centred pedagogies. However, a common finding in several research papers was that teachers need to acquire more classroom experiences with technology integration modelling or teaching tryouts or learning design tasks before they can exhibit a more sophisticated approach to their application of TPACK (Agyei & Voogt, 2012; Angeli & Valanides, 2013). Pre-service teachers especially, are reported to face difficulties developing synthetic knowledge domains due to insufficient teaching experience (Nguyen & Bower, 2018; Pamuk, 2012). For example, in Pamuk’s Australasian Journal of Educational Technology, 2022, 38(6). 55 (2012) study, pre-service teachers were asked to develop educational materials for teaching their choice of subject matter to their chosen target population. Pamuk’s (2012) proposal was to start from PCK, as technology integration requires a deep understanding of core knowledge and interpretation of the teaching context and its dynamics. Similarly, Nguyen and Bower (2018) claimed that novice designers may benefit from participating in reflective activities about their underlying teaching beliefs, pedagogical support while designing, extensive scaffolding for selecting technologies, concrete and authentic design contexts, and support for group work processes. Lastly, the learning technology by design approach has aimed to simultaneously cultivate PCK and TPACK by involving student teachers in authentic design problem-solving with technology (Koehler et al., 2014; Nguyen et al., 2022). Although a primary challenge of this approach was the cognitive load that students experience, it has been found that participants significantly developed knowledge in each of the seven TPACK domains (Koehler et al., 2014). TPACK measurement Research on TPACK measurement investigates mainly two evaluation methods (Chai et al., 2016; Koehler et al., 2014). The first evaluation method aims to promote teachers’ self-assessment drawn through questionnaires, rubrics, interviews, and reflective journals. Instruments developed to promote teachers’ self-assessment were general (Schmid et al., 2020; Schmidt et al., 2009), for specific technology such as interactive whiteboards (Koh & Divaharan, 2013), for specific pedagogy such as meaningful learning (Chai et al., 2013), or for specific content such as STEM (Chai et al., 2019). The second evaluation method aims to collect evidence of teachers’ involvement in learning design by analysing learning design discussions (Nguyen et al., 2022) or evaluating the artefacts produced. Examining peer artefacts has been considered a promising method that promotes dialogue among the designers and, if appropriately organised, reflection and awareness on the design process, extending teachers’ design experience. In this context, TPACK may provide a common language facilitating design decision-making and evaluation as well as designers’ interaction (Chai et al., 2016). Learning design practices integrating peer evaluation Learning design research has recorded many practices that have taken up peer evaluation as an inherent element of the underlying need for learning design evaluation in teacher education contexts. Peer evaluation has been valued as a practical method of formative evaluation when the instructor’s workload permits only providing a summative evaluation (Søndergaard & Mulder, 2012). It has been praised for bringing the constructivist learning principles into play, coupling the provision and use of feedback (Er et al., 2020; Nicol et al., 2014; Topping, 2021). In particular, a twofold approach to learning has been reported (Topping, 2021): first while formulating and delivering feedback, and then while receiving and evaluating the feedback, deciding what aspects to implement. However, concerns about peer evaluation involved the inevitable effect of friendship bonds, sympathy, antipathy, or even a peer’s popularity (Topping, 2009). Peers were often considered to show tolerance and lenience while reviewing to avoid conflicts and preserve social relationships (Friedman et al., 2008). Reviewees doubted the validity of peer assessment compared to an experienced instructor and believed that their peers would not mark them fairly (Karami & Rezaei, 2015). A critical issue relating to the quality and transparency of reviewers’ evaluation was the criteria used and the rating format (Friedman et al., 2008). Also, researchers aiming to improve the feedback quality called attention to the issue of structuring the peer evaluation process. For example, Er et al. (2020) structured dialogic feedback as a three-phase collaborative activity involving different levels of regulation. Several learning design practices incorporated peer evaluation. They mainly approached peer evaluation as an open process in which authentication or evaluation criteria were not considered. For instance, Sagy and Kali (2014) proposed a framework including three phases: (1) developing a design, (2) enacting it with learners, and (3) exploring its impact in various contexts. Peer evaluation is incorporated in the first two phases of this framework, either as oral discussion or in a written form, without following any particular criteria. Also, Bjælde et al. (2019) proposed a model for designing a course in higher education by incorporating free-form feedback loops realised by peers in moderated discussions. Asensio-Pérez et al. (2017) built a teacher professional development model around a learning design tool, ILDE, (Hernández- Australasian Journal of Educational Technology, 2022, 38(6). 56 Leo et al., 2018), including a main workshop phase for training ending with a peer review activity using ILDE “commenting” feature. TPACK's potential to inform developing learning design tools In line with recent proposals highlighting the complementary relationship of learning design with TPACK (Boschman et al., 2015), we acknowledge TPACK’s potential as an established framework and a practical aid for teachers while designing for technology enhanced learning. Furthermore, the recently proposed notion of design-TPACK (Nguyen et al., 2022), aims to support teachers’ learning design practices and reflect the importance of teachers’ understanding of technologies and pedagogical content knowledge for effective technology integration while designing learning interventions. Although several studies in the learning design area have provided pedagogical bases for constructing digital environments supporting the learning design process (Bennett et al., 2015), there is no systematic approach to integrating TPACK in these environments. TPACK is neither considered in the design process nor the evaluation process. Furthermore, although literature promotes peer evaluation as an effective method, it is mainly realised in general-purpose tools and not incorporated in learning design tools (Laurillard et al., 2018). This paper focuses on how TPACK may consistently address teachers’ learning design knowledge development and evaluation. We introduce an online environment named PeerLAND in teacher education to guide, in an integrated way, developing and evaluating learning designs as reflective activities. TPACK is adopted as a practical aid for teachers as designers on two levels: the level of actual design practice and the level of evaluation of learning designs, which we view as two interweaving mechanisms, the one feeding the other, in an integrated process. The aim is to encourage teachers to articulate and reflect upon their beliefs about learning and teaching with technology throughout the design process. In particular, PeerLAND: (1) offers a layer-based representation of the design, (2) scaffolds student teachers in selecting appropriate technology according to their pedagogical decisions to cultivate various types of TPACK knowledge progressively, (3) provides a peer feedback mechanism that promotes interaction among designers’ decisions and reviewers’ evaluation based on TPACK criteria, and (4) produces review reports that include comparative data of multiple reviewers, as a means of learning designs’ analysis. The following section elaborates how PeerLAND fosters thinking, designing, and evaluating learning designs through TPACK. PeerLAND interweaves design with peer evaluation in an integrated process PeerLAND (Peer Evaluation of LeArNingDesigns) is a free online platform (Figure 1) developed by the researhers in PHP using MySql DBMS. PeerLAND supports the design and evaluation process of technology enhanced learning artefacts, that is, learning designs. In line with the learning technology by design approach, participants work on authentic design problems undertaking the roles of designers and reviewers. In particular, the learning design process in PeerLAND starts with a modular design approach and ends with peer evaluation. It supports users as designers in explicitly representing their design ideas and making design decisions. It also supports users as reviewers in reflecting on the design process through a peer evaluation mechanism that supports sharing and evaluating peer designs, using criteria in line with the TPACK framework. Australasian Journal of Educational Technology, 2022, 38(6). 57 Figure 1. The start screen of PeerLAND PeerLAND supports the development of learning designs: The design environment The design environment of PeerLAND is minimal, supporting various design decisions related to pedagogy, technology, content, and their synthesis. It adopts a modular approach promoting teachers as designers to gradually develop a learning design in a multi-layer structure and cultivate various TPACK knowledge accordingly (Figure 2). Initially, as an author/designer, a teacher needs to bring to the surface or even cultivate PCK and make preliminary design decisions. Since the designer aims to develop a technology- enhanced design, technological decisions are inherent throughout the design process, although not explicitly required at the first two layers. Progressively, the designer has to start making more informed and complex decisions on the design structure and the technology enhanced learning activities based on the initial learning outcomes. Finally, the whole design rationale must unfold at the third layer and be explicit (TPACK). Figure 2. The multi-layer design process supported by PeerLAND At Layer 1, the designer needs to provide a general overview of a learning design, such as the target group, the learning outcomes, and the pedagogical approach adopted. The latter will affect the structure of the design. Thus, if the inquiry-based learning approach is adopted (Pedaste et al., 2015), the learning design is organised in phases such as orientation, conceptualisation, conclusion, and discussion. If the WebQuest approach (Dodge, 2001) is followed, then the learning design is organised in fields such as introduction, task, process, evaluation, and conclusion. Alternatively, the learning design may be organised around the Australasian Journal of Educational Technology, 2022, 38(6). 58 main concepts/themes of a project. For example, in a project on environmental pollution, the main themes may be environmental problems, causes and impacts, and environmental protection. Figure 3 depicts how the above decisions are implemented in PeerLAND for the learning design entitled “Let there be Rock”. The designer is prompted to provide the title of the learning design (1), a description (2) and its scope (3), define the learning outcomes (4), and the target group it addresses (5) by filling in the education level and selecting the discipline and sub-discipline. At (6), the designer outlines the structure of the learning design in phases or fields/themes that depend on the adopted pedagogical approach. The learning design shown in Figure 3 adopts the project-based approach organised in five themes. Finally, the designer who created this learning design can assign co-authors/designers (7) and reviewers (8), providing relevant rights to the design. These decisions aim at triggering designers’ PCK. In particular, we assume that PCK is triggered by the decisions the designer needs to make to align the target group with the scope of the learning design based on the curriculum by defining specific learning outcomes and deciding on the pedagogical approach to adopt for organising the structure of the design. Figure 3. Screenshot of PeerLAND with the overview of the “Let there be Rock” learning design At Layer 2, specific technology enhanced learning activities are designed (Figure 4). The designer is prompted to elaborate on each theme's scope and learning outcomes, considering the pedagogical approach adopted and initially articulated learning outcomes. They also need to propose the particular activities. The screenshot of Figure 4 shows the configuration of the second theme “Rock and Racism”, of the learning design “Let there be rock”: title definition (1), scope and outcomes (2), and learning activities (3). Designers proposing technology enhanced learning activities covering the scope and learning outcomes of the particular theme are expected to start synthesising PCK with TCK, in TPACK. At Layer 3, the main challenge is integrating content, pedagogy, and technology. Here, designers must explicitly represent their design ideas by integrating digital technologies and resources with teaching techniques and outcomes. Such a holistic approach aims at cultivating design skills by adopting a TPACK perspective. Australasian Journal of Educational Technology, 2022, 38(6). 59 Figure 4. Screenshot of PeerLAND with the configuration of the theme “Rock and Racism” Figure 5 shows a screenshot of PeerLAND while a designer articulates an activity. Here the designer fills in the title of the activity (1) and its scope/outcomes (2). Lists of teaching techniques (3) and digital tools/resources (4) are proposed to support designers, especially novice ones, select the most appropriate combination for each activity and argue about it. Moreover, the designer characterises the type of activity selecting in (6), one of the categories, Acquisition, Inquiry, Collaboration, Discussion, Produce, or Practice, according to the activity typology of the Conversational Framework (Laurillard, 2012). The designer also selects in (5) the specific knowledge processes (Experiencing, Conceptualising, Analyzing, and Applying) that the activity cultivates based on the New Learning framework (Kalantzis & Cope, 2012) in order to reach the expected outcomes. Australasian Journal of Educational Technology, 2022, 38(6). 60 Figure 5. Screenshot of an activity’s articulation Besides selecting the teaching techniques in (3) and the digital tools and/or resources in (4), the designer is prompted to explain their selection rationale in separate forms (Figures 6 and 7). Figure 6. Screenshot for explaining design decisions about teaching techniques adopted in a learning activity Australasian Journal of Educational Technology, 2022, 38(6). 61 Figure 7. Screenshot for explaining design decisions about digital tools/resources adopted in a learning activity A basic assumption guiding the rationale behind PeerLAND is that teachers who use it are mostly novices in learning design. This target group has various expertise in pedagogical-knowledge or PCK. To account for this varying expertise and support teachers with a structured and guided aid to learning design, PeerLAND adopts specific theoretical frameworks and conceptual tools. Adopting the Conversational Framework (Laurillard, 2012) and the New Learning Theory (Kalantzis & Cope, 2012), aims to stimulate designers think about the activities types they design and the cultivated knowledge processes. This supports primarily novice designers who tend to prematurely jump to design solutions without profoundly exploring their pedagogical rationale (Ronen-Fuhrmann & Kali, 2015). As designing is a complex process dealing with ill-defined problems and evolving constraints (Maina et al., 2015), applying such filtering of the vast range of ideas, theories, tools, and techniques in the field of technology enhanced learning, although limiting the range of choices, could be liberating for teachers’ design ventures. Moreover, pedagogical terms used at the three layers of the design process in PeerLAND form the basis of a design meta-language, necessary for teachers to use as common reference points. As designers, they reflect on what actually fits their design practice. As reviewers, they go one step further and attempt to match others’ learning designs to the available, commonly known items of their pedagogical and technological toolset, extending their design experience. Finally, in Figure 8, we summarise how PeerLAND implements the flow of learning design decision- making, aiming to scaffold the design process and progressively evoke cultivating designers’ TPACK. Figure 8. Design decisions taken through the design process in PeerLAND Australasian Journal of Educational Technology, 2022, 38(6). 62 PeerLAND supports the evaluation of learning designs: A peer evaluation mechanism PeerLAND adopts a structured and supportive peer evaluation mechanism aligned with the design process. It aims to promote reflection on design decisions taken around criteria aligned with the TPACK domains. This process also stimulates designers to compare their own decisions with the artefacts they review (Topping, 2021). The current version of PeerLAND supports authenticated evaluation for designers and reviewers, allowing designers to select their reviewers. Reviewers in PeerLAND can inspect, in view mode, all the layers of a design and provide two types of feedback: (1) criteria-based quantitative evaluation along with qualitative comments, (2) advice on the designers’ pedagogical and technological decisions taken at the third layer for an activity design. Reviewers can evaluate a design according to criteria aligned with the TPACK framework. Figure 9 depicts how a learning design’s evaluation is implemented in PeerLAND using the particular criteria in the forms TPACK (1/3) and TPACK (2/3). To assist reviewers, making each criterion’s rationale more transparent, PeerLAND provides the question that reviewers have to answer to evaluate each criterion by right-clicking on it. Reviewers value most of these criteria from 1 to 5 (filling in the empty frames) as this range follows the TPACK questionnaire (Schmidt et al., 2009). Reviewers can also submit qualitative comments on the strengths and weaknesses of a design (Figure 9, form TPACK 3/3), explaining the quantitative evaluations given in the forms TPACK (1/3) and TPACK (2/3). Figure 9. Evaluation criteria measuring the knowledge dimensions of TPACK organised in three tabs in PeerLAND reviewer environment: TPACK (1/3), TPACK (2/3), and TPACK (3/3) The evaluation framework adopted in PeerLAND allows reviewers to reflect on several criteria for each knowledge domain of the TPACK framework, apart from content knowledge, which is a prerequisite related to the learning design process (Figure 10). These criteria align with each TPACK knowledge domain scope by adapting the factors proposed by Schmidt et al. (2009) to include the particular technological and pedagogical tools. Such technological tools are Web 2.0 tools and tools of the e-learning platform. Also, such pedagogical tools are a typology of learning activities based on the Conversational Framework (Laurillard, 2012) and the knowledge processes proposed by the New Learning Theory (Kalantzis & Cope, 2012) employed in PeerLAND. Moreover, the value of each criterion given by the reviewers is associated with a weight (wi) that reflects the current context and priorities of the learning design project. For example, if the designers’ knowledge of curriculum standards is out of the scope of the current project, then the criterion x4 of PCK can be eliminated by giving the value zero (0) to weight w4 (Figure 10, PCK). Australasian Journal of Educational Technology, 2022, 38(6). 63 Figure 10. Evaluation framework based on criteria aligned with the TPACK framework In the case of the pedagogical knowledge criteria, their value is estimated automatically based on the comparison of the designer’s and the reviewer’s perspectives (Figure 11), reflecting the level of agreement between the two. In particular, the reviewers evaluate the designer’s decisions (reflecting agreement/disagreement) concerning the third layer of a learning design, where each activity is represented in terms of the activity type, the knowledge processes cultivated, the teaching techniques employed, and the digital tools/resources proposed. Comparing the designer’s point of view (Figure 5) with the reviewer’s point of view (Figure 11), we note that the reviewer can inspect and confirm by ticking in relevant boxes on the (R) column if they agree with the designer’s articulation appearing in column (A) as well as by ticking or not the relevant boxes in (1) and (2). Australasian Journal of Educational Technology, 2022, 38(6). 64 Figure 11. Screenshot for reviewers to provide advice on a designer’s decisions for an activity Evaluation reports PeerLAND produces a report with the numerical values of the reviewers’ evaluations per criterion and knowledge domain of TPACK and the mean value from all the reviewers. This report includes the qualitative revision comments and several comparative visualisations of the reviewers’ quantitative evaluations. The comparative visualisations aim to support the designer(s) in acknowledging the agreement level among the reviewers and interpreting the evaluation data from multiple reviewers. In Figure 12, the visualisation depicts the scores (x-axis: 1-5) provided by three reviewers for each knowledge domain of TPACK (y-axis: TK, PK, TPK, TCK, PCK, TPACK) for a specific design. The particular visualisation, illustrating the reviewers’ evaluations, provides the designer with a quick view on the knowledge domains mostly covered or not, stimulating them to search for the qualitative comments and the relevant reviewers’ argumentation. Australasian Journal of Educational Technology, 2022, 38(6). 65 Figure 12. Comparative visualisation of a learning design’s evaluation per TPACK knowledge domain from three reviewers Figures 13 and 14 illustrate the level of agreement between each reviewer and the designer related to the pedagogical knowledge elements, such as the knowledge processes covered and the types of activities developed for each phase of the learning design. As far as the knowledge processes are concerned (Figure 13), although the first reviewer’s view seems to match the designer’s view on the first phase of the design, then at the rest of the phases and especially at the fourth phase, they seem to diverge. Consequently, the designer’s design decisions should be reconsidered at Phase 4, where the agreement with all the reviewers has the lowest values. In Figure 14, the visualisations of the reviewers’ evaluations concern the types of activities developed at each design phase. In this case, the third reviewer seems to be the more divergent one compared to the designer’s selections through the five phases of the design, whilst the designer needs to reconsider the type of activities defined in the fourth phase of the design where again the agreement with all the reviewers has the lowest values. Figure 13. Comparative visualisation of a learning design’s evaluation from three reviewers according to the degree of agreement of each reviewer with the designer concerning the knowledge process(es) cultivated. Australasian Journal of Educational Technology, 2022, 38(6). 66 Figure 14. Comparative visualisation of a learning design’s evaluation from three reviewers according to the degree of agreement of each reviewer with the designer concerning the type(s) of activities proposed. Aim and research questions Aiming to evaluate the integration of PeerLAND in teacher education and specifically how interweaving learning design with peer evaluation empowers teachers to work as designers, we address the following questions: 1. How does student teachers' knowledge develop through the learning design and peer evaluation processes supported by PeerLAND? 2. How do student teachers value integrating peer evaluation in the learning design process through PeerLAND? Method Setting and participants We conducted an empirical study to evaluate PeerLAND’s integration into teacher education, specifically during the second year of a postgraduate course on technology-enhanced distance education at the National and Kapodistrian University of Athens. The 20 course-attendees, 14 males and 6 females, came from various disciplinary areas, such as computer science and mathematics, intending to follow a teaching career (we call them student teachers from now on). They all consented to participate in the research that followed the ethical standards of the institutional research and ethics committee. The course was organised based on the main design principles of the framework for constructivist pre- service teacher training on technology enhanced learning proposed in Papanikolaou et al. (2017). This framework aims to gradually cultivate synthetic knowledge domains of TPACK through active involvement in authentic design problem solving with technology. In this course, technology is perceived as a three- dimensional scheme (Papanikolaou, Makrh, et al., 2016). Participants were acquainted with: (1) means for developing learning designs (PeerLAND), (2) means for implementing the learning design as a virtual learning environment for students (Moodle), and (3) the digital tools for serving particular learning purposes such as various types of Web 2.0 and standalone applications. Participants were introduced to various pedagogical tools in order to be able to set learning outcomes and design activities of various types to accomplish them. Aiming to get participants acquainted with essential elements of learning designs, they were initially assigned to analyse several pedagogical and technological elements of exemplar learning designs. Afterwards, the student teachers were assigned an learning design project that included: (1) development of a learning design, (2) reviewing learning designs of their peers, and (3) implementing the learning design in Moodle. The learning design project was organised into two phases. In the first phase, the student teachers were introduced to PeerLAND. They were then assigned to develop a learning design for technology enhanced learning in PeerLAND and implement it in Moodle. The subject matter/disciplinary area was not an issue, Australasian Journal of Educational Technology, 2022, 38(6). 67 since the student teachers were considered fully conversant with their discipline. Thus, content knowledge is considered high. In the second phase, they participated in a peer evaluation activity using PeerLAND. They were assigned to review two learning designs of their peers. Therefore, each student teacher as a designer received two evaluation reports for their learning design from peers acting as reviewers. Data collection and analysis We followed a quantitative design to address the first research question and investigate how student teachers' knowledge develops through the learning design and peer evaluation processes supported by PeerLAND. Before and after the learning design project, we collected and analysed the participants’ answers on the adapted version of the TPACK questionnaire (Schmidt et al., 2009) to identify participants’ progress related to their involvement in the design, implementation, and evaluation of learning designs. Data analysis was performed in SPSS. We followed a qualitative design based on thematic analysis (Braun & Clarke, 2006) of participants’ responses to open-ended questions to address the second research question and determine how they value integrating peer evaluation in the learning design process through PeerLAND. We collected participants’ perceptions in self-reflective accounts, asking them to respond in three open-ended questions addressing, the advantages, the drawbacks, and their preferences regarding the peer evaluation process supported by PeerLAND. We applied thematic analysis following a deductive coding process in NVivo to identify themes. One of the researchers performed the thematic analysis while systematically consulting the rest of the researchers. We excluded data from two participants who did not fulfil all the course activities in the data analysis process. Thus, our findings refer to 18 out of 20 participants. Findings Research question 1: Student teachers' knowledge development through learning design and peer evaluation processes supported by PeerLAND First, we tested for significant differences between the before and after TPACK questionnaires by applying the non-parametric Wilcoxon signed rank test for every TPACK domain (Table 1). The Wilcoxon test was used as the sample size was small (< 20). The results show a significant improvement after carrying out the learning design project in every domain, except content-knowledge, where p > .001. Table 1 Wilcoxon test results comparing the median scores between the two TPACK questionnaires (n = 18) Knowledge domain Score before the learning design project Score after the learning design project z p Median Median Technology 3.78 4.55 -3.66 .000 Content 4.10 4.40 -.48 .628 Pedagogy 3.32 4.14 -3.47 .001 Technology-Pedagogy 3.50 4.30 -3.51 .000 Technology-Content 4.00 4.00 -2.92 .001 Pedagogy-Content 3.67 4.17 -3.15 .001 Technology-Pedagogy-Content 3.30 4.10 -3.25 .001 To further comprehend how the student teachers’ knowledge developed, we considered the data range of before and after scores per type of knowledge by producing boxplots of the dataset of Table 1. In the middle of the tinted box of the boxplots (Figure 15), the slightly thicker horizontal line represents the median value. This line is surrounded by a box, the top and bottom of which are the boundaries within which the middle 50% of the scores fall. The boxplots show that for each domain of TPACK, except the technology content- knowledge, the median score of the student teachers (before) was lower than the median score of the student teachers (after). Regarding technology content knowledge, although the median scores before and after do not differ, it appears that 50% of the scores (before) are lower than 50% of the scores (after). The above observations indicate that participating in the learning design project cultivated the student teachers’ TPACK knowledge (Finding 1). Australasian Journal of Educational Technology, 2022, 38(6). 68 Figure 15. Boxplots of knowledge domain scores on before and after TPACK questionnaires Research question 2: The value of integrating peer evaluation in the learning design process through PeerLAND Tables 2, 3, and 4 include the thematic analysis of the responses to the open-ended questions addressing student teachers’ perceptions of the corresponding advantages, drawbacks, and preferences. Frequencies and indicative quotes complement the analysis. Table 2 Student teachers’ perceptions (n=18, responses included more than one theme). Q1. How was the peer evaluation integrated into designing for learning advantageous to you? Theme Frequency Indicative quotes Q1.1 TPACK based evaluation criteria are appropriate for reviewing learning designs. 15 (84%) I appreciated the structured evaluation method that PeerLAND provided. The criteria based on TPACK are appropriate for reviewing learning designs. (Participant 11) Q1.2 Peers give constructive criticism and suggestions. 10 (56%) I received valuable feedback. My peers suggested several corrections and improvements for my design. (Participant 8) Q1.3 Stimulation of reviewers’ reflection. 10 (56%) Reviewing your peers’ designs triggers your reflection on your own work. I feel that the evaluation process ‘matured’ me as a designer. I realised my design’s shortcomings and omissions. It enlightened me towards improving my design. (Participant 4) Australasian Journal of Educational Technology, 2022, 38(6). 69 Q1.4 Stimulation of designers’ reflection. 2 (11%) The visual representations of the peer evaluation results provided by PeerLAND stimulated reflection on my design. (Participant 1) Q1.5 Promotion of learning design practice. 8 (44%) The evaluation process, acting as both a reviewer and a reviewee of learning designs, contributed towards improving my learning design and will guide my future practice. (Participant 10) Q1.6 Practical formative assessment that the instructor could not support. 4 (22%) Getting peer feedback was a practical formative assessment that boosted my work. Getting such an assessment from instructors is rare, almost impossible. (Participant 5) Q1.7 Cultivation of peer evaluation skills. 3 (17%) Through the systematic evaluation process followed, I cultivated peer evaluation skills. (Participant 2) Table 3 Student teachers’ perceptions (n=18, responses included more than one theme). Q2. In your opinion, what are the drawbacks of integrating learning design with peer evaluation? Theme Frequency Indicative quotes Q2.1 Peers give inadequate evaluation. 10 (56%) I am not sure whether my peers gave adequate evaluation. They obviously lack of experience and knowledge compared to an instructor. (Participant 12) Q2.2 Peers give favourable evaluation. 9 (50%) I think that I got favourable evaluation. I feel my peers hesitated to be hard on me due to authenticated evaluation. (Participant 7) Q2.3 Peers give inattentive evaluation. 4 (22%) I received one inattentive evaluation. It seems that my peer just wanted to fulfil the assignment as quick as possible. (Participant 1) Q2.4 Peers give subjective evaluation. 5 (28%) I feel that peer reviews are not objective. I am not sure whether I should trust mine or my peers’ point of view. (Participant 1) Q2.5 Copy effect / Design standardisation. 2 (11%) I feel that reviewing my peers’ design caused me a ‘copy effect’. I copied elements of the design to my design. Subsequently, I think this caused a standardisation of designing in our course. (Participant 18) Table 4 Student teachers’ perceptions (n=18, responses included more than one theme). Q3. What improvements would you suggest for the context of the peer evaluation adopted? Theme Frequency Indicative quotes Q3.1 Designer anonymity to address reviewer bias. 9 (50%) I would prefer being anonymous as a designer. If the designer maintains his anonymity, then the reviewer does not exercise a conscious or unconscious bias. (Participant 11) Q3.2 Reviewer anonymity to address peer burden. 10 (55%) Due to my interpersonal relationships with my peers, I admit that I was a bit soft on evaluating them. (Participant 6) Q3.3 Documentation mechanism to prevent inattentive evaluation. 3 (17%) I received one inattentive evaluation. Quantitative evaluation should be complemented with appropriate justifying comments. Maybe a structured documentation system would suffice. (Participant 15) Q3.4 Fewer evaluation criteria. 2 (11%) There were a lot of evaluation criteria organised around the knowledge domains of TPACK. Some of them could be omitted or merged. (Participant 14) Q3.5 Evaluation practice to cultivate review skills. 2 (11%) To be effective reviewers, I think we need to practice evaluating learning designs before the actual peer review process. I suggest organising a practice phase of evaluating sample learning designs to cultivate review skills. (Participant 1) Australasian Journal of Educational Technology, 2022, 38(6). 70 With regard to advantages (Finding 2), adopting TPACK criteria is highly valued for reviewing learning designs (Q1.1). Student teachers seem to have valued the peer reviews they received (Finding 3), as more than half of the participants considered constructive their peers’ criticism and suggestions (Q1.2). In this line, some participants have valued peer evaluation as a formative evaluation form (Finding 4) when it is not available from the instructor (Q1.6). The stimulation of student teachers’ reflection is twofold (Finding 5). As reviewers, studying and evaluating their peers’ designs intrigued them to reflect on their work (Q1.3). As designers, they reflected on the visual representations of the evaluations they received (Q1.4). Moreover, the participants reported the positive value of the review process on their future learning design practice (Q1.5) (Finding 6). They also appreciated cultivating peer evaluation skills (Q1.7) (Finding 7). Regarding the drawbacks, participants questioned the validity of peer evaluation (Finding 8), raising concerns about peers giving: • inadequate evaluation as they are not qualified as experts (Q2.1), • favourable evaluation to maintain friendly relationships (Q2.2), • inattentive evaluation as a cursory just to fulfil the assignment (Q2.3), and • subjective evaluation that may not be trusted. Also, it is worth mentioning that some participants noted the implication of copying design ideas and subsequently causing design standardisation (Q2.5) (Finding 9). Regarding their preferences for peer evaluation, half of the participants argued about preventing the reviewers’ bias through designer’s anonymity (Q3.1) (Finding 10). Likewise, more than half of the participants argued in favour of reviewer’s anonymity (Finding 11) to address peer burden due to interpersonal relationships (Q3.2). Some participants suggested: (1) having fewer criteria based on TPACK (Q3.4) (Finding 12), (2) incorporating a documentation mechanism that complements quantitative evaluation with explanatory comments (Q3.4) (Finding13), and (3) practising the evaluation of sample learning designs before the actual peer review process (Q3.5) (Finding 14). Discussion and conclusions Regarding the development of student teachers’ TPACK through the learning design project, our findings (Finding 1) suggest that the student teachers scored higher after participating in the learning design project, indicating a significant improvement in every TPACK domain, except for content knowledge, which was expected since the particular learning design project didn’t focus on this particular domain. This is an important finding providing evidence about the effectiveness of: (1) supporting the modular development of learning designs based on TPACK, (2) promoting the gradual synthesis of PCK with TCK into TPACK, (3) guiding the selection of pedagogical and technological tools that could be used in the design of technology enhanced learning activities, (4) promoting students to argue on their design decisions, and (5) structuring reflection on peers’ designs. This way, student teachers using PeerLAND that facilitates a modular learning design, design sharing and evaluation based on criteria aligned with the design process, managed to face the cognitive load of authentic design problem solving with technology. This is in line with Nguyean et al. (2022), who also underlined the need to encourage pre-service teachers articulate more of the integrated TPACK constructs (PCK, TPK, TCK) based on their available technological knowledge, pedagogical knowledge and content knowledge competencies, for instance, through promoting them to justify particular design decisions such as technology selections in terms of their content and pedagogy choices. Regarding how student teachers valued integrating peer evaluation in the learning design process through PeerLAND, findings showed that it stimulates reflection and promotes review skills (Finding 5, Finding 7). Similar to Søndergaard and Mulder (2012), peer evaluation was considered a practical formative evaluation form (Finding 4). The student teachers reported that studying, comparing, and evaluating peer designs intrigued them to reflect on their learning design practice and promoted their learning design skills by eliciting and refining their design ideas (Finding 6). Like previous research (Karami & Rezaei, 2015; Topping, 2009), student teachers acknowledged several deficiencies in peer evaluation. A recurrent concern was about peers not being experts. Another concern was about providing or getting a favourable evaluation to preserve interpersonal relationships. Student teachers doubted the veracity of the peer evaluations. They Australasian Journal of Educational Technology, 2022, 38(6). 71 also thought peer evaluation of being a cursory when participants cared only about fulfilling assessment requirements rather than engaging more deeply in the review process. Lastly, they noted a trend toward design standardisation due to copying design ideas (Finding 8). However, they also acknowledged having received constructive criticism and/or suggestions (Finding 3). Concerning the criteria used to guide the reviewing process, the student teachers considered appropriate the quantitative criteria supported by PeerLAND that underlie the TPACK framework, as opposed to (Finding 2) alternative options such as user-defined. However, some of them suggested having fewer criteria (Finding 12). Concerning the authentication approach adopted, the findings showed that student teachers were divided between the designer being anonymous or identifiable and the reviewer providing an anonymous or authenticated evaluation (Finding 10-Finding11). They provided strong arguments in favour of anonymity. They suggested that a designer’s identity should remain anonymous so that the reviewer does not exercise a conscious or unconscious bias. They mentioned the likelihood of an identifiable reviewer hesitating or avoiding giving a negative evaluation due to reservation or even fear of reciprocation. They claimed that a designer was likelier to accept an anonymous review without being influenced by biases or interpersonal relations with the reviewer. A main challenge that emerged from the research findings was ensuring the quality of the evaluation provided and addressing student teachers' concerns about the deficiencies of the peer evaluation process. As the participants seemed to have mixed perceptions regarding the authenticated evaluation supported for both designers and reviewers, it is worth considering how PeerLAND may also support anonymous reviews. Although having a modular design approach that encompasses TPACK as a means for integrating design with evaluation seemed to have students’ approval, we intend to investigate further students’ perceptions of the value and ease of using the TPACK criteria. This way, we anticipate cultivating a peer evaluation mindset addressing issues related to evaluation as cursory and design copying. In this line, a preparatory session on peer evaluation based on TPACK criteria with sample designs and a documentation mechanism for the reviews are interesting proposals that we shall further consider (Finding 13-Finding14). This research contributes to the design and the introduction of learning design tools in teacher education. They especially contribute to the design of teacher education programs that have struggled to find effective program-level and instructional-level strategies for adequately preparing pre-service teachers to integrate technology in their future classrooms (Wang et al., 2018). The implications of this study refer to operationalising TPACK for developing learning design tools and considering learning design development and evaluation holistically. Εspecially for teacher education programs addressed to pre-service teachers with a varying background in pedagogy, the experience of gradually cultivating various TPACK knowledge types through the modular design process of PeerLAND, seems quite successful. Through the peer evaluation process, pre-service teachers can reflect on artefacts and evaluate them using TPACK-based criteria, extending their experience in design and establishing the recognition of design qualities. The significance of cultivating PCK as a prerequisite for TPACK development has also been pinpointed by Pamuk (2012), Koehler et al. (2014), and Nguyean et al. (2022). The limitations of this study refer to the small sample of participants. However, our focus grounded findings on teachers’ rich experience over an learning design project that evolved over an entire academic semester rather than short training sessions and workshops lasting between a few hours and a few days. Future research goals should focus on building a teacher community around the learning design notion and put this into practice by extending PeerLAND in order to support all the features needed for incubating an online community: (a) scaffolding teachers as technology enhanced learning designers based on their particular needs, (b) allowing co-editing and sharing of designs, (c) supporting peer evaluation of the designs to promote dialogue and reflection among designers, and (d) promoting collaboration and exchange of good practices among peers. References Agyei, D. D., & Voogt, J. (2012). Developing technological pedagogical content knowledge in pre- service mathematics teachers through collaborative design. Australasian Journal of Educational Technology, 28(4). https://doi.org/10.14742/ajet.827 Angeli, C., & Valanides, N. (2013). Technology mapping: An approach for developing technological pedagogical content knowledge. Journal of Educational Computing Research, 48(2). https://journals.sagepub.com/doi/10.2190/EC.48.2.e https://doi.org/10.14742/ajet.827 https://journals.sagepub.com/doi/10.2190/EC.48.2.e Australasian Journal of Educational Technology, 2022, 38(6). 72 Asensio-Pérez, J. I., Dimitriadis, Y., Pozzi, F., Hernández-Leo, D., Prieto, L. P., Persico, D., & Villagrá- Sobrino, S. L. (2017). Towards teaching as design: Exploring the interplay between full-lifecycle learning design tooling and teacher professional development. Computers and Education, 114. 92- 116. https://doi.org/10.1016/j.compedu.2017.06.011 Bennett, S., Agostinho, S., & Lockyer, L. (2015). Technology tools to support learning design: Implications derived from an investigation of university teachers’ design practices. Computers and Education, 81, 211-220. https://doi.org/10.1016/j.compedu.2014.10.016 Bjælde, O. E., Hougaard, R. F., Caspersen, M. E., Lindberg, A. B., & Godsk, M. (2019). Learning design for science teacher training and educational development. Proceedings of ASCILITE 2015 - Australasian Society for Computers in Learning and Tertiary Education, Perth, 21-30. Boschman, F., McKenney, S., & Voogt, J. (2015). Exploring teachers’ use of TPACK in design talk: The collaborative design of technology-rich early literacy activities. Computers and Education, 82, 250- 262. https://doi.org/10.1016/j.compedu.2014.11.010 Braun, V., & Clarke, V. (2006). Qualitative research in psychology using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77- 101. https://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa Chai, C. S., Jong, M. S. Y., Yin, H., Chen, M., & Zhou, W. (2019). Validating and modelling teachers’ technological pedagogical content knowledge for integrative science, technology, engineering and mathematics education. Educational Technology and Society, 22(3), 61-73. https://www.jstor.org/stable/26896710 Chai, C. S., Koh, J. H. L., & Tsai, C. C. (2016). A review of the quantitative measures of technological pedagogical content knowledge (TPACK). In M. C. Herring, M. J. Koehler, & P. Mishra (Eds.) Handbook of technological pedagogical content knowledge (TPACK) for educators: Second edition. Routledge. https://doi.org/10.4324/9781315771328 Chai, C. S., Ng, E. M. W., Li, W., Hong, H. Y., & Koh, J. H. L. (2013). Validating and modelling technological pedagogical content knowledge framework among asian preservice teachers. Australasian Journal of Educational Technology, 29(1). https://doi.org/10.14742/ajet.174 Dodge, B. (2001). FOCUS: Five rules for writing a great WebQuest. Learning & Leading with Technology, 28(8), 6-9. Er, E., Dimitriadis, Y., & Gašević, D. (2021). A collaborative learning approach to dialogic peer feedback: A theoretical framework. Assessment and Evaluation in Higher Education, 46(4), 586-600. https://doi.org/10.1080/02602938.2020.1786497 Friedman, B. A., Cox, P. L., & Maher, L. E. (2008). An expectancy theory motivation approach to peer assessment. Journal of Management Education, 32(5), 580-612. https://doi.org/10.1177/1052562907310641 Harris, J., Phillips, M., Koehler, M., & Rosenberg, J. (2017). TPCK/TPACK research and development: Past, present, and future directions. Australasian Journal of Educational Technology, 33(3). https://doi.org/10.14742/ajet.3907 Hernández-Leo, D., Asensio-Pérez, J. I., Derntl, M., Pozzi, F., Chacón, J., Prieto, L. P., & Persico, D. (2018). An integrated environment for learning design. Frontiers in ICT. https://doi.org/10.3389/fict.2018.00009 Kalantzis, M., & Cope, B. (2012). New learning: Elements of a science of education (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9781139248532 Karami, A., & Rezaei, A. (2015). An overview of peer-assessment: The benefits and importance. Journal for the Study of English Linguistics, 3(1). https://doi.org/10.5296/jsel.v3i1.7889 Koehler, M. J., Mishra, P., Kereluik, K., Shin, T. S., & Graham, C. R. (2014). The technological pedagogical content knowledge framework. In J. Spector, M. Merrill, J. Elen, & M. Bishop (Eds) Handbook of research on educational communications and technology. Springer34. https://doi.org/10.1186/s41239-017-0072-z Koh, J. H. L., & Divaharan, S. (2013). Towards a TPACK-fostering ICT instructional process for teachers: Lessons from the implementation of interactive whiteboard instruction. Australasian Journal of Educational Technology, 29(2). https://doi.org/10.14742/ajet.97 Laurillard, D. (2012). Teaching as a design science: Building pedagogical patterns for learning and technology. Routledge. https://doi.org/10.4324/9780203125083 Laurillard, D. (2018). Teaching as a design science: Teachers building, testing, and sharing pedagogic ideas. In J. Voogt, G. Knezek, R. Christensen, & K. W. Lai (Eds), Second handbook of information technology in primary and secondary education. Springer International Handbooks of Education. Springer. https://doi.org/10.1007/978-3-319-71054-9_108 https://doi.org/10.1016/j.compedu.2017.06.011 https://doi.org/10.1016/j.compedu.2014.10.016 https://doi.org/10.1016/j.compedu.2014.11.010 https://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa https://www.jstor.org/stable/26896710 https://doi.org/10.4324/9781315771328 https://doi.org/10.14742/ajet.174 https://doi.org/10.1080/02602938.2020.1786497 https://doi.org/10.1177/1052562907310641 https://doi.org/10.14742/ajet.3907 https://doi.org/10.3389/fict.2018.00009 https://doi.org/10.1017/CBO9781139248532 https://doi.org/10.5296/jsel.v3i1.7889 https://doi.org/10.1186/s41239-017-0072-z https://doi.org/10.14742/ajet.97 https://doi.org/10.4324/9780203125083 Australasian Journal of Educational Technology, 2022, 38(6). 73 Laurillard, D., Kennedy, E., Charlton, P., Wild, J., & Dimakopoulos, D. (2018). Using technology to develop teachers as designers of TEL: Evaluating the learning designer. British Journal of Educational Technology, 49(6), 1044–1058. https://doi.org/10.1111/bjet.12697 Maina M., Craft, B., Mor, Y. (2015). The art & science of learning design. Technology enhanced learning. Sense Publishers. https://doi.org/10.1007/978-94-6300-103-8 Mishra, P. (2019). Considering contextual knowledge: The TPACK diagram gets an upgrade. Journal of Digital Learning in Teacher Education, 35(2), 76-78. https://doi.org/10.1080/21532974.2019.1588611 Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017–1054. https://doi.org/10.1111/j.1467- 9620.2006.00684.x Nguyen, G. N. H., & Bower, M. (2018). Novice teacher technology-enhanced learning design practices: The case of the silent pedagogy. British Journal of Educational Technology, 49(6), 1027-1043. https://doi.org/10.1111/bjet.12681 Nguyen, G. N. H., Bower, M., & Stevenson, M. (2022). The discourse of design: Patterns of TPACK contribution during pre-service teacher learning design conversations. Education and Information Technologies, 27, 8235–8264. https://doi.org/10.1007/s10639-022-10932-w Nicol, D., Thomson, A., & Breslin, C. (2014). Rethinking feedback practices in higher education: A peer review perspective. Assessment and Evaluation in Higher Education, 39(1), 102-122. https://doi.org/10.1080/02602938.2013.795518 Pamuk, S. (2012). Understanding preservice teachers’ technology use through TPACK framework. Journal of Computer Assisted Learning, 28(5), 425-439. https://doi.org/10.1111/j.1365- 2729.2011.00447.x Papanikolaou, K., Gouli, E., Makrh, K., Sofos, I., & Tzelepi, M. (2016). A peer evaluation tool of learning designs. In K. Verbert, M. Sharples, T. Klobučar (Eds) Adaptive and Adaptable Learning. EC-TEL 2016. Lecture Notes in Computer Science, 9891. Springer. https://doi.org/10.1007/978-3- 319-45153-4_15 Papanikolaou, K., Makrh, K., Magoulas, G. D., Chinou, D., Georgalas, A., & Roussos, P. (2016). Synthesizing technological and pedagogical knowledge in learning design. International Journal of Digital Literacy and Digital Competence, 7(134). https://doi.org/10.1186/s41239-017-0072-z Papanikolaou, K., Makri, K., & Roussos, P. (2017). Learning design as a vehicle for developing TPACK in blended teacher training on technology enhanced learning. International Journal of Educational Technology in Higher Education, 14(1). https://doi.org/10.1186/s41239-017-0072-z Pedaste, M., Mäeots, M., Siiman, L. A., de Jong, T., van Riesen, S. A. N., Kamp, E. T., Manoli, C. C., Zacharia, Z. C., & Tsourlidaki, E. (2015). Phases of inquiry-based learning: Definitions and the inquiry cycle. In Educational Research Review, 14, 47-61. https://doi.org/10.1016/j.edurev.2015.02.003 Ronen-Fuhrmann, T., & Kali, Y. (2015). Concretization of design ideas in the context of educational technology design. In M. Maina, B. Craft, & Y. Mor (Eds.), The art and science of learning design. Technology enhanced learning. SensePublishers. https://doi.org/10.1007/978-94-6300-103-8_3 Rosenberg, J. M., & Koehler, M. J. (2015). Context and technological pedagogical content knowledge (TPACK): A systematic review. Journal of Research on Technology in Education, 47(3), 186-210. https://doi.org/10.1080/15391523.2015.1052663 Sagy, O., & Kali, Y. (2014). Teachers as design-researchers of technology-enhanced learning. In Y. Eshet-Alkalai, A. Caspi, N. Geri, Y. Kalman, V. Silber-Varod, & Y. Yair (Eds.) Proceedings of the 9th Chais Conference for the Study of Innovation and Learning Technologies: Learning in the Technological Era, Raanana: The Open University of Israel. Saubern, R., Henderson, M., Heinrich, E., & Redmond, P. (2020). TPACK-time to reboot? Australasian Journal of Educational Technology, 36(3), 1-9. https://doi.org/10.14742/AJET.6378 Schmid, M., Brianza, E., & Petko, D. (2020). Developing a short assessment instrument for technological pedagogical content knowledge (TPACK.xs) and comparing the factor structure of an integrative and a transformative model. Computers and Education, 157, 103967. https://doi.org/10.1016/j.compedu.2020.103967 Schmidt, D. A., Baran, E., Thompson, A. D., Mishra, P., Koehler, M. J., & Shin, T. S. (2009). Technological pedagogical content knowledge (TPACK): The development and validation of an assessment instrument for preservice teachers. Journal of Research on Technology in Education, 42(2), 123-149. https://doi.org/10.1080/15391523.2009.10782544 https://doi.org/10.1111/bjet.12697 https://doi.org/10.1007/978-94-6300-103-8 https://doi.org/10.1080/21532974.2019.1588611 https://doi.org/10.1111/j.1467-9620.2006.00684.x https://doi.org/10.1111/j.1467-9620.2006.00684.x https://doi.org/10.1111/bjet.12681 https://doi.org/10.1007/s10639-022-10932-w https://doi.org/10.1080/02602938.2013.795518 https://doi.org/10.1111/j.1365-2729.2011.00447.x https://doi.org/10.1111/j.1365-2729.2011.00447.x https://doi.org/10.1007/978-3-319-45153-4_15 https://doi.org/10.1007/978-3-319-45153-4_15 https://doi.org/10.1186/s41239-017-0072-z https://doi.org/10.1186/s41239-017-0072-z https://doi.org/10.1016/j.edurev.2015.02.003 https://doi.org/10.1007/978-94-6300-103-8_3 https://doi.org/10.1080/15391523.2015.1052663 https://doi.org/10.14742/AJET.6378 https://doi.org/10.1016/j.compedu.2020.103967 https://doi.org/10.1080/15391523.2009.10782544 Australasian Journal of Educational Technology, 2022, 38(6). 74 Søndergaard, H., & Mulder, R. A. (2012). Collaborative learning through formative peer review: Pedagogy, programs and potential. Computer Science Education, 22(4), 343-367. https://doi.org/10.1080/08993408.2012.728041 Topping, K. J. (2009). Peer assessment. Theory into Practice, 48(1), 20-27. https://doi.org/10.1080/00405840802577569 Topping, K. J. (2021). Digital peer assessment in school teacher education and development: A systematic review. Research Papers in Education. https://doi.org/10.1080/02671522.2021.1961301 Wang, W., Schmidt-Crawford, D., & Jin, Y. (2018). Preservice teachers’ TPACK development: A review of literature. Journal of Digital Learning in Teacher Education, 34(4), 234-258. https://doi.org/10.1080/21532974.2018.1498039 Corresponding author: Kyparisia Papanikolaou, kpapanikolaou@aspete.gr Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY-NC- ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY- NC-ND 4.0. Please cite as: Papanikolaou, K. A., Makri, K., Sofos, I., Tzelepi, M. G., & Zalavra, E. (2022). Putting TPACK into action in learning design: The case of PeerLAND. Australasian Journal of Educational Technology, 38(6), 53-74. https://doi.org/10.14742/ajet.7556 https://doi.org/10.1080/08993408.2012.728041 https://doi.org/10.1080/00405840802577569 https://doi.org/10.1080/02671522.2021.1961301 https://doi.org/10.1080/21532974.2018.1498039 mailto:kpapanikolaou@aspete.gr https://creativecommons.org/licenses/by-nc-nd/4.0/ https://creativecommons.org/licenses/by-nc-nd/4.0/ https://doi.org/10.14742/ajet.7556