(page number not for citation purpose) 1 *Corresponding author. Email: horvath.laszlo@ppk.elte.hu Research in Learning Technology 2023. © 2023 L. Horváth. Research in Learning Technology is the journal of the Association for Learning Technology (ALT), a UK-based professional and scholarly society and membership organisation. ALT is registered charity number 1063519. http://www.alt.ac.uk/. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 ORIGINAL RESEARCH ARTICLE Adaptive pedagogical strategies responding to emergency remote teaching – immediate responses of Hungarian primary school teachers László Horváth* Institute of Education, ELTE Eötvös Loránd University, Budapest, Hungary (Received: 15 December 2022; Revised: 8 March 2023; Accepted: 15 March 2023; Published: 11 April 2023) Digital disruption is not a new phenomenon in education; however, it has become more prominent during the COVID-19 pandemic due to school closures and the related emergency remote teaching (ERT) period. Our study aims to explore the different pedagogical strategies that primary school teachers adopted during this period and determine how successful these strategies were in involving and engag- ing students. Altogether, 4028 teachers from 343 primary schools answered our online survey from all the regions of Hungary. The sample adequately represents the Hungarian primary school teacher population in terms of gender and age. We used cluster analysis and identified four clusters of pedagogical strategies; then, we used analysis of variance to explore how teachers’ digital competence and their ability to involve students in online learning varied across different clus- ters. Our analysis grasps the complexity of the issue, as it shows that two rather distinct strategies were both successful in involving students, and thus, there is no single solution best suited to digital learning. Overall, digitally competent teachers loosened the originally strict structure of education and provided more feedback, which proved to be an important element in successfully involving students in dig- ital learning during ERT. The framework validated in our research can be used by policymakers and school administrators in different national and educational con- texts, enabling them to understand the complexity of online teaching and learning. Furthermore, our results can offer some practical pointers for school teachers on how to combine different pedagogical strategies. Keywords: COVID-19; ICT integration in education; teachers’ digital competence; cluster analysis. Research in Learning Technology Vol. 31, 2023 Introduction The education sector globally has experienced increased exposure to digital dis- ruption in recent years, particularly during the emergency remote teaching (ERT) period that occurred as a response to COVID-19 and the related school closures. The central question of this paper is how the education sector can adapt and react to the changes and challenges of digital disruption. More specifically, we would like to examine how teachers adapt their teaching strategies by considering tech- nology integration. This paper chose to analyse one country, examining empirical data from Hungary, focusing on the ERT period starting in March 2020. mailto:horvath.laszlo@ppk.elte.hu http://www.alt.ac.uk/ http://creativecommons.org/licenses/by/4.0/ http://dx.doi.org/10.25304/rlt.v31.2978 https://orcid.org/0000-0003-0003-1045 L. Horváth 2 Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) Digital disruption is not a new phenomenon in education; however, it has increased during the ERT period and related school closures. Digital disruption can be defined as ‘the rapidly unfolding processes through which digital innovation comes to fundamentally alter historically sustainable logics for value creation and capture by unbundling and recombining linkages among resources or generating new ones’ (Skog, Wimelius, and Sandberg 2018, p. 432). It is especially interesting to examine the effects of digital disruption on the field of education because digital disruption has a two-fold impact. It not only affects the process of teaching and learning and the operation of schools, but it also affects the wider socio-economic contexts of students newly entering education and the world of work, as education must equip students with the necessary working skills. In our article, we first describe the context of our study: the ERT period (in gen- eral) and the Hungarian education system (in particular, related to issues of digital disruption). The theoretical background of our article builds on the intersection of technology integration and innovation theory; therefore, we then introduce this related context. After establishing the theoretical background of our article, we present the research methodology. We employed a quantitative research strategy to explore how teachers adapt their teaching strategies to the challenges of digital disruption. We conducted an online survey among Hungarian primary school teachers (N = 4028) to gather their perceptions regarding changes in their pedagogical practice and digital competence. We first present the descriptive data; then, we try to establish typical ped- agogical strategies via cluster analysis using the introduced variables. Finally, we pres- ent the details of the established clusters in relation to teachers’ digital competence (TDC) and their ability to involve students in the teaching and learning processes. To conclude, we discuss the relevance of our research, practical implications, possibilities to generalise the research, and limitations and further research opportunities. The context of this study: ERT There is empirical evidence from all around the world about how teachers have reacted to ERT. A narrative study by Nilsberth et al. (2021) emphasised how assessment in teaching shifted from a summative to a formative approach. Another study by Beattie, Wilson, and Hendry (2022) referred to the challenges of providing personalised learn- ing experiences during ERT, while others focused on the realisation of professional development needs and the lack of teacher competencies (Castañeda-Trujillo et al. 2021; Yong et al. 2021). The term ERT refers to a sudden change in the teaching context whereby stake- holders do not have enough time to plan for this transition or the necessary infra- structure and competencies (Barbour et al. 2020). ERT provides a unique context to explore how teachers integrate technology into their teaching and learning practices. It is especially important to explore how teachers reacted to ERT in the case of Hungary. The 2016 Hungarian Digital Education strategy concluded that teachers rarely used digital tools: less than 20% of teachers used digital solutions in more than a quarter of their classes (Digital Success Programme 2016). In 2019, the European Commission (2019) found that Hungary is one of the lowest-ranking EU member states in terms of the percentage of classes supported by ICT tools. Suddenly, in March 2020, the situation changed from 1 week to the next, as schoolteachers were required to ensure the continuity of teaching via remote tools. This global phenomenon allows http://dx.doi.org/10.25304/rlt.v31.2978 Research in Learning Technology Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 3 (page number not for citation purpose) researchers to gauge teachers’ initial considerations and strategies related to technol- ogy integration. The circumstances for technology integration were not ideal; thus, these responses should be considered sudden responses in an emergency and not well- planned strategic approaches. Nevertheless, we believe that our data can provide an in-depth understanding of technology integration processes in education. First, we will review existing empirical research on technology integration. After discussing the relevant dimensions of the phenomenon, we will focus on TDC as an important factor determining successful technology integration. Technology integration through the lens of innovation theory According to the National Center for Educational Statistics (2005), technology inte- gration can be defined as ‘the incorporation of technology and technology-based practices into the daily routines, work, and management of an organization’. Our study focuses on daily routines and work rather than management. Daily routines and work are the core business of schools: the processes and organisation of teach- ing and learning. To encompass both the top-down and bottom-up approaches, we interpret technology integration through the lens of educational innovation theory. Using innovation theory also aligns with the broader theme of digital disruption, as it can be understood from the perspectives of the Schumpeterian notion of creative destruction, putting the dynamics of disruptive innovation (Christensen 1997) into focus. The OECD (2019) defines innovation using the Oslo Manual’s definition (OECD 2018): ‘a new or improved product or process (or a combination thereof) that differs significantly from the unit’s previous products or processes and that has been made available to potential users (product) or brought into use by the unit (process)’. The broad definition of innovation encompasses various types of innovation. Furthermore, the innovation theory perspective distinguishes between the output, product, process, and nature of the phenomenon, where researchers’ focus could shift between the specifics of the given innovation and the characteris- tics of the agent responsible for creating/adapting the given innovation. The Oslo Manual (OECD 2018) defines the former as an ‘object’ and the latter as a ‘subject’ method for measuring innovation. Our analysis adopts a subject method as we do not focus on the specific digital innovations, technologies, and solutions adopted but on the agent (teachers) who are adopting the innovations and changing their practices and processes. Innovation theory (which is predominant in public sector innovation studies; Arundel, Bloch, and Ferguson 2019) could shed light on the usually invisible (Fugl- sang 2010) and hidden (NESTA 2007) everyday innovation processes (Lippke and Wegener 2014) driven by employees –in this case, teachers (Darsø and Høyrup 2012). Innovation processes can also be described using the notion of bricolage or tinkering; that is, those with lived experience of the process can create new structures through intrinsic actions that allow new ways of working (Fuglsang 2010). This latter perspec- tive is especially relevant for analysing emergent strategies in ERT as teachers had little to no time to plan and prepare for this situation; therefore, they had to rely on their existing experiences to try to solve the challenge. In the next section, we will review recent research results regarding technology integration in education to provide a theoretical base for our analysis of emergent http://dx.doi.org/10.25304/rlt.v31.2978 L. Horváth 4 Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) strategies. We will examine technology integration through the lens of innovation theory, which was presented in this section. Technology integration in education The previously cited definition of technology integration does not encompass the complex nature of the phenomenon. Amiel, Kubota, and Wives (2016) also draw attention to the importance of context in the emergence, diffusion, and adoption of (digital) innovations in an educational setting: technology integration does not hap- pen in isolation; it must be understood from a systemic perspective. Following an ecosystem approach, Amiel, Kubota, and Wives (2016) emphasise (after Nardi and O’Day 1999) that the focus should not be on the technology itself but on the activ- ities executed with the help of the technology and the context of implementation. It is important to investigate issues related to technology integration from a critical perspective as this integration does not always have positive results and can even lead to the reproduction of traditional methods (Chand, Deshmukh, and Shukla 2020). Drumm (2019) emphasises that teachers often rationalise their digital teaching with folk pedagogies (personalised mental models of teaching) and pseu- do-theories of learning, which Drumm partly considers a hindrance but also sees as a starting point for teachers. Considering that teachers’ beliefs play an import- ant role in successful technology integration, a failed or unsuccessful attempt at integration could reinforce traditional beliefs about the importance of ‘knowing content’ and about inequalities in learning (Chand, Deshmukh, and Shukla 2020). Technology integration without the necessary pedagogical considerations could easily facilitate behaviourist approaches to learning through automation and inter- activity (Drumm 2019). Based on TALIS data, Gil-Flores, Rodríguez-Santero, and Torres-Gordillo (2017) highlight that high ICT use is not only linked with con- structivist beliefs about learning but also a higher level of collaboration among teachers. Besides individual factors (digital competence, beliefs), group and organ- isational level factors also play an important role in explaining technology integra- tion. School leaders play an important role in supporting teachers in technology integration by improving the ICT infrastructure and creating a positive learning culture (Vermeulen et al. 2017). There are many models of technology integration, which change depending on the different elements they highlight. Among the most prominent ones are the technology acceptance model (TAM) and its updated version, the unified theory of acceptance and use of technology (UTAUT; Davis 1989; Davis, Bagozzi, and Warshaw 1989; Venkatesh et al. 2003). These models are based on different theo- ries, including the theory of reasoned action, the theory of planned behaviour, the innovation diffusion theory, and the social cognitive theory. The models predict technology use by considering factors such as performance and effort expectancy, social factors, facilitating conditions, and moderating variables such as gender, age, and experience. There are also other models that are more pedagogically focused, such as the Substitution – Augmentation – Modification – Redefinition (SAMR) model (Puentedura 2003) or the Passive-Interactive-Creative – Replace-Ampli- fy-Transform (PIC-RAT) model (Kimmons, Graham, and West 2020). However, these models do not have widespread, validated tools that would enable researchers to use them as valid and reliable instruments in explaining technology integration. http://dx.doi.org/10.25304/rlt.v31.2978 Research in Learning Technology Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 5 (page number not for citation purpose) Although these are prominent models widely used in the literature and educational practice, they do not address an important factor explaining technology integration in education: TDC. Teachers’ digital competence Falloon’s (2020) article summarises the different frameworks for understanding TDC: the Technological, Pedagogical, and Content Knowledge (TPACK) framework, the Distributed Thinking and Knowing, Engagement, Communication and Community, Knowledge Building (DECK) framework, the Critical Digital Literacy Framework (CDL), the Teacher Education Information Literacy (TEIL) framework, the Interna- tional Society for Technology in Education (ISTE) standards, the Digital Competence for Educators (DigCompEdu) framework and the TDC framework. The majority of studies on TDC use one or a combination of these frameworks (see, for example, Akturk and Ozturk 2019; Cerratto Pargman, Nouri, and Milrad 2018; Chou et al. 2020). This long list of different frameworks shows that there is ambiguity surrounding TDC in the scientific discourse. For this study, we selected a recently developed frame- work, the DigCompEdu, which has been, so far, validated in a few countries (Benali, Kaddouri, and Azzimani 2018; Cabero-Almenara et al. 2020; Ghomi and Redecker 2019). The framework describes six areas of TDC (using 22 indicators), considering both teachers’ professional and pedagogical competencies and learners’ competencies. The six areas are (1) professional engagement, (2) digital resources, (3) teaching and learning, (4) assessment, (5) empowering learners, and (6) facilitating learners’ digital competence (Redecker 2017). Research methodology This study uses the results of an online survey of Hungarian primary school teachers to explore the response of the school system to the digitally disruptive phenomenon caused by COVID-19 in the form of ERT. The aim of our study is to explore the dif- ferent pedagogical strategies that primary school teachers used during this period. By analysing what strategies digitally competent teachers used and how successful these strategies were in involving and engaging students, we can discuss the effectiveness of these pedagogical strategies. As our study is explorative, we used a quantitative research strategy to answer our research questions. We created a database using our online survey and analysed the data with Jamovi in alignment with our research questions. The following table (Table 1) describes the main aim of this study with its sub-questions, along with the data analysis method used to answer the research questions. The next subsections detail the sample and the instrument used in this study to answer our research questions, and then the study presents the results. Sample The survey was sent out to 450 primary schools in Hungary that participated in the EFOP-3.1.7 project (focusing on supporting primary schools’ capacity to combat early school leaving) in May 2020 (just a few months after the sudden shift to ERT). The final sample consists of 4028 teachers from 343 primary schools from all the regions http://dx.doi.org/10.25304/rlt.v31.2978 L. Horváth 6 Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) in Hungary. In each responding school, nearly half (47%) of the teachers completed our survey, which is overall a high response rate (76% of schools participated). The sample has the following demographic characteristics. The majority of the respondents are female (85.2%), which aligns with the proportion of women in the teacher population as a whole (85%; Hajdu et al. 2022). Hungarian public education is characterised by an aging teacher workforce: 47.7% of teachers are above 50 years old (Hajdu et al. 2022). In our sample, 43.7% of teachers are above 51 years old; therefore, we consider our sample as representative in terms of age. Respondents were also asked about their position and disciplinary focus (Table 2). Overall, our sample adequately characterises the Hungarian primary school teacher population in terms of gender and age. Instrument To assess the changes in the pedagogical practices and the organisation of educa- tion, we created a framework of items where respondents could indicate their chosen approach in their previous routine. Respondents could then indicate the direction of the change on a scale of 0–100. A point ranging from 0 to 49 meant a shift towards the element indicated on the left side of the scale, while a point between 51 and 100 indicated a shift towards the element on the right side of the scale. The following items were used: • Structure of learning (tight – loose): whether respondents opted for keeping disciplinary subjects separate in 45-minute classes or moved towards a looser structure utilising cross-curricular teaching. • Role of teacher (support – control): whether respondents opted to have a more supportive learning environment as opposed to a more controlled and struc- tured one. • Quantity of feedback (less – more): whether respondents gave less or more feedback than before. Table 1. Research questions and data analysis methods. Aim: Explore how teachers adapt their teaching strategies considering technology integration. RQ1: What are the different pedagogical strat- egies that primary school teachers adopted during the early days of ERT? Cluster analysis and a description of clusters along the main variables. RQ2: What pedagogical strategies did digitally competent teachers adopt in their response to the sudden shift to online learning? Analysis of variance: Comparing the previ- ously identified clusters (pedagogical strat- egies) with teachers’ digital competence. RQ3: How successful was each pedagogical strategy in terms of student involvement in online learning? Analysis of variance: Comparing the previously identified clusters (pedagogical strategies) with the reported percentage of involved students. ERT, emergency remote teaching. http://dx.doi.org/10.25304/rlt.v31.2978 Research in Learning Technology Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 7 (page number not for citation purpose) • Organisation of learning (asynchronous – synchronous): whether classes were scheduled asynchronously or respondents tried to meet regularly with their students online. • Type of assessment (formative – summative): whether respondents’ assessment practices shifted more towards a formative or summative approach. Several pedagogical strategies exist depending on the combination of these items. Therefore, this study aimed to explore the different pedagogical strategies that primary school teachers in Hungary followed during the early days of ERT. Additionally, it is important to understand how teachers’ digital competencies influenced their choice of pedagogical strategy during the ERT period. Another aim of our study was to explore what pedagogical strategies digitally competent teachers followed in response to the sudden shift to online learning. To assess TDC, we used Dig- CompEdu. The instrument measures the level of TDC by summing a set of items. We used the instrument by offering a 5-point Likert scale where respondents could assess each item and clarify how often they use it in their everyday practices. The instrument was translated into Hungarian by Horváth et al. (2020) for a teacher-educator sample; their translation was reworded in this study for a primary education sample. Table 2. Background information on the sample. Background variables N % of valid responses Gender Female 3364 85.2 Male 584 14.8 Age 25 years old or younger 46 1.2 25–30 years old 161 4.1 31–40 years old 739 18.8 41–50 years old 1268 32.3 51–60 years old 1467 37.3 61–65 years old 235 6 65 years old or older 15 0.4 Position Teacher with managerial roles (e.g. princi- pal, vice-principal, head of committee) 1271 31.7 Teachers without managerial roles 2551 63.6 Other (e.g. school psychologist, pedagogi- cal assistant) 190 4.7 Disciplinary area based on the National Core Curricu- lum (multiple choice) Physical education and sports 1092 27.9 Lifestyle and practice 915 23.3 Informatics 319 8.1 Arts 1217 31 Our earth and our environment 740 18.9 Human and nature 819 20.9 Human and society 513 13.1 Mathematics 1484 37.9 Ethnic language and literature 71 1.8 Foreign languages 453 11.6 Hungarian language and literature 1626 41.5 http://dx.doi.org/10.25304/rlt.v31.2978 L. Horváth 8 Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) Finally, student involvement in online learning during ERT was measured by teachers’ approximation of the percentage of students they were able to actively involve in the teaching and learning process. Thus, we wanted to explore how successful each pedagogical strategy was in terms of student involvement in online learning. This was measured separately for grades 1 to 4 (ISCED 1) and grades 5 to 8 (ISCED 2) as we felt that younger students could be considered a different target group as they are in the process of learning how to read, which could prove an important factor in using online learning tools and connecting to online lessons. Results Descriptive statistics To provide an overall picture of Hungarian primary schools’ responses to ERT, we present the main descriptive statistics, the level of TDC, and the percentage of stu- dents involved in digital learning as reported by teachers. First, the results regarding the changes in pedagogical practice in terms of the examined dimensions show (Figure 1) that the main strategy was to have a looser structure of classes with teachers providing more control than in face-to-face settings. Teachers provided feedback more frequently and reported using a summative type of assessment while shifting towards asynchronous solutions in the organisation of learning. To measure TDC, the 22 items of the DigCompEdu (assessed on a 5-point Likert scale) were summed. The reliability statistic according to McDonald’s ω is high (0.930); therefore, there is internal consistency. We also checked the validity of the one-factor model with Confirmatory Factor Analysis (CFA) using a Diagonally Weighted Least Squares (DWLS) method and robust standard errors (considering the ordered nature of the items). The CFA yielded acceptable model fit measures (χ2 (203) 59.0 55.7 62.4 48.7 53.2 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0 Structure of learning (tight - loose) Role of teacher (support - control) Quantity of feedback (less - more) Organization of learning (asynchronous - synchronous) Type of assessment (formative - summative) Figure 1. The mean and standard deviation of the scales (0–100) showing the shift in pedagogical strategies of Hungarian primary education teachers during the emergency remote teaching (ERT) period. Points lower than 50 mean a shift towards the left side of the scale, and points higher than 50 mean a shift towards the right side of the scale. http://dx.doi.org/10.25304/rlt.v31.2978 Research in Learning Technology Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 9 (page number not for citation purpose) = 6367; p < 0.001; CFI = 0.978; TLI = 0.975; RMSEA = 0.092; 95% CI [0.090; 0.094]; SRMR = 0.079) according to the thresholds proposed (CLI and TLI > 0.95; RMSEA < 0.10, and SRMR < 0.08) by Boateng et al. (2018). The summed-up TDC variable ranges from 22 to 110, with a mean of 88 and a standard deviation of 14.8. By cal- culating the levels of TDC according to the original instrument (similar to the levels of the Common European Framework of Reference for Languages), we can see that primary school teachers reported a high level of digital competency (45% [N = 1699] C1 – leader; 24.6% [N = 929] B2 – expert). Finally, the teachers believed they were able to involve around 75%–80% of their students in digital learning in the ERT period. They reported higher success with involving younger children (ISCED 1; M = 79.5%; SD = 23%) as compared to ISCED2 students (M = 74.6%; SD = 20.6%). Identifying typical pedagogical strategies responding to ERT (RQ1) The main aim of our study is to explore pedagogical strategies that primary school teachers used immediately after shifting to ERT. In accordance with our first research question (RQ1: What are the different pedagogical strategies that primary school teachers followed during the early days of ERT?), we conducted a cluster analysis on the variables used to describe changes in pedagogical practices and the organisation of education. The following variables were included in the cluster analysis: • structure of learning (tight – loose) • role of the teacher (support – control) • quantity of feedback (less – more) • organisation of learning (asynchronous – synchronous) • type of assessment (formative – summative) A K-means cluster analysis method was used with the Hartigan-Wong algorithm using Jamovi. By looking at the gap statistic (k) using the method developed by Kas- sambara and Mundt (2020), we established that the optimal number of clusters was four. The cluster membership variable was saved to the database, providing opportu- nities to analyse and compare the different groups. The following diagram (Figure 2) shows the descriptive statistics of the cluster variables for each of the four clusters with the number of respondents for each cluster. Teachers in Cluster 1 loosened up the structure of learning, deviating from the usual schedule of 45-minute classes and opting for a more flexible arrangement. Respondents in this group reported that they acted in more supportive ways and pro- vided more formative feedback than before. They also opted for an asynchronous type of teaching. Teachers in Cluster 2 adopted the opposite strategy of Cluster 1 in some regards. They tightened the structure, strictly sticking to the previous schedule, and were more controlling (as in providing more structure). While they did not change the organisation of learning nor the quantity of feedback provided, they still noted a shift towards summative assessments. Teachers in Cluster 3 were similar to those in Cluster 1 in that they opted for a loosened structure, but they accompanied this with stricter control in learning. They opted for a synchronous approach with a summative type of feedback and a stronger controlling role. Finally, Cluster 4 teachers provided http://dx.doi.org/10.25304/rlt.v31.2978 L. Horváth 10 Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) a tighter structure of learning and emphasised asynchronous learning opportunities; they also noted they took on a more supportive teacher role with less, although for- mative, feedback. Table 3 describes and highlights the key characteristics of each cluster to provide a comprehensive overview of each pedagogical strategy that we identified through our cluster analysis. Once we identify the characteristics of the pedagogical strategies of Hungarian primary school teachers, it is worth it to explore the demographic composition of these clusters. Table 4 indicates the gender, age, and disciplinary composition of the clusters. Although the contingency table analysis shows there are no significant asso- ciations between variables (except for gender, which has a weak association), looking at the descriptive percentages provides insights regarding the composition of the clus- ters. According to the results, most male teachers are in Cluster 4, while Cluster 3 has the most senior teachers. Regarding disciplinary composition, Cluster 2 has the most PE teachers, while Cluster 1 has humanistic-oriented teachers, and Cluster 4 mostly hosts natural science-oriented teachers. Our analysis successfully identified and described the pedagogical strategies that Hungarian primary school teachers followed in the early days of ERT. This enables us to answer our first research question (RQ1: What are the different pedagogical strate- gies that primary school teachers followed during the early days of ERT?), identifying and describing four pedagogical strategies based on different characteristics. In the next section, these clusters represent groups of teachers following distinc- tive pedagogical strategies. According to our research aims, we explored what charac- terises digitally competent (based on the DigCompEdu) teachers’ behaviour and how effective these strategies were in involving and engaging students. 67.4 28.5 76.9 36.9 31.9 48.0 66.4 63.6 54.5 62.0 76.5 80.9 79.3 78.0 80.9 46.4 41.1 30.3 32.4 39.1 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0 Structure of learning (tight - loose) Role of teacher (support - control) Quantity of feedback (less - more) Organization of learning (asynchronous- synchronous) Type of assessment (formative- summative) Cluster1 (N=377) Cluster2 (N=546) Cluster3 (N=486) Cluster4 (N=453) Figure 2. Means and standard deviation of the scales (0–100) for each cluster describing the shift in Hungarian primary education teachers’ pedagogical strategies during the emergency remote teaching (ERT) period. Points lower than 50 mean a shift towards the left side of the scale, and points higher than 50 mean a shift towards the right side of the scale. http://dx.doi.org/10.25304/rlt.v31.2978 Research in Learning Technology Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 11 (page number not for citation purpose) Table 3. Basic characteristics of pedagogical strategies employed by Hungarian primary school teachers during ERT. Cluster no. Cluster 1 (N = 377) Cluster 2 (N = 546) Cluster 3 (N = 486) Cluster 4 (N = 453) Structure of learning (tight – loose) tight tight tight tight no change no change no change no change loose loose loose loose Role of teacher (support – control) support support support support no change no change no change no change control control control control Quantity of feedback (less – more) less less less less no change no change no change no change more more more more Organisation of learn- ing (asynchronous – synchronous) asynchronous asynchronous asynchronous asynchronous no change no change no change no change synchronous synchronous synchronous synchronous Type of assessment (formative – summative) formative formative formative formative no change no change no change no change summative summative summative summative ERT, emergency remote teaching. Table 4. Demographic composition of clusters. Descriptive indicators Cluster1 Cluster2 Cluster3 Cluster4 Contingency table analysis Gender (% of male teach- ers within clusters) 13.3 17.5 14.0 20.1 χ2 (3) = 9.46; p = 0.024 Cramer’s V = 0.0717 Age (% of teachers above 50 within clusters) 43.6 43.5 49.2 43.8 χ2 (18) = 14.6; p = 0.690 Cramer’s V = 0.0515 D is ci p li n ar y fi el d ( % o f te ac h er s te ac h in g [… ] w it h in c lu st er s) Physical education and sports 24.20 29.80 25.10 28.20 χ2 (3) = 4.90; p = 0.179 Cramer’s V = 0.0517 Lifestyle and practice 23.60 21.10 20.50 21.20 χ2 (3) = 1.38; p = 0.711 Cramer’s V = 0.0274 Informatics 10.30 8 10.20 8.60 χ2 (3) = 2.36; p = 0.500 Cramer’s V = 0.0359 Arts 34.80 25.70 30.70 26.60 χ2 (3) = 10.6; p = 0.014 Cramer’s V = 0.0761 Our earth and our environment 16.80 18 18.60 19.90 χ2 (3) = 1.30; p = 0.728 Cramer’s V = 0.0267 Human and nature 23.60 17.60 21.70 19.40 χ2 (3) = 5–78; p = 0.123 Cramer’s V = 0.0562 Human and society 15.50 13.10 11.30 12.40 χ2 (3) = 3.43; p = 0.330 Cramer’s V = 0.0433 Mathematics 35.60 37 34.40 39.10 χ2 (3) = 2.31; p = 0.511 Cramer’s V = 0.0355 Ethnic language and literature 1.60 1.50 1.50 2.50 χ2 (3) = 1.88; p = 0.599 Cramer’s V = 0.0320 Foreign languages 14.10 11.50 12.50 12.40 χ2 (3) = 1.41; p = 0.704 Cramer’s V = 0.0277 Hungarian lan- guage and literature 42.10 39.60 40.30 38.40 χ2 (3) = 1.22; p = 0.747 Cramer’s V = 0.0259 Note: Largest percentages within clusters are highlighted in bold. http://dx.doi.org/10.25304/rlt.v31.2978 L. Horváth 12 Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) Assessing TDC and teachers’ ability to involve students in ERT (RQ2 and RQ3) To answer our second research question (RQ2: What pedagogical strategies did dig- itally competent teachers adopt in their response to the sudden shift to online learn- ing?), we investigated how digitally competent teachers acted during ERT and what kind of pedagogical strategies they followed. As discussed, we summed up the indica- tors of the DigCompEdu scale to get an overall indicator of TDC. The sub-dimensions of the DigCompEdu framework show that Hungarian pri- mary school teachers are advanced in selecting, creating, and managing digital resources (M = 4.35; SD = 0.68) and in professional engagement (M = 4.21; SD = 0.66), focusing on organisational communication, professional collaboration, reflec- tive practice, and digital continuous professional development. There are areas that need development, such as teaching and learning (M = 3.89; SD = 0.79), where teach- ers reported lower levels of competence in teaching, guidance, collaborative learning, and self-regulated learning with digital solutions. Another area that needs improve- ment is facilitating learners’ digital competence (M = 3.77, SD = 0.90), meaning developing learners’ information and media literacy, communication, content cre- ation, responsible use, and problem-solving. The summed-up variable of DigCompEdu measuring TDC failed to demon- strate normality (Shapiro-Wilk test: W = 0.969; p < 0.001). Therefore, we used a non- parametric one-way analysis of variance (Kruskal-Wallis) to explore differences between the clusters of pedagogical strategies. We used the Dwass-Steel-Critchlow- Fligner method to compare the different groups pairwise. Figure 3 shows the results (descriptive data and statistically significant comparisons). According to the results of the Kruskall-Wallis test (χ2 (3) = 107.2; p < 0.001), there are significant and intermedi- ate (ε2 = 0.058) differences between the clusters. Furthermore, all pairwise comparisons are significant. Teachers in Cluster 3 (M = 93.5; SD = 12.2) and Cluster 1 (M = 91.1; 91.1 88.1 93.5 84.2 12.0 32.0 52.0 72.0 92.0 112.0 132.0 152.0 ** * *** *** *** *** Cluster 1 Cluster 2 Cluster 3 Cluster 4 Figure 3. Means and standard deviations of teachers’ digital competence (TDC) (mea- sured by the sum of DigCompEdu scores) for each cluster, indicating statistically signif- icant differences between groups according to the results of the Dwass-Steel-Critchlow- Fligner pairwise comparisons after the non-parametric ANOVA (Kruskal-Wallis). *p < 0.05; **p < 0.01; ***p < 0.001. http://dx.doi.org/10.25304/rlt.v31.2978 Research in Learning Technology Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 13 (page number not for citation purpose) SD = 12.8) had higher levels of digital competence compared to those in Cluster 2 (M = 88.1; SD = 13.5) and Cluster 4 (M = 84.2; SD = 15.4). Thus, in answer to our second research question (RQ2: What pedagogical strat- egies did digitally competent teachers adopt in their response to the sudden shift to online learning?), the results show that teachers who were more digitally competent reacted to ERT by loosening the structure but providing stricter control over the teaching and learning process and using summative assessments and synchronous learning opportunities. As all the DigCompEdu means for the clusters are between 82.5 and 100 points (C1 – leader domain), we must treat this conclusion with caution as it can mean, in reality teachers are basically on the same level of expertise. It is important to examine the effectiveness of teachers in each cluster in terms of their ability to involve and engage students during ERT periods, which is our third research question (RQ3: How successful can each pedagogical strategy be considered in terms of student involvement in online learning?). As stated, primary school teachers reported that they could involve, on average, 79.5% (SD = 23%) of students on the ISCED1 level and, on average, 74.6% (SD = 20.6%) of students on the ISCED2 level. However, as before, both variables failed the prerequisite of normality (Shapiro-Wilk test: ISCED1: W = 0.835; p < 0.001; ISCED2: W = 0.917; p < 0.001); therefore, the non-para- metric ANOVA was used. Figure 4 shows the descriptive statistics and the results of pairwise comparisons. The results and comparisons show that the pedagogical strategies used by teachers in Cluster 3 (ISCED1 M = 83.3%; SD = 20.3%; ISCED 2 M = 80.4%; SD = 18.9%) and Cluster 1 (ISCED 1 M = 85.2%; SD = 19.8%; ISCED2 M = 78.8%; SD = 18.5%) are successful. There are no significant differences between the percentage of involved stu- dents for Cluster 3 and Cluster 1; however, both percentages are significantly different 85.2 78.877.0 73.0 83.3 80.4 73.7 68.7 0.0 20.0 40.0 60.0 80.0 100.0 120.0 140.0 % of students involved (ISCED 1) % of students involved (ISCED 2) *** *** *** *** *** *** *** *** Cluster1 Cluster2 Cluster3 Cluster4 Figure 4. Means and standard deviations of the percentage of involved students (ISCED 1 and 2) in emergency remote teaching (ERT) (based on self-declaration of teachers) for each cluster, indicating statistically significant differences between groups accord- ing to the results of the Dwass-Steel-Critchlow-Fligner pairwise comparisons after the non-parametric ANOVA (Kruskal-Wallis). *p < 0.05; **p < 0.01; ***p < 0.001. http://dx.doi.org/10.25304/rlt.v31.2978 L. Horváth 14 Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) from Cluster 2 (ISCED1 M = 77%; SD = 24%; ISCED2 M = 73%; SD = 20.1%) and Cluster 4 (ISCED1 M = 73.7%; SD = 25.6%; ISCED2 M = 68.7%; SD = 22.5%). The same pattern can be seen for both ISCED1 and ISCED2 students. However, on the descriptive level, it seems that the strategies applied by Cluster 1 teachers are slightly more successful for ISCED1 students, while strategies used by Cluster 3 teachers are slightly more successful for ISCED2 students. Thus, in answer to our third research question (RQ3: How successful can each pedagogical strategy be considered in terms of student involvement in online learn- ing?), it seems that the most successful approach for ISCED1 students is a loosened structure with more supportive teacher behaviour, more formative-type feedback, and more asynchronous learning opportunities. Instead, more control, summative feed- back, and synchronous learning opportunities worked better for ISCED2 students. Discussion and conclusion The main aim of our study was to explore how the education sector adapted and reacted to the changes and challenges of digital disruption. We used the example of the COVID-19-induced ERT period to gauge teachers’ emergency responses. To identify the different pedagogical strategies that primary school teachers followed during the early days of ERT (RQ1), we clustered respondents based on their answers regarding the change in practice in the structure of education, the role of the teacher, quantity of feedback, organisation of learning and type of assessment. At the end of our analysis, four distinctive groups emerged. To better understand the different strategies primary school teachers used, we examined what strategies digitally competent teachers followed (RQ2) and which strategy led to the highest percentage of students being involved in digital learning (RQ3). According to our results, the successful strategies involved teachers loosen- ing the strict frameworks, teaching structure, and learning processes and providing more feedback than usual (common elements of Cluster 3 and 1). Digitally competent teachers (RQ2) exercised more control over the teaching and learning processes than usual and relied on synchronous teaching solutions and more summative assessments during digital learning (Cluster 3). The same strategies (more control, synchronous teaching solutions, and summative assessments) were also best at successfully involv- ing students in the learning process (RQ3; Cluster 3 and 1). The study found that a pedagogical strategy that is more supportive and provides more formative assessment with dominantly asynchronous teaching methods can also be successful. It seems this latter strategy was more successful for ISCED1 students, while the former strategy (Cluster 3) was more successful for ISCED2 students. Our results show that both synchronous and asynchronous strategies can be successful. In Cluster 1, teachers paired asynchronous learning with more support, while in Cluster 3, teachers paired the synchronous approach with more control. Our results emphasise that online teaching and learning is a complex issue that cannot be described using one dimension (e.g. asynchronous or synchronous) but requires dif- ferent characteristics to be considered together. This finding is in line with Fernandez, Ramesh, and Manivannan (2022), who emphasised that synchronous learning was more stressful for students, but asynchronous learning put a greater burden on them. Therefore, our findings indicate that the complex pedagogical strategy linking teach- ers’ role (support/control) to their learning organisation (synchronous/asynchronous) http://dx.doi.org/10.25304/rlt.v31.2978 Research in Learning Technology Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 15 (page number not for citation purpose) can have important practical implications: teachers’ control behaviours can help man- age and structure workload, while their supportive behaviours can help manage stress. Our cluster analysis approach can be generalised and should be tested in other educational and national contexts as it seems to grasp the complexity of the issue. Adding to this complexity, the results’ implications are that two distinct strategies can both be successful, emphasising the importance of personalisation (reiterating the claims of Beattie, Wilson, and Hendry 2022), and that there is no one-size-fits-all approach to digital learning. The literature suggests that technology integration can lead to the reproduction of traditional methods (such as in the case of Cluster  3), as confirmed by Chand, Deshmukh, and Shukla (2020), but it can also link to con- structivist beliefs (such as in the case of Cluster 1) as concluded by Gil-Flores, Rodrí- guez-Santero, and Torres-Gordillo (2017). Overall, loosening the originally strict structure of education and providing more feedback (either summative or formative – Nilsberth et al. 2021, supported the effective- ness of the latter form) was important for digitally competent teachers. It also proved to be an important element in successfully involving students in digital learning during ERT. Finally, this study has some limitations. Although our sample is representative of Hungarian primary school teachers, it is unclear how it can be generalised to other national and educational contexts. Although the international literature discussed in this paper provides some support for the applicability of our claims to other contexts, the results should be further explored. Additionally, as the study employed a quan- titative approach using a self-reporting questionnaire, further data could ensure the validity of our results by conducting qualitative studies to understand the different pedagogical strategies in practice. Overall, our research sheds light on the complexity of online teaching and learn- ing during emergencies (in this case, the COVID-19 pandemic). Although teachers’ strategies were immediate responses to an external phenomenon, it seems there were deliberate pedagogical considerations behind these choices. Our research adds to the discourse on online teaching and learning by introducing a framework to identify different pedagogical strategies for online teaching and learning. Policymakers and school administrators can use the framework and underlying results to better under- stand their approach to online teaching and learning and introduce supporting pol- icies that consider the complexity of the model. Finally, the results can be useful for school teachers as they can help them see the complex interrelations and different elements that a pedagogical strategy can contain, highlighting possible pitfalls and opportunities. Funding Project no. 134206 has been implemented with the support provided from the National Research, Development and Innovation Fund of Hungary, financed under the OTKA Postdoctoral Excellence funding scheme. References Akturk, A. O. & Ozturk, H. S. (2019) ‘Teachers’ TPACK levels and students’ self-efficacy as pre- dictors of students’ academic achievement’, International Journal of Research in Education and Science, vol. 5, no. 1, pp. 283–294. http://dx.doi.org/10.25304/rlt.v31.2978 L. Horváth 16 Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) Amiel, T., Kubota, L. C. & Wives, W. W. (2016) ‘A systemic model for differentiating school technology integration’, Research in Learning Technology, vol. 24, p. 31856. doi: 10.3402/ rlt.v24.31856 Arundel, A., Bloch, C. & Ferguson, B. (2019) ‘Advancing innovation in the public sector: align- ing innovation measurement with policy goals’, Research Policy, vol. 48, no. 3, pp. 789–798. doi: 10.1016/j.respol.2018.12.001 Barbour, M., et al., (2020) Understanding Pandemic Pedagogy: Differences between Emergency Remote, Remote, and Online Teaching, Canadian ELearning Network, Halfmoon Bay. doi: 10.13140/RG.2.2.31848.70401 Beattie, M., Wilson, C. & Hendry, G. (2022) ‘Learning from lockdown: examining Scottish pri- mary teachers’ experiences of emergency remote teaching’, British Journal of Educational Studies, vol. 70, no. 2, pp. 217–234. doi: 10.1080/00071005.2021.1915958 Benali, M., Kaddouri, M. & Azzimani, T. (2018) ‘Digital competence of Moroccan teach- ers of English’, International Journal of Education and Development using Information and Communication Technology, vol. 14, no. 2, pp. 99–120. Boateng, G. O., et al., (2018) ‘A novel household water insecurity scale: procedures and psycho- metric analysis among postpartum women in western Kenya’, PLoS One, vol. 13, no. 6, p. e0198591. doi: 10.1371/journal.pone.0198591 Cabero-Almenara, J., et al., (2020) ‘Development of the teacher digital competence validation of DigCompEdu check-in questionnaire in the University Context of Andalusia (Spain)’, Sustainability, vol. 12, no. 5, p. 6094. doi: 10.3390/su12156094 Castañeda-Trujillo, J. E., et al., (2021) ‘Pedagogical strategies used by English teacher educators to overcome the challenges posed by emergency remote teaching during the COVID-19 pandemic’, Íkala, Revista de Lenguaje y Cultura, vol. 26, no. 3, pp. 697–713. doi: 10.17533/ udea.ikala/v26n3a12 Cerratto Pargman, T., Nouri, J. & Milrad, M. (2018) ‘Taking an instrumental genesis lens: new insights into collaborative mobile learning’, British Journal of Educational Technology, vol. 49, no. 2, pp. 219–234. doi: 10.1111/bjet.12585 Chand, V. S., Deshmukh, K. S. & Shukla, A. (2020) ‘Why does technology integration fail? Teacher beliefs and content developer assumptions in an Indian initiative’, Educational Technology Research and Development, vol. 68, no. 5, pp. 2753–2774. doi: 10.1007/s11423-020-09760-x Chou, C., et al., (2020) ‘Developing and validating a scale for measuring teachers’ readiness for flipped classrooms in junior high schools’, British Journal of Educational Technology, vol. 51, no. 4, pp. 1420–1435. doi: 10.1111/bjet.12895 Christensen, C. M. (1997) Innovator’s Dilemma, Harvard Business Review Press, Boston, MA. Darsø, L. & Høyrup, S. (2012) ‘Developing a framework for innovation and learning in the workplace’, in Practice-Based Innovation: Insights, Applications and Policy Implications, eds H. Melkas & V. Harmaakorpi, Springer-Verlag, Berlin, pp. 135–154. Davis, F. D. (1989) ‘Perceived usefulness, perceived ease of use, and user acceptance of informa- tion technology’, MIS Quarterly, vol. 13, no. 3, pp. 319–340. doi: 10.2307/249008 Davis, F. D., Bagozzi, R. P. & Warshaw, P. R. (1989) ‘User acceptance of computer technology: a comparison of two theoretical models’, Management Science, vol. 35, no. 8, pp. 982–1003. doi: 10.1287/mnsc.35.8.982 Digital Success Programme (2016) Digital Education Strategy of Hungary, Digital Success Programme, [online] Available at: https://digitalisjoletprogram.hu/files/d4/6b/d46bf17fdef- 3c9b5c1d38bd6db64c2a7.pdf Drumm, L. (2019) ‘Folk pedagogies and pseudo-theories: how lecturers rationalise their digital teaching’, Research in Learning Technology, vol. 27. doi: 10.25304/rlt.v27.2094 European Commission (2019) 2nd Survey of Schools: ICT in Education: Hungary Country Report, European Commission, Luxembourg. doi: 10.2759/76901 Falloon, G. (2020) ‘From digital literacy to digital competence: the teacher digital competency (TDC) framework’, Educational Technology Research and Development, vol. 68, no. 5, pp. 2449–2472. doi: 10.1007/s11423-020-09767-4 http://dx.doi.org/10.25304/rlt.v31.2978 https://doi.org/10.3402/rlt.v24.31856 https://doi.org/10.3402/rlt.v24.31856 https://doi.org/10.1016/j.respol.2018.12.001 https://doi.org/10.13140/RG.2.2.31848.70401 https://doi.org/10.1080/00071005.2021.1915958 https://doi.org/10.1371/journal.pone.0198591 https://doi.org/10.3390/su12156094 https://doi.org/10.17533/udea.ikala/v26n3a12 https://doi.org/10.17533/udea.ikala/v26n3a12 https://doi.org/10.1111/bjet.12585 https://doi.org/10.1007/s11423-020-09760-x https://doi.org/10.1111/bjet.12895 https://doi.org/10.2307/249008 https://doi.org/10.1287/mnsc.35.8.982 https://digitalisjoletprogram.hu/files/d4/6b/d46bf17fdef3c9b5c1d38bd6db64c2a7.pdf https://digitalisjoletprogram.hu/files/d4/6b/d46bf17fdef3c9b5c1d38bd6db64c2a7.pdf https://doi.org/10.25304/rlt.v27.2094 https://doi.org/10.2759/76901 https://doi.org/10.1007/s11423-020-09767-4 Research in Learning Technology Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 17 (page number not for citation purpose) Fernandez, C. J., Ramesh, R. & Manivannan, A. S. R. (2022) ‘Synchronous learning and asyn- chronous learning during COVID-19 pandemic: a case study in India’, Asian Association of Open Universities Journal, vol. 17, no. 1, pp. 1–14. doi: 10.1108/AAOUJ-02-2021-0027 Fuglsang, L. (2010) ‘Bricolage and invisible innovation in public service innovation’, Journal of Innovation Economics & Management, vol. 5, no. 1, pp. 67–87. doi: 10.3917/jie.005.0067 Ghomi, M. & Redecker, C. (2019) ‘Digital competence of educators (DigCompEdu): devel- opment and evaluation of a self-assessment instrument for teachers’ digital competence’, Proceedings of the 11th International Conference on Computer Supported Education, SCITEPRESS – Science and Technology Publications, Heraklion, pp. 541–548. doi: 10.5220/0007679005410548 Gil-Flores, J., Rodríguez-Santero, J. & Torres-Gordillo, J.-J. (2017) ‘Factors that explain the use of ICT in secondary-education classrooms: the role of teacher characteristics and school infra- structure’, Computers in Human Behavior, vol. 68, pp. 441–449. doi: 10.1016/j.chb.2016.11.057 Hajdu, T., et al., (2022) A közoktatás indikátorrendszere 2021 [Indicator System of the Hungarian Public Education 2021], Eötvös Loránd Kutatási Hálózat – Közgazdaság- és Regionális Tudományi Kutatóközpont, Közgazdaság-tudományi Intézet, Budapest. Horváth, L., et al., (2020) ‘Tanárképzők digitális kompetenciájának mérése – a DigCompEdu adaptálása a hazai felsőoktatási környezetre [Measuring the digital competence of teacher educators – adapting the DigCompEdu in Hungarian higher education]’, Neveléstudomány, vol. 2020, no. 2, pp. 5–25. doi: 10.21549/NTNY.29.2020.2.1 Kassambara, A. & Mundt, F. (2020) factoextra: Extract and Visualize the Results of Multivariate Data Analyses, [online] Available at: https://CRAN.R-project.org/package=factoextra Kimmons, R., Graham, C. R. & West, R. E. (2020) ‘The PICRAT model for technology inte- gration in teacher preparation’, Contemporary Issues in Technology and Teacher Education, vol. 20, no. 1, [online] Available at: https://citejournal.org/volume-20/issue-1-20/general/ the-picrat-model-for-technology-integration-in-teacher-preparation Lippke, L. & Wegener, C. (2014) ‘Everyday innovation – pushing boundaries while maintain- ing stability’, Journal of Workplace Learning, vol. 26, no. 6/7, pp. 376–391. doi: 10.1108/ JWL-10-2013-0086 Nardi, B. A. & O’Day, V. (1999) Information Ecologies: Using Technology with Heart, MIT Press, Cambridge, MA. National Center for Educational Statistics (2005) Forum Unified Education Technology Suite, National Center for Educational Statistics, [online] Available at: https://nces.ed.gov/ pubs2005/tech_suite/index.asp NESTA (2007) Hidden Innovation. How Innovation Happens in Six ‘Low Innovation’ Sectors, London, [online] Available at: https://media.nesta.org.uk/documents/hidden_innovation.pdf Nilsberth, M., et al., (2021) ‘Digital teaching as the new normal? Swedish upper secondary teachers’ experiences of emergency remote teaching during the COVID-19 crisis’, European Educational Research Journal, vol. 20, no. 4, pp. 442–462. doi: 10.1177/14749041211022480 OECD (2018) The Measurement of Scientific, Technological and Innovation Activities, Oslo Manual 2018, 4th edn, OECD, Paris. doi: 10.1787/24132764 OECD (2019) Measuring Innovation in Education 2019 What Has Changed in the Classroom? OECD, Paris. doi: 10.1787/9789264311671-en Puentedura, R. R. (2003) ‘A matrix model for designing and assessing network-enhanced courses’, Hippasus [Preprint], [online] Available at: http://www.hippasus.com/resources/matrixmodel/ Redecker, C. (2017) Digital Competence of Educators, ed Y. Punie, Publications Office of the European Union, Luxembourg. Skog, D. A., Wimelius, H. & Sandberg, J. (2018) ‘Digital disruption’, Business & Information Systems Engineering, vol. 60, no. 5, pp. 431–437. doi: 10.1007/s12599-018-0550-4 Venkatesh, V., et al., (2003) ‘User acceptance of information technology: toward a unified view’, MIS Quarterly, vol. 27, no. 3, p. 425. doi: 10.2307/30036540 Vermeulen, M., et al., (2017) ‘The role of transformative leadership, ICT-infrastructure and learning climate in teachers’ use of digital learning materials during their classes: teachers’ http://dx.doi.org/10.25304/rlt.v31.2978 https://doi.org/10.1108/AAOUJ-02-2021-0027 https://doi.org/10.3917/jie.005.0067 https://doi.org/10.5220/0007679005410548 https://doi.org/10.1016/j.chb.2016.11.057 https://doi.org/10.21549/NTNY.29.2020.2.1 https://CRAN.R-project.org/package=factoextra https://citejournal.org/volume-20/issue-1-20/general/the-picrat-model-for-technology-integration-in-teacher-preparation https://citejournal.org/volume-20/issue-1-20/general/the-picrat-model-for-technology-integration-in-teacher-preparation https://doi.org/10.1108/JWL-10-2013-0086 https://doi.org/10.1108/JWL-10-2013-0086 https://nces.ed.gov/pubs2005/tech_suite/index.asp https://nces.ed.gov/pubs2005/tech_suite/index.asp https://media.nesta.org.uk/documents/hidden_innovation.pdf https://doi.org/10.1177/14749041211022480 https://doi.org/10.1787/24132764 https://doi.org/10.1787/9789264311671-en http://www.hippasus.com/resources/matrixmodel/ https://doi.org/10.1007/s12599-018-0550-4 https://doi.org/10.2307/30036540 L. Horváth 18 Citation: Research in Learning Technology 2023, 31: 2978 - http://dx.doi.org/10.25304/rlt.v31.2978 (page number not for citation purpose) use of digital learning materials’, British Journal of Educational Technology, vol. 48, no. 6, pp. 1427–1440. doi: 10.1111/bjet.12478 Yong, K. L., et al., (2021) ‘Challenges in emergency remote teaching among Malaysian public elementary school teachers’, International Journal of Emerging Technologies in Learning (iJET), vol. 16, no. 24, pp. 74–90. doi: 10.3991/ijet.v16i24.27453 http://dx.doi.org/10.25304/rlt.v31.2978 https://doi.org/10.1111/bjet.12478 https://doi.org/10.3991/ijet.v16i24.27453