Bozkuş, K. (2021). Digital devices and student achievement: The relationship in PISA 2018 data. International Online Journal of Education and Teaching (IOJET), 8(3), 1560-1579. Received : 15.02.2021 Revised version received : 16.04.2021 Accepted : 17.04.2021 DIGITAL DEVICES AND STUDENT ACHIEVEMENT: THE RELATIONSHIP IN PISA 2018 DATA Research article Kıvanç Bozkuş Artvin Çoruh University kbozkus@artvin.edu.tr Biodata Dr. Bozkuş is an Assistant Professor working at Artvin Coruh University in Turkey. His research interests are professional development of teachers, school development, leadership, and organizational behavior. Copyright © 2014 by International Online Journal of Education and Teaching (IOJET). ISSN: 2148-225X. Material published and so copyrighted may not be published elsewhere without written permission of IOJET. http://orcid.org/0000-0002-4787-3664 mailto:kbozkus@artvin.edu.tr International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1560-1579. 1561 DIGITAL DEVICES AND STUDENT ACHIEVEMENT: THE RELATIONSHIP IN PISA 2018 DATA Kıvanç Bozkuş kbozkus@artvin.edu.tr Abstract This research aims to find how the infrastructure of digital devices within the school and teachers’ capacity using digital devices affect student achievement defined as PISA 2018 reading, math, and science scores. The ex post facto co-relational causal research design was employed. The data were obtained from the school questionnaire administered to school principals who participated in PISA (Program for International Student Assessment) 2018. A scale was developed from the PISA 2018 data available on OECD PISA website. To correlate factor means with PISA 2018 scores, a matrix was constructed by the researcher. To predict PISA 2018 scores with factor means, linear regression analyses were done. The results showed that the infrastructure of digital devices within the school affected PISA 2018 reading, math and science scores more than teachers’ capacity using digital devices. It was also revealed that there was a strong relationship between the infrastructure of digital devices within the school and teachers’ capacity using digital devices, and developing the infrastructure of digital technologies could provide practical benefits for students. Keywords: PISA exams, technology, skills, infrastructure 1. Introduction Technological advances change the way organizations do business. Education is also affected by these advances. If schools do not include the equipment that students will use in real life, schools cannot prepare students to perform the functions that society expects from them. Students should be able to use technology effectively if they want to live and work successfully in an increasingly complex and more knowledge-driven society. If schools do not use technology adequately, they cannot increase students' academic achievement. Many studies indicate that technology has a positive contribution to the academic achievement of students (Sezer, 2017; Sivin-Kachala & Bialo, 2000; Weathersbee, 2008). When educational technologies are used correctly, they increase students' self-confidence, motivation, and classroom participation (Passey, Rogers, Machell, McHugh & Allaway, 2003; Sivin-Kachala & Bialo, 2000). The use of technology in student-centered teaching positively affects student performance (Brown, 2007; Machin, McNally & Silva, 2007). In the learning environments where technology is used, student achievement increases, and students' high-level thinking skills develop (Becker, 2007; Ebuara, 2012; Simpson, 2010). Educational technology is thought to have a positive effect on the graduates’ professional life (Cradler, McNabb, Freeman, & Burchett, 2002; OECD, 2005). Technology alone is not enough for the development of education. The use of technology in lessons does not mean that it is integrated (Koehler & Mishra, 2005). It is only possible to realize the potential of technology by integrating it into the educational environment (Muir- Herzig, 2004; Voogt & Knezek, 2008). Technology integration is achieved by making mailto:kbozkus@artvin.edu.tr Bozkuş 1562 technology a seamless, routinely used, and almost invisible part of the learning environment (such as board and pen). If technology is effectively integrated, then it provides opportunities for students to search and find available information and apply their academic skills to real- world problems. Traditional educational practices, on the other hand, do not provide students with all the necessary skills they will need in today's world. Many research (Brenner & Brill, 2016; Njiku, Maniraho, & Mutarutinya, 2019; Tosuntaş, Çubukçu, & İnci, 2019) focuses on technology integration in education. However, research (Estapa, & Nadolny, 2015) that investigates the effect of technology integration on student achievement is scarce. Because “the use of technology in the classroom is not a critical issue facing education in the 21st century. The issue of foremost importance is to develop thinking skills in our students so that they will be able to utilize the power of technological tools to solve problems and do useful work” (McCain, 2005, p. 84). This research tries to close this gap by analyzing a very large data set. This research aims to find how the infrastructure of digital devices within the school and teachers’ capacity using digital devices affect student achievement defined as PISA 2018 reading, math, and science scores. Therefore, these two questions were asked: 1. How the infrastructure of digital devices within the school and teachers’ capacity using digital devices are correlated? 2. How the infrastructure of digital devices within the school and teachers’ capacity using digital devices predict PISA 2018 reading, math and science scores? 1.1. Technology Integration in Education Technology integration is a multidimensional, complex, slow, challenging, and dynamic process (Harris, Mishra, & Koehler, 2009; Hsu, 2010a; Mishra & Koehler, 2006; Roblyer, 2006). Hew and Brush (2007) define technology integration as teachers' use of technology to improve students' thinking skills. There are different uses of technology integration in the literature. While it is defined as teachers' use of computers in their classrooms (Redish & Chan, 2007), some researchers (Bebell, Russell & O'Dwyer, 2004; Hennessy, Ruthven & Brindley, 2005) define it according to how teachers use technology to perform their usual daily activities productively. There are many barriers to technology integration in education. Hew and Brush (2007) found that there are 123 obstacles to technology integration, and these are grouped under 6 categories: resources, knowledge and skills, school/institution, attitudes and beliefs, measurement, and subject culture. Hope (1997) classified the barriers in two dimensions; institutional factors (lack of vision and leadership, insufficient technology resources, lack of staff development, etc.) and individual factors (teachers' self-confidence, time barrier, traditional teaching styles, teachers' negative attitudes towards technology, etc.). Like Hope (1997), Ertmer (1999) classified two types of barriers: first- and second-order barriers of technology integration. First-order barriers refer to the lack of resources within the school. Second-order barriers are derived from teachers' beliefs, confidence, and attitudes. Institutional factors such as the insufficient number of computers, lack of hardware, and software are important factors affecting integration (Chen, 2010; Hsu, 2010b; Teo & van Schaik, 2009; Williams, Coles, Wilson, Richardson &Tuson, 2000). The availability of technological infrastructure and resources in schools is very important for the integration of technology in education (Buabeng-Andoh, 2012; Plomp, Anderson, Law, & Quale, 2009). International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1560-1579. 1563 Teachers are an important factor in technology integration (Granger, Morbey, Lotherington, Owston, & Wideman, 2002; Njiku, Maniraho, & Mutarutinya, 2019; Zhao & Frank, 2003). Fear of technology and lack of technical skills are crucial to technology integration (Tosuntaş, Çubukçu, & İnci, 2019; Lawson & Comber, 1999). The teacher's confidence and experience in the use of technology are important (Bovée, Voogt& Meelissen, 2007). Tondeur, Valcke and van Braak (2008) found that although teachers gradually began to integrate technology into their teaching strategies, there were significant differences in how they were integrated. 1.2. The Usage of Digital Technologies in Education There are enablers and barriers identified by researchers in the usage of digital technologies to determine which factors increase the usage and which factors prevent these technologies being used in education (Göktaş, Yıldırım & Yıldırım, 2009). These enablers can be exemplified as technology strategies, in-service training, strong infrastructures, technical support, and role models. In terms of barriers, Kim, Kim, Lee, Spector, and DeMeester (2013) claims that there are two sets of barriers that cause the difference in technology integration among teachers that have the relevant knowledge. Firstly, it is environmental readiness, which means the availability of technologies such as computers or internet access. However, this paper identifies the second barrier as the reason for technology integration for not being able to achieve and does not provide a discussion on the first barrier. The second barrier is the essential factors that hinder technology integration, even if the first barriers are being overcome. According to them, the second barrier consists of teachers’ beliefs in education, and these beliefs should be studied and identified for technology integration to be achieved. This is caused by the assumption that as the teachers do not change their beliefs, they will not be changing their behaviors in classrooms towards integrating digital technologies. Therefore, the authors put forward identifying and overcoming teachers’ beliefs towards the place of digital technologies in education as the main aim to accomplish. Lim and Khine (2006) bring an additional barrier into question, which is the effect of the compatibility of the curriculum of schools with digital technologies. According to them, a strong connection between higher education and what is happening in schools for pre-service teachers to put their training into practice should be established. According to them, even though teachers are trained in digital technologies and enough infrastructures are established in classrooms, it does not mean that they will be using these technologies. This is caused by the curriculums are not being updated in a way that they can be compatible with digital technologies to be used in class. Therefore, even though the teachers have the necessary training and classrooms are provided with the technology, teachers gravitate back to non- technology use and traditional pedagogy methods. Because they do not believe that the curriculum set by the schools can be taught effectively while using digital technologies. Petko (2012) takes a similar stance on Lim and Khine’s (2006) research and refuses the previous studies that solely focus on the beliefs of teachers and the infrastructure of classrooms in digital technology integration. In his research, he divides the process of effective integration of digital technologies into classrooms has three main factors. Firstly, on an individual level, teachers should be competent and their beliefs towards digital technology use should change. Secondly, on the school level, the necessary infrastructure should be provided, and the school culture should be changed. Thirdly, on an educational system level, educational policies and curriculum should be re-evaluated. Petko (2012) describes this theoretical model as the “will, skill, tool” in which teachers are most likely to employ Bozkuş 1564 information and communication technologies. According to this research, within each factor, improving some aspects is more effective than others. However, overall, integration of digital technologies in education cannot be achieved, if one of these factors is overlooked and the problem is trying to be solved by only focusing on one or two factors contributing to the lack of technological integration. Inan and Lowther (2010) pointed out to the low success rates in mathematics and reading announced by the Department of Education in the United States (NCES, 2008). By these results, they infer that an empirical correlation between access to technologies and success in test scores or improved quality of learning cannot be made. Therefore, they claim that the first barrier in digital education integration, which is access to the technologies, does not have a significant positive effect on test scores of students. According to them, integration of the technology into education and resulting in success is a more complicated process than having the digital technologies available to students and teachers. Factors such as computer literacy and age of teachers, the belief of teachers into the importance of technology in education and teachers’ willingness to incorporate technology in their classes play a more important role in technology integration into classrooms than the availability of the technologies. Among the barriers and factors contributing to the lack of integration of digital technologies in education, most researchers focus on the second barrier, individual-level factors and give less importance to the digital technologies’ infrastructure. Sang, Valcke, van Braak, Tondeur, and Zhu (2011) claim that integration of digital technologies into education cannot be limited with the first barriers, such as technological infrastructure and for a more comprehensive understanding of technology integration, second barriers should be examined. According to this paper, educators’ beliefs can be changed by taking certain measures on a school level that improves the motivation of them to incorporate digital technologies in classrooms. These measures can be exemplified by involving teachers in policymaking and planning processes on the usage of the said technologies in the classroom and curriculum and increasing the educators’ involvement in digital technologies. As another researcher who focuses on the importance of the teachers’ abilities to utilize the technology access provided in the classrooms, Depta (2015) claims that technology infrastructure in classrooms is not “a catalyst, but rather an accelerator”. According to her, having technology networks in a classroom is a positive contributor, but the effectiveness of it lies in the teachers’ capacities to utilize them for students to gain new skills and capacities. Moreover, she defines the success of technologies in classrooms by their ability to integrate with the education, collaboration between educator and technology staff, network stability, and communication and feedback. So, therefore, not just technologies becoming accessible and teachers being able to utilize them, but a rethinking of all education processes is necessary for digital technologies to be successful in classrooms. In this process, how the classrooms are designed, how many students are educated in a classroom, and how technology is being perceived should be looked over. In terms of the second barriers, Ertmer (2005) sees the beliefs of teachers as the main obstacle in front of digital technology integration. She claims that digital technologies in education cannot be used effectively for two main reasons: prejudice and belief of teachers. Firstly, teachers have a pedagogical prejudice against the usage of digital technologies in the classroom. Usually, teachers tend to feel more comfortable teaching in traditional pedagogical methods and to avoid the usage of high-technology devices in the classroom. However, the research leaves an open door for this prejudice to change over time if methods are being applied to change this prejudice. Secondly, she claims that as the teachers already have a pedagogical belief that traditional teaching methods are better, they would never use digital technologies in classrooms, if they are not trained enough. So, according to her, International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1560-1579. 1565 classrooms can have more advanced hardware and software installed, if the teachers who are going to use them are not able to use these tools, they become obsolete. Ertmer, Leftwich, and Tondeur (2015) found that prejudices of the teachers continue to exist as a problem for digital technologies being used in classrooms and additional “enablers”, such as training, administrative, and peer support should be established for effective digital technology integration to be achieved. In another study, first and second barriers have been examined by Li, Worch, Zhou, and Aguiton (2015) to evaluate the current conditions of the barriers, as the “digital native” young generation who grew up with digital technologies and uses in their daily life enter the workforce as a teacher. In their findings, they observed that even though digital technologies have improved over the years and have become more widespread and accessible, the digital infrastructure remains to be a factor as the first barrier in integration. Moreover, contrary to expectations, digital natives joining the workforce did not significantly improve the second barrier in digital technology integration since the beliefs of teachers in digital technologies’ application in education did not significantly change. Therefore, in their research, they concluded that even though young teachers feel more comfortable with their abilities in their capabilities in the use of digital technologies and these technologies are more available, first and second barriers are yet to be overcome. There is another factor that contributes to the second barrier as it impacts the teachers’ beliefs in digital technologies in education. Gilakjani (2013) states that self-efficacy plays an important role in the improvement of teachers’ usage of computer technology. Self-efficacy affects educators’ confidence in them to effectively use digital technology for education. As teachers do not feel in their abilities to use the technologies, their belief in digital technologies as a reliable education tool decreases. Therefore, according to his research, teachers should be aware of the aims, effectiveness, and technological tools of digital technologies to ensure better learning and teaching in their classrooms. 1.3. The Impact of Digital Technology in Education Approaches on how to achieve digital technology integration into education are not limited to the role of environmental conditions, teachers’ beliefs, and curriculum. Lim, Zhao, Tondeur, Chai, and Tsai (2013) point out an imbalance between the amount of investment that is done in digital technologies in education and the benefits of it in return. According to their research, an investment into providing access to technologies and training for students and teachers respectively has fewer returns than an investment in the usage of technology in the private sector, such as banking. In their paper, they draw attention to a factor in schools that has not been studied enough: the principals. According to the authors, education of principals in usage and importance of digital technologies plays a critical role as they are the ones who can provide the necessary guidelines to the teachers and can create a commitment to change towards a “learning environment” at school. Another study had been made by Pegrum, Oakley, and Faulkner (2013) on the usage of mobile handheld technologies in primary and secondary levels of education in Australia. In their findings, they emphasized that these devices alone by themselves cannot provide learning for the students and cannot be enough to improve the test success of students. According to them, only by integrating these devices into the “ecology of learning”, these devices can be effective education tools. The authors found that mobile devices can be beneficial in shifting teacher-based learning to student-based learning if they are used in education. Bozkuş 1566 Researches have shown the positive impact of having the necessary digital technology infrastructure for students as well. According to Wastiau, Blamire, Kearney, Quittre, Gaer and Monseur (2013), students who have more access to information and communication technologies both at home and at school are more confident in their digital abilities than the students who have little access to the technologies. Therefore, the study found that “digitally confident students” are more likely to use digital technologies better in classrooms. Moreover, these students believe in the positive impact of information and communication technologies in their learning processes, compared to their counterparts. However, students not having the necessary technological access at home and school do not only affect their abilities in the use of technology but also create an ethical problem for teachers who want to use digital technologies in their classrooms. According to Davies and West (2014), one of the reasons why teachers have moral or ethical objections to the usage of digital technologies in education is because not every student has the same level of access to digital technologies out of the classroom. Due to this discrepancy between students, teachers hesitate to give homework or assignment that requires the application of digital technologies because every student might not have the same opportunity. Therefore, this paper claims that future efforts to improve learning using educational technologies will need to prioritize equipping teachers and students with new technologies and educational resources. 2. Method 2.1. Research Design This study employed the ex post facto co-relational causal design (Cohen, Manion & Morrison, 2007). This design is used when research variables already exist in nature, leaving no room for manipulation (Fraenkel, Wallen & Hyun, 2012). Since there is a lack of manipulation, this design generates results which have weaker causality comparing to the experimental design. Yet, it can reveal the causality when an experimentation is not possible. Thus, it can be considered as an alternative method to the experimental design. 2.2. Data Collection The data were obtained from the school questionnaire administered to school principals who participated in PISA 2018. The questionnaire was developed by the PISA Governing Board. Principals were asked the following main question about the school's capacity using digital devices: “To what extent do you agree with the following statements about your school’s capacity to enhance learning and teaching using digital devices?” These included 11 Likert type items related to this question. Possible responses to each item were strongly disagree (01), disagree (02), agree (03), and strongly agree (04). For the sampling procedure, please refer to the PISA manuals. A total of 21903 school principals answered to this question. However, 20504 principals fully responded to all items. To reveal the factor structure of 11 items, an exploratory factor analysis based on the maximum likelihood extraction method was done. Kaiser Meyer Olkin measure of sampling adequacy was found high enough (.92) to consider the sample size is adequate. Bartlett's test of sphericity was significant (χ2=116826, df=55, p< .001). There were 2 factors that have eigenvalues bigger than 1. Since the correlation between factors was high and significant (r=.74, p<.001), oblimin rotation was used. The resulting factor structure is presented in Table 1. International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1560-1579. 1567 Table 1. Factor statistics of data Items Mean Factor 1 Factor 2 Uniqueness Variance α The number of digital devices connected to the Internet is sufficient 2.74 0.88 0.28 32.6 0.91 The school’s Internet bandwidth or speed is sufficient 2.69 0.72 0.46 The number of digital devices for instruction is sufficient 2.61 0.87 0.24 Digital devices at the school are sufficiently powerful in terms of computing capacity 2.65 0.84 0.28 The availability of adequate software is sufficient 2.69 0.72 0.36 Teachers have the necessary technical and pedagogical skills to integrate digital devices in instruction 2.77 0.67 0.56 22.2 0.81 Teachers have sufficient time to prepare lessons integrating digital devices 2.71 0.77 0.49 Effective professional resources for teachers to learn how to use digital devices are available 2.75 0.72 0.42 An effective online learning support platform is available 2.52 0.41 0.60 Teachers are provided with incentives to integrate digital devices in their teaching 2.57 0.54 0.68 The school has sufficient qualified technical assistant staff 2.60 0.48 0.57 The first 5 items formed factor 1 with loadings from .72 to .88. The other 6 items formed factor 2 with loadings from .41 to .77. Factor 1 explained 32.6% variance and had a Cronbach α reliability coefficient of .91. Factor 2 explained 22.2% variance and had a Cronbach α reliability coefficient of .81. Factor 1 had items related to the infrastructure of digital devices within the school, and Factor 2 had items related to teachers’ capacity using digital devices. Discrimination indexes of the items were between .49 and .76. To confirm the 2-factor structure of 11 items, a confirmatory factor analysis based on maximum likelihood estimation was done (Appendix 2). Results indicated a good model fit (CFI=.95, NFI=.95, IFI=.95, AGFI=.93, RMSEA=.08, SRMR=.04) according to Hooper, Coughlan, and Mullen (2008). 2.3. Data Analysis The study analyzed the PISA 2018 data available on OECD PISA website. To correlate factor means with PISA 2018 scores using Pearson product-moment correlation technique, a matrix was constructed by the researcher (Appendix 1). A total of 76 countries were included in the matrix along with their factor 1 and 2 statistics and PISA 2018 reading, math, and science scores. To predict PISA 2018 scores with factor means, linear regression analyses were done. Bozkuş 1568 3. Results The correlations between factor means and PISA 2018 reading, math, and science scores are presented in Table 2. Table 2. Correlations between factor means and PISA 2018 scores Reading Math Science The infrastructure of digital devices within the school 0.62** 0.66** 0.64** Teachers’ capacity using digital devices 0.17 0.25* 0.20 * p<0.05, ** p<0.001 Mean of factor 1 was moderately correlated to reading, math and science scores (r=0.62, r=0.66, r=0.64 respectively, p<0.001). The mean of factor 2 was only correlated to math score at a low level (r=0.25, p<0.05). The regression coefficients of factor means to predict PISA 2018 reading, math, and science scores are presented in Table 3. Table 3. Regressions between factor means and PISA 2018 scores Reading Math Science The infrastructure of digital devices within the school R .62 .66 .64 R2 .38 .44 .41 F 45.51** 58.23** 51.35** b0 (Intercept) 214.79 187.32 215.80 b1 (mean) 89.36 101.78 90.79 Teachers’ capacity using digital devices R .17 .25 .20 R2 .03 .06 .04 F 2.06 4.77* 3.01 b0 (Intercept) 357.03 308.48 346.41 b1 (mean) 36.41 56.82 42.23 * p<0.05, ** p<0.001 International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1560-1579. 1569 The linear regression results of factor 1 mean as the predictor of reading score suggests that the infrastructure of digital devices within the school can account for 38% of the variance in reading score and the regression model predicts reading score significantly well (F(1, 73)=45.51, p<0.001). If the school had no infrastructure of digital devices then the reading score would be as low as 214.79 (b0 =214.79, t=5.99, p<0.001) and each 1 increase in infrastructure mean also increases the reading score by 89.36 (b1 =89.36, t=6.75, p<0.001). The linear regression results of factor 1 mean as the predictor of math score suggests that the infrastructure of digital devices within the school can account for 44% of the variance in math score and the regression model predicts math score significantly well (F(1, 74)=58.23, p<0.001). If the school had no infrastructure of digital devices then the math score would be as low as 187.32 (b0 =187.32, t=5.20, p<0.001) and each 1 increase in infrastructure mean also increases math score by 101.78 (b1 =101.78, t=7.63, p<0.001). The linear regression results of factor 1 mean as the predictor of science score suggests that the infrastructure of digital devices within the school can account for 41% of the variance in science score and the regression model predicts science score significantly well (F(1, 74)=51.35, p<0.001). If the school had no infrastructure of digital devices then the science score would be as low as 215.80 (b0 =215.80, t=6.30, p<0.001) and each 1 increase in infrastructure mean also increases the science score by 90.79 (b1 =90.79, t=7.17, p<0.001). The linear regression results of factor 2 mean as the predictor of math score suggests that teachers’ capacity using digital devices can account for only 6% of the variance in math score and the regression model predicts math score significantly well (F(1, 74)=4.77, p<0.05). If teachers had no capacity using digital devices then the math score would be as low as 308.48 (b0 =308.48, t=4.44, p<0.001), and each 1 increase in teachers’ capacity mean also increases math score by 56.82 (b1 =56.82, t=2.18, p<0.05). 4. Discussion This research has contributed to the area by developing a valid standardized scale. Although the items of the scale have been written and applied to school principals by the PISA Governing Board, the author put considerable effort to make factor analyzes to establish construct validity. The scale can be used to assess the infrastructure of digital devices within the school, and teachers’ capacity using digital devices. Researchers can make use of the scale to measure these factors and relate them to other variables like student achievement, teacher self-efficacy, instruction methods used, administrative support, etc. This research has shown that the infrastructure of digital devices within the school affects PISA 2018 reading, math and science scores more than teachers’ capacity using digital devices. This result contradicts with many results of previous research (Ertmer, 2005, 2015; Hermans, Tondeur, van Braak, & Valke, 2008; Inan & Lowther, 2010; Kim, Kim, Lee, Spector, & DeMeester, 2013; Petko, 2012; Sang, Valcke, van Braak, Tondeur, & Zhu, 2011). However, previous research mostly investigates the success of educational technology integration. Moreover, previous research has aimed to formulate a method to achieve educational technology integration. This research is different, for it investigates the effect of the infrastructure of digital devices within the school and teachers’ capacity using digital devices on student achievement. Many pieces of research overlook the impact of digital technology infrastructure in the integration of technology in education. However, Batane and Ngwako (2017) emphasize the importance of establishing the necessary networks and overcoming the first barrier in digital technology integration. According to them, one way to combat the second barrier and to Bozkuş 1570 change teachers’ values on the use of digital technologies in classrooms is to widen the digital technologies provided in the classrooms, such as tablets, smart boards, and laptops for teachers and students. By increasing the available digital technologies in classes, firstly, teachers who try to integrate digital technologies in their teaching methods would not be discouraged by the lack of infrastructure. Secondly, having these technologies encourage the teachers, even if they do not feel comfortable in their capacities to use those technologies. Therefore, findings of this research back the position of this paper that having the necessary infrastructure of digital technologies in the classrooms has a more significant impact than the teachers’ capacity to be able to use them. Currently, digital technologies are becoming more available to students and teachers. In research conducted by Hew and Brush (2007), they found that the most common reported technology integration barrier, with 40% of the participants replying, has been the resources. In the research, resources have been identified as limited hardware, access, time, and technical support. Therefore, the findings of the research have been aligning with other scholars who put forward the lack of digital infrastructure as the main reason for problems in digital integration. Ertmer and Leftwich (2013) claim that significant progress in terms of removing the first and second barriers has been achieved as access to technology has become more widespread and the teachers had better training on digital technologies. Therefore, according to them, the focus should not be on technology or the teachers themselves, rather on students and their thinking skills. According to them, the focus should be on teaching the students how to utilize the technological tools to solve problems and do useful work. Therefore, the authors defend the necessity of having the necessary technologies available in the classrooms. However, unlike the many previous research that focused on teaching educators how to use technology and incorporate into education by changing their beliefs, their research gives the students more responsibility and importance in the integration of technologies in education. This may explain why the finding that the infrastructure of digital devices within the school affects scores more than teachers’ capacity using digital devices. Students may make use of digital infrastructure of the school and thus achieve better educational gains. A systematic literature review by Sun, Siklander, and Ruokamo (2018), supports this assertion. They found that “employing a computer environment was considered to be the most efficient way to trigger” students’ interest in digital learning environments (p. 77). This research has shown that both the infrastructure of digital devices within the school and teachers’ capacity using digital devices affects PISA 2018 math scores more than reading and science scores. This is an interesting finding that needs to be researched. Studies should employ different methodologies to reveal the reasons behind this. Especially, qualitative methods could be of help. An assumption for this may be that students who are already prone to math are also more likely to use digital devices. Another assumption can be made that there is more digital content for math. The impact of technology on mathematics achievement is well-proved by previous research. A meta-analysis by Higgins, Huscroft-D’Angelo, and Crawford (2019) investigated effects of technology in mathematics on achievement, motivation, and attitude. Results from 24 studies indicated a significant impact of technology in mathematics on student achievement, motivation, and attitudes. Similarly, Saal, Ryneveld, and Graham (2021) found that the availability of computers at school and using computers in the mathematics classroom positively associated with the mathematics achievement of students. However, Lee, Longhurst, and Campbell (2017) found that student achievement in science was not correlated to teachers’ capacity using digital devices. This research has also shown that there is a strong relationship between the infrastructure of digital devices within the school and teachers’ capacity using digital devices. We can infer International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1560-1579. 1571 this from the fact that the factors of the scale have a strong correlation. Digital infrastructure of the school may have provided opportunity of interacting with them, thus increasing teachers’ capacity using digital devices. This result is similar to that of Petko, Prasse, and Cantieni (2018) who suggest that school readiness has a strong impact on teacher readiness. According to them, teacher readiness is about the confidence of teachers in their abilities to use digital technologies and willingness to utilize them in education. School readiness, on the other hand, encompasses a larger network of curriculum, principal support, and infrastructure for available digital technologies. Even though their research sees both teacher and school readiness as crucial in technology education, they emphasize the fact that school providing the necessary digital infrastructure is a factor that pushes teachers to familiarize themselves and use these technologies in education. It is worth mentioning that this and their study collected data from different groups (i.e. principals and teachers), yet the findings are similar. Further research should be conducted on the effect of the infrastructure of digital devices within the schools as its impacts have been disregarded by the previous research. This research has shown that developing the infrastructure of digital technologies can have practical benefits to students. Debate on how full integration will be achieved and the place of teachers in this process has been done since computers have been invented and as the technology advances, this debate will change its form. Moreover, teaching will be shifting from teacher-centered learning to student-centered learning and even tertiary exams will shift shapes to accommodate the new learning environment in a better way. However, all these changes will take time and it is clear from the results of this research that providing the necessary digital technology infrastructure will be helping the students to achieve better results under the current education environment. Therefore, digital technology infrastructure is a concrete step that will benefit the students. Educators should focus on developing the infrastructure of digital technologies for better student achievement. Governments, families, and other stakeholders should find ways to improve the school's capacity using digital devices. Equipping classes with smart boards, tablets, computers and high-speed internet access may increase teacher capacity of using them, but the increase in student achievement will be much more than that. Because students are more engaged and comfortable with technology (Carstens, Mallon, Bataineh, & Al-Bataineh, 2021). This study had some limitations that should be considered when interpreting the findings. Since the samples were comprised of school principals, the general findings in this study may not apply to other groups of educators like teachers which could result in different outcomes if implemented. Future studies should include teachers to address this issue. Second, the ex post facto design and correlational statistics used in this study limit the link of causality among variables. Future studies should facilitate experimental designs in order to establish a stronger link of causality. Third, cross-sectional data, as was the case in this study, may result in unidirectional inferences among variables. Therefore, it is suggested that future studies should use longitudinal data to get an understanding of multiple relationships among the variables. Fourth, the self-reported data collected in this research may be affected by respondents’ subjectivity. Using alternative data collection techniques may help to solve this. Lastly, this study examined only 2 variables relating to student achievement. There could be other variables that may play a role in this relationship. 5. Conclusion This research showed that students do perform better in reading, math, and science overall when they are provided with the digital technology devices in school. To evaluate this, two separate factors have been taken into account. Firstly, the impact of the digital technology Bozkuş 1572 infrastructure of the school has been researched. Secondly, teachers’ capacity to use digital technology infrastructure has been researched. It has been found that both factors, school providing digital technology infrastructure and teachers’ capacity to use digital technology infrastructure had a positive impact on student’s test scores. However, it has also been found that the existence of digital technology infrastructure had a significantly higher impact on student success in test scores than teachers’ capacity to use the digital structure. International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1560-1579. 1573 References Batane, T., & Ngwako, A. (2017). Technology use by pre-service teachers during teaching practice: Are new teachers embracing technology right away in their first teaching experience? Australasian Journal of Educational Technology, 33(1). Bebell, D., Russell, M., & O’Dwyer, L. (2004). Measuring teachers’ technology uses: Why multiple-measures are more revealing. Journal of Research on Technology in Education, 37(1), 45-63. Becker, K. (2007). Digital game-based learning once removed: Teaching teachers. British Journal of Educational Technology, 38(3), 478-488. Bovée, C., Voogt, J., & Meelissen, M. (2007). Computer attitudes of primary and secondary students in South Africa. Computers in Human Behavior, 23(4), 1762-1776. Brenner, A. M., & Brill, J. M. (2016). Investigating practices in teacher education that promote and inhibit technology integration transfer in early career teachers. TechTrends 60, 136–144. Brown, C. (2007). Learning through multimedia construction: A complex strategy. Journal of Educational Multimedia and Hypermedia, 16(2), 93-124. Buabeng-Andoh, C. (2012). Factors influencing teachers’ adoption and integration of information and communication technology into teaching: A review of the literature. International Journal of Education and Development using Information and Communication Technology, 8(1), 136-155. Carstens, K. J., Mallon, J. M., Bataineh, M., & Al-Bataineh, A. (2021). Effects of technology on student learning. TOJET: The Turkish Online Journal of Educational Technology, 20(1), 105-113. Chen, R. J. (2010). Investigating models for preservice teachers’ use of technology to support student-centered learning. Computers and Education, 55(1), 32-42. Cohen, L., Manion, L., & Morrison, K. (2007). Research methods in education (6. ed.). New York: Routledge. Cradler, J., McNabb, M., Freeman, M., & Burchett, R. (2002). How does technology influence student learning? Learning and Leading with Technology, 29(8), 46-56. Davies, R. S. & West, R. E., (2014). Technology Integration in Schools. In J. M. Spector, M. D. Merrill, J. Elen, M. J. Bishop (Eds.) Handbook of Research on Educational Communications and Technology, (pp. 84-854). New York: Springer. Depta, M. (2015). Best Practices in Implementation of Technology Change in the K-12 Context. American Journal of Educational Research, 3(12B), 41-56. Ebuara, V. O. (2012). Information communication technology (ICT) as a factor in knowledge creation in Cross River Nigeria. Journal of Educational Review, 5(1), 43-49. Ertmer, P. (1999). Addressing first- and second-order barriers to change: Strategies for technology integration. Educational Technology Research and Development, 47(4), 47- 61. Ertmer, P. A. (2005). Teacher pedagogical beliefs: The final frontier in our quest for technology integration? Educational Technology Research and Development, 53, 25- 39. Ertmer, P. A. (2015). Technology integration. In J. M. Spector (Ed.), The SAGE encyclopedia of educational technology (pp. 748-751). Thousand Oaks, CA: Sage. Ertmer, P. A., & Ottenbreit-Leftwich, A. (2013) Removing obstacles to the pedagogical changes required by Jonassen's vision of authentic technology-enabled learning. Computers & Education,64, 175-182. Bozkuş 1574 Ertmer, P., Ottenbreit-Leftwich, A., &Tondeur, J. (2016). Teacher Beliefs and Uses of Technology to Support 21st Century Teaching and Learning. In H. Fives & M.G. Gill (Eds.) International Handbook of Research on Teachers’ Beliefs. Routledge. Estapa, A. & Nadolny, L. (2015). The effect of an augmented reality enhanced mathematics lesson on student achievement and motivation. Journal of STEM Education, 16(3), 40- 48. Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2012). How to design and evaluate research in education (8. ed.). New York: McGraw-Hill. Gilakjani, A. P. (2013). Factors Contributing to Teachers’ Use of Computer Technology in the Classroom. Universal Journal of Education Research, 1(3), 262-267. Goktas, Y., Yildirim, S., & Yildirim, Z. (2009). Main Barriers and Possible Enablers of ICTs Integration into Pre-service Teacher Education Programs. Educational Technology & Society, 12 (1), 193–204. Granger, C. A., Morbey, M. L., Lotherington, H., Owston, R. D., & Wideman, H. H. (2002). Factors contributing to teachers’ successful implementation of IT. Journal of Computer Assisted Learning, 18(4), 480-488. Guoyuan, S., Valcke, M., van Braak, J., Tondeur, J., & Zhu, C. (2011). Predicting ICT integration into classroom teaching in Chinese primary schools: Exploring the complex interplay of teacher-related variables. Journal of Computer Assisted Learning. 27. 160- 172. Harris, J., Mishra, P., & Koehler, M. (2009). Teachers’ technological pedagogical content knowledge and learning activity types: Curriculum-based technology integration reframed. Journal of Research on Technology in Education, 41(4), 393-416. Hennessy, S., Ruthven, K., & Brindley, S. (2005). Teacher perspectives on integrating ICT into subject teaching: Commitment, constraints, caution, and change. Journal of Curriculum Studies, 37(2), 155-192. Hermans, R., Tondeur, J., van Braak, J., & Valke, M. (2008). The impact of primary school teachers’ educational beliefs on the classroom use of computers. Computers & Education, 51, 1499-1509. Hew, K. F., & Brush, T. (2007). Integrating technology into K-12 teaching and learning: Current knowledge, gaps and recommendations for future research. Educational Technology Research and Development, 55(3), 223-252. Higgins, K., Huscroft-D’Angelo, J., & Crawford, L. (2019). Effects of technology in mathematics on achievement, motivation, and attitude: A meta-analysis. Journal of Educational Computing Research, 57(2), 283–319. Hooper, D., Coughlan, J., & Mullen, M. (2008). Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods, 6(1), 53- 60. Hope, W. C. (1997). Teachers, computer technology, and the change process. Clearing House, 70, 191-193. Hsu, S. (2010a). Developing a scale for teacher integration of information and communication technology in grades 1-9. Journal of Computer Assisted Learning, 26(3), 175-189. Hsu, S. (2010b). The relationship between teacher's technology integration ability and usage. Journal of Educational Computing Research, 43(3), 309-325. Inan, F. A., & Lowther, D. L. (2010). Factors affecting technology integration in K-12 classrooms: A path model. Educational Technology Research and Development, 58, 137-154. International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1560-1579. 1575 Kim, C., Kim, M. K., Lee, C., Spector, J. M., & DeMeester, K. (2013). Teacher beliefs and technology integration. Teaching and Teacher Education, 29, 76-85. Koehler, M. J. & Mishra, P. (2005). What happens when teachers design educational technology? The development of technological pedagogical content knowledge. Journal of Educational Computing Research, 32(2), 131-152. Lawson, T., & Comber, C. (1999). Superhighways technology: Personnel factors leading to successful integration of information and communications technology in schools and colleges. Journal of Information Technology for Teacher Education, 8(1), 41-54. Lee, H., Longhurst, M., & Campbell, T. (2017). Teacher learning in technology professional development and its impact on student achievement in science. International Journal of Science Education, 39(10), 1282-1303. Li, L., Worch, E., Zhou, Y., & Aguiton, R. (2015) How and Why Digital Generation Teachers Use Technology in the Classroom: An Explanatory Sequential Mixed Methods Study. International Journal for the Scholarship of Teaching and Learning, 9(2). Lim, C. P., & Khine, M. S. (2006). Managing teachers' barriers to ICT integration in Singapore schools. Journal of Technology and Teacher Education, 14(1), 97-125. Lim, C.P., Zhao, Y., Tondeur, J., Chai, C. S., & Tsai, C.C. (2013). Bridging the gap: Technology trends and use of technology in schools. Journal of Educational Technology & Society, 16(2), 59-68. Machin, S., McNally, S., & Silva, O. (2007). New technology in schools: Is there a payoff? The Economic Journal, 117(522), 1145-1167. McCain, T. (2005). Teaching for tomorrow: Teaching content and problem-solving skills. Thousand Oaks, CA: Corwin. Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017-1054. Muir-Herzig, R. G. (2004). Technology and its impact in the classroom. Computers & Education, 42, 111-131. National Center for Education Statistics. (2008). Digest of Education Statistics 2007 (No. NCES 2008–022). Washington, DC: U.S. Government Printing Office. Njiku, J., Maniraho, J. F. & Mutarutinya, V. (2019). Understanding teachers’ attitude towards computer technology integration in education: A review of literature. Education and Information Technologies, 24, 3041–3052. OECD. (2005). Are Students Ready for a Technology-Rich World? What PISA Studies Tell Us. Paris: OECD. Passey, D., Rogers, C., Machell, J., McHugh, G. & Allaway, D. (2003). The Motivational Effect of ICT on Pupils: emerging findings. London: DfES. Pegrum, M., Oakley, G., & Faulkner, R. (2013). Schools going mobile: A study of the adoption of mobile handheld technologies in Western Australian independent schools. Australasian Journal of Educational Technology, 29(1). Petko, D. (2012). Teachers’ pedagogical beliefs and their use of digital media in classrooms: Sharpening the focus of the ‘will, skill, tool’ model and integrating teachers’ constructivist orientations. Computers & Education, 58, 1351-1359. Petko, D., Prasse, D., & Cantieni, A. (2018). The interplay of school readiness and teacher readiness for educational technology integration: A structural equation model. Computers in the Schools, 35(1), 1-18. Plomp, T., Anderson, R. E., Law, N., & Quale, A. (2009). Cross-national information and communication technology: policies and practices in education. Charlotte, NC: Information Age Publishing. Bozkuş 1576 Redish, T. & Chan, T. C. (2007). Technology leadership: Aspiring administrators’ perceptions of their leadership preparation program. Electronic Journal for the Integration of Technology in Education, 6, 123-139. Roblyer, M. D. (2006). Integrating educational technology into teaching. Upper Saddle River: Pearson. Saal, P. E., Ryneveld, L. V., & Graham, M. A. (2021). Comparing the relationship between using educational technology in mathematics and student achievement in South Africa and Germany. Journal of Research on Technology in Education. Sang, G., Valcke, M., van Braak, J., Tondeur, J., & Zhu, C. (2011). Predicting ICT integration into classroom teaching in Chinese primary schools: Exploring the complex interplay of teacher-related variables. Journal of Computer Assisted Learning, 27, 160- 172. Sezer, B. (2017). The effectiveness of a technology-enhanced flipped science classroom. Journal of Educational Computing Research, 55(4), 471–494. Sivin-Kachala, J. & Bialo, E. R. (2000). Research report on the effectiveness of technology in schools. Washington, DC: Software and Information Industry Association. Simpson, A. (2010). Integrating technology with literacy: Using teacher-guided collaborative online learning to encourage Critical thinking. Research in Learning Technology, 18(2), 119–131. Sun, L. P., Siklander, P., & Ruokamo, H. (2018). How to trigger students’ interest in digital learning environments: A systematic literature review. Seminar.Net, 14(1), 62–84. Teo, T., & van Schaik, P. (2009). Understanding technology acceptance in pre-service teachers: A structural-equation modeling approach. The Asia-Pacific Education Researcher, 18(1), 47-66. Tondeur, J., Valcke, M., & van Braak, J. (2008). A multidimensional approach to determinants of computer use in primary education: Teacher and school characteristics. Journal of Computer Assisted Learning, 24(6), 494-506. Tosuntaş, Ş, Çubukçu, Z, & İnci, T. (2019). A Holistic view to barriers to technology integration in education. Turkish Online Journal of Qualitative Inquiry, 10(4), 439-461. Voogt, J. & Knezek, G. (2008). International handbook of information technology in primary and secondary education. New York: Springer USA. Wastiau, P., Blamire, R., Kearney, C., Quittre, V., Gaer, E., & Monseur, C. (2013). The Use of ICT in Education: A survey of schools in Europe. European Journal of Education, 48, 11-27. Weathersbee, J. C. (2008). Impact of Technology Integration on Academic Performance of Texas School Children. San Marcos: Applied Research Projects, Texas State University. Williams, D., Coles, L., Wilson, K., Richardson, A., & Tuson, J. (2000). Teachers and ICT: Current use and future needs. British Journal of Educational Technology, 31(4), 307- 320. Zhao, Yong, & Frank, K. A. (2003). Factors affecting technology uses in schools: An ecological perspective. American Educational Research Journal, 40(4), 807-840. International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1560-1579. 1577 Appendix 1. Matrix used to correlate factor means with PISA 2018 Scores Country Factor 1 Factor 2 PISA 2018 Scores N Mean SD SE N Mean SD SE Reading Math Science Albania 325 2.25 0.80 0.04 325 2.67 0.61 0.03 405.43 437.22 416.73 Australia 701 3.13 0.66 0.02 700 2.72 0.54 0.02 502.63 491.36 502.96 Austria 288 3.13 0.75 0.04 289 2.97 0.59 0.03 484.39 498.94 489.78 Baku (Azerbaijan) 130 2.52 0.60 0.05 131 2.69 0.44 0.04 389.39 419.64 397.65 Belarus 234 2.71 0.56 0.04 234 2.79 0.42 0.03 473.79 471.87 471.26 Belgium 276 2.91 0.72 0.04 276 2.64 0.50 0.03 492.86 508.07 498.77 Bosnia and Herzegovina 213 2.30 0.65 0.04 213 2.57 0.47 0.03 402.98 406.38 398.50 Brazil 561 1.90 0.77 0.03 562 2.25 0.62 0.03 412.87 383.57 403.62 Brunei Darussalam 55 2.54 0.70 0.09 55 2.48 0.55 0.07 408.07 430.11 430.98 B-S-J-Z (China) 361 3.30 0.58 0.03 361 3.25 0.53 0.03 555.24 591.39 590.45 Bulgaria 191 2.69 0.58 0.04 191 2.79 0.48 0.03 419.84 436.04 424.07 Canada 808 3.01 0.63 0.02 808 2.61 0.54 0.02 520.09 512.02 518.00 Chile 232 2.68 0.73 0.05 232 2.56 0.53 0.04 452.27 417.41 443.58 Chinese Taipei 187 3.10 0.61 0.04 188 2.99 0.52 0.04 502.60 531.14 515.75 Colombia 245 2.08 0.81 0.05 245 2.37 0.60 0.04 412.30 390.93 413.32 Costa Rica 205 2.23 0.79 0.05 205 2.26 0.65 0.05 426.50 402.33 415.62 Croatia 181 2.75 0.62 0.05 181 2.75 0.48 0.04 478.99 464.20 472.36 Czech Republic 325 2.74 0.54 0.03 325 2.69 0.42 0.02 490.22 499.47 496.79 Denmark 272 3.15 0.65 0.04 272 2.95 0.44 0.03 501.13 509.40 492.64 Dominican Republic 214 2.35 0.81 0.06 216 2.74 0.60 0.04 341.63 325.10 335.63 Estonia 230 2.93 0.57 0.04 230 2.68 0.43 0.03 523.02 523.41 530.11 Finland 213 2.73 0.57 0.04 213 2.60 0.39 0.03 520.08 507.30 521.88 France 214 2.78 0.65 0.04 214 2.62 0.46 0.03 492.61 495.41 492.98 Georgia 318 2.79 0.55 0.03 318 2.87 0.44 0.02 379.75 397.59 382.66 Germany 190 2.30 0.71 0.05 190 2.30 0.50 0.04 498.28 500.04 502.99 Greece 238 2.48 0.70 0.05 238 2.28 0.57 0.04 457.41 451.37 451.63 Hong Kong 113 2.86 0.54 0.05 113 2.52 0.48 0.05 524.28 551.15 516.69 Hungary 230 2.57 0.65 0.04 230 2.51 0.50 0.03 475.99 481.08 480.91 Iceland 130 2.94 0.65 0.06 130 2.69 0.45 0.04 473.97 495.19 475.02 Indonesia 339 2.89 0.73 0.04 341 2.82 0.69 0.04 370.97 378.67 396.07 Ireland 156 2.70 0.65 0.05 156 2.28 0.56 0.04 518.08 499.63 496.11 Israel 169 2.40 0.72 0.06 171 2.52 0.57 0.04 470.42 463.03 462.20 Italy 523 2.72 0.60 0.03 523 2.51 0.42 0.02 476.28 486.59 468.01 Japan 183 2.33 0.65 0.05 183 2.05 0.45 0.03 503.86 526.97 529.14 Jordan 313 2.37 0.75 0.04 311 2.41 0.62 0.04 419.06 399.76 429.25 Kazakhstan 616 2.65 0.72 0.03 616 2.93 0.54 0.02 386.91 423.15 397.10 Korea 187 2.97 0.63 0.05 188 2.53 0.52 0.04 514.05 525.93 519.01 Kosovo 203 1.87 0.65 0.05 203 2.57 0.49 0.03 353.07 365.88 364.88 Latvia 298 2.88 0.55 0.03 299 2.54 0.44 0.03 478.70 496.13 487.25 Lebanon 300 2.43 0.76 0.04 300 2.47 0.66 0.04 353.36 393.45 383.72 Lithuania 362 3.08 0.61 0.03 362 3.08 0.55 0.03 475.87 481.19 482.07 Luxembourg 43 3.00 0.54 0.08 42 2.62 0.53 0.08 469.99 483.42 476.77 Macao 45 2.90 0.61 0.09 45 2.93 0.52 0.08 525.12 557.67 543.59 Malaysia 191 2.31 0.62 0.04 191 2.56 0.46 0.03 414.98 440.21 437.62 Malta 49 2.96 0.57 0.08 49 2.69 0.56 0.08 448.23 471.72 456.59 Mexico 284 2.15 0.89 0.05 284 2.42 0.64 0.04 420.47 408.80 419.20 Moldova 236 2.39 0.52 0.03 236 2.60 0.38 0.02 423.99 420.60 428.49 Montenegro 61 2.52 0.61 0.08 61 2.95 0.51 0.07 421.06 429.61 415.17 Morocco 175 1.97 0.75 0.06 175 2.30 0.67 0.05 359.39 367.73 376.60 Netherlands 147 3.04 0.53 0.04 147 2.75 0.39 0.03 484.78 519.23 503.38 New Zealand 183 3.11 0.54 0.04 183 2.65 0.46 0.03 505.73 494.49 508.49 Bozkuş 1578 North Macedonia 104 2.32 0.78 0.08 104 2.70 0.54 0.05 392.67 394.45 413.04 Norway 243 3.06 0.69 0.04 243 2.92 0.44 0.03 499.45 500.96 490.41 Panama 191 2.14 0.81 0.06 192 2.49 0.64 0.05 376.97 352.85 364.62 Peru 339 2.05 0.77 0.04 338 2.29 0.59 0.03 400.51 399.84 404.22 Philippines 187 2.38 0.77 0.06 187 2.79 0.54 0.04 339.69 352.57 356.93 Poland 240 2.61 0.56 0.04 240 2.67 0.36 0.02 511.86 515.65 511.04 Portugal 274 2.33 0.67 0.04 274 2.37 0.46 0.03 491.80 492.49 491.68 Qatar 187 3.26 0.63 0.05 187 3.16 0.57 0.04 407.09 414.23 419.13 Romania 169 2.60 0.66 0.05 169 2.49 0.53 0.04 427.70 429.92 425.76 Russian Federation 247 2.74 0.61 0.04 248 2.86 0.46 0.03 478.50 487.79 477.72 Saudi Arabia 233 2.52 0.87 0.06 232 2.70 0.71 0.05 399.15 373.24 386.25 Serbia 185 2.51 0.66 0.05 185 2.68 0.53 0.04 439.47 448.28 439.87 Singapore 165 3.43 0.51 0.04 165 3.05 0.41 0.03 549.46 569.01 550.94 Slovak Republic 358 2.70 0.59 0.03 358 2.69 0.43 0.02 457.98 486.16 464.05 Slovenia 317 3.11 0.63 0.04 317 2.94 0.39 0.02 495.35 508.90 507.01 Spain 1055 2.56 0.69 0.02 1055 2.33 0.52 0.02 481.39 483.25 Sweden 218 3.33 0.62 0.04 218 3.04 0.53 0.04 505.79 502.39 499.44 Switzerland 226 3.06 0.68 0.05 226 2.80 0.57 0.04 483.93 515.31 495.28 Thailand 290 2.77 0.73 0.04 290 2.94 0.57 0.03 392.89 418.56 425.81 Turkey 186 3.02 0.74 0.05 186 2.91 0.59 0.04 465.63 453.51 468.30 Ukraine 250 2.29 0.59 0.04 250 2.75 0.43 0.03 465.95 453.12 468.99 United Arab Emirates 697 3.15 0.64 0.02 698 3.10 0.54 0.02 431.78 434.95 433.64 United Kingdom 386 2.68 0.64 0.03 387 2.61 0.51 0.03 503.93 501.77 504.67 United States 149 3.15 0.66 0.05 149 2.81 0.55 0.05 505.35 478.24 502.38 Uruguay 189 2.22 0.69 0.05 189 2.35 0.56 0.04 427.12 417.66 425.81 International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1560-1579. 1579 Appendix 2. Factor structure according to confirmatory factor analysis