Polat M, & Turhan, N.S. (2022). A meta-analysis study on the relationship between mathematical literacy and mathematics achievement in PISA tests. International Online Journal of Education and Teaching (IOJET), 9(2). 661-676. Received : 12.01.2022 Revised version received : 13.03.2022 Accepted : 19.03.2022 A META-ANALYSIS STUDY ON THE RELATIONSHIP BETWEEN MATHEMATICAL LITERACY AND MATHEMATICS ACHIEVEMENT IN PISA TESTS Research Article Murat Polat http://orcid.org/0000-0001-5851-2322 Anadolu University, Turkey mpolat@anadolu.edu.tr Nihan Sölpük Turhan https://orcid.org/0000-0002-8672-3853 Fatih Sultan Mehmet Foundation University, Turkey nsolpuk@fsm.edu.tr Biodata: Murat Polat received his Ph.D. from Osmangazi University, Graduate School of Educational Sciences. Currently, he is working as a language instructor at Anadolu University, School of Foreign Languages. His research interests include language assessment, IRT and educational Statistics Nihan Sölpük Turhan received her Ph.D. from Osmangazi University, Graduate School of Educational Sciences. Currently, she is working at Fatih Sultan Mehmet University Her research interests include assessment & evaluation and educational statistics. Copyright © 2014 by International Online Journal of Education and Teaching (IOJET). ISSN: 2148-225X. Material published and so copyrighted may not be published elsewhere without written permission of IOJET. http://orcid.org/0000-0001-5851-2322 mailto:mpolat@anadolu.edu.tr https://orcid.org/0000-0002-8672-3853 mailto:nsolpuk@fsm.edu.tr Polat & Turhan 662 A META-ANALYSIS STUDY ON THE RELATIONSHIP BETWEEN MATHEMATICAL LITERACY AND MATHEMATICS ACHIEVEMENT IN PISA TESTS Murat Polat mpolat@anadolu.redu.tr Nihan Sölpük Turhan nsolpuk@fsm.edu.tr Abstract This research is an extensive meta-analysis investigating the link between mathematical literacy and mathematics achievement which are tested mainly in PISA exams. With this aim in mind, first, the studies related to the research objective between the years 2003-2021 were examined. In the second stage, eight studies which met the research criteria and 13 quantitative data categories that were common in those studies were determined. In the end of the process, the data set of this meta-analysis study consisted data obtained from 34146 participants in total. Based on the findings driven from the study, a heterogeneous data structure was observed; thus, the Random Effect model was used in the analyses and it was summoned that the link between mathematical literacy and mathematics achievement had a moderately significant effect on the effect size value (Cohen’s d value: 0.66). The analysis of the subgroups revealed that there was no noteworthy difference between the effect levels considering the independent variables including the publication type, sample group and publication year. Finally, the results of this meta-analysis revealed that there was a positive and moderate relationship among students’ mathematical literacy and mathematics achievement measured in PISA tests. Keywords: PISA tests, mathematical literacy, mathematics achievement, meta-analysis 1. Introduction Accurate measurement of mathematical literacy and mathematics achievement are among the main challenges of mathematics education, and the ability tests related to this education are important factors defining the reliability of this measurement. The path and power of the relationship between these two concepts can also affect mathematics education and its further steps. For this reason, large-scale international assessment of nations' mathematical achievement and literacy are made systematically and periodically. The organization that carries out the most comprehensive international evaluation in this field is the Program for International Student Assessment (PISA) which is affiliated with the Organization for Economic Cooperation and Development (OECD). The OECD announced that the most comprehensive educational research in terms of international large-scale measurement is the PISA exam (OECD, 2010). In this respect, the PISA test makes statistical comparisons by obtaining data on different fields from students aged 15, parents and teachers (OECD, 2010, 2013). In the evaluations made within the scope of this program, analyses and comparisons are made regarding the level of mathematics achievement and literacy. Evaluations within the scope of this program started in 2000 and are carried out every three years. Within the scope of this program, every three years either science, maths or reading is determined as the basic mailto:mpolat@anadolu.redu.tr mailto:nsolpuk@fsm.edu.tr International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 661-676. 663 subject in testing, and this subject is paid more attention than the others in the PISA tests (OECD, 2007, 2010). In this way, large-scale exams with a high reliability and content validity are conducted by concentrating on diverse educational goals (Taş et al., 2016). In time, the importance of mathematics achievement and mathematical literacy have been recognized more vividly in local education systems as a result of standardized international exams and evaluations; thus, the prominence of mathematical literacy, which means the capacity to put mathematical awareness into practice in daily life, is naturally increasing (Freudenthal, 1991). The objectives and scope of the mathematics course have been updated for the development and advancement of mathematical literacy in the education systems of many countries (Gravemeijer &Terwel, 2000). Doing this, the main goal is to train students who can solve the problems met in daily life by using the mathematics knowledge. According to Breakspear (2012), the biggest contribution to the intensification in the importance of mathematical literacy is made by the Program of International Student Assessment (PISA), testing the individual’s mathematical literacy. In this context, the evaluations made within the scope of the PISA program can directly affect and shape the mathematics education policies of many countries (Breakspear, 2012). 1.1. Literature Review OECD (2016) reports emphasized the fact that student success is an important issue for the educational planning of countries and necessary to investigate which components in this process are highly significant for teaching specific skills more effectively. For this reason, considering the role of classroom teacher in student success, it is inevitable to conduct research on a particular education system, teacher training circle and teaching competencies, and check how students can learn better (Polat et al., 2022). Klieme et al. (2009) grouped teaching quality into three main categories in the theoretical model concerning the effect of education on learning, considering the dimension of teaching quality (Polat, 2022). These categories are cognitive activation, classroom management and supportive classroom climate, respectively. Thus, the mediators observed in the quality of teaching can be described as the affective characteristics of students, and the effects can be the knowledge obtained from lectures and learners’ understanding the subject (Polat, 2020). Precisely for this reason, the PISA program evaluation is built on mental stimulation, learner focused training, mirroring, and training perceptions (OECD, 2013). In this context, it has been emphasized by many researchers that cognitive activation, classroom climate and classroom management are basic determinants of teaching quality in maths classes (Chen et al., 2011; Frenzel et al., 2010; Van den Heuvel- Panhuizen, 2001). 1.1.1. Mathematical Literacy According to the PISA program, the level of mathematical literacy is a student’s mastery of maths in elucidating the difficulties they come across in their daily routines, and the success of using their mathematical perception and comprehension capacity to meet their daily needs (Di Martino & Zan, 2011). Also, according to the PISA program, mathematical literacy is described as a student’s ability to convey, use and deduce mathematics in various settings (OECD, 2013). In the change and development process of the concept of mathematical literacy in the last century, this skill has been especially emphasized and has become a research topic with the findings of various international exams (PISA, TIMSS) (Toraman et al., 2022). Mathematical literacy helps the individual to be aware of the role that maths plays in the contemporary realm and understanding, making applications related to daily life, interpreting Polat & Turhan 664 in numerical and spatial thinking. It can be said that it provides critical analysis and problem- solving skills in real life conditions (Özgen & Bindak, 2008). The term, mathematical literacy is the capacity to convert a daily problem into a mathematical equation, the mathematical coding or decoding ability to unravel difficulties and structured reasoning. It allows individuals to see and use relationships (Tekin & Tekin, 2004). In other words, mathematical literacy is necessary in students' adaptation to new technologies, defining mathematical problems, solving them and establishing communication tools using various representations (Çolak, 2006). Such an ability expands after gaining mathematical literacy which includes using mathematical concepts, tools, operations, steps, instruments, and stages to reason at the mathematical level, describe, explain, and predict phenomena (OECD, 2016). McCrone and Dossey (2007) report mathematical literacy as the ability to grasp the function of maths and to acquire skills where and when necessary. There are many studies on testing and conceptualising mathematical literacy in the literature. Among them, Saenz (2009) listed the mathematical literacy cyphers as circumstantial, technical and theoretical while investigating Spanish teacher candidates’ problems while-teaching maths. In their study, which aimed to define mathematical literacy skills of university students, Güneş and Gökçek (2010) found a substantial connection between pre-service teachers' departments and their mathematical literacy levels, which indicates that there is a significant relationship between those variables. Again, in the study conducted by Tekin and Tekin (2004) for the same purpose, mathematics pre-service teachers showed the best performance in questions about mathematical processes and it was also observed that they showed the worst performance in questions about the history of mathematics. Since 2000, when PISA studies began, many studies have been conducted on mathematical literacy. Some of these studies compare countries with each other by examining many variables. In line with this information obtained, it is aimed to find out which of these variables have effects on mathematical literacy (Yenilmez & Ata, 2013). Generally, the country in focus is compared with the country or countries with a high level of mathematical literacy. Likewise, another international exam related to mathematical literacy is the Trends in International Mathematics and Science Study (TIMSS) application. The TIMSS application is an educational screening study conducted every 5 years to test the ability of pupils in different countries in subjects like mathematics and science (Foy et al., 2008). 1.1.2. Mathematical Achievement Basically, mathematics is a language that uses some symbols, a system that develops logical thinking in humans, an aid we use to understand the world and improve the environment we live in. Therefore, it can be said that mathematics is a system which is interwoven with our daily life. This makes mathematics an abstract notion. However, no matter how abstract it is, acquired mathematical knowledge will one day find an application area for anyone (Baloğlu, 2010) even for those who claim their academic weakness or failure in maths. Today’s digital world requires more or less practical mathematics knowledge and particularly reasoning and problem-solving skills thinking in almost every profession, including academic life as well (Toraman & Korkmaz, 2022). Furthermore, employers expect their staff to solve the routine problems as fast as they can besides the complex ones which have never been encountered before. This desire creates the need to find solutions to the problem through reasoning rather than some detached mathematical skills. Therefore, the new understanding of mathematics education puts learning by internalizing mathematics instead of learning just a pre-determined set of codes and formulas (Olkun & Toluk, 2003). Similarly, it is known that majority of International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 661-676. 665 Turkish people perceive mathematics as "a lesson about numbers and calculations seen at school, using unique signs and symbols". However, mathematics has already taken its place in our daily lives in shopping, checking the clocks or calculations, considering the financial concepts and in the functioning of our general reasoning skills (Turgut, 2021). Because of this importance, achievements related to mathematics are observed at every age and in every field, even academically from pre-school education schools to higher education programs or socially (Baykul, 2003). Therefore, mathematics achievement is critical for the individual’s career and has a vital role in gaining the 21st century skills. In many studies, considering the ones related to PISA tests, it is agreed that mathematics achievement is divided into four subgroups: numbers, algebra, geometry, data, and probability (Le, 2006). Most of the math test questions are based on students' use of these subgroups when solving problems, beyond making basic definitions and simple calculations. In terms of mental processes, questions in standard mathematics tests cover knowledge, using conventional mathematical skills, using complex mathematical formulas, solving problems, and applying for a great number of reasoning skills (Wood, 2007). In practice, besides the mathematics test achievement, data about the attitudes of the students towards the mathematics lesson and learning, the number of educational resources in their homes, how much time they spend on the homework for the mathematics lesson, and the education level of the teachers who teach maths are obtained in order to be able to reason the success of the countries (Ross, 2008). In one of these studies, Baki (2013) explains the failure of Turkish students in maths with the traditional teaching approach adopted by most of the Turkish teachers. In other words, since mathematics is taught with this traditional understanding, it is seen as a skill that is disconnected from our daily needs, a fixed structure which could never change, a set which consists of definite rules and equations that must be learned separately (Pesen, 2003). Consequently, maths, which is taught in this context, is considered a tough, unpleasant and difficult course which is hard to get through and achieve by the Turkish learners especially for the ones who lack supplementary education, resources and training. 2. Methodology 2.1. Research Objective It is noteworthy that there are many national and international studies examining the effects of mathematical literacy, which is tried to be measured in the PISA exams, on mathematics achievement (Breakspear, 2012; Güneş & Gökçek, 2010; Ma & Kishor, 1997; Turgut, 2021; Wang, 2007). Therefore, it is intended to compile the link between mathematical literacy and mathematics achievement through a meta-analysis study by bringing together the findings of research articles and theses circulated on this topic. 2.2. Research Design This study aimed to make inferences by using meta-analysis method on studies examining the relationship between mathematical literacy and mathematics achievement. For this purpose, a limited subject was determined in the research, and it was aimed to bring together the studies on this subject and to produce a generalizable result (Littel, et al., 2008). Therefore, while conducting meta-analysis, effect sizes and values of the data acquired from the studies conducted with quantitative data on a subject are calculated and analyses are made in Polat & Turhan 666 accordance with the purpose of the research (Cohen et al., 2002). As a result, a numerical value is reached by using statistical methods with quantitative findings created by using certain criteria on a determined subject in the meta-analysis study. Accordingly, such studies are broad-spectrum approaches that enable to reveal findings related to larger sample groups (Cooper et al., 2009). 2.3. Literature Scanning and Inclusion/Exclusion Criteria In order to determine the studies for meta-analysis, an extensive literature review was made in Google Academic, ULAKBIM National Database, YOK National Academic Thesis Centre, Dergipark, EBSCO, Science Direct, ProQuest, ERIC databases. While conducting this review in those databases, necessary survey studies were carried out by the researchers based on mathematical literacy and mathematics achievement. The deadline for the data collection process was set as November 2021. The studies obtained by the scanning process were determined as reliable and valid studies in peer-reviewed journals or master's and doctoral theses. First of all, the studies focusing on the basic concepts included in the current research were determined, and then the studies with quantitative values suitable for the meta-analysis pattern were selected. Accordingly, thirteen different quantitative data findings included in the studies reached were counted in the analysis. The followings were used as the criteria for the inclusion of a particular study in the analyses, the research should be conducted before November 2021, presented the statistical values necessary for the calculation of the effect size value, the sample size and correlation coefficient values all of which were clear and quantitative, and finally, the full text of the relevant scientific sources could be accessed. On the other hand, the reasons why other studies on the same issue were not included in the research were as follows: not specifying the quantitative data necessary for calculating the effect size value, not performing the correlation analysis, and/or not obtaining the full text of the scientific publication. Table 1. Publication data of the studies included in the meta-analysis Publication year 2013-2017 2018-2019 Total n 7 6 13 % 0,54 0,46 100 Publication Type Article Thesis n 4 9 13 % 0,31 0,69 100 Sample Student Academic n 11 2 13 % 0,85 0,15 100 2.4. Coding The data gained from the coding process from meta-analysis studies should be organized by bringing the relevant pieces of data together before the analysis. The descriptive features of the coding form developed for the research are presented in the Table 2. International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 661-676. 667 Table 2. Descriptive features of the studies included in the meta-analysis Author(s) Year Type Sample n Coşkuner, 2013 2014 Thesis Student 4996 Dibek, Yavuz, Demirtaşlı, Yalçın, 2017 2017 Article Student 6928 Ilhan, Tutak, Celik, 2019 2019 Article Academic 232 İlhan, 2019 2019 Thesis Academic 384 Kukey, Tutak, T, Aydoğdu, 2019 2019 Article Student 334 Efe, 2019 2019 Thesis Student 214 Rivera, 2018 2018 Thesis Student 400 Önal, Yorulmaz, Gökbulut, Altıner, 2017 2017 Article Student 274 In this descriptive chart, the coding was done separately from the components of the research bibliography, publication year, type of research (thesis, article), sample information (student and academic) and quantitative values (sample size, correlation coefficients etc.). Eight different studies were included in the study in which different correlation coefficient values were given on the research subject. For this reason, in this meta-analysis, analyses were made on 13 quantitative value data in total. Among these studies, five datasets of Coskuner (2013) and two datasets from Rivera (2018) were used. Furthermore, in order to ensure validity and reliability in meta-analysis studies, three other researchers who are experts in the field are asked to check the coding process (Açıkel, 2009). Therefore, in this study, the dataset created in data coding process was controlled and approved by 3 different researchers, one professor and two associate professors working in the field. After the expert opinion was taken, some differences were determined, and after the necessary arrangements were made, a complete consensus was achieved, and the data coding process was completed. 2.5. Data Analysis In the meta-analysis model, it is essential to reach generalizable results over the effect size value on quantitative values about a subject limited by the researcher (Littell et al., 2008). According to this model, the effect size value is a fixed quantity used to define the power and path of the relationship in research (Borenstein et al., 2009). Moreover, meta-analysis studies consist of three groups which can be listed as studies focused on standardized mean difference, correlation and risk ratio (odd ratio/risk ratio) (Littell et al., 2008). In addition, one of two different impact models is considered as the basis of the meta-analysis method (Lipsey & Wilson, 2001). These are called fixed effects and random effects model. The reason for choosing an effect model is that the studies suitable for the research are not functionally equal and/or it is aimed to generalize from the determined effect size (Borenstein et al., 2009). In this study, analyses were made using the random effects model. Finally, the statistical package programs used in this meta-analysis study are Microsoft Excel and Comprehensive Meta-Analysis. Finally, the subgroups that could be effective in this research were determined as publication type (thesis, article), publication year (2013-2017,2018-2021) and sample type (student, academic). In this study, analyses were made on Q values and it was investigated whether there was a difference between the subgroups according to the Q statistic. Polat & Turhan 668 2.6. Publication Bias In this research, publication bias is one of the essential issues directing the analysis results. Different statistical values related to publication bias can be calculated with appropriate statistical programs for Fisher’s Z analysis (Ellis, 2010). The funnel plot method was preferred to test publication bias, and the effect size funnel obtained according to the research data on publication bias was presented in Figure 1. Figure 1. Effect size funnel on publication bias According to the distribution observed in Figure 1, it can be stated that the studies are distributed symmetrically in the graphical display of the publication bias analysis. Accordingly, the black circles in the funnel plot are related studies. The black tiles at the bottom show the overall effect of the original study at the 95% confidence interval, and the white ones show the overall effect obtained by adjusting for publication bias. According to this finding, a symmetrical distribution was determined for the data in the study, and this was accepted as evidence that there was no publication bias in the study. According to the literature, it is not sufficient to explain the publication bias based on the funnel plot. This is because this finding stems from subjective studies (Sterne & Harbord, 2004; Terrin et al., 2005). For this reason, it is necessary to reveal the finding on this subject with different statistical tests. In this context, different analyses were made for publication bias in the study. Duval, Tweedie's trim and fill and Egger test results were presented in Table 3. Table 3. Duval Tweedie's trim and fill and Egger test results Studies trimmed Point estimate Lower limit Upper limit Q p Observed values 0 .72 .71 .72 2374.97 Adjusted values 0 .72 .71 .72 2374.97 Egger Test (2-tailed) .57 International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 661-676. 669 The analysis displayed in Table 3 proves that the number of studies to be included to the data set to make the funnel plot be symmetrical according to Duval, Tweedie's trim and fill was 0, and it was seen that there was no need to include any studies in the research to correct the asymmetrical state of the funnel plot. According to Egger's regression test finding, p=0.59 (p>.05) value emerged in the study, and it was concluded that there was no publication bias, which means that we have a good data set for meta-analysis (Wang et al., 2007). In order to complete the data set to be used in this research, the included and excluded articles were organized by the researchers, and quantitative data and analysis results of the selected articles were compared. The limits to be regarded for the interpretation of the standardized mean difference effect size were determined. These were defined as insignificant if Cohen d ≤ .20, small effect if .20 ≤ Cohen d ≤ .50, moderate effect if .50 ≤ Cohen d ≤ .80, and large effect if Cohen d ≥ .80 (Cohen, 1988). All the findings obtained from the meta- analysis were compared according to this effect size value. In addition, test statistics of heterogeneity were examined, and the limits of the findings were determined according to the I2 value in addition to the Q statistic. According to these limits: I2 < 25 means low heterogeneity, 25 < I2 < 50 level means moderate heterogeneity, 75 < I2 and above means high heterogeneity (Cooper et al., 2009). Finally, since there were sub-group variables in the study, the sub-group analysis was done via Analog ANOVA. The main purpose of analogue ANOVA is to analyse whether the effect size changes in subgroups in meta-analyses, and since it contains multiple comparisons, it has similarities to the classical ANOVA test because this test is a subgroup analysis based on Chi-square test (Lipsey & Wilson, 2001). In the studies using meta-analysis method, the reliability and validity of the studies included in the analysis are determined in the first step of the study. It is emphasized as a prerequisite for meta-analysis that the reliability and validity level of the studies included in the analysis are specified in the study (Petitti, 2000). In sum, reliability of the data collection tools involved in this research met the desired reliability criteria. 3. Findings The findings of this study, regarding the effect size of the relationship between mathematical literacy and mathematics achievement, were obtained through the meta-analysis including the related research results carried out between 2013 and 2021. This research consisted of a total of eight studies determined according to the previously mentioned criteria, and a total of 13762 participants formed the sample of the meta-analysis. Table 4. Mathematical literacy & achievement effect size and heterogeneity test n Effect size z CI (Reliability) df Q p 𝐼2 SE Lower Limit Upper Limit Fixed .72 166.461 .71 .72 12 2374.97 .00 99.49 Random 8 .66 9.802 .05 .56 .73 According to the data obtained from the studies included in the meta-analysis, it was revealed that the effect size was 0.72 in the fixed effects model and 0.66 in the random effects model. In this case, since there is a difference in effect sizes, it is recommended to perform homogeneity/heterogeneity test on the data set in order to explore which of the two different Polat & Turhan 670 effect sizes better reveals the relationship between mathematical literacy and mathematics achievement (Borenstein et al., 2009). Accordingly, as can be seen from Table 4, it was concluded that the data set had a heterogeneous structure (Q= 2374.97, p= .00, I2= 99.49). In addition, the I2 statistic, which was developed as a supplementary analysis to the Q statistic, helps to obtain clearer results regarding heterogeneity (Petticrew & Roberts, 2006). The obtained I2= 99.49 statistical value revealed a high level of heterogeneity in the data set and supports the findings in the relevant literature (Cooper et al., 2009). Based on this result, it was deemed appropriate to consider the effect size value in the study according to the random effects model. Figure 2. Meta-analysis findings According to the results of the analysis in Figure 2, it can be seen that there is a positive and moderate relationship between the mathematical literacy and mathematics achievement of the participant students measured in the PISA tests. In addition to the analysis of determining the common effect value, a subgroup analysis was performed to determine the source of the heterogeneity in the findings, and the relevant findings were presented in Table 5. Table 5. Random effects model subgroup analysis Variable n Effect size SE CI (Reliability) df Qb 𝑝 Lower Limit Upper Limit Publication Article 4 .73 .27 .38 .89 3 493.31 Thesis 9 .63 .02 .55 .69 8 628.88 Total 13 .63 .05 .71 .72 1 .44 .50 International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 661-676. 671 For this analysis, the Analog ANOVA test was preferred, which shows whether the relationship between mathematical literacy and mathematics achievement differs considering the subgroups considering the random-effects model. In the research, subgroup-analysis on publication type (thesis and article) and sample group (student, educator) was made. The findings were presented by considering the Qb (Q-between) values in the homogeneity test in the analysis of subgroups. Thinking of the results of the subgroup-test, it was revealed that the relationship between mathematical literacy and mathematics achievement did not change significantly considering the type of publication (p>.05). Table 6. Subgroup analysis in the random effects model n Effect size SE CI (Reliability) df Qb 𝑝 Variable Lower Limit Upper Limit Sample Academic 2 .66 1.07 -.38 .97 Student 11 .65 .05 .55 .74 Total 13 1 .00 .97 In the analysis made for the sample group according to the results of the subgroup, the link between mathematics achievement and mathematical literacy was not significant (p=0.97), and compared to the sample group (either student or academic), it was found that there was no significant difference (p>.05). Table 7. Subgroup analysis in random effects model According to the results of the subgroup-analysis presented in Table 7, the relationship between mathematical literacy and mathematics achievement in the test performed for the sample group was p=0.34. Thus, according to the publication year, it was found that there was no noteworthy difference among mathematical literacy and mathematics achievement scores considering the publication year of the research papers (p>.05). n Effect size SE CI (Reliability) df Q 𝑝 Variable Lower Limit Upper Limit Publ. Year 2013-2017 7 .71 .04 .60 .79 2018-2022 6 .58 .19 .22 .80 Total 13 .69 .05 .59 .77 1 .91 .34 Polat & Turhan 672 4. Discussion & Conclusion In this study, quantitative research focusing on the relationship between mathematics literacy and mathematics achievement measured in PISA exams between 2013 and 2021 was examined, and a meta-analysis study was conducted on the data set consisting of thirteen quantitative variables that were determined in these studies. According to the statistical analysis made on the relationship between mathematical literacy and mathematics achievement in the PISA tests, it was revealed that the available data set had a heterogeneous structure (Q= 2374.97, p= .00, I2= 99.49). Considering this finding, the model chosen for the meta-analysis was determined as Random Effect Model because it was stated by different researchers that the fixed effects model was suitable for homogeneously distributed data sets (Ellis, 2010; Littell et al., 2008). In addition, bearing in mind the results of the statistical analysis regarding the effect size, it was concluded that the effect size of the data set was d=0.66, and it was determined that this value indicates a moderate relationship. Cohen (1988) stated in his study that values between 0.5 and 0.8 showed a significant, positive and moderate relationship between two variables. Wang et al. (2007) underlined the role of effect size in their study and reported that their research findings driven from fixed and random effects designs were not descriptive at all. Their research findings based on the fixed effects model revealed that the application type didn’t have any significant impact on learners’ maths scores. In addition, in the subgroup analysis of the variables (moderator), there was no statistically significant difference between the effect levels according to the type of publication type, sample group and publication year. Furthermore, when meta-analysis studies on the link between mathematics achievement and mathematical literacy are investigated, it can be found that these studies have been particularly conducted on the relationship of these variables with the same dependent variables in both small scale and big scale studies. Tabuk’s (2019) research, in which 30 data sets of 8292 participants were used, mathematics achievement and students’ attitudes towards mathematics, reported a positive and weak effect size (0.278) was obtained according to the Random Effects Model. A big scale meta-analysis study on the same variables was conducted by Ma and Kishor (1997), and it was seen that there was an important and strong relationship amid pupils’ attitudes towards mathematical literacy and their mathematics achievement. In addition to these studies, especially when the results obtained from the research on mathematical literacy and mathematics achievement are investigated, another critical issue is to define and classify the skills embodied with mathematical literacy. Saenz (2009) classified mathematical literacy demands as contextual, procedural and conceptual, in a study which investigates the category Spanish teacher candidates had found the most challenging while teaching. In this study, contextual knowledge was found to be more difficult than other demands, and it was also reported that the measurement of the student's ability to reproduce, relate and reflect, which was emphasized in the PISA tests, and the low success rates observed were an indicator of the complexity of the questions in the test. This finding may reveal that the mathematical literacy and mathematical skills revealed in this study are actually cognitive skills that are intertwined with each other, and that the low achievement, especially in the skills of making connections and reflecting, may actually be directly related to mathematical literacy and skills. International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 661-676. 673 Moreover, İlbağı (2012) also reported that using open access PISA questions in his study, different school types and participant independent variable created significant differences in achievement observed in studies based on the PISA test results. This finding actually contradicts with the conclusion that the participant variable obtained from the study did not make a significant difference. Such a difference can also be seen when the results of the study by Akyüz and Pala (2010) are examined. According to these researchers, family and classroom atmosphere in Turkey, Greece and Finland create significant differences, and this finding was demonstrated by factor analysis examining the strengths of variables affecting mathematical literacy. Another meta-analysis study on the causes of student success in the Pisa tests was conducted on the role of homework in students' academic achievement. It was revealed that pupils who received more assistance from their family members on doing the homework had poorer academic success and the effect was calculated as d = 0.23, 95% CI [0.21, 0.25]. When analysed by country, it was concluded that mathematical literacy had a better impact on math scores in Europe compared to Asia, but insignificant differences were observed regarding other independent variables (Fernández, et al. 2022). To conclude, studies focusing on the effects of mathematical literacy on mathematics achievement have gained more importance in the field of mathematics education and especially after the evaluations of international PISA tests. However, the limited number of studies on this subject, especially the results of the PISA test, leads to the emergence of very few scientific publications in the context of examining different skills in the field of mathematics education and the relationship between these skills; therefore, most often insufficient and weak estimations are made. The importance of academic literacy and especially mathematical literacy, which is a critical and important skill, examining different variables, conducting new and multidimensional research, and making more meta-analysis especially on cognitive domain skills, examining more variables in mathematics education and identifying meaningful methodological correlations and relationships will eventually help mathematics teachers. It will enable them to analyse, discuss, test and theorize those findings (Marzano & Kendall, 2007). In addition, performing broad-based meta-analyses, including such research topics, will enable researchers to reach more reliable results with statistical comparisons on larger sample groups in countries with abundance in publications but difficulties in inferencing, theory development and practice, and doing this, more in-depth data and analysis in terms of the effectiveness of education can be provided. The importance of the findings and results obtained from this meta-analysis study are noteworthy since its results will contribute to revealing the importance of mathematics literacy, especially in mathematics education, and how important mathematics literacy is while teaching maths in a classroom experiencing the hands-on practice. Polat & Turhan 674 References Açıkel, C. (2009). Meta analiz ve tıptaki yeri. Klinik Psikofarmoloji Bülteni, 19(2), 164-172. Akyüz, G., & Pala, N. M. (2010). PISA 2003 sonuçlarına göre öğrenci ve sınıf özelliklerinin matematik okuryazarlığına ve problem çözmeye etkisi. İlköğretim Online, 9(2), 668-678. Baloğlu, M. (2010). An Investigation of the Validity and Reliability of the Adapted Mathematics Anxiety Rating Scale-Short Version (mars-sv) among Turkish Students. European Journal of Psychological Education, 25, 507-518. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to Meta-Analysis. UK: Wiley. Breakspear, S. (2012). The Policy Impact of PISA: An Exploration of the Normative Effects of International Benchmarking in School System Performance. OECD Education Working Papers, No. 71, OECD Publishing, Paris. https://doi.org/10.1787/5k9fdfqffr28- en. Chen, H.Y., Thompson, M. S., Kromrey, J. D., & Chang, G. H. (2011). Relations of student perceptions of teacher oral feedback with teacher expectancies and student self-concept. The Journal of Experimental Education, 79(4), 452-477. Cohen, J. (1988). Statistical power analysis for the behavioural sciences. Hillsdale, Erlbaum. Cohen, L., Manion, L., & Morrison, K. (2002). Research methods in education. Routledge. Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage Publication. Çolak, S. K. (2006). Materyal kullanımının altıncı sınıf öğrencilerinin geometri kavramları bağlamında matematiksel okuryazarlığına etkisi üzerine deneysel bir çalışma. Unpublished Master’s Thesis, Gazi University, Ankara. Di Martino, P., & Zan, R. (2011). Attitude towards mathematics: A bridge between beliefs and emotions. ZDM Mathematics Education, 43(4), 471-482. https://doi.org/10.1007/s11858-011-0309-6 Ellis, P. D. (2010). The essential guide to effect sizes. Statistical power, meta-analysis, and the interpretation of research result. New York: Cambridge University Press. Fernández-Alonso, R., Álvarez-Díaz, M., García-Crespo, F. J., Woitschach, P., & Muñiz, J. (2022). Should we Help our Children with Homework? A Meta-Analysis Using PISA Data. Psicothema, 34(1), 56-65. Frenzel, A. C., Goetz, T., Pekrun, R., & Watt, H. M. G. (2010). Development of mathematics interest in adolescence: Influences of gender, family, and school context. Journal of Research on Adolescence, 20(2), 507-537. Freudenthal, H. (1991). Revisiting mathematics education; China lectures. The Netherlands, Dordrecht: Kluwer Academic. Foy, P., Martin, M. O., & Mullis, I. V. (2008). TIMSS 2007 international technical report. https://timss.bc.edu/TIMSS2007/PDF/TIMSS2007_InternationalMathematicsReport.pd f Gravemeijer, K., & Terwel, J. (2000). Hans Freudenthal: A mathematician on didactics and curriculum theory. Journal of Curriculum Studies, 32(6), 777- 796. https://doi.org/10.1080/00220270050167170 Güneş, G., & Gökçek, T. (2010). Öğretmen adaylarının matematik okuryazarlık düzeyleri. IX. Ulusal Fen Bilimleri ve Matematik Eğitimi Kongresi, Bildiri Özetleri, 144. İlbağı, E. A. (2012). PISA 2003 matematik okuryazarlığı soruları bağlamında 15 yaş grubu öğrencilerinin matematik okuryazarlığı ve tutumlarının incelenmesi (Unpublished doctoral dissertation). Atatürk University, Erzurum. Klieme, E., Pauli, C., & Reusser, K. (2009). The Pythagoras study: Investigating effects of teaching and learning in Swiss and German mathematics classrooms. T. Janik ve T. https://doi.org/10.1787/5k9fdfqffr28-en https://doi.org/10.1787/5k9fdfqffr28-en https://doi.org/10.1007/s11858-011-0309-6 https://timss.bc.edu/TIMSS2007/PDF/TIMSS2007_InternationalMathematicsReport.pdf https://timss.bc.edu/TIMSS2007/PDF/TIMSS2007_InternationalMathematicsReport.pdf https://doi.org/10.1080/00220270050167170 International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 661-676. 675 Seidel (Ed.), The power of video studies in investigating teaching in the classroom (s. 137-160). Münster: Waxmann. Le, T. A. (2006). Applying realistic mathematics education in Vietnam: Teaching middle school geometry (Doctoral dissertation). Potsdam: Mathematisch- Naturwissenschaftlichen Fakultät der Universität Potsdam. Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage. Littell, J. H., Corcoran, J., & Pillai, V. (2008). Systematic reviews and meta-analysis. OUP. Ma, X. & Kishor, N. (1997). Assessing the relationship between attitude towards mathematics and achievement: A meta-analysis. Journal for Research in Mathematics Edu, 28(1), 26- 47. Marzano, R. J., & J.S. Kendall (2007). The New Taxonomy of Educational Objectives. Corwin Press, Thousand Oaks, CA. McCrone, S. S., & Dossey, J. A. (2007). Mathematical literacy - it’s become fundamental. Principal Leadership, 7(5), 32-37. OECD (2007). PISA 2006: Science competencies for tomorrow’s World. Paris: OECD Pub. OECD (2010). PISA 2009 Results: What students know and can do: Student performance in reading, mathematics and science. Paris: OECD Publishing. OECD (2013). PISA 2012 Results: What students know and can do: Student performance in mathematics, reading and science. Paris: OECD Publishing. OECD, (2016). PISA 2015 assessment and analytical framework. Science, reading, mathematic and financial literacy. Paris: OECD Publishing. Olkun, S., & Toluk, Z. (2003). Activity-based mathematics teaching in primary education. Ankara: Anı Publishing. Özgen, K. & Bindak, R. (2008). Matematik Okuryazarlığı Özyeterlik Ölçeğinin Geliştirilmesi, Kastamonu Eğitim Dergisi, 16(2), 517-528. Pesen, C. (2003). Mathematics Teaching. Ankara: Nobel Publishing. Petitti, D. B. (2000). Meta-analysis, decision analysis, and cost-effectiveness analysis (2nd ed.). New York: Oxford University Press. Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences.US, Blackwell. Polat, M. (2020). Analysis of multiple-choice versus open-ended questions in language tests according to different cognitive domain levels. Novitas-ROYAL (Research on Youth and Lang.), 14(2), 76 96. Polat, M. (2022). Comparison of performance measures obtained from foreign language tests according to item response theory vs classical test theory. International Online Journal of Education and Teaching (IOJET), 9(1). 471-485. Available from: https://www.researchgate.net/publication/357529426 Polat, M., Toraman, Ç. & Turhan, N.S. (2022). Reliability analysis of PISA 2018 reading literacy student questionnaire based on Item Response Theory (IRT): Turkey sample. International Journal of Curriculum and Instruction, 14(1), 1004–1028. Ross, S. (2008). Motivation correlates of academic achievement: Exploring how motivation influences academic achievement in the PISA 2003 dataset. (Unpublished Doctoral Thesis). http://hdl.handle.net/1828/3209 Saenz, C. (2009). The role of contextual, conceptual and procedural knowledge in activating mathematical competencies (PISA). Educational Studies in Mathematics, 71(2), 123- 143. Sterne, J., & Harbord, R. M. (2004). Funnel plots in meta-analysis. The Stata Journal, 4(2), 127-141. Tabuk, M. (2019). Matematik Başarısı ve Matematiğe İlişkin Tutum: Meta-analiz Çalışması: Marmara Üniversitesi. Atatürk Eğitim Fakültesi Eğitim Bilimleri Dergisi, 49(49), 167- 186. https://www.researchgate.net/publication/357529426 http://hdl.handle.net/1828/3209 Polat & Turhan 676 Taş, U. E., Arıcı, Ö., Ozarkan, H. B., & Özgürlük, B. (2016). Uluslararasi öğrencı̇ değerlendı̇rme programı PISA 2015 ulusal raporu. Ankara: Millî Eğitim Bakanlığı. Tekin, B., & Tekin, S. (2004). Matematik öğretmen adaylarının matematiksel okuryazarlık düzeyleri üzerine bir araştırma. Retrieved in November 2021, from: http://www.matder.org.tr/index.php?option=com_content&view=article&id=77:matem atik-ogretmenadaylarinin-matematiksel-okuryazarlik-duzeyleri-uzerine-bir-arastirma- &catid=8:matematik-kosesimakaleleri&Itemid=172 Terrin, N., Schmid, C. H., & Lau, J. (2005). In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. Journal of Clinical Epidemiology, 58(9), 894-901. Toraman, Ç., Karadağ, E. & Polat, M. (2022). Validity and reliability evidence for the scale of distance education satisfaction of medical students based on item response theory (IRT). BMC Med Education, 22, (94). https://doi.org/10.1186/s12909-022-03153-9 Toraman, Ç., & Korkmaz, G. (2022). The great barrier to teaching profession: Technicism, rethinking the meaning of professionalism through teachers’ experience. International Online Journal of Education and Teaching (IOJET), 9(1). 486-505. Turgut, S. (2021). A meta-analysis of the effects of realistic mathematics education-based teaching on mathematical achievement of students in Turkey. Journal of Computer and Education Research, 9 (17), 300-326. DOI: 10.18009/jcer.844906 Wang, S., Jiao, H., Young, MJ., Brooks, T., & Olson, J. (2007). A Meta-Analysis of Testing Mode Effects in Grade K-12 Mathematics Tests. Educational and Psychological Measurement, 67(2):219-238. doi:10.1177/0013164406288166 Wood, B. (2007). The impact of students’ skills in self-regulated learning on mathematics literacy and problem solving scales as measured by PISA 2003: A comparison of the US and Finland (Unpublished Doctoral Thesis). California State University, Long Beach, the USA. Yenilmez, K., & Ata, A. (2013). Matematik okuryazarlığı dersinin öğretmen adaylarının matematik okuryazarlığı özyeterliğine etkisi. The Journal of Academic Social Science Studies,6(2),1803-1816. https://arastirmax.com/en/system/files/dergiler/79201/makaleler/6/2/arastrmx_79201_6 _pp_1803-1816.pdf http://www.matder.org.tr/index.php?option=com_content&view=article&id=77:matematik-ogretmenadaylarinin-matematiksel-okuryazarlik-duzeyleri-uzerine-bir-arastirma-&catid=8:matematik-kosesimakaleleri&Itemid=172 http://www.matder.org.tr/index.php?option=com_content&view=article&id=77:matematik-ogretmenadaylarinin-matematiksel-okuryazarlik-duzeyleri-uzerine-bir-arastirma-&catid=8:matematik-kosesimakaleleri&Itemid=172 http://www.matder.org.tr/index.php?option=com_content&view=article&id=77:matematik-ogretmenadaylarinin-matematiksel-okuryazarlik-duzeyleri-uzerine-bir-arastirma-&catid=8:matematik-kosesimakaleleri&Itemid=172 https://doi.org/10.1186/s12909-022-03153-9 https://arastirmax.com/en/system/files/dergiler/79201/makaleler/6/2/arastrmx_79201_6_pp_1803-1816.pdf https://arastirmax.com/en/system/files/dergiler/79201/makaleler/6/2/arastrmx_79201_6_pp_1803-1816.pdf