Sebuah Kajian Pustaka: Journal of Mathematics Education p-ISSN 2089-6867 Volume 12, No. 1, February 2023 e–ISSN 2460-9285 https://doi.org/10.22460/infinity.v12i1.p101-116 101 META-ANALYSIS OF COMPUTER-BASED MATHEMATICS LEARNING IN THE LAST DECADE SCOPUS DATABASE: TRENDS AND IMPLICATIONS Maximus Tamur1*, Sabina Ndiung1, Robert Weinhandl2, Tommy Tanu Wijaya3, Emilianus Jehadus1, Eliterius Sennen1 1Universitas Katolik Indonesia Santu Paulus Ruteng, Indonesia 2Linz School of Education, STEM Education, Johannes Kepler Universität Linz, Austria 3School of Mathematical Sciences, Beijing Normal University, Beijing, China Article Info ABSTRACT Article history: Received Jan 11, 2023 Revised Feb 10, 2023 Accepted Feb 26, 2023 Computer-Based Mathematics Learning (CBML) has gone global in the last decade and is making a substantial impact for educational purposes. But the fact is that in the scientific literature, it is found that studies aimed at testing these theoretical assumptions have inconsistent results. In this regard, this meta-analysis was conducted to determine the effect of CBML and to analyze categorical variables to consider the implications. Data were retrieved from the Scopus database using Publish or Perish between 2010 and 2023. This study examined 29 independent samples from 28 eligible primary studies with 1179 subjects. The population estimate was based on a random effects model, and the CMA software was used as a calculation aid. The study's results provide an overall effect size of 1.03 (large effect). This indicates that applying CBML significantly affects students' mathematical abilities. The four categorical variables considered in the study are discussed to clarify research trends. Furthermore, the research implications are outlined and contribute to future CBML implementation arrangements. Keywords: Computer-Based Mathematics Learning, Mathematical Abilities, Meta-Analysis, Scopus Database This is an open access article under the CC BY-SA license. Corresponding Author: Maximus Tamur, Department of Mathematics Education, Universitas Katolik Indonesia Santu Paulus Ruteng Jl. Ahmad Yani 10 Ruteng, Manggarai, East Nusa Tenggara 86511, Indonesia. Email: maximustamur@unikastpaulus.ac.id How to Cite: Tamur, M., Ndiung, S., Weinhandl, R., Wijaya, T. T., Jehadus, E., & Sennen, E. (2023). Meta-analysis of computer-based mathematics learning in the last decade scopus database: Trends and implications. Infinity, 12(1), 101-116. 1. INTRODUCTION Computer-based Mathematics Learning (CBML) has been widely implemented and included in the teaching system in almost every country. The use of CBML in the classroom is associated with students' academic abilities because teachers provide exciting learning opportunities and broader content (de Mendivil et al., 2019; McLaren et al., 2017; Xin et al., 2020; Zhang & Wang, 2020). Applying CBML significantly influences students' mathematical abilities (Kim et al., 2022; Nurjanah et al., 2020; Pereira et al., 2021; https://doi.org/10.22460/infinity.v12i1.p101-116 https://creativecommons.org/licenses/by-sa/4.0/ Tamur et al., Meta-analysis of computer-based mathematics learning in the last decade scopus … 102 Shyshkina et al., 2018). It can be said that the use of CBML has become a trend in learning mathematics in the last decade. The use of CBML in learning is increasingly widespread because it presents content numerically, graphically and symbolically without the additional time burden of manually calculating complex computational problems (Juandi, Kusumah, Tamur, Perbowo, Siagian, et al., 2021; Juandi, Kusumah, Tamur, Perbowo, & Wijaya, 2021; Tamur, Kusumah, Juandi, Kurnila, et al., 2021; Tamur, Kusumah, Juandi, Wijaya, et al., 2021). Integrating computer technology into learning mathematics will help students make connections in mathematics by making the learning process more realistic and effective (Zaldívar-Colado et al., 2017). CBML implementation will be more interesting, inventive, and exploratory (Aungamuthu, 2013; Foster et al., 2016; Ochkov & Bogomolova, 2015; Tamur et al., 2022). These conditions allow students to be more active and successful in learning (Das et al., 2021; Tatar et al., 2014; Timmers et al., 2013). CBML provides useful feedback to support students' representation, visualization, and critical abilities (Granberg & Olsson, 2015). Applying CBML can improve Students' Mathematical Ability (SMA). However, in the scientific literature, studies testing these theoretical assumptions provide inconsistent findings. Several studies provide findings stating the superiority of the CBML group (e.g., Ishartono et al., 2022; Jelatu et al., 2018; Kariadinata et al., 2019; Ningsih & Paradesa, 2018; Nurjanah et al., 2020; Takači et al., 2015), whereas other studies have found that CBML implementation does not outperform the group that does not use it (e.g., del Cerro Velázquez & Morales Méndez, 2021; Hamid et al., 2020). Various studies that analyze the same problem sometimes provide findings that vary and contradict each other. As a result, making decisions or conclusions can be subjective (Wicherts, 2020). The problem is that educators and other related parties need accurate information for making educational decisions and arrangements under what conditions the use of CBML achieves a high level of effectiveness in SMA. The problems described above can be overcome by summarizing the various results of primary research. Quantitative research procedures can be chosen to provide accurate policy-making information (Higgins & Katsipataki, 2015). In this regard, the meta-analysis study is the choice because of its role specifically to integrate the findings of the primary study and investigate the reasons for the variation in results to be considered in its implementation in the future. Meta-analysis provides in-depth and accurate conclusions (Fadhli et al., 2020; Siddaway et al., 2019). Thus, when there is a need to draw conclusive conclusions, it is necessary to take into account the results of various individual studies using meta-analysis methods (Wicherts, 2020). Previously, meta-analysis studies have been found in the literature regarding the effect of CBML implementation on SMA (e.g., Demir & Basol, 2014; Juandi, Kusumah, Tamur, Perbowo, & Wijaya, 2021; Tamur, Kusumah, Juandi, Kurnila, et al., 2021). However, these studies analyzed primary data from all databases, such as Google Scholar, ERIC, repository, library database, and others. In contrast to previous studies, this study analyzes data taken from the Scopus database. Apart from the fact that no studies specifically chose Scopus as the only search database, Scopus was also chosen because it applies consistent standards in selecting documents to be included in its index (Phan et al., 2022). Scopus also displays more documents than other top databases, such as the Web of Science, specifically for research reviews in education and social sciences (Hallinger & Chatpinyakoop, 2019; Hallinger & Nguyen, 2020). This study also fills a gap in the literature regarding a comparative picture of CBML implementation in Indonesia and abroad. Previous studies have also shown inconsistent trends in the variables studied in terms of year of publication, source of publication, and level of education (e.g., Juandi, Kusumah, Volume 12, No 1, February 2023, pp. 101-116 103 Tamur, Perbowo, & Wijaya, 2021; Tamur et al., 2020; Tamur, Kusumah, Juandi, Wijaya, et al., 2021). This meta-analysis is necessary to evaluate the implementation of CBML to see the overall trend clearly and also consider the implications. Thus, this study will directly contribute to CBML science and practice in the future, especially concerning educational settings. This research answers explicitly two questions, namely; (1) whether the mean effect size representing the intervention for each study on the effect of CBML on SMA is significantly different from zero, and (2) whether there is a difference in the magnitude of the effect of CBML on SMA based on categorical variables. 2. METHOD This study focuses on analyzing the influence of the quantitative profile of the study group from the Scopus database, which specifically investigates the effectiveness of CBML on students' mathematical abilities (SMA). This work was conducted to determine the magnitude of the overall effect of CBML and to analyze the role of categorical variables in the differences in effect sizes of each study. To achieve this goal, a meta-analysis approach was applied. Meta-analysis was developed to compare the results of various primary studies and its role is very vital for scientific development, namely providing a more objective method of drawing conclusions or decisions when various primary studies provide varying results (Cooper, 2017; Schmidt & Hunter, 2015). In general, meta-analysis research begins with formulating research problems and hypotheses, followed by a literature search, then coding variables, then statistical analysis, and ends with the interpretation of findings (Çoğaltay & Karadağ, 2015). This work also follows these stages. 2.1. Literature Search The Scopus database was chosen as the document search location in this study. Furthermore, the Publish or Perish program is used to download a study on the application of CBML with the keyword combination used being geogebra mathematical ability; Cabri mathematical ability; maple mathematical abilities; mathematical ability algebrator, Matlab Mathematical Ability, MatchCAD, and Wingeom. As an example, Figure 1 presents the process of tracing studies related to implementing CBML on the Scopus database using the PoP application. The search results obtained 193 related studies. Figure 1. The process of tracing data using the PoP application. Tamur et al., Meta-analysis of computer-based mathematics learning in the last decade scopus … 104 2.2. Literature Inclusion Criteria In this study, studies that were successfully identified using the PoP application were then shared according to the following criteria: a. Study in an English setting, and retrieved from Scopus database between 2010-2023. b. Provide statistical information to obtain effect size values. Studies that do not meet this will be excluded from the analysis (e.g., Dockendorff & Solar, 2018; Jacinto & Carreira, 2017; Zulnaidi et al., 2020). c. Study with a quantitative approach and must involve a control group as a comparison. Studies that used only one sample or used a qualitative approach were excluded from the analysis. Furthermore, in this study, the data filtering process used the PRISMA (Preferred Reporting Items for Systematic Review and Meta-Analyses) protocol, as shown in Figure 2. Figure 2. Filtering data through PRISMA Figure 2 presents the process of filtering data using PRISMA. Finally, there are 28 studies, and because there are studies that investigate more than one independent sample, in the study conducted by Apriatna et al. (2020) then, 29 effect sizes were examined. 2.3. Coding All eligible studies were coded using a detailed coding scheme based on the coding manual and coding protocol. Each preliminary study included in the analysis was coded. In this work, the research instrument is a coding sheet specially created to extract information from individual studies converted into numerical data. The coding was carried out by two Volume 12, No 1, February 2023, pp. 101-116 105 students who had previously been specifically trained according to guidelines from Cooper (2017). To determine reliability, a random sample of 5 from the eligible studies was duplicated and distributed among the coders. Each coder was provided with a copy of the article, coding form and coding protocol. To assess the reliability between coders, the percent agreement or abbreviated PA is used. The PA method is the easiest and most intuitive approach to building reliability (Syed & Nelson, 2015). PA is simply the ratio of items that the two coders agree on to the total number of items assessed and is calculated using the following formula: 𝑃𝐴 = 𝑁𝐴 𝑁𝐴 + 𝑁𝑃 𝑥 100 Where NA is the total number of agreements and NP is the total number of disagreements. An agreement rate of 85% or greater is predetermined to be considered high. From the calculation results, the level of both PAs is 94.26%. The figure indicates there is a substantial to near-perfect match between the coders. Thus the instrument to be used in this study is reliable. 2.4. Statistical Analysis The parameter used to estimate the population in a meta-analysis study is the effect size. In this study, the effect size is defined as the magnitude of the effect of CBML on SMA. CMA software stands for Comprehensive Meta-Analysis used to transform the effect size of each study, including the overall effect size, P-value, Q statistics, and confidence intervals. The program also draws funnel plots and research forest plots. Hedges' g metrics were applied in this study, and the interpretation of effect sizes was based on the classification of Cohen et al. (2017) i.e., less than 0.2 (negligible), 0.2 to 0.5 (small effect), 0.5 to 0.8 (medium effect), 0.8 to 1.3 (large effect), and more than 1.3 (very large effect). The random effects model was chosen as the estimation method because it does not assume that all the studies analyzed have the same true effect (Pigott & Polanin, 2020). In this study, the random effect model was determined after fulfilling the heterogeneity test. This test is carried out by observing the p-value. The null hypothesis, which states that all studies are the same (homogeneous), is rejected if the p-value <0.05. Rejecting the null hypothesis indicates that effect sizes between studies or study groups may not measure the same population parameters (Çoğaltay & Karadağ, 2015). Conditions show that differences in study categories affect study effect sizes. An analysis of publication bias was performed to prevent misrepresentation of the findings. Publication bias reflects the fact that it is more likely that articles deemed statistically significant are published. In addition to the scientific facts that about 6% of researchers rarely try to publish research that is not significant (Cooper, 2017). As a result, the aggregate effect size obtained may be exaggerated (Juandi et al., 2022; Park & Hong, 2016). To anticipate this, funnel plots were examined to assess the possible amount of bias, and Rosenthal's FSN statistic was used to assess the impact of bias (Çoğaltay & Karadağ, 2015). In this study, it is said to be resistant to publication bias if the distribution of effect sizes is symmetrical around the vertical line. However, if the fact is that the effect size scatter is not completely symmetrical, then a trim and fill procedure is used. Then, if it is found that the observed effect and the virtual effect created according to the random effects model are the same then the study is resistant to the influence of publication bias. Tamur et al., Meta-analysis of computer-based mathematics learning in the last decade scopus … 106 3. RESULT AND DISCUSSION 3.1. Results First, the results of research aimed at answering question one are presented. After screening the primary studies, 28 articles from the Scopus database and 29 independent samples were analyzed in this study. Figure 3 presents the research forest plot consisting of the study name, ES (effect size), standard error, variance per the study, confidence intervals, z values, and p values. Figure 3. Research forest plot Figure 3 shows the broad confidence levels and inconsistent response rates. Descriptively this shows the effect size of each study is heterogeneous. Statistically, this needs to be checked so that the estimation method is in accordance with the initial assumptions. Table 1 presents a summary of the overall analysis to answer question 1 as well as to determine the estimation method. Table 1. The meta-analysis results according to the estimation model Model N Hedges’s g Standard error Test of null Q P Decision Z-value P-value Fixed-effects 29 0.81 0.04 18,38 0.00 186.87 0.00 Reject H0 Random-effects 29 1.03 0.01 8,57 0.00 When Table 1 is observed, it appears that the P value <0, which means that the distribution of effect sizes for each study is heterogeneous. This means that the estimation method for the population in this study is in accordance with the random effect model. Table Study name Statistics for each study Hedges's g and 95% CI Hedges's Standard Lower Upper g error Variance limit limit Z-Value p-Value Priatna, 2017a 0,79 0,26 0,07 0,27 1,31 2,97 0,00 Priatna, Martadiputra & Wibisono, 2018 0,92 0,27 0,07 0,40 1,45 3,44 0,00 Munandar, Usman, & Saminan, 2020 1,97 0,32 0,10 1,34 2,61 6,11 0,00 Juandi, & Priatna, 2018 0,36 0,25 0,06 -0,13 0,86 1,43 0,15 Nurafni, & Ningrum, 2021 0,50 0,25 0,06 -0,00 1,00 1,95 0,05 Takaci, Stankov, & Milanovic, 2015 0,68 0,11 0,01 0,46 0,89 6,24 0,00 Septian, Darhim, & Prabawanto, 2020 2,33 0,38 0,15 1,58 3,08 6,09 0,00 Rhamawati, 2019 2,32 0,37 0,13 1,60 3,04 6,32 0,00 Saha, Ayub, & Tarmizi, 2010 0,60 0,28 0,08 0,05 1,14 2,15 0,03 Maryono, Rodiah, & Syaf, 2020 1,10 0,27 0,07 0,57 1,63 4,05 0,00 Ridha, Pramiarsih, & Widjajani, 2019 0,73 0,30 0,09 0,14 1,32 2,44 0,01 Negara, Wahyudin, Nurlaelah, & Herman, 2022 0,72 0,24 0,06 0,26 1,19 3,05 0,00 Murni, Sariyasa, & Ardana, 2017 3,77 0,43 0,18 2,93 4,61 8,81 0,00 Khalil, Farooq, Çak?ro?lu, & Khali, 2018 0,82 0,32 0,10 0,19 1,45 2,53 0,01 Zetriuslita, Nofriyandi, & Istikomah, 2019 0,01 0,31 0,10 -0,59 0,62 0,04 0,97 Zulnaidi, & Zamri, 2017 0,53 0,11 0,01 0,31 0,74 4,81 0,00 Nurjanah, Latif, Yuliardi, & Tamur, 2020 0,43 0,24 0,06 -0,03 0,90 1,82 0,07 Kusumah, Kustiawati, & Herman, 2020 0,79 0,23 0,05 0,34 1,23 3,47 0,00 Jelatu, Sariyasa, & Ardana, 2018 1,12 0,27 0,08 0,58 1,66 4,08 0,00 Ningsih, & Paradesa, 2018 1,53 0,29 0,08 0,97 2,10 5,32 0,00 Apriatna, Budiyono, & Indriati, 2020a 0,60 0,26 0,07 0,10 1,11 2,35 0,02 Apriatna, Budiyono, & Indriati, 2020b 0,63 0,26 0,07 0,13 1,14 2,45 0,01 Kariadinata et al., 2019 1,35 0,31 0,10 0,75 1,96 4,38 0,00 Muntazhimah, & Miatun, 2018 0,97 0,32 0,10 0,34 1,60 3,02 0,00 Ishartono et al., 2022 0,15 0,26 0,07 -0,35 0,65 0,59 0,55 Velázquez, & Méndez, 2021 0,32 0,22 0,05 -0,12 0,75 1,44 0,15 Priatna, 2017b 2,85 0,33 0,11 2,19 3,50 8,56 0,00 Zulnaidi et al., 2019 0,48 0,22 0,05 0,04 0,92 2,16 0,03 Septian et al., 2020 1,57 0,34 0,12 0,91 2,24 4,63 0,00 1,01 0,12 0,01 0,78 1,25 8,58 0,00 -4,00 -2,00 0,00 2,00 4,00 Fav ours A Fav ours B Plot forest research Meta Analysis Volume 12, No 1, February 2023, pp. 101-116 107 1 also shows the results of hypothesis testing to answer the first research question. When Table 1 is observed it appears that the P value for the test of null is less than 0 based on the random effects model. This means that the mean effect size representing the intervention for each study of the effect of CBML on SMA is significantly different from zero. In other words, the overall results of the study clarify the superiority of the CBML group. The overall effect size based on the random effects model is 1.03 which is classified as a large effect according to Cohen et al. (2017). It is then necessary to calculate whether this overall effect size is related to publication bias then the funnel plot of the study is observed. Figure 4 presents the funnel plot of the study. Figure 4. Funnel plot of 29 independent samples When it is observed in Figure 4, the study ESs are spread not entirely symmetrically in the center of the funnel plot. Therefore, it is necessary to observe whether these results resist publication bias's influence. For this reason, the Trim and Fill testing procedure is carried out as illustrated in Figure 5. Figure 5. Trim and fill test The trim and fill test results as illustrated in Figure 5 show that there is no difference between the observed effect sizes and the virtual effects created according to the random- effects model. Thus, there was no publication bias in this study or no studies were pruned or added due to publication bias. So the overall effect size calculated as 1.03 and categorized as large effect is not associated with publication bias. This value is not exaggerated. Tamur et al., Meta-analysis of computer-based mathematics learning in the last decade scopus … 108 Second, the research results are presented to answer question two. It has previously been shown that the estimation method fits the random effects model. This shows that the effect size of each study is heterogeneous so that categorical variables that affect the relationship between CBML and SMA must be investigated (Arik & Yilmaz, 2020). The summary of the analysis related to these catogri variables is illustrated in Table 2. Table 2. Summary of category variable analysis Category Variables Category N Hedge's g Heterogeneity Decision (Qb) df(Q) P Sample Size 30 or less 18 0.92 4.32 1 0,03 Reject H0 31 or over 11 0.73 Educational stage Junior high school (JHS) 17 0.87 6.99 2 0.03 Reject H0 Senior high school (SHS) 5 0.76 College 7 0.58 Publication Sources Journal 12 0.57 48.6 2 0.00 Reject H0 Proceeding 17 1.21 Country Indonesia 23 0.99 22.75 1 0.00 Reject H0 Foreign 6 0.57 3.2. Discussion The first objective of this research is to determine the magnitude of the overall effect of applying CBML on SMA. The analysis results gave the overall effect size of 1.03 and were accepted as a large effect. These results are supported by a previous meta-analysis conducted by Tamur et al. (2020) regarding the effect of implementing mathematical software in Indonesia. Although the database and population used differ from this study, the overall effect size is almost the same, namely 1.16. In addition, the results of this study are almost the same as the findings of studies conducted by Juandi, Kusumah, Tamur, Perbowo and Wijaya (2021) and Juandi, Kusumah, Tamur, Perbowo, Siagian, et al. (2021) where the overall study effect sizes were 1.07 and 0.96, respectively. This shows a nearly similar overall trend with respect to CBML adoption. In this study, the number of subjects was 1127 people, and the average sample size was 44 people. Related to that, the effect size of 1.03 can be interpreted as students who are ranked 22 in the experimental class are equivalent to students who are ranked 8 in the control class. This illustration illustrates the magnitude of the influence of CBML on SMA. This condition is explained in the scientific literature that the application of CBML helps increase students' knowledge of mathematical concepts and procedures (Zulnaidi & Zamri, 2017). In CBML, students are motivated to learn because of technology's wide and interesting content (Zulnaidi et al., 2020). CBML implementation produces concept organizing principles which are then used to think about various possible solutions (Santos-Trigo & Reyes-Rodriguez, 2016). Furthermore, from the analysis of categorical variables, as illustrated in Table 2, it can be seen that there are categorical variables in this study that clarify the effect size of the study. First, the analysis results show that the difference in sample size affects the study's effect size. It appears that a small sample size (30 or less) gives a larger effect size than the other categories. These results directly contribute to future classroom arrangements. The Volume 12, No 1, February 2023, pp. 101-116 109 literature also supports this result (e.g., Juandi, Kusumah, Tamur, Perbowo, Siagian, et al., 2021; Juandi, Kusumah, Tamur, Perbowo, & Wijaya, 2021) that small sample size is defined as 30 or less great effect than in the other categories. The analysis results also show that differences in educational level affect the effect size of CBML on SMA. The findings in this study illustrate that the implementation of CBML is more effective in JHS than in SHS and College. This result is surprising because it differs from previous studies' findings (e.g., Juandi, Kusumah, Tamur, Perbowo, & Wijaya, 2021). This difference may be due to the database used is not the same. However, further studies are needed to verify and clarify the consistency of the results regarding this variable. Furthermore, Table 2 presents the analysis results related to the publication source variable. The results of the analysis show that differences in publication sources also explain the variation in effect sizes between studies. Statistically, the mean effect size of the study group from proceedings was greater than that of the journal. This indicates that this study is not associated with publication bias, as supported by the trim and fill test. In this study, there was no indication that journal editors chose to publish only significant articles (Cooper, 2017) due to the fact that the study group originating from proceedings was even more significant. This study also considers country differences as a categorical variable. In the literature, six studies were found from abroad that examined CBML for SMA, and the rest were from Indonesia. There is a significant difference in the mean effect size between categories. This result is not surprising because the implementation of CBML abroad has been going on for a long time, as Sugandi and Delice (2014) reported. These results were influenced by the Hawthorne effect, as investigated by Tamur, Kusumah, Juandi, Wijaya, et al. (2021) that students feel enthusiastic about learning because of the novelty of the treatment. If the same treatment is given continuously, the effect will decrease. However, this difference in results is interesting for further investigation through further research. Finally, given the importance of implementing CBML, teaching teachers how to manage technology-enabled learning (Dockendorff & Solar, 2018). Continuous training for teachers is needed to integrate technology effectively (Kartal & Çınar, 2022). These results have implications for the importance of mathematics teacher training and its impact on developing technological pedagogical content knowledge. In this study, many studies were excluded because the categories were not measured. Hartatiana et al. (2017) for example conducted an experiment regarding the effectiveness of CBML by only considering gender differences. Although there is statistical information that supports the effect size transformation, it does not lead to the purpose of this study. Thus, further studies that specifically consider gender differences as a categorical variable need to be carried out. Furthermore, of the 193 identified studies, only 28 met the inclusion requirements. Of the number that do not meet the analysis requirements, it examines more about analyzing student abilities related to CBML implementation (e.g., Mendrek et al., 2018), development studies (e.g., Khalil et al., 2018) and review research (e.g., Juandi, Kusumah, Tamur, Perbowo, Siagian, et al., 2021; Tamur et al., 2020; Wang & Tahir, 2020). From the distribution of data extracted, it was identified that several studies specifically investigated differences in initial ability as a categorical variable (e.g., Kariadinata et al., 2019; Muntazhimah & Miatun, 2018), whereas more studies did not measure it. The direct implication of this condition is that in the future, it is necessary to investigate whether these categorical variables influence the effectiveness of CBML on students' mathematical performance. Tamur et al., Meta-analysis of computer-based mathematics learning in the last decade scopus … 110 4. CONCLUSION This meta-analysis analyzed 29 independent samples from 28 primary studies. Based on the random effects model, it was found that the application of CBML had a large effect on SMA. The variation in the effect size of each primary study was moderated by the categorical variables observed in this study. However, these findings are only based on the Scopus database searched through the PoP application. The fact is that there are still many related studies that cannot be downloaded, especially publications from the IEEE, which must go through an affiliate and be paid specifically. Further cooperation procedures are needed to obtain more related studies. ACKNOWLEDGEMENTS The authors would like to appreciate the technical assistance from two students of the mathematics education study program at the Catholic University of Indonesia, Santu Paulus Ruteng, who were involved as variable coders. REFERENCES Apriatna, E. J., Budiyono, & Indriati, D. (2020). The effectiveness of problem based learning assisted by cabri 3D on student’s mathematical communication writing and drawing skills. Journal of Physics: Conference Series, 1581(1), 012060. https://doi.org/10.1088/1742-6596/1581/1/012060 Arik, S., & Yilmaz, M. (2020). The effect of constructivist learning approach and active learning on environmental education: a meta-analysis study. International Electronic Journal of Environmental Education, 10(2), 44-84. Aungamuthu, Y. (2013). Towards a responsive pedagogy: Using ICT as a tool to engage access students’ academic identities in mathematics. Alternation, 8, 66-85. Çoğaltay, N., & Karadağ, E. (2015). Introduction to Meta-Analysis. In E. Karadağ (Ed.), Leadership and Organizational Outcomes: Meta-Analysis of Empirical Studies (pp. 19-28). Springer International Publishing. https://doi.org/10.1007/978-3-319-14908- 0_2 Cohen, L., Manion, L., & Morrison, K. (2017). Research methods in education. Routledge. Cooper, H. (2017). Research synthesis and meta-analysis: A step-by-step approach (Fifth Edition ed.). https://doi.org/10.4135/9781071878644 Das, A. K., Das, S., & Mukherjee, J. (2021). Largest triangle inside a terrain. Theoretical Computer Science, 858, 90-99. https://doi.org/10.1016/j.tcs.2020.12.018 de Mendivil, Í. S. M., Crespo, R. G., González-Castaño, A., Ruiz, A. A. M., & Palma, L. O. (2019). Herramienta pedagógica basada en el desarrollo de una aplicación informática para la mejora del aprendizaje en matemática avanzada [A pedagogical tool based on the development of a computer application to improve learning in advanced mathematics]. Revista Española de Pedagogía, 77(274), 457-485. https://doi.org/10.22550/REP77-3-2019-06 del Cerro Velázquez, F., & Morales Méndez, G. (2021). Application in augmented reality for learning mathematical functions: A study for the development of spatial intelligence in secondary education students. Mathematics, 9(4), 369. https://doi.org/10.3390/math9040369 https://doi.org/10.1088/1742-6596/1581/1/012060 https://doi.org/10.1007/978-3-319-14908-0_2 https://doi.org/10.1007/978-3-319-14908-0_2 https://doi.org/10.4135/9781071878644 https://doi.org/10.1016/j.tcs.2020.12.018 https://doi.org/10.22550/REP77-3-2019-06 https://doi.org/10.3390/math9040369 Volume 12, No 1, February 2023, pp. 101-116 111 Demir, S., & Basol, G. (2014). Effectiveness of computer-assisted mathematics education (CAME) over academic achievement: A meta-analysis study. Educational Sciences: Theory and Practice, 14(5), 2026-2035. Dockendorff, M., & Solar, H. (2018). ICT integration in mathematics initial teacher training and its impact on visualization: the case of GeoGebra. International Journal of Mathematical Education in Science and Technology, 49(1), 66-84. https://doi.org/10.1080/0020739X.2017.1341060 Fadhli, M., Brick, B., Setyosari, P., Ulfa, S., & Kuswandi, D. (2020). A meta-analysis of selected studies on the effectiveness of gamification method for children. International Journal of Instruction, 13(1), 845-854. https://doi.org/10.29333/iji.2020.13154a Foster, M. E., Anthony, J. L., Clements, D. H., Sarama, J., & Williams, J. M. (2016). Improving mathematics learning of kindergarten students through computer-assisted instruction. Journal for Research in Mathematics Education JRME, 47(3), 206-232. https://doi.org/10.5951/jresematheduc.47.3.0206 Granberg, C., & Olsson, J. (2015). ICT-supported problem solving and collaborative creative reasoning: Exploring linear functions using dynamic mathematics software. The Journal of Mathematical Behavior, 37, 48-62. https://doi.org/10.1016/j.jmathb.2014.11.001 Hallinger, P., & Chatpinyakoop, C. (2019). A bibliometric review of research on higher education for sustainable development, 1998–2018. Sustainability, 11(8), 2401. https://doi.org/10.3390/su11082401 Hallinger, P., & Nguyen, V.-T. (2020). Mapping the landscape and structure of research on education for sustainable development: A bibliometric review. Sustainability, 12(5), 1947. https://doi.org/10.3390/su12051947 Hamid, H., Angkotasan, N., Jalal, A., Muhtadi, D., & Sukirwan. (2020). Students’ mathematical proficiency in solving calculus problems after Maple implementation. Journal of Physics: Conference Series, 1613(1), 012025. https://doi.org/10.1088/1742-6596/1613/1/012025 Hartatiana, H., Darhim, D., & Nurlaelah, E. (2017). Student’s spatial reasoning through model eliciting activities with Cabri 3D. Journal of Physics: Conference Series, 895(1), 012075. https://doi.org/10.1088/1742-6596/895/1/012075 Higgins, S., & Katsipataki, M. (2015). Evidence from meta-analysis about parental involvement in education which supports their children’s learning. Journal of Children's Services, 10(3), 280-290. https://doi.org/10.1108/JCS-02-2015-0009 Ishartono, N., Nurcahyo, A., Waluyo, M., Prayitno, H. J., & Hanifah, M. (2022). Integrating GeoGebra into the flipped learning approach to improve students' self-regulated learning during the COVID-19 pandemic. Journal on Mathematics Education, 13(1), 69-86. https://doi.org/10.22342/jme.v13i1.pp69-86 Jacinto, H., & Carreira, S. (2017). Mathematical problem solving with technology: The techno-mathematical fluency of a student-with-GeoGebra. International Journal of Science and Mathematics Education, 15(6), 1115-1136. https://doi.org/10.1007/s10763-016-9728-8 https://doi.org/10.1080/0020739X.2017.1341060 https://doi.org/10.29333/iji.2020.13154a https://doi.org/10.5951/jresematheduc.47.3.0206 https://doi.org/10.1016/j.jmathb.2014.11.001 https://doi.org/10.3390/su11082401 https://doi.org/10.3390/su12051947 https://doi.org/10.1088/1742-6596/1613/1/012025 https://doi.org/10.1088/1742-6596/895/1/012075 https://doi.org/10.1108/JCS-02-2015-0009 https://doi.org/10.22342/jme.v13i1.pp69-86 https://doi.org/10.1007/s10763-016-9728-8 Tamur et al., Meta-analysis of computer-based mathematics learning in the last decade scopus … 112 Jelatu, S., Sariyasa, S., & Ardana, I. M. (2018). Effect of GeoGebra-Aided REACT Strategy on Understanding of Geometry Concepts. International Journal of Instruction, 11(4), 325-336. https://doi.org/10.12973/iji.2018.11421a Juandi, D., Kusumah, Y. S., & Tamur, M. (2022). A meta-analysis of the last two decades of realistic mathematics education approaches. International Journal of Instruction, 15(1), 381-400. https://doi.org/10.29333/iji.2022.15122a Juandi, D., Kusumah, Y. S., Tamur, M., Perbowo, K. S., Siagian, M. D., Sulastri, R., & Negara, H. R. P. (2021). The effectiveness of dynamic geometry software applications in learning mathematics: A meta-analysis study. International Journal of Interactive Mobile Technologies (iJIM), 15(02), 18-37. https://doi.org/10.3991/ijim.v15i02.18853 Juandi, D., Kusumah, Y. S., Tamur, M., Perbowo, K. S., & Wijaya, T. T. (2021). A meta- analysis of Geogebra software decade of assisted mathematics learning: what to learn and where to go? Heliyon, 7(5), e06953. https://doi.org/10.1016/j.heliyon.2021.e06953 Kariadinata, R., Yaniawati, R. P., Juariah, J., Sugilar, H., & Muthmainah, A. (2019). Spatial thinking ability and mathematical character students through Cabri 3D with a scientific approach. Journal of Physics: Conference Series, 1402(7), 077094. https://doi.org/10.1088/1742-6596/1402/7/077094 Kartal, B., & Çınar, C. (2022). Preservice mathematics teachers’ TPACK development when they are teaching polygons with geogebra. International Journal of Mathematical Education in Science and Technology, 1-33. https://doi.org/10.1080/0020739X.2022.2052197 Khalil, M., Farooq, R., Çakıroğlu, E., Khalil, U., & Khan, D. (2018). The development of mathematical achievement in analytic geometry of grade-12 students through GeoGebra activities. Eurasia Journal of Mathematics Science and Technology Education, 14(4), 1453-1463. https://doi.org/10.29333/ejmste/83681 Kim, S. J., Kastberg, S. E., Xin, Y. P., Lei, Q., Liu, B., Wei, S., & Chen, Y. (2022). Counting strategies of students struggling in mathematics in a computer-based learning environment. The Journal of Mathematical Behavior, 68, 101007. https://doi.org/10.1016/j.jmathb.2022.101007 McLaren, B. M., Adams, D. M., Mayer, R. E., & Forlizzi, J. (2017). A computer-based game that promotes mathematics learning more than a conventional approach. International Journal of Game-Based Learning (IJGBL), 7(1), 36-56. https://doi.org/10.4018/IJGBL.2017010103 Mendrek, M., Grzesik, N., Krzyżak, A., & Kuźma, K. (2018). Different defuzzification methods in Guimbal Cabri G2 helicopter takeoff possibility evaluation. Transport Problems, 13(2), 27-38. Muntazhimah, M., & Miatun, A. (2018). Cabri 3D - assisted collaborative learning to enhance junior high school students’ spatial ability. Journal of Physics: Conference Series, 948(1), 012042. https://doi.org/10.1088/1742-6596/948/1/012042 Ningsih, Y. L., & Paradesa, R. (2018). Improving students’ understanding of mathematical concept using maple. Journal of Physics: Conference Series, 948(1), 012034. https://doi.org/10.1088/1742-6596/948/1/012034 https://doi.org/10.12973/iji.2018.11421a https://doi.org/10.29333/iji.2022.15122a https://doi.org/10.3991/ijim.v15i02.18853 https://doi.org/10.1016/j.heliyon.2021.e06953 https://doi.org/10.1088/1742-6596/1402/7/077094 https://doi.org/10.1080/0020739X.2022.2052197 https://doi.org/10.29333/ejmste/83681 https://doi.org/10.1016/j.jmathb.2022.101007 https://doi.org/10.4018/IJGBL.2017010103 https://doi.org/10.1088/1742-6596/948/1/012042 https://doi.org/10.1088/1742-6596/948/1/012034 Volume 12, No 1, February 2023, pp. 101-116 113 Nurjanah, N., Latif, B., Yuliardi, R., & Tamur, M. (2020). Computer-assisted learning using the Cabri 3D for improving spatial ability and self- regulated learning. Heliyon, 6(11), e05536. https://doi.org/10.1016/j.heliyon.2020.e05536 Ochkov, V. F., & Bogomolova, E. P. (2015). Teaching mathematics with mathematical software. Journal of Humanistic Mathematics, 5(1), 265-285. https://doi.org/10.5642/jhummath.201501.15 Park, S., & Hong, S. (2016). The empirical review of meta-analysis published in Korea. Asia Pacific Education Review, 17(2), 313-324. https://doi.org/10.1007/s12564-016- 9433-x Pereira, J., Tang, J., Wijaya, T. T., Chen, J., Hermita, N., & Tamur, M. (2021). Modeling the interior angles of a triangle using Hawgent Dynamic Mathematics Software. In 2021 Universitas Riau International Conference on Education Technology (URICET). Phan, T. T., Do, T. T., Trinh, T. H., Tran, T., Duong, H. T., Trinh, T. P. T., Do, B. C., & Nguyen, T.-T. (2022). A bibliometric review on realistic mathematics education in scopus database between 1972-2019. European Journal of Educational Research, 11(2), 1133-1149. https://doi.org/10.12973/eu-jer.11.2.1133 Pigott, T. D., & Polanin, J. R. (2020). Methodological guidance paper: High-quality meta- analysis in a systematic review. Review of Educational Research, 90(1), 24-46. https://doi.org/10.3102/0034654319877153 Santos-Trigo, M., & Reyes-Rodriguez, A. (2016). The use of digital technology in finding multiple paths to solve and extend an equilateral triangle task. International Journal of Mathematical Education in Science and Technology, 47(1), 58-81. https://doi.org/10.1080/0020739X.2015.1049228 Schmidt, F. L., & Hunter, J. E. (2015). Methods of meta-analysis: Correcting error and bias in research findings (Third Edition ed.). London: SAGE Publications, Ltd. https://doi.org/10.4135/9781483398105 Shyshkina, M., Kohut, U., & Popel, M. (2018). The systems of computer mathematics in the cloud-based learning environment of educational institutions. In CEUR Workshop Proceedings. https://doi.org/10.48550/arXiv.1807.01770 Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019). How to do a systematic review: A best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annual Review of Psychology, 70(1), 747-770. https://doi.org/10.1146/annurev-psych-010418-102803 Sugandi, B., & Delice, A. (2014). Comparison of Turkish and Indonesian secondary mathematics curricula; Reflection of the Paradigms. Procedia - Social and Behavioral Sciences, 152, 540-545. https://doi.org/10.1016/j.sbspro.2014.09.240 Syed, M., & Nelson, S. C. (2015). Guidelines for establishing reliability when coding narrative data. Emerging Adulthood, 3(6), 375-387. https://doi.org/10.1177/21676968155876 Takači, D., Stankov, G., & Milanovic, I. (2015). Efficiency of learning environment using GeoGebra when calculus contents are learned in collaborative groups. Computers & Education, 82, 421-431. https://doi.org/10.1016/j.compedu.2014.12.002 https://doi.org/10.1016/j.heliyon.2020.e05536 https://doi.org/10.5642/jhummath.201501.15 https://doi.org/10.1007/s12564-016-9433-x https://doi.org/10.1007/s12564-016-9433-x https://doi.org/10.12973/eu-jer.11.2.1133 https://doi.org/10.3102/0034654319877153 https://doi.org/10.1080/0020739X.2015.1049228 https://doi.org/10.4135/9781483398105 https://doi.org/10.48550/arXiv.1807.01770 https://doi.org/10.1146/annurev-psych-010418-102803 https://doi.org/10.1016/j.sbspro.2014.09.240 https://doi.org/10.1177/21676968155876 https://doi.org/10.1016/j.compedu.2014.12.002 Tamur et al., Meta-analysis of computer-based mathematics learning in the last decade scopus … 114 Tamur, M., Juandi, D., & Kusumah, Y. S. (2020). The effectiveness of the application of mathematical software in Indonesia; A meta-analysis study. International Journal of Instruction, 13(4), 867-884. https://doi.org/10.29333/iji.2020.13453a Tamur, M., Kusumah, Y. S., Juandi, D., Kurnila, V. S., Jehadus, E., & Samura, A. O. (2021). A meta-analysis of the past decade of mathematics learning based on the computer algebra system (CAS). Journal of Physics: Conference Series, 1882(1), 012060. https://doi.org/10.1088/1742-6596/1882/1/012060 Tamur, M., Kusumah, Y. S., Juandi, D., Wijaya, T. T., Nurjaman, A., & Samura, A. O. (2021). Hawthorne effect and mathematical software based learning: a meta-analysis study. Journal of Physics: Conference Series, 1806(1), 012072. https://doi.org/10.1088/1742-6596/1806/1/012072 Tamur, M., Weinhandl, R., Sennen, E., Ndiung, S., & Nurjaman, A. (2022). The effect of Cabri Express in geometry learning on students' mathematical communication ability. JTAM (Jurnal Teori Dan Aplikasi Matematika), 6(4), 1027-1033. https://doi.org/10.31764/jtam.v6i4.10865 Tatar, E., Kağizmanli, T. B., & Akkaya, A. (2014). The effect of a Dynamic Software on the success of analytical analysis of the circle and prospective mathematics teachers opinions. Necatibey Faculty of Education Electronic Journal of Science & Mathematics Education, 8(1), 153-177. Timmers, C. F., Braber-van den Broek, J., & van den Berg, S. M. (2013). Motivational beliefs, student effort, and feedback behaviour in computer-based formative assessment. Computers & Education, 60(1), 25-31. https://doi.org/10.1016/j.compedu.2012.07.007 Wang, A. I., & Tahir, R. (2020). The effect of using Kahoot! for learning – A literature review. Computers & Education, 149, 103818. https://doi.org/10.1016/j.compedu.2020.103818 Wicherts, J. (2020). Meta-analysis. In V. Zeigler-Hill & T. K. Shackelford (Eds.), Encyclopedia of Personality and Individual Differences (pp. 2859-2860). Springer International Publishing. https://doi.org/10.1007/978-3-319-24612-3_1326 Xin, Y. P., Park, J. Y., Tzur, R., & Si, L. (2020). The impact of a conceptual model-based mathematics computer tutor on multiplicative reasoning and problem-solving of students with learning disabilities. The Journal of Mathematical Behavior, 58, 100762. https://doi.org/10.1016/j.jmathb.2020.100762 Zaldívar-Colado, A., Alvarado-Vázquez, R. I., & Rubio-Patrón, D. E. (2017). Evaluation of using mathematics educational software for the learning of first-year primary school students. Education Sciences, 7(4), 79. https://doi.org/10.3390/educsci7040079 Zhang, Y., & Wang, Q. (2020). Content learning opportunities, computer-based instruction, and students’ mathematics and science achievement. International Journal of Mathematical Education in Science and Technology, 51(8), 1164-1180. https://doi.org/10.1080/0020739X.2020.1717659 Zulnaidi, H., Oktavika, E., & Hidayat, R. (2020). Effect of use of GeoGebra on achievement of high school mathematics students. Education and Information Technologies, 25(1), 51-72. https://doi.org/10.1007/s10639-019-09899-y https://doi.org/10.29333/iji.2020.13453a https://doi.org/10.1088/1742-6596/1882/1/012060 https://doi.org/10.1088/1742-6596/1806/1/012072 https://doi.org/10.31764/jtam.v6i4.10865 https://doi.org/10.1016/j.compedu.2012.07.007 https://doi.org/10.1016/j.compedu.2020.103818 https://doi.org/10.1007/978-3-319-24612-3_1326 https://doi.org/10.1016/j.jmathb.2020.100762 https://doi.org/10.3390/educsci7040079 https://doi.org/10.1080/0020739X.2020.1717659 https://doi.org/10.1007/s10639-019-09899-y Volume 12, No 1, February 2023, pp. 101-116 115 Zulnaidi, H., & Zamri, S. N. A. S. (2017). The effectiveness of the GeoGebra software: The intermediary role of procedural knowledge on students’ conceptual knowledge and their achievement in mathematics. Eurasia Journal of Mathematics, Science and Technology Education, 13(6), 2155-2180. https://doi.org/10.12973/eurasia.2017.01219a https://doi.org/10.12973/eurasia.2017.01219a Tamur et al., Meta-analysis of computer-based mathematics learning in the last decade scopus … 116