Southeast Asian Mathematics Education Journal Volume 12, No. 1 (2022) 51 The Effect of Implementing STEAM and 4Dframe Learning in Developing Students’ Computational Thinking Skills 1 Rio Mardani Suhardi & 2 Gusnandar Yoga Utama 1 SMP Negeri 8 Batam, Indonesia 2 SEAMEO QITEP in Mathematics, Indonesia 1 riomardanisuhardi@gmail.com 2 gusnandaryogautama@gmail.com Abstract Computational thinking skills have been a popular term for teachers worldwide, and PISA 2022 will become the first PISA in evaluating them. Computational thinking helps students enhance their potential in contributing to other disciplines. However, students’ computational thinking skills at SMP Negeri 8 Batam were low. In overcoming the problem, the teacher employed STEAM learning as an alternative approach in stimulating students’ computational thinking skills. A teaching aid, named 4Dframe, was utilised to support the STEAM-based teaching. The objective of this action research study is to illustrate the effect of employing STEAM approach and the 4Dframe as the teaching assistance in developing students' computational thinking skills. The study involved 40 students of 9 th grade in SMP Negeri 8 Batam, Indonesia. Three STEAM activities incorporating Warka water tower, Batam-Bintan straw bridge, and planting machine were performed in eight online meetings. In each activity, the students administered decomposition, abstraction, pattern recognition, and algorithm as the cornerstones of computational thinking. The data were gathered through observational forms during the learning and test to evaluate students’ computational thinking skills. The results present that 73% and 88% of students acquired the minimum score for the computational thinking post-tests on the first and second cycle respectively. Although sample and methodology limitations prevent any claim to generalisation, this learning strategy could be implemented as an alternative for conducting mathematics learning activities in elevating students’ computational thinking skills with students in similar contexts. Keywords: 4Dframe, action research, computational thinking skills, STEAM Introduction PISA (2019) presented that, on average, Indonesian 15-year-olds scored 379 points in mathematics compared to an average of 489 points in OECD countries which places Indonesia in the rank of 70 out of the 77 OECD countries that is indeed significantly low. As a future challenge, for the first time, the PISA 2021 framework incorporated an appreciation of the collaboration between mathematical and computational thinking engendering a similar set of perspectives, thought processes and mental models which learners are necessary succeed in an increasingly technological world (PISA, 2018). Computational thinking is an innovative thinking ability in identifying life phenomena to provide various practical solutions toward the investigated problems (Fajri, Yurniwati, & Utomo, 2019). With the rapid flow of technological developments which elevate the economy competitiveness, research planners have concerned their efforts on equipping the younger generation to encounter future challenges through the development of computational thinking in the last 10 years (Khine, 2018). mailto:1riomardanisuhardi@gmail.com The Effect of Implementing STEAM and 4Dframe Learning in Developing Students' Computational Thinking Skills 52 Based on the researchers’ observations while teaching grade 9 th in SMPN 8 Batam, Indonesia, students were not used to problems and activities which aim at improving their computational thinking skills. It can be identified from the pre-test results of most students which were still low in solving problems associated with computational thinking. It is corroborated by the results of interviews with mathematics teachers who were teaching 9 th grade in SMPN 8 Batam in the previous school year. They asserted that students' computational thinking abilities were still low. Cuny, Snyder and Wing (2010) elaborated that computational thinking is a thinking process involved in formulating problems and solutions. Hence, it is easily performed solutions by an information processing agent. Others explained that computational thinking helps students to enhance their potential in contributing to other disciplines, particularly STEM (Sands, Yadav, & Good, 2018). There is a necessity to train student’s computational thinking, one of which is by implementing STEM learning. Science, Technology, Engineering, and Mathematics (STEM) are trending in 21 st century education. STEM education is crucial because it accommodates an interdisciplinary approach which plays a pivotal role for the future of a country (Gülhan & Şahin, 2018). While Baines (2019) argued that students possess a perception of STEM which tends to be boring and confusing. Therefore, the proper and wise utilisation of A (art) can make the STEAM learning more enjoyable for students. STEM learning is the current teaching method that the researcher performs in the classroom. Based on several observation results in grade 8 th for the 2020/2021 academic year, in general, students are tremendously interested in STEM, but some students seem less active during the activities. However, STEAM integrates artistic design, expression, reflective and multi-sensory appeal which requires art to associate (Daugherty, 2013). Therefore, the utilization of art increases the students’ attractiveness in being active during learning activities. This research was formulated to enhance students’ computational thinking skills through STEAM learning. Based on the researcher’s experience in teaching in the classroom, students experience difficulties developing their imagination. There is a need for teaching aids which are able to generate imagination and stimulate student's computational abilities. Therefore, the researcher attempts to employ the 4Dframe as a STEAM-based learning media. The 4Dframe was initiated by Hogul Park, a Korean engineer and model maker originally inspired by classical Korean architecture. The 4Dframe produces an advantage on its utilisation which is suitable to be administered as a learning media in schools which integrating science, technology, engineering, arts (including architecture or design), and mathematics (Park, 2018). The 4Dframe is an educational toolkit which assists students to develop creativity with their imagination. Thus, applying STEM and the 4Dframe learning is expected to enhance students’ computational thinking skills. Computational thinking is a cognitive or thinking process incorporating logical reasoning by which problems are solved, and artefacts, procedures and systems are better to comprehend (Csizmadia, Curzon, Dorling, Humphreys, Ng, Selby, & Woollard, 2015). It encompasses: (1) the ability to think algorithmically; (2) the ability to think in terms of decomposition; (3) the ability to think in generalisations, identifying and formulating Rio Mardani Suhardi & Gusnandar Yoga Utama 53 patterns; (4) the ability to think in abstractions, selecting appropriate representations; and (5) the ability to think in terms of evaluation. STEAM Education defines a variable as a characteristic which conveys a feature, useful or critical parameters, system elements when identifying a system, or when evaluating its performance, status, and condition. The application of STEAM educators incorporates three domains of teaching and learning which are pedagogy, assessment, and technology integration (Anito, Elipane, Sarmiento, & Butron, 2019). Because recognising design is the prior concern to understanding engineering, student engagement in this problem-solving process is crucial. The EiE (Engineering is Elementary) project generated a simple five-step engineering design process for children: Ask, Imagine, Plan, Create, and Improve (Hester & Cunningham, 2007). It also produced a series of questions to assist students to get through every step. Moving through the Engineering Design Process (EDP) involves asking the following questions or making the following decisions as displayed in Figure 1. Figure 1. Moving through the Engineering Design Process (EDP) (Hester & Cunningham, 2007). The 4Dframe is suitable for both individual and group activities. In addition to being a complex structural material, the 4Dframe has also been associated with various software advanced with the objective of providing visualising attractive geometrical models. When utilising the 4Dframe sets to implement geometric modelling in schools, students are able to learn various interdisciplinary topics in an active, meaningful and fun way. These topics accommodates the fields of art, architecture, global or local issues, socio-cultural, or transdisciplinary of all these with the implementation phenomena-based learning methods (Fenyvesi, Park, Choi, Song, & Ahn, 2016). The question which is formulated in this study is how can the implementation of STEAM and 4Dframe learning improve students’ computational thinking skills? Hence, this research The Effect of Implementing STEAM and 4Dframe Learning in Developing Students' Computational Thinking Skills 54 concentrated on how the implementation of STEAM and the 4Dframe learning can effectively enhance students’ computational thinking skills of 9 th graders at SMPN 8 Batam. Methods This research employed classroom action research by Stephen Kemmis and Robin McTaggart (Arikunto et al., 2017) with a qualitative approach. The objective of this study is to elaborate the thinking processes and student behaviour during the learning process, and to determine the effect of applying STEAM and the 4Dframe on student’s development of computational thinking skills. This classroom action research encompasses two cycles, each of which was conducted by employing four online meetings. Before the implementation phase, students first administered a pre-test to examine their initial abilities. The STEAM and the 4Dframe learning were performed in three meeting topics incorporating Warka Water Tower, Building Straw Bridge, and the Planting Machine. In the final meeting in every cycle, a post-test was performed to calculate the improvement of students’ computational thinking skills. In collecting the data, we developed several instruments such as Computational Thinking Skills Test obtained from Bebras, Indonesia Challenge and mathematics textbook grade 9. The test was utilised to gather data on students’ computational thinking skills. Furthermore, we also formulated questions for the interview to explore things which were not monitored during the observation and to identify the obstacles experienced by students during the phase of implementation. Interviews were conducted with several students based on the interview guidelines until the required data were fulfilled. To portray teachers’ actions and students’ responses during classroom activities, we distributed an observation sheet for teachers’ actions and students’ responses. The teachers’ actions refer to the steps of the STEAM EDP during learning activities. Meanwhile, students’ response is the response conveyed by students after receiving action from the teacher. The implementation of STEAM and the 4Dframe learning to enhance students’ computational thinking skills is considered successful if the average score of the students’ computational thinking skills has elevated from one cycle to the next cycle and at least, 80% of students pass the minimum score of 71. Results and Discussion Pre-test Result The pre-test was conducted on Wednesday, July 21 st 2021 in two hours. The test questions were distributed through Google Classroom along with work instructions. The results presented that the average score of students was 63.00 and the percentage of students who passed the minimum score was 38%. It implies that many students could not answer the questions based on computational thinking skills. Then, the researcher organized an orientation to the students by introducing the 4Dframe and explaining the application during STEAM learning activity. Rio Mardani Suhardi & Gusnandar Yoga Utama 55 Student Activities with STEAM and 4Dframe Learning Learning with STEAM and 4Dframe was conducted for six meetings. The topics provided to stimulate students’ computational thinking skills encompassed the Warka water tower, Batam-Bintan straw bridge and planting machines. The learning was performed in two cycles and the following is a summary of the learning. Warka Water Tower The Warka water tower is a water harvesting system which was designed by Arturo Vittori and Andreas Vogler to help Ethiopians produce clean water. This topic was selected as an initial form of learning orientation with 4Dframe teaching assistance to students. Figure 2. Students’ Warka water tower design process. At the beginning of the teaching and learning activities, the teacher directed students to practice the phases of the EDP. From the problem, several students asked questions about how Warka tower produces clean water. Based on the teacher’s explanation, students were instructed to imagine by making an ideal Warka water tower in accordance with the criteria and constrains. In the create phase, students were demanded to generate an ideal Warka water tower design plan illustrated by Figure 2 (a). Then, they employed 4Dframe to develop the two models of the water tower as displayed in Figure 2 (b) and (c). During this activity, students generated algorithmic thinking in which they establish Warka water tower step by step by implementing four frames by adjusting the designs they have formulated, and by generating abstraction thinking skills where students selected the right tubes and connectors of the 4Dframe. Algorithmic thinking is a method to obtain a solution through clear definition of steps, while the abstraction skill makes it easier to think about making artefacts more understandable by decreasing unnecessary details (Csizmadia et al., 2015). In the improve phase, the students performed a trial design, conducting reflection and improvement for the next cycle as presented in Figure 2 (c). Batam-Bintan Straw Bridge The topic of Batam-Bintan Straw Bridge was selected by considering the local issue. Kepulauan Riau provincial government plans to build a bridge which connects two islands, Batam and Bintan. The Effect of Implementing STEAM and 4Dframe Learning in Developing Students' Computational Thinking Skills 56 At the beginning of the lesson, students in groups were provided directions to determine the actual distance between Batam and Bintan island by utilising Google Maps. This stage stimulates students’ abstraction skills in which students chose necessary features on Google Maps app to identify the distance between two points. The learning was continued in accordance with the principles of the EDP in STEAM learning. It began with students observing the problem and asking some questions, such as how to connect the two regions to access links and economic equality. Then, in groups, students conducted brainstorming to provide several alternative solutions, until the best solution was selected to bridge the two areas. Figure 3. Students’ Batam-Bintan Bridge design process. Based on Figure 3 (a), students were instructed to design a bridge. Then, they made a prototype of the first design as displayed from Figure 3 (b). During this EDP stage, students developed their algorithmic thinking skills in creating ideal bridge. Furthermore, in improving the phase, they compared the two design (c), and then evaluated the experimental results, hence, they could be enhanced in the next cycle (d). These steps are tremendously beneficial for training students’ evaluation skills. Csizmadia et al. (2015) emphasised that evaluation is a process to ensure that a solution, whether an algorithm, system or process is fit for the purpose. Planting Machine To inspire sustainable living in agriculture, we required machines which are able to distribute seeds precisely. Therefore, the last topic selected was planting machines to make students more active during learning process. Rio Mardani Suhardi & Gusnandar Yoga Utama 57 Figure 4. Students’ planting machine design process. In the beginning, students were provided the opportunity to ask questions associated with the problems which are what the right solution is, and what the constraints are. Students in groups were discussing what the best solution was, until the planting machine was selected as the solution to create an environmental seed planting car as displayed in Figure 4 (a). With the contextual problems, students were asked to provide simpler solutions by creating a planting machine which is able to save natural resources as presented in Figure 4 (b). This design stage helps students to enhance decomposition skills. On the other hand, students also attempted to sharpen the generalisation skills through identifying problems and associating the solutions provided with the implementation of several disciplines in the fields of science, engineering, mathematics, and art (Csizmadia et al., 2015). In the create phase, students made two prototypes suitable with their designs by utilising the 4Dframe toolkit. Finally, both designs were then examined, compared and improved again in the next cycle as illustrated in Figure 4 (c) and (d). Reflection of Cycle I The objective of the reflection was to determine the success rate of the actions. This research was considered to be successful if the STEAM and the 4Dframe were at least in good category. The second criterion is that at least 80% of students who administered the test passed the minimum test score of 71. Based on the observer notes, the application of learning at the first, second, and third meetings was in good category. Thus, the implementation of learning has fulfilled the indicators of research success. Perceiving the implementation of learning, the results of the post-test in the form of students’ scores became a consideration for the success of the research. The test administered was a computational thinking ability test. The researcher determined a score to the students’ test results with a scale of 0 to 100. Students were considered complete if they acquired a minimum score of 71. Here, 29 students completed the final test and 11 students did not complete. Hence, the percentage of students who obtained minimum score was 73%. The Effect of Implementing STEAM and 4Dframe Learning in Developing Students' Computational Thinking Skills 58 As less than 80% of students passed the minimum score on the post-test of cycle I, the study did not fulfil the success indicators. Therefore, the study was continued to the cycle II. Before continuing the research to cycle II, researchers and observers evaluated the strengths and weaknesses of the learning in cycle I. Strengths in cycle I were maintained, while deficiencies in cycle I were enhanced. In the learning activities, students seemed enthusiastic. It can be identified from students who have attempted to create product designs before the learning started. Most of the students in the group also looked quite active during the group discussion. The application of 4Dframe was tremendously helpful for students to generate designs and to conduct design trials. By employing 4Dframe in implementing geometric modelling in school, students learn various interdisciplinary topics in an active, meaningful, and fun way (Fenyvesi et al., 2016). Thus, the utilisation of 4Dframe was maintained in cycle II. During group activities, there were some students who did not obtain a role and contribute to the group. It was caused by the large number of group members. Each group consisted of 5-6 people with heterogeneous skills. During the discussion, the active and high-achiever students were dominating, while others were passive. In overcoming this problem, the teacher demanded group members to pay attention to the group’s goals and to assign duties or responsibilities to every member. Another problem encountered in this cycle was the limited 4Dframe toolkit which was distributed to students. They could only create one product design. It was difficult to identify comparisons when a trial was conducted. Hence, the teacher provided an alternative solution by providing plastic straws and connectors such as 4Dframe made with thick paper. Reflection Cycle II The reflection cycle was conducted by observers. Researchers examined the results of observations, student interview data, and the results of the post-test. The percentage of students who acquired the minimum score was 88%. It presented an increase of 15% of students who passed minimum score. Therefore, the indicators of research success were fulfilled. Hence, the research was not continued to the next cycle. In the implementation of learning, there was no problem in implementing EDP steps and utilising the 4Dframe toolkit. Based on the results of observations on the teacher actions and student responses, the implementation of learning at the fifth and sixth meetings were considered very good. It can be observed from all the learning steps provided in the lesson plans which were performed by researchers very well. Research Findings The implementation of learning in cycle II was elevated in accordance with findings in cycle I. The results of observations on the application of learning in cycle I for the first, second, and third meetings were in the good category. Meanwhile, the observation results of the implementation in the second cycle for the fifth, sixth and seventh meetings were in the very good category. Rio Mardani Suhardi & Gusnandar Yoga Utama 59 The results of the post-test in the cycle II enhanced compared to the cycle I. In the cycle I, the average score of the students’ computational thinking test was 73%. Meanwhile, in the second cycle, the average score of the students’ computational thinking test was 88%. The 4Dframe-assisted STEAM learning had encouraged students to be enthusiastic. Furthermore, the learning was effective to encourage students in completing the worksheets. Based on interviews with some students, they enjoyed learning by utilising 4Dframes. STEAM learning is a lesson which was time consuming. It requires consideration from the teacher in presenting the number of problems in every meeting. Moreover, it was difficult for teachers to manage online classes with 40 students at the same time. Students with high abilities were more dominant in group activities and caused students with low abilities to be passive as they were not involved in the learning activities. In overcoming this problem, the teacher explained the groups of students about the importance of working in team and sharing roles in group activities. Improving Students' Computational Thinking Ability Computational thinking skills in this study were students’ abilities in (1) algorithmic thinking, (2) decomposition, (3) providing explanations, (4) generalisation, and (5) evaluation to determine the conjectures they formulated in solving related problems to mathematics material in STEAM learning of the three topics. Students' computational thinking ability was evaluated by providing a final test after students were provided an action in the form of a STEAM and the 4Dframe learning strategy. The objective of the test was to determine the success of the teacher’s actions in enhancing students’ computational thinking skills. The data for the final test of cycle I presented that 73% students or 29 of 40 students obtained a minimum score of 71. The test result of the first cycle increased compared to the results of the initial test conducted before performing the action. The results of the initial test revealed that only 38% of students passed the minimum score. Although there was an increase from the initial test to the final test of cycle I, the study was considered unsuccessful because the percentage of minimum passing score did not acquire 88%. In the final test of the second cycle, 88% of students or 35 of 40 students obtained a minimum score of 71. Thus, after being provided action in the second cycle, the research fulfilled the established research success indicators. One of the weaknesses in the first cycle is presented in Figure 5, which displays students’ answers in the post-test. Based on Figure 5 (a), student 1 experienced difficulty in abstracting questions. Meanwhile, in Figure 5 (b), student 2 was quite good at abstracting and generalising the questions provided. Student 2 performed basic algorithmic by following instructions by formulating a step-by-step solution. Algorithmic thinking is a method to identify a solution through a clear definition of steps (Csizmadia et al., 2015). The Effect of Implementing STEAM and 4Dframe Learning in Developing Students' Computational Thinking Skills 60 (a) (b) Figure 5. Students’ answers in cycle I. Translation Problem: Robi goes to Mira’s birthday. He cannot see colour very well. Yellow (C) will be seen as green (A). While blue (D) is seen as red (B). Robi holds up a row of balloons to welcome his guests. Challenge: Select two rows that look the same for Robi! Answer of the students in Figure 4 (a): answer A is the same as answer B Answer of the students in Figure 4 (b): If all C is replaced with A, you will obtain: A). A D A E D A F A B B). A B A E B A F A D C). A D A E B B F A A D). A B A E A B F A D If all D is replaced with B, you will obtain: A). A B A E B A F A B B). A B A E B A F A B C). A B A E B B F A A D). A B A E A B F A B Then, the correct answer is A and B In the first stage illustrated by Figure 5 (a), student 2 changed C to A with a series of balloons, then continued by changing D to B. It also presented the recursive strategy (decomposition) that student 2 had performed. Students 2’s abstraction ability was identified when she merely concerned on the requested balloon and ignored other balloons. At the end, student 2 formulated generalisations by adjusting the pattern she created with the answer choices. Rio Mardani Suhardi & Gusnandar Yoga Utama 61 Another given problem is about cone. Given a chocolate shaped cone divided into four parts A, B, C, and D. The height for each part is x. The students will determine (a) the ratio between area surface A to area surface B, (b) the ratio between area surface B to area surface C, and (c) the ratio between area surface C to area surface D. Figure 6 illustrates one of answers of the students. Figure 6. The answer of student 3 for the first question. Based on the problem, student 3 obtained the equation to identify the surface area of A which is the area of the blanket of cone A minus the area of the blanket of cone B (algorithmic thinking) that is Other than obtaining the equation, the student was also explaining the work in details as shown by Figure 7. Figure 7. Student 3 elaborated the first question. Translation Explanation: We understand that: height of cone A:B:C:D = 4x:3x:2x:x or it can be simplified to 4:3:2:1. By employing the principle of congruence, it identified that the ratio between the radius and The Effect of Implementing STEAM and 4Dframe Learning in Developing Students' Computational Thinking Skills 62 the height of each cone is the same. The ratio of the radius of the cone A:B:C:D = 4:3:2:1 or it could be the same as 4x:3x:2x:x. A = 4, B = 3, C = 2, D = 1. With all this information, it can be implied that the surface areas of A, B, C, and D are the area of the blanket. Area A = Area of blanket of cone A - Area of blanket of cone B Area B = Area of blanket of cone B - Area of blanket of cone C Area C = Area of blanket of cone C - Area of blanket of cone A On the other hand, one of the successful factors of this research is displayed in the Figure 8. The first thing that student 3 performed in answering the question was to simplify the equation (decomposition) of the comparison of shapes A, B, C and D: to By utilizing the concept of similarity (abstraction), student 3 obtained the ratio of the radii of the A, B, C, and D planes as well: . Figure 8. The answer of student 3 for the second and third question. These solutions were also applied to solve the second and third question by replacing each corresponding variable (generalisation). The ability to generalise can be observed when students are able to adapt a solution, or part of a solution, to be implemented to a whole class of similar problems (Csizmadia et al., 2015). Conclusion Based on the data exposure and discussion, and the methodological limitations due to the size of the sample, the steps of STEAM and the 4Dframe learning were able to enhance students’ thinking and computation abilities by implementing each stage in the EDP. Based on this research, the following suggestions may be of interest to readers with similar teaching-learning contexts. 4Dframe-assisted STEAM learning is a learning strategy which is able to elevate students’ computational thinking skills and to be employed as an alternative in learning mathematics. Learning with the 4Dframe-assisted STEAM strategy is easier to apply if the learning is conducted face-to-face. When it is performed virtually, there are several problems might occur such as limited student access to the internet, and the students were reluctant to discuss in groups. Rio Mardani Suhardi & Gusnandar Yoga Utama 63 References Anito, J. J., Elipane, L., Sarmiento, C., & Butron, B. (2019). The Philippines STEAM education mode. Philippines: PNU University Publication Office. Arikunto, S., Suhardjono, & Supardi. (2017). Penelitian tindakan kelas. PT. Bumi Aksara. Baines, A. S. (2019). STEAM education theory and practice. Cham: Springer. Csizmadia, A., Curzon, P., Dorling, M., Humphreys, S., Ng, T., Selby, C., & Woollard, J. (2015). Computational thinking a guide for teachers. London: Computing at School. Cuny, J., Snyder, L., & Wing, J. (2010). Demystifying computational thinking for non- computer. Retrieved from http://www.cs.cmu.edu/~CompThink/resources/TheLinkWing.pdf Daugherty, M. K. (2013). The Prospect of an “A” in STEM Education. Journal of STEM Education: Innovations and Research, 14(2), 10–15. https://www.jstem.org/jstem/index.php/JSTEM/article/view/1744/1520 Fajri, M., Yurniwati, & Utomo, E. (2019). Computational thinking, mathematical thinking. Dinamika Matematika Sekolah Dasar, 1(1), 1–18. Fenyvesi, K., Park, H.G., Choi, T., Song, K., & Ahn, S. (2016). Modelling environmental problem-solving through STEAM activities: 4Dframe’s Warka water workshop. Bridges Finland Conference Proceedings, pp. 601–605. Retrieved from https://jyx.jyu.fi/handle/123456789/50739 Gülhan, F., & Şahin, F. (2018). Activity implementation intended for STEAM (STEM+Art) education: mirrors and light. Journal of Inquiry Based Activities, 8(2), 111–126. Hester, K., & Cunningham, C. (2007). Engineering is elementary: an engineering and technology curriculum for children. Presented at ASEE Annual Conference and Exposition. Retrieved from https://www.eie.org/sites/default/files/research_article/research_file/ac2007full8.pdf Khine, M. S. (2018). Computational thinking in the STEM disciplines. Springer. https://doi.org/10.1007/978-3-319-93566-9. Park, H. (2018). Mathematics learning through arts, technology. 8th ICMI-East Asia Regional Conference on Mathematics Education, p. 111. Taiwan. PISA. (2018). PISA 2022 Mathematics Framework. Retrieved from https://pisa2021- maths.oecd.org/files/PISA%202021%20Mathematics%20Framework%20Draft.pdf PISA. (2019). PISA 2018 results. Retrieved from https://www.oecd.org/pisa/publications/pisa-2018-results.htm Sands, P., Yadav, A., & Good, J. (2018). Computational thinking in K-12. In M. S. Khine (Eds.), Computational Thinking in the STEM Disciplines, (pp. 154–164). USA: Springer International Publishing. The Effect of Implementing STEAM and 4Dframe Learning in Developing Students' Computational Thinking Skills 64