Turk, N., Kalayci, N., Yamak, H. (2021). Evaluation of the effectiveness of science, technology, engineering and mathematics (stem) curriculum designed and implemented for undergraduate programs of faculty of education. International Online Journal of Education and Teaching (IOJET), 8(3). 1401-1428. Received : 12.08.2020 Revised version received : 04.04.2021 Accepted : 07.04.2021 EVALUATION OF THE EFFECTIVENESS OF SCIENCE, TECHNOLOGY, ENGINEERING AND MATHEMATICS (STEM) CURRICULUM DESIGNED AND IMPLEMENTED FOR UNDERGRADUATE PROGRAMS OF FACULTY OF EDUCATION Research article Corresponding author Nilay Türk (https://orcid.org/0000-0001-5936-9003) Ministry of National Education turknilay@gmail.com Nurdan Kalaycı (https://orcid.org/0000-0003-1982-2410) Gazi University nurdankal@yahoo.com Havva Yamak (https://orcid.org/0000-0002-1666-0746). Gazi University havvademirelli@gmail.com Nilay Türk is currently math teacher at the Ministry of National Education, Turkey. She conducts research on teacher training, learning and teaching methods, active teaching methods, STEM education and STEM teacher education. Nurdan Kalaycı is currently Professor of Curriculum and Instruction at the Department of Educational Sciences, Faculty of Education, Gazi University, Turkey. She conducts researches on learning and teaching processes, active teaching methods, higher education, quality in higher education, gender equality and education. Havva Yamak is currently Professor of Science Education at the Department of Mathematics and Science Education, Faculty of Education, Gazi University, Turkey. She conducts researches on teacher training, science education, chemistry education, technological pedagogical content knowledge, STEM education and STEM activities. Copyright © 2014 by International Online Journal of Education and Teaching (IOJET). ISSN: 2148-225X. Material published and so copyrighted may not be published elsewhere without written permission of IOJET. https://orcid.org/0000-0001-5936-9003 mailto:turknilay@gmail.com https://orcid.org/0000-0003-1982-2410 mailto:nurdankal@yahoo.com mailto:havvademirelli@gmail.com http://orcid.org/xxxx http://orcid.org/xxxx http://orcid.org/xxxx Turk, Kalayci, & Yamak 1402 EVALUATION OF THE EFFECTIVENESS OF SCIENCE, TECHNOLOGY, ENGINEERING AND MATHEMATICS (STEM) CURRICULUM DESIGNED AND IMPLEMENTED FOR UNDERGRADUATE PROGRAMS OF FACULTY OF EDUCATION Nilay Türk turknilay@gmail.com Nurdan Kalaycı nurdankal@yahoo.com Havva Yamak havvademirelli@gmail.com Abstract The aim of this study was to evaluate the effectiveness of STEM Curriculum designed and implemented for undergraduate programs of faculties of education. The research model is experimental. The design is a single group pre-test post-test experimental design. The study group consisted of 18 pre-service teachers studying in the Department of Science Education. In order to determine the effectiveness of STEM Curriculum, two data collection tools were prepared: "Education Module Evaluation Form" and "Interview Form". The Wilcoxon Signed Ranks Test was used to analyze the quantitative data and the content analysis was used to analyze the qualitative data. According to the findings, it was concluded that the STEM curriculum applied positively influenced the development of the competency of pre-service teachers to prepare education module (lesson plans, work sheets, measurement-evaluation tools) according to STEM education approach. It was concluded that the STEM curriculum contributed to the knowledge and skills, professional competencies and awareness of pre- service teachers about preparing educational material according to STEM education approach. Keywords: STEM education, STEM curriculum, teacher education, pre-service teacher education, program development 1. Introduction Education represents an open and dynamic system by its nature, therefore it is in contact with other systems that guide society. As the need of the individual and society is met with education, changes and other systems affect the education system (Okçabol, 2007, p. 25). One of the main objectives of education is to educate the workforce capable of maintaining the social life order in harmony with economic, social, scientific and technological developments and changes and ensuring the development of the society. One of the most important tools that enable a country to achieve this goal is curriculums. Curriculums should be prepared to meet the social, economic, cultural and global needs of both individuals and society and to prepare individuals to adapt to changes and developments occurring at local mailto:turknilay@gmail.com mailto:nurdankal@yahoo.com mailto:havvademirelli@gmail.com International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1401-1428. 1403 and global scale (Gömleksiz et al., 2005, p. 2; Özdemir, 2011, p. 94). With the increasing importance given to curriculums, changes and developments in the world have led to changes in the education approach and teaching methods in the programs and brought along new approaches and methods. When the new teaching practices used throughout the world are examined, it is seen that student-centered approaches/ methods/ techniques are preferred where interdisciplinary approach is predominant. Recently, STEM education has come to the forefront with the introduction of engineering education in schools, considering that engineering will provide a good environment for mathematics, science and technology education (Akgündüz et al., 2015, p. 5). The main purpose of STEM education is to enable students to learn about the world in which they live in an integrated way instead of decomposed learning (Dugger, 2010, p. 2). STEM education is an approach based on the integration of STEM fields (Bybee, 2013; Moore et al., 2014; Sanders, 2009). STEM training is also preferred because it provides the preparation of the workforce that will develop and sustain national economies in a continuously expanding and globalizing economy (Kelley & Knowles, 2016, p. 2). Higher education institutions are important because they train teachers to implement STEM education, provide academic support to the trainers involved in the implementation of STEM education approach, and train qualified workforce needed by the society with pre-service training provided in STEM fields (Erdem, 2013, p. 114). Most of the responsibility for training the workforce of a country belongs to teachers. The ability of teachers to fulfill this responsibility properly depends on the quality of the their education. Therefore, the education faculties of universities and experts and institutions that develop the programs of education faculties have great responsibilities. The quality of workforce will increase at the same rate as pre-service teachers being educated in high quality and in a good environment (Erişen, 2001, p. 7). Since curriculums are one of the important elements that make higher education a powerful, successful and effective one (Barnett & Coate, 2005), the development of undergraduate programs of the faculty of education in a way that will enable pre-service teachers to acquire the competencies expected from them will positively affect the success of STEM education applied in educational institutions. The role of the teacher in the classroom has started to change with the application of approaches/ methods such as STEM education that center the student and design the learning process with an interdisciplinary/ transdisciplinary approach in the classrooms. However, some studies on the teachers show that teachers have no information about new methods and they need in-service training on these methods (Asilsoy, 2007, pp. 112-114; Memişoğlu, 2008, p. 138). But teachers work in different learning environments today, they have different and current professional needs (Livingston, 2017, p. 141). Teachers who will implement STEM education need lessons and workshops that demonstrate how to integrate STEM areas while collaboratively solving the real-life problem (Shernoff et al., 2017, p. 13). For the success of STEM, it is very important that the teachers who will design the learning process from beginning to end have content knowledge and pedagogical content knowledge about STEM. On the other hand, Çorlu (2012) states that the teachers who will apply STEM education should have integrated teaching knowledge. This knowledge may help teachers to teach in the best way students STEM disciplines that they can use in their daily lives and occupational choices (Holdren et al., 2010, p. 57). Thus, students will be able to learn the concepts of science, technology, engineering and mathematics successfully and find solutions to the problems of the real world. Turk, Kalayci, & Yamak 1404 Within the scope of the project which was carried out by STEM centers of universities, educations that promote the STEM education and workshops are organized. With the STEM center established by Istanbul Aydın University, it aims to increase the competence of teachers and students in STEM fields and to contribute to the transformation of schools into STEM schools (STEM School, 2018). In the STEM Center (BAUSTEM) established in Bahçeşehir University, a STEM curriculum has been developed for primary schools. In addition, trainings are given to increase the STEM competencies of teachers and STEM research is carried out (Bahçeşehir University, 2018). Hacettepe Science, Technology, Engineering and Mathematics Education and Applications Laboratory (Hacettepe STEM & Maker Lab) since 2009, such as Advanced Applications in Science Teacher Education (S- TEAM), Assessment Strategies in Research-Based Science Learning (SAILS) and Mathematics and Science for Life (MASCIL) projects based on innovative approaches were carried out. In addition, STEM & MakersFest Expo is held every year, hosted by Hacettepe University, where conferences with participants from different institutions and workshops for teachers and students are held (H-STEM Lab, 2018). Teacher workshops and projects are carried out in the BILTEMM Education Application and Research Center within the Middle East Technical University (METU) to improve the educational opportunities offered to schools, teachers and students (METU, 2018). STEM laboratory was established within Muş Alparslan University in the fall term of 2013-2014. Within the scope of this laboratory, pre- service science teachers are trained (Yıldırım, 2016, p.36). The fact that the studies are teacher-dominated causes the criticism that the pre-service teachers are not educated in accordance with the requirements of the age. Bers and Portsmore (2005) emphasized the importance of pre-service teachers' learning new approaches and methods in their study. After the announcement of STEM education approach, it started to be applied in schools and the idea that pre-service teachers should be trained with the necessary professional knowledge, skills and experience to apply this approach started to prevail. In some published reports (Akgündüz et al., 2015; Holdren et al., 2010; MOE, 2016; TÜSİAD, 2014;) and in researches (Akaygün & Aslan Tutak, 2016; Altunel, 2018; Gökbayrak & Karışan, 2017; Kim et al, 2015; Koehler, Feldhaus, Fernandez, & Hundley, 2013; Marulcu & Sungur, 2012; Yildiz, Alkan, & Cengel, 2019) it was stated that STEM education approach should be included in the undergraduate programs of the education faculty in order for pre-service teachers to successfully apply STEM education. When the curriculums of the lessons related to STEM fields published by the Ministry of National Education in 2018 are examined, it is seen that the interdisciplinary approach is taken into consideration, and the activities aiming to the development of products by integrating the information about the learning outcomes of each unit under the name of “Science, Engineering and Entrepreneurship Applications" are included in the Science Curriculum. With this study, the Ministry of National Education initiated the transition to STEM education in the curriculum. The undergraduate programs of the Faculty of Education, which are of great importance in the training of pre-service teachers who will be the implementers of the programs, are also gaining importance in this context. When the course names and course contents of the updated undergraduate programs (Science, Mathematics, Computer and Instructional Technology, Physics, Chemistry, Biology) were examined, it was determined that there were no courses related to STEM education (Council of Higher Education, 2018). Çolakoğlu and Günay Gökben (2017) state that although various studies have been carried out within the STEM centers/ laboratories in universities, no progress has been achieved in terms of STEM education yet, and no undergraduate or graduate education programs related to STEM have been opened in any faculty of education. This situation necessitates the opening of a course on STEM education for the undergraduate programs of faculties of education and designing a curriculum that includes all aspects of this course. For International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1401-1428. 1405 this reason, STEM Curriculum was designed by researchers for undergraduate programs of faculties of education in the fall semester of 2016-2017 and applied in the spring semester of 2016-2017. The curriculum is applicable worldwide, as it includes international STEM activities that can be used in all education faculties, and it is prepared with international literature in mind. The aim of this study is to evaluate the effectiveness of STEM Curriculum designed for undergraduate programs of faculties of education. The research questions related to the aim of the research are given below. 1. Is there a significant difference between the pre-test and post-test scores of the education modules prepared by pre-service teachers according to STEM education approach? 2. What are the pre-service teachers' opinions on the STEM curriculum? 3. What are the pre-service teachers' suggestions for STEM curriculum? 4. What are the pre-service teachers' opinions on the contribution of STEM curriculum to their future teaching career? 2. Methodology 2.1. Model The research model is experimental. The experimental design is a single group pre-test and post-test experimental design. This design is often used in research on the benefits of new curriculum (the design of curriculum) or the importance of the new teaching method (Cohen et al., 2006, p. 212). In this design, the significance of the difference between the pre- test and post-test values of a single group is tested (Büyüköztürk et al., 2014, p. 201). In this context, quantitative data were obtained within the scope of pre-test and post-test. The change between these data was examined. The independent variable of the research was the STEM curriculum and the dependent variable was the competence to design the learning-teaching process according to the STEM education approach. After the curriculum was implemented, qualitative data were obtained through interviews with pre-service teachers. The interview technique was used in order to determine the opinions and suggestions of the pre-service teachers on the dimensions of the STEM curriculum (content, learning-teaching process, measurement-evaluation) and their professional contribution. Interviews using semi- structured interview forms allow the questions to be re-organized and discussed by providing flexibility during the interview, even though the questions are ready in advance, they allow specific information to be collected from the participants, and the majority of the form consists of questions to be clarified (Ekiz, 2003, p.62; Merriam, 2009, p.87). While these interviews provide fixed choice answering, they also enable in-depth research in the relevant field (Büyüköztürk et al., 2014, p.150-152). Since qualitative and quantitative data were needed in the research, the mixed method was used in the study. The study was conducted using a multiphase design of mixed method research. The multiphase design is based on examining the problem through the cycle of quantitative and qualitative research in which the next method is performed sequentially, building on the learning from the previous method to examine the goal of a program. The multiphase design provides an inclusive methodology framework for long-term multiphase research aimed at developing a research or evaluation program (For example, needs analysis, program development and program evaluation studies in the context of program evaluation are multiple stages) (Creswell & Plano Clark, 2011, p. 100). The doctoral dissertation in which this research was produced consisted of three stages: The first stage was the needs analysis conducted in order to design the STEM curriculum, the second stage was application Turk, Kalayci, & Yamak 1406 of the STEM curriculum and the third stage was the interviews with the pre-service teachers, for whom the STEM curriculum was applied. 2.2. Study Group The study group of the research consists of all 18 pre-service teachers who choose Chemistry Applications (Elective) course from the second grade students of Gazi University Faculty of Education, Department of Science Education in the spring semester of the 2016- 2017 academic year. Appropriate sampling method, one of the random sampling methods, was preferred in the formation of the study group. The advantage of this sampling is that it provides speed, practicality and convenience to the researcher (Fraenkel & Wallen, 2009, pp. 98-99; Yıldırım & Şimşek, 2013, p. 141). As the department thought that taking pre-service teachers from the main courses would create a problem, it was deemed appropriate to select pre-service teachers who took elective courses for the study as the study group. It was decided to conduct the study at the second grade level, since the education faculty undergraduate programs only have elective courses in the second and fourth grade and fourth grade students concentrate on the KPSS exam. In the first week, in order to learn the personal information of pre-service teachers, Pre- service Teacher Information Form was used. The form consisted of four questions. The first question was about identity information, the second question was gender, the third question was about the high school they graduated from, and the fourth question was about the state of being aware of the concept of STEM education (Have you heard about STEM education concept before? Yes/ No. If your answer is yes, explain this concept.). If the pre-service teachers marked yes in the fourth question, they were asked to explain the reason for the answer. The pre-service teachers' information is shown in Table 1. Table 1. Information of study group pre-service teacher Total Pre- service Teacher (n) Gender Type of High School Graduated Awareness of STEM Education F e m a le M a le N o rm a l H ig h S c h o o l A n a to li a n H ig h S c h o o l A n a to li a n T e a c h e r H ig h S c h o o l S c ie n c e H ig h S c h o o l P ri v a te H ig h S c h o o l O th e r Y e s N o 18 15 3 1 12 4 - 1 - - 18 All of the pre-service teachers answered "No" to the fourth question. For this reason, it was determined that all pre-service teachers did not have any knowledge of STEM education before the STEM curriculum was applied. International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1401-1428. 1407 2.3. Data Collection Tools and Data Collection 2.3.1. Education module evaluation form The form was prepared by the researcher in order to determine the suitability of the education modules (they include lesson plans, worksheets, measurement and evaluation tools, and related explanation and content information) prepared by the pre-service teachers to the STEM education approach. The form consists of three parts: “general evaluation form”, “lesson plan evaluation form” and “supplementary opinion”. The explanations for these sections are given below. The General Evaluation Form was prepared in the form of a three-level checklist consisting of three criteria in order to assess the general information status of the module prepared. The Lesson Plan Evaluation Form consisted of two parts. The first part was prepared as a two-level checklist consisting of ten criteria in order to evaluate the prepared lesson plans structurally and the second part was prepared as a five-level scoring key consisting of forty-seven criteria in order to determine the suitability of the prepared lesson plans to STEM education approach. The Supplementary Opinion section was designed to allow assessors to express their opinions if they wish to make evaluations or provide opinions outside the specified criteria for the module. In order to ensure the content validity of the evaluation form, a table of specifications was prepared including the dimensions that should be included in STEM lesson plan and the characteristics of these dimensions. Then, five field experts (one measurement and evaluation expert, two curriculum and instructions, two STEM education experts) were asked to evaluate by the giving the evaluation form and the table of specifications. Experts stated whether each item in the scale met the characteristics in the table, and if not, how it could be improved. Necessary corrections were made on the scale within the framework of the opinions of the experts. In order to ensure the face validity of the evaluation form, the opinions of the five field experts mentioned above were taken. As a result of the interviews, experts expressed that the expressions of the criteria in the form should be similar. Similar expressions were used in all criteria, taking into account expert opinions. Before the form was used in the study, a pilot application was performed to test its reliability. Using this form, two field experts-studying in the field of STEM education- evaluated ten STEM lesson plans published for free on the web. Shapiro-Wilk analysis was used to test whether the scores obtained from the experts showed normal distribution (Expert 1 = 0.605; Expert 2 = 0.955; p> .05). After determining the normal distribution of the total evaluation scores of the experts, "Pearson Moments Multiplication Correlation Coefficient" (r = 0.87; p <0.01) and "Class Correlation Coefficient" (r = 0.86; p <0, 01) were calculated. When the scores obtained were examined, it was found that the correlation was high and therefore there was consistency between the total scores of the experts. In order to investigate the reliability of the form in more detail, Cohen's Kappa value, which enables the determination of the compliance ratio of the experts in each criterion, was also calculated. When the obtained value (k = 0.81) was examined, it was determined that the level of consistency among the scores of the experts for each criterion was high. According to the results mentioned above, the evaluation form was found to be reliable and it was decided that it was suitable for the study. In order to ensure reliability, before using the evaluation form, experts were informed on how to use the form. 2.3.2. Interview form The form was prepared as a semi-structured interview form by the researcher in order to determine the opinions and suggestions of the pre-service teachers about the dimensions of Turk, Kalayci, & Yamak 1408 STEM curriculum and professional contribution. The interviews conducted using semi- structured interview forms are directed around the questions and the subject(s) to be investigated are revealed (Merriam, 2009, p. 88). While preparing the interview questions, in order to ensure the content validity, the opinions of experts from each of the fields of measurement and evaluation, curriculum and instruction, and science education were consulted, and care was taken to ensure that the questions to be included in the form were in accordance with the general objective and sub-objectives of the study. In line with the opinions of experts, a question regarding the professional contribution of the program was added. After the interview form was prepared, the form was finalized according to the feedback received as a result of the pilot application on three pre-service teachers. 2.4. Data Analysis 2.4.1. Analysis of quantitative data Only quantitative data were obtained from the Education Module Evaluation Form. SPSS 20 package program was used in the analysis of quantitative data. The Shapiro-Wilk test was used to check whether the scores obtained from the form meet the assumption of normality. According to the data obtained, it was determined that the data were not distributed normally (Pre-test = 0.808; Post-test = 0.799; p <.05). Therefore, Wilcoxon Signed Ranks Test, which is one of the nonparametric tests, was used in the analysis of the data. In this test, the scores obtained from measurements made on the same subjects at two different times are analyzed (Büyüköztürk, 2014, p. 174). When the education module evaluation form was used in the research process, the reliability of the research data was tested again by reliability analyzes. The form was used as a pre-test and post-test by three experts (two experts in curriculum and instruction, one expert in STEM education and developing lesson plans) within the scope of the research. Correlation coefficients of the scores obtained from three experts for pre-test and post-test were calculated. The intraclass correlation coefficient of the scores obtained from three experts within the scope of the pre-test and post-test was calculated. When the obtained correlation coefficients (rpre = 0.90; rpost = 0.91; p <0.01) were examined, it was found that there was consistency between the scores given by the experts. In addition, the Fleiss' Kappa value, which allows the experts to look at the compliance rate in each criterion, was calculated for a more detailed investigation of the reliability of the scores obtained from the form. When the values obtained (kpre = 0.84; kpost = 0.83) were examined, it was determined that the level of agreement was high among the scores given by the experts for each criterion. According to the results, it was determined that the data obtained from the evaluation form for the research were reliable. 2.4.2. Analysis of qualitative data Qualitative data analysis is a process that involves combining and interpreting the statements of the participants with the findings of the researcher regarding those statements. This process is also a process of making sense (Merriam, 2009, p.167). Content analysis was preferred among the qualitative data analysis approaches in the study. In content analysis, similar data are brought together within the framework of certain codes/ concepts and themes and they are comprehensively edited and interpreted (Yıldırım & Şimşek, 2013, p. 259). Content analysis was performed on the data obtained by following the steps indicated below: The data obtained for each research question were combined, the answers given were grouped according to their similarities, and codes similar to the original views were created. The codes were determined by considering the sub-problems of the research and the International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1401-1428. 1409 literature. After all the codes were determined, the data set was re-read and reviewed and coding was made to reflect the data in the best way. Later, the interrelated ones among these codes were brought together, and themes were determined in a way that could explain, cover and reflect all related codes. This process is thematic coding. The data collected in thematic coding is categorized through codes (Yıldırım & Şimşek, 2013, p. 268-269). After the codes and themes were created, their frequencies were determined. Codes were listed in descending order under the themes they belong to according to their frequencies. In this research, all steps related to the selection of the study group, preparation of data collection tools, data collection and analysis were explained in detail and thus external validity was tried to be ensured. Three types of data triangulation (triangulation) -the use of different data collection methods (two different data collection tools were used), multiple data sources (data collected from pre-service teachers at different times) and comparison of findings (findings obtained from different data collection tools were compared and interpreted)- determined by Denzin (1978) were done. The opinions of the two experts were taken about the tables created in the research. After the data analysis, the findings were examined by the participants and their opinions about reflecting the opinions of the participants were obtained. Some statements made to ensure the validity of the research are effective in ensuring reliability (Lincoln & Guba, 1985; Miles et al., 2013). External reliability was tried to be provided by making a detailed explanation about the process of the study. In order to ensure internal reliability, the data obtained by qualitative methods were tried to be explained with direct excerpts from the verbal and written statements of the participants. The data obtained from the study were analyzed by two experts. The consistency of the codes used by the two researchers was determined using Miles and Huberman's (1994) Consensus / (Consensus + Disagreement) x100 formula. The reliability percentage of the pre-service teacher interview form among the coders was calculated as 87%. 2.5. Designing the STEM Curriculum The STEM Curriculum applied in the research was designed according to Taba's curriculum development model (Ornstein & Hunkins, 2009, p. 199). Firstly, the needs analysis, which is the first stage of the program development process, was conducted. In the study, democratic approach, which is one of the needs analysis approaches, was preferred. It was also made of document analysis examining the faculty of education graduate programs in Turkey and abroad. As a result of the needs analysis, it was found out that pre-service teachers did not know STEM education approach and teacher education programs did not include courses aimed at acquiring integrated teaching knowledge (Türk et al., 2018). Based on the identified needs, the learning outcomes of curriculum were determined. The content covering the theoretical knowledge required for the pre-service teachers of science teaching department to associate their fields with other STEM fields was selected. Student-centered approaches, strategies, learning models, teaching methods and techniques were identified for use in the learning-teaching process, and collaborative studies and practices were included. After the implementation of the curriculum, how to evaluate pre-service teacher was determined and related data collection tools were prepared. While designing the curriculum, Robert's (2013 ) suggestions for using it in the training of pre-service teachers who can apply integrated STEM education were taken into consideration (planning an integrated STEM lesson, choosing design tasks that integrate STEM content, finding solutions to problems using the engineering design process, supporting the Turk, Kalayci, & Yamak 1410 experiential learning environment, providing a cooperation environment for solving problems by applying STEM concepts etc.). A total of four experts, two from the field of curriculum and instruction and two from the field of science education (who work on STEM education), evaluated the STEM curriculum designed. As a result of the evaluation, the program was finalized according to the opinions of the experts. 2.6. Implementation of the STEM Curriculum The STEM curriculum was designed in accordance with the modular programming approach. The 14-week curriculum included 11 modules for two hours of courses per week. Modules: “Introduction”, “STEM Education”, “Inquiry Based Learning and Problem Based Learning”, “Project Based Learning, Creativity and Innovation”, “Educational Technologies”, “Mathematical Modeling”, “Engineering Design Process”, “STEM Activity Application-1”, “STEM Activity Application-2”, "Measurement and Evaluation in STEM Education”, “Preparing the Education Module”. The prepared curriculum was applied to the study group in weekly order and the effectiveness of the curriculum was evaluated according to the data obtained. 3. Findings 3.1. Change in Education Module Evaluation Scores The products prepared by the study group were evaluated by experts before and after the STEM curriculum was implemented. Table 2 shows the Wilcoxon signed rankings test to determine whether there is a significant difference between the total scores obtained from the evaluation form. Table 2. Comparison of pre-test and post-test total scores of pre-service teachers' module evaluation form Post-test – Pre-test N Rank average Z P D Negative sequence 0 ,00 -3,750* ,000 0,88 Positive sequence 18 9,50 Equal 0 - Total 18 * Based on negative sequences When the analysis results given in Table 2 were examined, it was found that there was a statistically significant difference between the pre-test and post-test total scores of the education modules of the study group (Z = 3,750; p <, 01). Considering the mean of difference scores, the observed difference was found to be in favor of the post-test score. When the effect size value (d = 0.88>, 80) was examined, it was found that the applied curriculum affected the pre-service teachers at a high level. In addition, the median values of the pre-test and post-test total scores were calculated, and when these values were compared (Median pretest = 23.00; Median posttest = 95.50), it was determined that the post-test scores of the pre-service teachers increased significantly. Table 3 shows the Wilcoxon signed rankings test to determine whether there is a significant difference between the pre-test and post-test scores obtained from the sub- dimensions of the education module evaluation form. International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1401-1428. 1411 Table 3. Comparison of pre-test and post-test scores of pre-service teachers on sub- dimensions of module evaluation form Sub-dimensions Post-Test–Pre-Test N Rank average Z P D 1. General Evaluation Form Negative sequence 0 ,00 - 3,750* ,000 0,88 Positive sequence 18 9,50 Equal 0 2.Lesson Plan Evaluation Form Section A Negative sequence 4 2,50 - 2,512* ,012 0,59 Positive sequence 9 9,00 Equal 5 Section B General Negative sequence 0 ,00 - 3,792* ,000 0,89 Positive sequence 18 9,50 Equal 0 Learning Outcome Negative sequence 0 ,00 - 3,792* ,000 0,89 Positive sequence 18 9,50 Equal 0 Content Negative sequence 0 ,00 - 3,792* ,000 0,89 Positive sequence 18 9,50 Equal 0 Learning- Teaching Process Negative sequence 0 ,00 - 3,750* ,000 0,88 Positive sequence 18 9,50 Equal 0 Measurement and Evaluation Negative sequence 0 ,00 - 3,750* ,000 0,88 Positive sequence 18 9,50 Equal 0 Additional Information Negative sequence 0 ,00 - 3,906* ,000 0,92 Positive sequence 18 9,50 Equal 0 * Based on negative sequences According to the analysis results given in Table 3, it was found that there was a statistically significant difference between the pre-test and post-test scores of the sub- Turk, Kalayci, & Yamak 1412 dimensions of the education module evaluation form of the study group. Considering the mean of difference scores, the observed difference was found to be in favor of the post-test score. When the effect size values of the pre-test and post-test scores of the sub-dimensions of the education module evaluation form were examined in Table 3, it was determined that the curriculum affected the pre-service teachers within the scope of the lesson plan evaluation form part A (d = 0.59) at the middle level, while it affected the other sub-dimensions at the high level (d> 0,80). According to the findings, it was found out that the designed curriculum was effective in the development of the pre-service teachers' competences in preparing the education module according to STEM education approach. Arslan (2018) also found that at the end of the course in which he conducted STEM-oriented laboratory practices, the pre-service science teachers improved their competence in planning and implementing activities for STEM education. Rinke et al. (2016) in the study, which examined the results of STEM teacher education model built on STEM-specific teacher education principles, they determined that pre-service teachers who took STEM course obtained more gains compared to pre-service teachers who took traditional course. König et al. (2020) found that pre-service teachers' lesson planning skills improved. 3.2. Pre-service Teachers' Opinions on STEM Curriculum Pre-service teachers were asked questions about the dimensions of STEM curriculum. The findings were presented as subheadings separately. 3.2.1. Pre-service teachers' opinions on the content of STEM curriculum Pre-service teachers were asked the question "What are your opinions on the content of STEM curriculum?". The findings are presented in Table 4. Table 4. Pre-service teachers' opinions on the content of STEM curriculum Theme Code f ft Effect of content on knowledge about STEM education It provided learning the knowledge needed to apply the STEM education 18 33 Learning to do research-based, problem-based, project- based and context-based learning enabled learning how to achieve learning in STEM education 8 Provided learning to relate STEM disciplines 4 Provided to learn preparing measurement and evaluation tools suitable for STEM education 2 Provided to learn the relationship among STEM fields 1 Effect of content on knowledge of STEM fields Provided to learn how to perform the engineering design process while the student was designing 10 22 Provided to learn mathematical modeling 9 Provided to learn the educational technologies 3 Effect of content on pedagogical/ professional competencies Provided to learn the expression of science subjects in different ways 2 12 Provided learning to do plan 2 Provided learning how to provide meaningful learning 1 Provided learning the student-centered education 1 Provided to learn how to teach the issues by embodying 1 Provided professional development 1 International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1401-1428. 1413 Acquiring information about other STEM fields gave different perspectives 1 Provided to learn applications and activities that will interest the student 1 Provided learning to create tables and graphics 1 Acquiring knowledge on other STEM fields will enable better teaching in the future 1 Other Theoretical knowledge and praxis complemented each other 8 9 Provided to prepare the module prepared at the end of the semester the better 1 Total 76 76 When Table 4 was examined, it was found that the pre-service teachers reported their opinions about the theme "Effect of content on knowledge about STEM education" at most. In the context of this theme, pre-service teachers stated the most (f = 18) "It provided learning the knowledge needed to apply the STEM education". Pre-service teachers stated that the content of the curriculum enabled learning "to relate STEM disciplines", "to prepare measurement and evaluation tools suitable for STEM education" and “to learn the relationship among STEM fields". In the context of the theme “Effect of content on knowledge of STEM fields”, pre-service teachers emphasized that the curriculum provided "to learn how to perform the engineering design process while the student was designing" (f=10), "to learn mathematical modeling" (f=9) and "to learn the educational technologies" (f=3). Pre-service teachers also stated that theoretical knowledge and praxis complemented each other. Lambert et al. (2018) similarly prepared a two-year professional development program for teachers of mathematics, science and special education. As a result of the program, it was determined that teachers' knowledge of mathematics and science increased, they learned to make connections between STEM fields, they became willing to implement engineering design activities and their teaching practices were more qualified. Nadelson et al. (2013) prepared a two-year professional development program called "SySTEMic Solution" for primary school teachers. The findings of the study showed that the focus should be on the development of content knowledge in order to ensure the professional development of teachers regarding STEM education. As a result of the two-year education, it was determined that teachers' knowledge, self-confidence and efficacy implementation competencies increased significantly. A pre-service teacher's opinion about the content of STEM curriculum is as follows: "... It is important for us to learn STEM because we cannot apply STEM effectively if I do not know the purpose of STEM, the learning process, concepts of creativity and innovation, mathematical modeling and engineering design process." - PT6 3.2.2. Pre-service teachers' opinions on the learning-teaching process of STEM curriculum Pre-service teachers were asked the question "What are your opinions on the learning- teaching process of STEM curriculum". The findings are presented in Table 5. Turk, Kalayci, & Yamak 1414 Table 5. Pre-service teachers' opinions on learning-teaching process of STEM curriculum Theme Code f ft The effect of materials used Worksheets provided better learning 7 20 The visuals and videos used in presentations made it better to listen because they were remarkable 7 Giving group files was effective for viewing all products and performing self-assessment 5 Worksheets enabled to see the level of learning of the subject 1 The effect of the practical of the learning-teaching process Provided better learning 7 15 Provided high motivation 3 Made the lesson fun 2 Provided to do research and experimentation without giving direct information 1 Provided understanding that the practical lesson is effective 1 Provided experience to work to be done to the student 1 The effect of learning-teaching process on learning The combination of theoretical knowledge and practice provided better learning 6 14 Being active in class enabled permanent learning 4 The use of station technique provided permanent learning 1 Provided practical learning of different methods and techniques 1 Good communication with the instructor was effective in learning 1 Making presentations for theoretical knowledge was effective in learning 1 The nature of learning-teaching process Presentations were effective 8 11 Activities were instructive 1 Activities were good 1 Practice-weighted weeks were more fun 1 The effect on the way of thinking of learning-teaching process Group work in activities/ practices enabled us to look at different perspectives 2 4 Enabled to think differently 2 Other Efficiency of the course enabled to come willingly each week 1 4 Presentations made at the end of each course improved the ability to make presentations 1 Sometimes time was not enough 1 Effective communication was established through eye contact and correct tone of voice 1 Total 68 68 When Table 5 was examined, it was determined that the pre-service teachers expressed their opinions on the theme of "The effect of materials used" at most (f = 20). Pre-service teachers related to materials; stated that the worksheets provide better learning (f = 7), the visuals and videos used in presentations made it better to listen because it was remarkable (f International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1401-1428. 1415 = 7), giving group files was effective for viewing all products and performing self-assessment (f = 5). Ormancı and Şaşmaz Ören (2010) emphasized that the worksheets provide information reinforcement, facilitate the identification of preliminary information and can be used for evaluation purposes. Demircioğlu and Atasoy (2006) stated that the worksheets prepared in accordance with the constructivist approach enable students to better structure their knowledge in their minds and actively participate in the course. The fact that the learning-teaching process was practical provided better learning (f = 7) and high motivation (f = 3) and made the lesson fun (f = 2). Pre-service teachers stated that the combination of theoretical knowledge and practice provided better learning (f = 6), and being active in the lesson provided permanent learning (f = 4). They also stated that the presentations were effective (f = 8). Kaya (2018) applied STEM activities to pre-service science teachers. The pre-service teachers stated that the activities gave them a different perspective, that the knowledge gained through the practices was learned permanently, that they increased their creativity skills, that they enjoyed the activities and that they found the lesson enjoyable. In the study conducted by Ensari (2017), pre-service teachers stated that STEM activities made the lesson more fun and remarkable, made the learning more permanent, enabled active participation in the lesson and made the subjects more comprehensible. Awad and Barak (2018) developed a STEM- based program for pre-service science teachers. The researchers found that the program plays an important role in promoting meaningful learning, motivating pre-service teachers to help each other, positively influencing their interests and self-efficacy beliefs. A pre-service teacher's view of the STEM curriculum is as follows: "From the first week to the last week, presentations, activity papers and products used in the narration of all subjects showed us what STEM is, what it contains, and how to apply it... Activities made with worksheets suitable for each subject helped to strengthen the subject, to learn better and to make the lesson more enjoyable. This enabled us to learn effectively." -PT6 3.2.3. Pre-service teachers' opinions on measurement-evaluation dimension of STEM curriculum Pre-service teachers were asked the question "What are your opinions on the measurement and evaluation dimension of STEM curriculum". The findings are presented in Table 6. Table 6. Pre-service teachers' opinions on measurement and evaluation dimension of STEM curriculum Theme Code f ft The effect of using different measurement and evaluation methods/ techniques The use of different evaluation methods (rubric, peer assessment, self-assessment, etc.) provided better learning to prepare forms appropriate to these methods. 10 19 The use of different measurement forms provided practical learning of the use of forms 5 The use of different measurement tools enabled evaluation from every aspect 1 The use of different measurement tools enabled to see deficiencies of learning with different aspect 1 Peer and self assessment methods enabled objective criticism of himself and his friend 1 Turk, Kalayci, & Yamak 1416 Group assessments provided a broad overview of the products 1 The effect of evaluations on the determination of learning Enabled to recognize the shortcomings and mistakes in learning 7 15 Self- assessment enabled to recognize own shortcomings 5 Evaluation of outcomes and process provided more comprehensive evaluation 2 Revealed the level of learning 1 The effect of evaluations on student studies Group evaluation enabled to study rigorously 3 8 Process evaluation enabled to be more careful and attentive when studying 3 Continuous evaluation and getting feedback made the studies more qualified 1 The evaluation of the prepared products provided a better preparation of the studies carried out in the subsequent weeks 1 Quality of measurements and evaluation Objective 1 4 Versatile 1 Suitable for the topics of the week 1 Evaluations were appropriate because the program is mainly focused on practice and group work 1 Other Feedbacks given during the evaluation provided correction of missing and mistakes 1 4 Weekly opinion forms were effective to inquiry the information learned 1 Knowing that it's being evaluated enabled more motivation 1 Lack of written and oral exams made the process easier 1 Total 50 50 When Table 6 was examined, it was determined that pre-service teachers expressed the most (f = 19) opinions on "The effect of using different measurement and evaluation methods/ techniques" theme. Pre-service teachers stated that using different assessment methods each week enables them to learn how to prepare (f = 10) forms in accordance with these methods (f = 5). Nowadays, process-based measurement and evaluation approaches that enable students to identify their learning in a multidimensional way by examining their learning in different dimensions and way come to the forefront. Therefore, alternative measurement-evaluation approach has started to replace traditional measurement-evaluation approach. Yurdabakan (2011) emphasized that peer and self-assessment enables the active participation of students in the learning and assessment process and improves their learning in his study. In some studies, it is pointed out that alternative assessment-evaluation approaches allow students to follow the development of the student, see assessment as a part of the learning process, and evaluate the learning outcomes both product and process-oriented (Acar & Anıl, 2009; Çil & Çepni, 2018). It was determined that the evaluations made according to the opinions of the pre-service teachers enabled them to realize their own learning deficiencies and mistakes (f = 7). Pre- service science teachers stated in Ören, Ormancı, and Evrekli's (2011) study that alternative assessment and evaluation approaches enable them to be aware of their own development, to International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1401-1428. 1417 follow this development easily, to be active in the lesson and to learn by doing. In addition, Mamlok et al. (2007) stated in their study that students' being active in their own assessment processes enabled them to take more responsibility. According to the opinions of the pre-service teachers, it was determined that the evaluations made pre-service teachers to be more careful and attentive while working (f = 6) and the studies to be more qualified (f = 1). Knowing that pre-service teachers will be evaluated using different assessment tools after completing their products/ performance tasks each week may have caused them to prepare their products meticulously and diligently. A pre-service teacher's opinion is as follows: "For example, we always had an evaluation in the process. You directed it this way. We saw our own mistakes. I have learned and applied peer and self-evaluation better. ... " PT12 3.3. Pre-service Teachers’ Suggestions for STEM Curriculum Pre-service teachers were asked the question "What are your suggestions for STEM curriculum?". The findings are presented in Table 7. Table 7. Pre-service teachers’ suggestions on STEM curriculum Theme Code f ft Content Activities to integrate art with STEM can be included 4 5 Number of STEM activity can be increased 1 Learning-teaching process Different methods and techniques can be used 2 2 Materiel Different materials may be provided 1 2 Products of secondary school students about STEM activities can be shown 1 Measurement- evaluation Individual evaluations can be made 1 1 Other Course duration can be increased 6 8 The classroom environment can be designed to suit the practice 1 Learned activities can be applied in a secondary school 1 Number of teachers can be increased in order to facilitate the counselling process 1 Total 21 21 When Table 7 was examined, it was found that the pre-service teachers proposed more (f = 5) opinions about the content dimension of the program. In the context of this theme, pre- service teachers made suggestions like "Activities to integrate art with STEM can be included" (f=4) and "Number of STEM activity can be increased" (f=1). While explaining different STEM education approaches to pre-service teachers, STEAM approach was also mentioned. However, the pre-service teachers might have wondered if this approach was not given importance in the activities. They might have proposed to increase the number of activities in order to increase the understanding of the subject. The code "Course duration can be increased" was with the highest frequency (f = 6). Pre- service teachers might have brought this suggestion because they had a shortage of time in the activities of the curriculum during the implementation process. Similarly, as a result of STEM-based environmental education of Kuvaç (2018), pre-service science teachers offered Turk, Kalayci, & Yamak 1418 suggestions to extend the duration of the course. A pre-service teacher’s suggestion on STEM curriculum are as follows: "Lesson hours could be longer. ... In order to spend time more efficiently when combined with theory, the teacher and the student should be given a longer period of time. "-PT3 3.4. Pre-service Teachers' Opinions on the Contribution of STEM Curriculum to Future Professional Life Pre-service teachers were asked the question "What are the contributions of STEM curriculum to your future professional life?". The findings are presented in Table 8. Table 8. Pre-service teachers' opinions on the professional contribution of STEM curriculum Theme Code f ft Contribution to the knowledge and skills of designing learning-teaching process Provided to learn what to do in order to motivate the student in class 5 25 Provided learning to apply STEM education 3 Provided learning the features of the lesson process that enables students to learn more permanently 3 Provided learning the features of the lesson process that enables students to learn better 3 Provided learning to teach using new methods and techniques 2 Provided learning 5E learning model 2 Provided learning to do activities/ practices that will improve students' imagination 2 Provided learning the elements that will enable the student to think creatively 2 Provided learning to design a student-centered learning environment with STEM activities 1 By linking science with technology, engineering and mathematics, provided to learn designing courses that enable students to think multi-faceted, develop products, innovate, and solve problems by research-questioning 1 Provided learning to relate the lesson with daily life 1 Contribution to the knowledge of integrating/ associating STEM fields Provided learning to integrate/ associate science, technology, engineering and mathematics 8 13 Provided learning to integrate science and mathematics by mathematical modeling 4 Provided learning to integrate science with technology 1 Contribution to the knowledge and skills related to STEM fields Provided learning the use of engineering in education 6 9 Provided acquiring knowledge and skills related to other STEM fields 1 Provided learning mobile applications and computer programs that can be used in science education 1 Provided to research and acquiring new knowledge on four disciplines 1 Contribution to the lesson planning competence Provided to learn preparing lesson and activity plan according to STEM 4 8 Provided to learn applying 5E learning model in lesson 2 International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1401-1428. 1419 plan Provided to learn student-centered lesson planning 1 Provided to learn lesson planning according to interdisciplinary approach 1 Contribution to the knowledge of measurement and evaluation Provided to learn preparing and applying alternative measurement and evaluation tools 5 6 Provided to learn evaluation tools used to get to know the student 1 Contribution to the professional development Provided learning how to collaborate with colleagues 2 4 Provided learning to do research 1 Provided learning the activities should be designed in a way that enables students to think creatively, conduct research and inquiry and find solutions by presenting products based on daily life problems. 1 Contribution to the competence of guiding the students Provided learning the principles of communicating with students 1 2 Provided learning how to be guided when the student experiences problems during activities 1 Contribution to awareness about STEM education Provided discerning the importance of having knowledge about other STEM fields except their field 7 15 Provided discerning the importance of technology in STEM education 6 Provided discerning that STEM activities offer significant opportunities to identify students' interests and to guide them 1 Provided to understand that the aim was not only to give theoretical knowledge, but also that it was to enable them to invent and innovate by guiding the student and making them think at higher order 1 Contribution to awareness about learning-teaching process design Provided realizing the positive effect of problem-oriented work on student's problem solving skills 2 6 Provided realizing that the teacher should guide the student during the lesson 2 Provided realizing the effectiveness of a student-centered course 1 Provided realizing the importance of making students think about their daily life problems 1 Contribution to awareness about measurement and evaluation Provided realizing the importance of giving feedback after the activity 1 1 Other Provided realizing the importance of responsibility of student training 2 5 Provided realizing what needs to be done to train more qualified students 1 Provided realizing that different and beautiful products can be presented with simple materials 1 Practical training provided empathy to understand students 1 Turk, Kalayci, & Yamak 1420 Total 94 94 When Table 8 was examined, it was determined that pre-service teachers gave the most opinions (f = 25) about "Contribution to the knowledge and skills of designing learning- teaching process" theme. Pre-service teachers stated that the program provided to learn what to do in order to motivate the student in class. Alsop and Watts (2000) stated that emotions related to the subject to be learned affect learning. The introduction phase of the course is very important because students' willingness for the learning process will affect their active participation in the course. This finding is important since the fact that pre-service teachers started to care about students' emotions will affect the overall lesson positively. The pre-service teachers also stated that the program provided learning to apply STEM education (f=3), the features of the lesson process that enables students to learn more permanently (f=3) and better (f=3), to teach using new methods and techniques (f=2), 5E learning model (f=2), the elements that will enable the student to think creatively (f=2), to do activities/ practices that will improve students' imagination (f=2) and to design a student- centered learning environment with STEM activities (f=2). The 5E learning model is seen as an effective learning model in the implementation of STEM-oriented teaching because there is a meaningful relationship between the stages of the learning model and all stages are connected to each other and these stages provide students with the opportunity to experience how to conduct scientific research and to solve problems such as engineers (Dass, 2015). The STEM approach ensures student-centered learning and the student to actively use the knowledge and skills acquired in different situations. Active learning is preferred in STEM education because of improving student's self-learning skills (Aydede & Kesercioğlu, 2012) and increasing success in science, engineering and mathematics (Freeman et al., 2014). It is important that pre-service teachers learned to design student-centered learning environments in order to apply the basic philosophy of STEM education. It has been found out that pre-service teachers learned to integrate/ associate science, technology, engineering and mathematics (f=8), to integrate science and mathematics by mathematical modeling (f=4), to integrate science with technology (f=1) and the use of engineering in education (f=6) with STEM education program within the scope of the integration/ association of STEM fields. Mathematical modeling process is the adaptation and interpretation of the result obtained by the mathematical realization of a real life problem (Karahan & Bozkurt, 2018). When mathematical modeling is made to solve real life problems, it is ensured that students' conceptual structures are created, developed or revised. Therefore modeling and STEM training are similar (Lesh & Doer, 2003). Engineering discipline enables the integration of science, technology and mathematics when appropriate learning environments are created. Engineers' use of science and mathematics, especially in model building and data analysis (Katehi et al., 2009), provides guidance in the association of this field with other STEM fields. In this respect, engineering discipline is important for STEM education. Another field of STEM is technology. Lantz (2009) argues that STEM education should be integrated using technology-supported learning tools. Technology helps students acquire knowledge and practice-based skills in different disciplines to solve real-life problems (Cavanagh & Trotter, 2008). Since STEM education is an approach based on the integration of STEM fields (Bybee, 2013; Moore et al., 2014; Sanders, 2009), it is important that pre-service teachers have knowledge about STEM fields except of science and the integration of these fields so that they can successfully implement this approach in the future. Pre-service teachers stated that they learned to prepare lesson and activity plan, apply 5E model in lesson plan and student-centered lesson planning during the implementation of the program. Preparing a lesson plan includes the selection of methods, techniques and materials International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1401-1428. 1421 appropriate for the learning outcomes stated in the curriculum, development of activities, editing real life situations and planning the lesson process (Kablan, 2012). As there is not a ready program for STEM education in the curriculum, the pre-service teachers themselves should select the learning outcomes related to STEM fields from the related curriculums and plan their education according to STEM education. In this respect, it is an important finding that pre-service teachers learned to prepare lesson/ activity plan in accordance with STEM. In addition, Buyruk and Korkmaz (2016) stated that in order for STEM education to be reflected in the education system in our country, pre-service teachers should be aware of STEM education while continuing their education in education faculties. The findings show that the applied curriculum raised the pre-service teachers awareness about STEM education. Some of the opinions of a pre-service teacher about the contribution of STEM curriculum to their future professional life are as follows: "I have learned how to apply STEM in my future teaching life and how to tell a topic with STEM. In STEM education, I learned how to apply science, technology, mathematics and engineering and how to combine them. ... I learned to prepare an evaluation tool according to the alternative approach. We have learned how to use the engineering design process in education." -PT1 4. Results and Suggestions The results obtained from the research findings were explained according to the sub- objectives of the research and presented below.  It was concluded that the STEM curriculum was effective in the development of the pre-service teachers' competences in preparing the education module according to the STEM education approach.  While designing the STEM curriculum, it was aimed to acquire the basic content knowledge necessary for the pre-service teachers to learn and apply STEM education approach, and the pre-service teachers were expected to experience a student-centered, practice/ activity-oriented, group-based learning process using materials to enable them to be active. In addition, It is aimed to determine the development of pre-service teachers by evaluating with process and product evaluation. As a result of the positive opinions of pre- service teachers about STEM curriculum and findings, it was determined that the dimensions of the program were prepared in accordance with these aims.  Pre-service teachers proposed to include activities that integrate art with STEM, to increase the number of STEM activities, to use different methods-techniques and materials, and to increase the duration of course. According to these findings, it was concluded that pre- service teachers wanted to learn different STEM approaches and activities, to work with different materials and to apply different methods, and that the course duration was insufficient.  Since there is no STEM education department in the undergraduate programs of the Faculty of Education, pre-service science teachers need to have the knowledge and skills related to STEM fields as well as the appropriate learning models, teaching methods and techniques for STEM to successfully implement the STEM education approach. It was concluded that the STEM curriculum contributed to the knowledge and skills, professional competencies and awareness of pre-service teachers about preparing educational material according to STEM education approach. Turk, Kalayci, & Yamak 1422 In line with the results of the research, suggestions were made for the university management, faculties, the institutions and researchers.  The classes of the courses aiming to learn STEM education approach should have physical equipment and design in which group and individual studies can be done, materials should be ready in the classroom, learning environment should be arranged in such a way that classroom activities can be applied.  Student-centered methods and techniques can be used in the learning-teaching process of the courses in education faculty programs.  STEM Curriculum can be included as compulsory or elective courses in the teacher education programs.  The subjects and applications related to integrating / associating STEM fields can be included in the content of the courses in which the STEM education approach can be associated in the education faculty programs.  STEM curriculum can be implemented in other STEM fields other than the Science Teaching Department and opinions can be obtained from the program. The STEM curriculum can be implemented within the context of a course with pre-service teachers from all STEM fields, or the STEM curriculum may be implemented in the context of a course in which all STEM faculties participate in the course. International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1401-1428. 1423 References Acar, M., & Anıl, D. (2009). Sınıf öğretmenlerinin performans değerlendirme sürecindeki değerlendirme yöntemlerini kullanabilme yeterlikleri, karşılaştıkları sorunlar ve çözüm önerileri. [Classroom teacher evaluation methods to use in the performance assessment process qualification of able, they comparison problems and solution proposals]. Journal of TUBAV Science,2(3), 354-363. Akaygun, S., & Aslan Tutak, F. (2016). STEM images revealing STEM conceptions of pre- service chemistry and mathematics teachers. International Journal of Education in Mathematics, Science and Technology,4(1), 56-71. Akgündüz, D., Aydeniz, M., Çakmakçı, G., Çavaş, B., Çorlu, M. S., Öner, T., & Özdemir, S. (2015). STEM eğitimi Türkiye raporu [STEM education report on Turkey]. İstanbul: İstanbul Aydın University. Alsop, S., & Watts, M. (2000). Facts and feelings: Exploring the affective domain in the learning of physics. Physics Education, 35(2), 132. Altaş, S. (2018). STEM eğitimi yaklaşımının sınıf öğretmeni adaylarının mühendislik tasarım süreçlerine, mühendislik ve teknoloji algılarına etkisinin incelenmesi. [Investigation of the effects of STEM education approach on the perceptions of classroom teaching candidates about engineerıng design processes and about engineering and technology] (Unpublished master’s thesis). Muş Alparslan University, Muş, Turkey. Altunel, M. (2018). STEM eğitimi ve Türkiye: Fırsatlar ve riskler [STEM education and Turkey: Opportunities and risks.]. Foundation for Political, Economic and Social Research, 207, 1-7. Arslan, Ö. (2018). Fen, teknoloji, mühendislik ve matematik (STEM) uygulamalarının farklı bağımlı değişkenler üzerinden incelenmesi. [Analyzing the effect of science technology engineering and mathematics (STEM) applications with respect to different dependent variıables] (Unpublished master’s thesis). Muş Alparslan University, Muş, Turkey. Asilsoy, Ö. (2007). Biyoloji öğretmenleri için proje tabanlı öğrenme yaklaşımı konulu bir hizmet içi eğitim kurs programı geliştirilmesi ve etkililiğinin araştırılması. [Developing an in-service education course on project based learning approach for biology teachers and testing its efficiency] (Unpublished master’s thesis). Karadeniz Technical University, Trabzon, Turkey. Awad, N., & Barak, M. (2018). Pre-service science teachers learn a science, technology, engineering and mathematics (STEM)-oriented program: The case of sound, waves and communication systems. Eurasia Journal of Mathematics, Science and Technology Education, 14(4), 1431-1451. doi:10.29333/ejmste/83680. Aydede, M. N., & Kesercioğlu, T. (2012). Aktif öğrenme uygulamalarının öğrencilerin kendi kendine öğrenme becerilerine etkisi [The Effect of Active Learning Applications on Students’ Self Direct Learning Skills]. Hacettepe University Journal of Education, 43(43), 37-49. Aydin, E., & Delice, A. (2007). Experiences of mathematics student teachers in a series of science experiments [Paper presentation]. The 6 th WSEAS International Conference on Education and Educational Technology, Italy. Bahçeşehir University (2018). BAUSTEM hakkında [About BAUSTEM.]. Retrieved from http://inteach.org/. doi:10.29333/ejmste/83680 Turk, Kalayci, & Yamak 1424 Barnett, R., & Coate, K. (2005). Engaging the curriculum in higher education. Maidenhead: Open University. Buyruk, B., & Korkmaz, Ö. (2014). FeTeMM farkındalık ölçeği (FFÖ): Geçerlik ve güvenirlik çalışması. [STEM awareness scale (SAS): Validity and reliability study]. Journal of Turkish Science Education, 11(1), 3-23. doi:10.24106/kefdergi.356829. Büyüköztürk, Ş. (2014). Sosyal bilimler için veri analizi el kitabı [Manual of data analysis for social sciences]. Ankara: Pegem. Büyüköztürk, Ş., Çakmak, E. K., Akgün, Ö. E., Karadeniz, Ş., & Demirel, F. (2014). Bilimsel araştırma yöntemleri [Scientific research methods]. Ankara: Pegem. Bybee, R. W. (2013). The case for STEM education: Challenges and opportunities. National Science Teacher Association [NSTA]. Cavanagh, S., & Trotter, A. (2008). Where’s the ‘T’in STEM? Technology counts, STEM: The push to improve sciecne, technology, engineering and maths. Retrieved from https://www.edweek.org/ew/articles/2008/03/27/30stemtech.h27.html. Çil, E., & Çepni, S. (2018). STEM eğitiminde ölçme-değerlendirme. [Measurement and evaluation in STEM education]. In S. Çepni (Ed.), Kuramdan uygulamaya STEM eğitimi [STEM education from theory to practice],(pp. 555-604). Ankara: Pegem. Cohen, L., Manion, L., & Morrison, K. (2006). Research methods in education. London: Routledge. Çolakoğlu, M. H., & Gökben, A. G. (2017). Türkiye’de eğitim fakültelerinde FeTeMM (STEM) çalışmaları. [FeTeMM (STEM) studies in faculties of education in Turkey]. Journal of Research in Informal Environments, 2(2), 46-69. doi:10.21733/ibad.548456. Çorlu, M. (2012). A pathway to STEM education: Investigating pre-service mathematics and science teachers at Turkish universities in terms of their understanding of mathematics used in science (Unpublished doctoral dissertation). Texas A&M University, Texas, USA. Çorlu, M. A., & Corlu, M. S. (2012). Scientific inquiry based professional development models in teacher education. Educational Sciences: Theory and Practice, 12(1), 514- 521. https://files.eric.ed.gov/fulltext/EJ978456.pdf. Çorlu, M. S. (2014). FeTeMM eğitimi makale çağrı mektubu. [Call for manuscripts on STEM education]. Turkish Journal of Education, 3(1), 4-10. doi:10.19128/turje.181071. Çorlu, M. S., Capraro, R. M., & Capraro, M. M. (2014). Introducing STEM education: Implications for educating our teachers in the age of innovation. Education and Science, 39(171), 74-85. http://egitimvebilim.ted.org.tr/index.php/EB/article/view/2142/651. Council of Higher Education. (2018). Teacher Education Undergraduate Programs. Cresswell, J. W., & Plano Clark, V. L. (2011). Desiging and conducting mixed method research. London: Sage. Dass, P. M. (2015). Teaching STEM effectively with the learning cycle approach. K-12 STEM Education, 1(1), 5-12. Demircioğlu, H., & Atasoy, Ş. (2006). Çalışma yapraklarının geliştirilmesine yönelik bir model önerisi. [A model proposal for the development of worksheets]. The Journal of doi:10.24106/kefdergi.356829 doi:10.21733/ibad.548456 International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1401-1428. 1425 Buca Faculty of Education, 19, 71-79. https://dergipark.org.tr/tr/pub/deubefd/issue/25441/268448. Denzin, N. K. (1978). The research act: A theoretical introduction to sociological methods. New York: McGraw-Hill. Dugger, W. E. (2010). Evolution of STEM in the United States [Paper presentation]. The 6th Biennial International Conference on Technology Education Research, Queensland, Australia. Ekiz, D. (2003). Eğitimde araştırma yöntem ve metodlarına giriş: Nitel, nicel ve eleştirel kuram metodolojileri [Introduction to research methods and methods in education: Qualitative, quantitative and critical theory methodologies]. Ankara: Anı. Ensari, Ö. (2017). Öğretmen adaylarının FeTeMM eğitimi ve FeTeMM etkinlikleri hakkındaki görüşleri. [Pre-service Teachers’ Views on STEM Education and STEM activitiesi] (Unpublished master’s thesis). Yüzüncü Yıl University, Van, Turkey. Erdem, A. R. (2013). Bilgi toplumunda üniversitenin değişen rolleri ve görevleri. [Changing roles and missions of university in information society]. Journal of Higher Education, 3(2), 109-120. doi:10.2399/yod.13.013. Erişen, Y. (2001). Öğretmen yetiştirme programlarına ilişkin kalite standartlarının belirlenmesi ve fakültelerin standartlara uygunluğunun değerlendirilmesi. [Determination the quality standards related to teacher training programs and evaluation of appropriateness of determinated standards] (Unpublished doctoral dissertation). Ankara University, Ankara, Turkey. Fraenkel, J. R., & Wallen, N. E. (2009). How to design and evaluate research ın education. New York: McGraw-Hill. Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410-8415. Gökbayrak, S., & Karışan, D. (2017). STEM etkinliklerinin fen bilgisi öğretmen adaylarının bilimsel süreç becerilerine etkisi [An Investigation of the Effects of STEM based Activities on Preservice science Teacher’s Science Process Skills]. The Western Anatolia Journal of Educational Sciences (WAJES), 8(2), 63-84. Gömleksiz, M., Yaşar, S., Sağlam, M., Hakan, A., Sözer, E., & Gözütok, D. (2005). Eğitim programları ve öğretim alanı profesörler kurulu ilköğretim 1-5. sınıflar öğretim programlarını değerlendirme toplantısı sonuç bildirisi [Curriculum and instruction professors board primary education 1-5 grades curriculum evaluation meeting final report]. Eskişehir, Anadolu University. Holdren, J. P., Lander, E., & Varmus, H. (2010). Prepare and inspire: K-12 science, technology, engineering, and math (STEM) education for America’s Future. President’s Council of Advisors on Science and Technology. H-STEM Lab. (2014). Bilim ve ı̇novasyon eğitiminde Hacettepe Üniversitesi'nin rolü [The role of Hacettepe University in science and innovation education.]. Hacettepe Bilim, Kültür ve Teknoloji Dergisi, Eylül-Ekim, 12-13. https://dergipark.org.tr/tr/pub/deubefd/issue/25441/268448 Turk, Kalayci, & Yamak 1426 Kablan, Z. (2012). The effects of level of cognitive learning and concrete experience on teacher candidates’ lesson planning and application skills. Education and Science, 37(163), 239-253. Karahan, E., & Bozkurt, G. (2018). STEM eğitiminde matematik odaklı gerçek dünya problemleri ve matematiksel modelleme [Mathematics-focused real-world problems and mathematical modeling in STEM education]. In S. Çepni (Ed.), Kuramdan uygulamaya STEM eğitimi [STEM education from theory to practice],(pp. 353-373). Ankara: Pegem. Katehi, L., Pearson, G., & Feder, M. (2009). Engineering in K-12 education: Understanding the status and improving the prospects. Washington DC: National Academy of Engineering. Kaya, M. E. (2018). STEM uygulamalarının fen bilgisi öğretmen adayları öz düzenleme ve yaratıcılığına etkisi. [Effects of STEM applications on scientıfıc arrangement and creativity of science teachers of science] (Unpublished master’s thesis). Erzincan Binali Yıldırım University, Erzincan, Turkey. Kelley, T. R., & Knowles, J. G. (2016). A conceptual framework for integrated STEM education. International Journal of STEM Education, 3(1), 11. doi:10.1186/s40594- 016-0046-z. Kim, C., Kim, D., Yuan, J., Hill, R. B., Doshi, P., & Thai, C. N. (2015). Robotics to promote elementary education pre-service teachers' STEM engagement, learning, and teaching. Computers & Education, 91, 14-31. Koehler, A. A., Feldhaus, C. R., Fernandez, E., & Hundley, S. (2013). STEM alternative certification programs & pre-service teacher preparedness. Journal of STEM Education: Innovations and Research, 14(4). König, J., Bremerich-Vos, A., Buchholtz, C., Fladung, I., & Glutsch, N. (2020). Pre–service teachers’ generic and subject-specific lesson-planning skills: On learning adaptive teaching during initial teacher education. European Journal of Teacher Education, 43(2), 131-150. doi:10.1080/02619768.2019.1679115. Kuvaç, M. (2018). Fen, teknoloji, mühendislik ve matematik (STEM) temelli çevre eğitimine yönelik öğretim tasarımının etkililiği. [The effectiveness of instructional design on science, technology, engineering and mathematics (STEM) based environmental education] (Unpublished doctoral dissertation). Istanbul University-Cerrahpasa, İstanbul, Turkey. Lambert, J., Cioc, C., Cioc, S., & Sandt, D. (2018). Making connections: Evaluation of a professional development program for teachers focused on STEM integration. Journal of STEM Teacher Education, 53(1), 3-25. Lantz, H. B. (2009). Science, technology, engineering, and mathematics (STEM) education: What form? What function? What is STEM education? Retrieved from https://dornsife.usc.edu/assets/sites/1/docs/jep/STEMEducationArticle.pdf. Lesh, R. A., & Doerr, H. M. (2003). Beyond constructivism: Models and modeling perspectives on mathematics problem solving, learning, and teaching. New York: Routledge. Lincoln, Y., & Guba, E. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage. International Online Journal of Education and Teaching (IOJET) 2021, 8(3), 1401-1428. 1427 Livingston, K. (2017). The complexity of learning and teaching: challenges for teacher education. European Journal of Teacher Education, 40(2), 141-143. doi:10.1080/02619768.2017.1296535. Mamlok Naaman, R., Hofstein, A., & Penick, J. E. (2007). Involving science teachers in the development and implementation of assessment tools for “science for all” type curricula. Journal of Science Teacher Education, 18(4), 497-524. Marulcu, İ., & Sungur K. (2012). Fen bilgisi öğretmen adaylarının mühendis ve mühendislik algılarının ve yöntem olarak mühendislik-dizayna bakış açılarının incelenmesi. [Investigating Pre-Service Science Teachers’ Perspectives on Engineers, Engineering and Engineering Design as Context]. Afyon Kocatepe University Journal of Sciences, 12(1), 13-23. Memişoğlu, H. (2008). Sosyal bilgiler dersi öğretiminde proje tabanlı öğrenme yaklaşımı. [Project based learning approach in teaching of the social sciences course] (Unpublished doctoral dissertation). Gazi University, Ankara, Turkey. Merriam, S. B. (2009). Qualitative research: A guide to design and implementation. (S. Turan, Ed.; & Trans.). San Francisco: Jossey-Bass. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Thousand Oaks, Califorinia: SAGE. Miles, M. B., Huberman, A. M., & Saldaña, J. (2013). Qualitative data analysis: A methods sourcebook. Los Angeles, CA: Sage. Ministry of National Education. (2016). STEM eğitimi raporu [STEM education report]. Ankara. Moore, T. J., Stohlmann, M. S., Wang, H. H., Tank, K. M., Glancy, A. W., & Roehrig, G. H. (2014). Implementation and integration of engineering in K-12 STEM education. In Ş. Purzer, J. Strobel, and M. E. Cardella (Eds.), Engineering in pre-college settings: synthesizing research, policy, and practices (pp. 35-60). West Lafayette: Purdue University. Nadelson, L. S., Hay, A., Pyke, P., Callahan, J., Dance, M., & Pfiester, J. (2013). Teacher STEM perception and preparation: Inquiry-based STEM professional development for elementary teachers. The Journal of Educational Research, 106(2), 157-168. doi:10.1080/00220671.2012.667014. Okçabol, R. (2007). Yükseköğretim sistemimiz [Our higher education system]. Ankara: Ütopya. Ören, F. Ş., Ormancı, Ü., & Evrekli, E. (2014). The alternative assessment-evaluation approaches preferred by pre-service teachers and their self-efficacy towards these approaches. Education and Science, 39(173), 103-117. Ormancı, Ü., & Şaşmaz Ören, F. (2010). Çalışma yapraklarının yararları, sınırlılıkları ve kullanımına ilişkin sınıf öğretmeni adaylarının görüşleri. [Opinions of elementary teacher candidates regarding the benefits, limitations and use of worksheets] [Paper presentation]. The International Conference on New Trends in Education and Their Implications, Antalya. Ornstein, A. C., & Hunkins, F. P. (2009). Curriculum foundations, principles and issues. Boston: Allynand Bacon. Turk, Kalayci, & Yamak 1428 Middle East Technical University (2018). About BİLTEMM. Retrieved from https://biltemm.metu.edu.tr/tr. Özdemir, S. M. (2011). Toplumsal değişme ve küreselleşme bağlamında eğitim ve eğitim programları: Kavramsal bir çözümleme. [Education and curricula within the context of social change and globalization: A conceptual analysis]. Kırşehir Faculty of Education Journal (JKEF), 12(1), 85-110. Rinke, C. R., GladstoneBrown, W., Kinlaw, C. R., & Cappiello, J. (2016). Characterizing STEM teacher education: Affordances and constraints of explicit STEM preparation for elementary teachers. School Science and Mathematics, 116(6), 300-309. Roberts, A. S. (2013). Preferred instructional design strategies for preparation of pre-service teachers of integrated STEM education (Unpublished doctoral dissertation). Old Dominion University, Norfolk, USA. Sanders, M. (2009). STEM, STEM education, STEMmania. Technology Teacher, 68(4), 20– 26. Shernoff, D. J., Sinha, S., Bressler, D. M., & Ginsburg, L. (2017). Assessing teacher education and professional development needs for the implementation of integrated approaches to STEM education. International Journal of STEM Education, 4(1), 13. doi:10.1086/596996. STEM School (2018). STEM okulu hakkında [About STEM school]. Retrieved from https://stemokulu.weebly.com/kurumsal.html. Türk, N., Kalayci, N., & Yamak, H. (2018). New trends in higher education in the globalizing world: STEM in teacher education. Universal Journal of Educational Research, 6(6), 1286-1304. doi:10.13189/ujer.2018.060620. Türk Sanayicileri ve İşinsanları Derneği [TÜSİAD]. (2014). STEM alanında eğitim almış işgücüne yönelik talep ve beklentiler araştırması [A research on demands and expectations for a STEM-trained workforce]. İstanbul. Yıldırım, B. (2016). 7. Sınıf fen bilimleri dersine entegre edilmiş fen teknoloji mühendislik matematik (STEM) uygulamaları ve tam öğrenmenin etkilerinin incelenmesi. [An Examination of the Effects of Science Technology Engineering Mathematics (STEM) Application and Mastery Learning Integrated into the 7th Grade Science Course] (Unpublished doctoral dissertation). Gazi University, Ankara, Turkey. Yıldırım, A., & Şimşek, H. (2013). Sosyal bilimlerde nitel araştırma yöntemleri [Qualitative research methods in the social sciences]. Ankara: Seçkin. Yildiz, E. P., Alkan, A. & Cengel, M. (2019). Teacher candidates attitudes towards the STEM and sub-dimensions of STEM. Cypriot Journal of Educational Science, 14(2), 322-344. Yurdabakan, İ. (2011). The view of constructivist theory on assessment: Alternative assessment methods in education. Ankara University Journal of Faculty of Educational Sciences, 44(1), 51-77.