a © 2022 Indonesian Society for Science Educator 79 J.Sci.Learn.2022.5(1).79-90 Received: 5 July 2021 Revised: 9 August 2021 Published: 1 March 2022 Development of the STEM-Pedagogical Content Knowledge Scale for Pre- service Teachers: Validity and Reliability Study Behiye Akçay1*, Filiz Avcı1 1Istanbul Universty - Cerrahpaşa, Turkey *Corresponding Author. behiye.akcay@iuc.edu.tr ABSTRACT This study aims to develop a valid and reliable scale to determine pre-service teachers' STEM-Pedagogical Content Knowledge (STEM-PCK) levels. This study was conducted in the 2018-2019 academic year with 322 pre-service teachers in Turkey. In the study, one of the mixed method typologies, Exploratory Sequential Design, was applied. The scale was submitted for evaluation by four experts to determine the content and face validity. Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) was performed to determine the scale's construct validity. As a result of EFA, the scale had five facto rs: STEM Pedagogical Knowledge, Pedagogical Knowledge, Engineering Pedagogical Knowledge, Mathematics Pedagogical Knowledge, and Science Pedagogical Knowledge. As a result of CFA χ²/df = 2.71, RMSEA = .07, RMR = 0.04, SRMR = .07, NFI = .94, NNFI = .96, CFI = .96, IFI = .96, RFI = .94 values were reached and the factor structure determined to be suitable. To determine the reliability of the scale, internal consistency and test-retest reliability analyzes were made. The internal consistency reliability value of the scale was found as .98. The final form of the STEM-PCK scale is a 5-point Likert type that includes 57 items and five factors. Keywords STEM, Pedagogical Content Knowledge, Validity, Reliability 1. INTRODUCTION Science, Technology, Engineering, and Mathematics (STEM) has been a growing area since President Barack Obama shared the United States national dialogue in 2011 (Epstein & Mille, 2011). They were improving STEM research studies focused mainly on preparing students for STEM-related careers and improving STEM literacy for all students. The studies on STEM is showed that teachers' effectiveness has a substantial effect on students’ achievement in STEM subjects (Knezek, Christensen & Tyler-Wood, 2015; York, 2018). In addition, Wahono, Lin, & Chang (2020) explained that STEM enactment positively affects Asian students’ learning outcomes in academic learning achievement, higher-order thinking skills, and motivation. Science teachers should know how to combine many different types of knowledge from science disciplines for effective science instruction for STEM areas (Schmidt & Fulton, 2015). Each science discipline requires important content of scientific knowledge and subject-specific pedagogy (York, 2018). Because teachers’ knowledge of a particular subject matter, which includes Pedagogical Content Knowledge (PCK) and Content Knowledge (CK), are vital components of teacher education programs (Kleickmann et al., 2013). STEM-related research showed that pre-service teachers are not experts in STEM subjects because of inadequate content knowledge and low self- confidence to teach STEM (Epstein & Miller, 2011; York, 2018). Therefore, pre-service teachers need a more robust STEM education (Srisawasdi, 2012). Most teacher education programs include content courses for chemistry, biology, physics, astronomy, technology, earth sciences, and science teaching methods systems. However, York (2018) argued that to know theoretical knowledge of different disciplines is not enough to understand how to integrate and teach in actual classroom instruction. Therefore teacher education programs must prepare pre-service teachers about how to lead a specific content knowledge at the level of their students in terms of pedagogy and inquiry. Also, teacher Journal of Science Learning Article DOI: 10.17509/jsl.v5i1.36293 80 J.Sci.Learn.2022.5(1).79-90 education programs should support teachers’ PCK for STEM (Srikoom, Faikhamta & Hanuscin, 2018). Srikoom, Faikhamta & Hanuscin (2018) explained that the components of PCK for STEM include how teachers comprehend the place of STEM in science education curriculum and educational materials, how teachers conceptualized STEM education. Integrating STEM into science education and teaching practices in return the complex multi-dimensional process with many dimensions such as Scientific Knowledge, Mathematical Knowledge, Technological Knowledge, Engineering Knowledge, Content Knowledge, Pedagogical Knowledge become important (Sarkim, 2020). Initially, Pedagogical Content Knowledge (PCK) propose by Shulman in 1986. Shulman explained three categories of content knowledge, including subject-matter (content) knowledge, subject-matter pedagogical knowledge, and curricular knowledge (Kind, 2009). Magnusson, Krajcik, & Borko (1999) explained that PCK requires “…teacher’s understanding of how to help students understand the specific subject matter. It includes knowledge of how particular subject matter topics, problems, and issues can be organized, represented, and adapted to the diverse interests and abilities of learners, and then presented for instruction.” (p. 96). PCK is unique to each subject, context, and teacher because they have different knowledge about teaching and how to teach the subject matter knowledge to their students (Park, Suh & Seo, 2018). Studies showed that qualified science subjects significantly affect students’ attitudes and career choices toward science-related fields (Knezek, Christensen & Tyler-Wood, 2015). Developing teachers’ STEM pedagogical knowledge and proficiency with STEM- specific pedagogical strategies would increase their self- confidence to teach STEM and, in turn, would positively affect their students’ career choices in STEM-related fields (York, 2018). In recent years, studies on STEM education (Corlu, Capraro, & Corlu, 2015; Derin, Aydın & Kirkiç, 2017) have focused mainly on teachers and pre-service teachers. El- Deghaidy & Mansour (2015) argued that to promotes effective STEM education, we need to identify teachers' content knowledge and pedagogical content knowledge to enact STEM education in class. Therefore, this study aims to develop a valid and reliable scale that can determine the Pedagogical Content Knowledge of pre-service teachers about what, when, why, and how they teach STEM subjects and contributes to the field. 2. METHOD 2.1 Research Model The study aimed to develop a valid and reliable scale to determine teacher candidates' Science, Technology, Engineering, and Mathematics-Pedagogical Content Knowledge (STEM-PCK) levels. To achieve this aim, Exploratory Sequential Design, one of the mixed-method research typologies in which qualitative and quantitative research methods were used together, was used in the study (Tashakkori & Creswell, 2007). In the Exploratory Sequential Design, after the qualitative data are collected and analyzed, the data are tested and diagnosed with quantitative methods (Creswell & Clark, 2011). The pattern scheme of the research design shows in Figure 1. Figure 1 Scale development process Collection of qualitative data • Determining the need • Literature research • Creating an item pool Analysis of qualitative data • Taking expert opinion • First form the scale Collection of quantitative data • Applying the Pilot Study for Item Selection • Determining the sample Analysis of quantitative data • Performing statistical analysis for the application of the rearranged scale and for item selection • Exploratory Factor Analysis (SPSS) Confirmatory Factor Analysis(LISREL) Reliability calculations Interpreta- tion • Creating the final form of the scale Journal of Science Learning Article DOI: 10.17509/jsl.v5i1.36293 81 J.Sci.Learn.2022.5(1).79-90 2.2. Scale Development Process In this study, initial scale development steps explained by Seçer (2015) were used. These stages are(1) Determining the need; (2) Literature search and creating an item pool; (3)Taking expert opinion; (4) First form of the scale; (5) Applying the pilot study for item selection; (6) Determining the sample group; (7) Performing statistical analysis for the application of the rearranged scale and item selection; (8) Creating the final form of the scale. 2.2.1. Determining the need When the literature examined, it is seen that in recent years, scale development studies in the fields of PCK and TPCK (Technological Pedagogical Content Knowledge) for teachers and pre-service teachers are frequently used (Graham et al., 2009; Schmidt et al., 2009; Landry, 2010; Marks, 1990; Sun & Strobel, 2014). However, when the literature was examined, a limited number of studies were found that measure the Pedagogical Content Knowledge of teachers and pre-service teachers towards STEM (Rigelman, 2014; Yildirim & Sahin-Topalcengiz, 2019). While Rigelman (2014) developed a rubric to assess teachers’ STEM Pedagogical Content Knowledge, Yildirim & Sahin-Topalcengiz (2019) developed a 5-point Likert type STEM PCK Scale for pre-service teachers. It has six factors, including 21st-Century Skills Knowledge, Pedagogical Knowledge, Mathematical Knowledge, Science Knowledge, Engineering Knowledge, and Technology Knowledge. However, this scale doesn’t include a specific dimension to assess pre-service teachers' STEM PCK. Instead, it has subject matter knowledge dimensions for science, mathematics, engineering, and technology. This study aimed to develop a Likert-type scale to measure the STEM Pedagogical Knowledge (STEM Knowledge for Teaching), Pedagogical Knowledge, Engineering Pedagogical Knowledge, Mathematics Pedagogical Knowledge, and Science Pedagogical Knowledge levels pre-service teachers. I hope that the STEM-PCK scale, which aims to develop in this study, will play a leading role in determining the STEM PCK levels of pre-service teachers. 2.2.2. Literature research and creating an item pool While creating the scale items developed to determine pre-service teachers' STEM PCK levels, first examined the theoretical structure with a wide-ranging literature review in PCK and TPCK. As a result of the literature research (Aydin-Gunbatar, Boz & Yerdelen-Damar, 2017; Schmidt et al. 2009; Tiryaki, 2018; Pamuk, Ulken, & Dilek 2012; Graham et al., 2009; Canbazoglu-Bilici, Yamak, Kavak & Guzey., 2013; Tyler-Wood, Knezek & Christensen 2010; Srikoom, Hanuscin & Faikhamta, 2017) an item pool consisting of 87 items was created. The draft form of the STEM-PCK scale was prepared as 5-Likert (1: Strongly Disagree, 2: Disagree, 3: Undecided, 4: Agree, 5: Strongly Agree) type presented to expert opinion. The item pool prepared to ensure the content and face validity was submitted to the evaluation of three field experts working in the science education department at three different universities in Turkey and a Turkish language expert who teaches Turkish in a public secondary school in Turkey to be examined in terms of compliance with the spelling rules (Table 1). Experts ask to indicate in Table 2 their answers regarding the eligibility of the items for the scale. Next, experts ask to mark the "Appropriate " section on the table if the thing is appropriate and the "Not Appropriate " area if it is not applicable and write their explanations in the "Explanation " section. Finally, it states that experts could add the new items suggested to the scale to the table. 2.2.3. First form of the scale In line with the experts, two items (6th and 53rd items) were excluded from the item pool as it was stated that they were not directed to the feature intended to measure. Items excluded from the scale: Item 6: “I have sufficient opportunities to work on each of the STEM fields.” Item 53: "I can apply/teach a lesson plan that includes (combines) Science, Technology, Engineering and Mathematics subjects." In the feedback from experts, it was also determined that some items were not clearly expressed and understood, some items could not measure the behavior they wanted to measure adequately, and some things could not focus on the behavior they wanted to measure, and 2, 4, 7, 8, 11, 12 ., 13., 14., 15., 16., 17., 18., 23., 24., 25., 26., 27., 28., 29., 30., 31., 32., 33., 34., 44., 54., 58., 75. and 79. İtems were rearranged. Examples of mixed items: Table 1 Demographic characteristics of the expert group Gender Title Department University Female Associate Professor Science Education Marmara University Female Associate Professor Science Education Boğaziçi University Male Assistant Professor Science Education Kafkas University Female Teacher Turkish MoNE Turkish Teacher Table 2 Form prepared for expert opinion Scale Items Appropriate Not Appropriate Explanation 1 2 Journal of Science Learning Article DOI: 10.17509/jsl.v5i1.36293 82 J.Sci.Learn.2022.5(1).79-90 Item 2: The item is written as "I know the importance of STEM in science education" was arranged as "I know the importance of STEM in education." Item 58: The item is written as "While evaluating the STEM teaching process, I can develop measurement/evaluation tools appropriate to the subject." It was arranged as "I can prepare the appropriate STEM measurement tools for the evaluation process." The draft scale form with 85 items was made ready for pilot application after the expert Turkish teacher checked the scale in terms of spelling and language. 2.2.4. Applying the pilot study for item selection A pilot study carries out to determine whether there were any unnoticed expressions or format problems on the form and items of the draft scale created before the general applications. The pilot application was carried out by the researchers on a group of 15 senior pre-service science teachers. According to the application results, necessary corrections made on the items decide that 20 minutes are required to implement the scale. 2.2.5. Determining the sample group The study population consists of pre-service teachers from 1., 2., 3. and 4. grades that's a study in Turkey during 2018-2019. The sample of the study consisted of 322 teacher candidates in different grade levels (1., 2., 3. and 4. grades) from six (6) other teaching departments in seven (7) different universities in Turkey using a purposeful sampling method. The scale was prepared online, for which the first draft form create after the pilot study. In the study, the STEM- PCK draft scale form link was sent as an e-mail to the pre- service teachers in the sample group. Pre-service teachers who received the invitation were provided with access to the online scale form to fill out on demand. As a result of the study, 322 teacher candidates filled the scale. The teacher candidates' 274 (85.1%) are female, and 48 (14.9%) are male. The information about the university and departments of the teacher candidates, who constitute the research study group, is given in Table 3. 3. RESULT AND DISCUSSION 3.1. Validity Studies 3.1.1. Content validity Within the scope of the content validity, three field experts consulted for the quantitative and qualitative evaluation of the items in the scale. As a result of an assessment of the field experts, the things they found appropriate, unsuitable, and suggested review examines. Intercoder reliability among experts is according to Miles & Huberman (1994). It calculates with the formula (Reliability = number of agreements / (number of the accords + disagreements) x 100), and this ratio determines as .87. 3.1.2. Face validity For the face validity of the scale, the scale evaluates by three experts. Relevant items remove from the scale according to their feedback. After the necessary corrections, an explanation section was prepared on the front page of the hierarchy about what purpose the ranking will be used for, what it aims to measure, and how many items it consists of. Later, the scale was applied to pre- Table 3 Descriptive information about the students in the study group f % University İstanbul University -Cerrahpasa 169 52.5 Marmara University 49 15.2 Yıldız Teknik University 55 17.1 Kafkas University 28 8.7 Boğaziçi University 12 3.7 Recep Tayyip Erdoğan University 5 1.6 Uludağ University 4 1.2 Gender Female 274 85.1 Male 48 14.9 Department Science Education 185 57.5 Mathematics Education 33 10.2 Primary Education 66 20.5 Social Studies Education 5 1.6 Physics Education 25 7.8 Chemistry Education 8 2.5 Grade Freshman 1 0.3 Sophomore 16 5 Junior 218 40.3 Senior 87 16.1 Total Participant 322 Journal of Science Learning Article DOI: 10.17509/jsl.v5i1.36293 83 J.Sci.Learn.2022.5(1).79-90 service teachers within the scope of a pilot study, and it was aimed to determine the incomprehensible items from the items in the hierarchy. 3.1.3. Construct validity (Factor analysis) Factor analysis was carried out to reveal the implicit structure of the measuring tool and to determine the sub- dimensions of the scale by bringing together the related items. In scale development studies, Exploratory Factor Analysis (EFA) is performed to reveal the implicit structure of the scale, while Confirmatory Factor Analysis (CFA) is committed to verifying this implicit structure. Exploratory factor analysis (EFA). Exploratory Factor Analysis (EFA) was carried out to determine the sub-dimensions of the items related to each other in the scale and the relationship between them. Developing the scale first determines whether the sample size to which the scale is applied is sufficient for factor analysis and whether multivariate normality is achieved (Buyukozturk, 2007). The EFA result calculated the Kaiser-Meyer-Olkin (KMO) test result as .94 (Table 4). According to Kaiser (1974), values between .00 and .49 are unacceptable, .50 and .59 are miserable, .60 and .69 are mediocre, .70 and .79 are middling, .80 and .89 are meritorious, and .90 and 1.00 are marvelous. For this reason, the KMO test result obtained shows that the sample size is suitable for factor analysis of the available data. Because as the value obtained approaches 1.00, the adequacy of the sample size increases (Seçer, 2015). Also, the significance of Barlett's test of sphericity (Bartlett's test of sphericity) indicates that the data set provides multivariate normality (14118.21, p <.05) (Seker & Gencdogan, 2014). There are 85 items in the first form of the STEM-PCK scale, creates due to the pilot study. As a result of the factor analysis, five factors were found with an eigenvalue more significant than one and explaining 60.38% of the total variance. The first factor explains 22.52%, the second factor 12.44%, the third factor 9.39%, the fourth factor 8.60%, and the fifth factor 7.43% (Table 5). Table 4 KMO and Bartlett’s test values for STEM-PCK Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO) Bartlett's Test of Sphericity Approx. Chi- Square df p .94 14118.21 1596 .00 Table 5 Total variance explained results for STEM-PCK scale Factors Total Eigenvalues Rotation sums of squared loadings (% of Variance) Rotation sums of squared loadings (Cumulative %) F1 12.84 22.52 22.52 F2 7.09 12.44 34.96 F3 5.35 9.39 44.36 F4 4.90 8.60 52.96 F5 4.23 7.43 60.38 *Extraction Method: Principal Component Analysis Table 6 Rotated component matrix Factors Item number F1 F2 F3 F4 F5 Q46 .80 Q53 .79 Q47 .78 Q45 .77 Q48 .77 Q43 .76 Q57 .74 Q49 .74 Q56 .72 Q52 .72 Q50 .72 Q61 .71 Q55 .70 Q59 .69 Q54 .69 Q62 .69 Q42 .68 Q60 .65 Q58 .61 Journal of Science Learning Article DOI: 10.17509/jsl.v5i1.36293 84 J.Sci.Learn.2022.5(1).79-90 The Scree Plot chart (Figure 2) examines five vertical breaks with an eigenvalue greater than 1. Therefore, according to these results, it evaluates that the scale had five factors. The resulting component matrix table was examined at the end of the analysis with a factor load value of at least .32 and above for each item. Since the sub-dimensions in the measurement tool were thought to be unrelated to each other, Varimax rotation was performed, which is one of the orthogonal rotation techniques. Items with a difference of less than .10 between factor loading values were evaluated as overlapping items, and as a result of the analysis, 1-2-3- 4-5-6-7-9-12-16-21-22-25-28-30- Items 33-37-41-74-75- 77-78-79-81-82-83-84 and 85 were excluded from the scale. When the Rotated Component Matrix table (Table 6) was Table 6 Rotated component matrix (Continued) Factors Item number F1 F2 F3 F4 F5 Q51 .59 Q63 .57 Q44 .57 Q80 .52 Q8 .49 Q70 .80 Q71 .79 Q73 .74 Q66 .72 Q65 .72 Q64 .70 Q72 .70 Q69 .69 Q67 .65 Q68 .63 Q76 .59 Q29 .78 Q17 .75 Q32 .75 Q24 .72 Q13 .71 Q36 .70 Q40 .69 Q20 .67 Q15 .80 Q18 .76 Q27 .73 Q34 .72 Q11 .71 Q39 .70 Q31 .66 Q23 .64 Q14 .77 Q19 .77 Q10 .76 Q26 .68 Q35 .65 Q38 .55 Figure 2 Scree plot chart of the scale Journal of Science Learning Article DOI: 10.17509/jsl.v5i1.36293 85 J.Sci.Learn.2022.5(1).79-90 examined, it was seen that the load value was .32 and above, and the items did not have overlapping properties. When the factor load values in the sub-dimensions are examined, they were determined that the load values in the first sub- dimension are between .49 and .80, the load values in the second sub-dimension are between .80 and .59, the load values in the third sub-dimensions are between .78 and .67, the load values in the fourth sub-dimensions are between .80 and .64, the load values in the fifth sub-dimensions are between .77 and .55. When the data obtained as a result of EFA are evaluated, the STEM-PCK scale, the first version of 85 items, revealed its implicit structure consisting of 57 items and five sub-dimensions (Table 7). Confirmatory Factor Analysis (CFA). Confirmatory Factor Analysis (CFA) is an analysis performed to test and verify sub-dimensions determined by EFA (Seçer, 2015). It is seen that the x2/df value of the model obtained as a result of the CFA analysis is less than 3, so it provides the acceptable fit value. In addition, NNFI (Non-Normed Fit Index), IFI (Incremental Fit Index), RMSEA (Root Mean Square Error of Approximation), and SRMR (Standardized Root Mean Square Residual) values meet the criteria of good fit; NFI (Normed Fit Index), RFI (Relative Fit Index), CFI (Comparative Fit Index) and RMR (Root Mean Square Residual) Values were evaluated to be in acceptable compliance ranges. These values show that the model tested for the five-factor structure of the STEM-PCK scale examined within the scope of CFA fits well (Table 8). Table 8 data has been interpreted according to the eligibility criteria of Seçer (2015). In figure 3, t values of the model are shown. The t values of the model obtained from the STEM-PCK scale as a result of CFA were examined. Since all the values are higher than 1.96, all variables determined that path in the model are significant. Therefore, it evaluates that no item was incompatible with other things and had an excellent validity (Figure 3). In figure 4, standardized solutions of CFA shows. In this figure, the factor load value of each item has a load value of at least .30 or more was examined. The path diagram given in Figure 4 below review shows that the scale factor load values are at the desired level, between .32 and .73. Therefore it concludes that standard factor loadings have an effect on each factor on variable’s variance. 3.2. Reliability Studies To determine the reliability of the STEM-PCK scale, the internal consistency coefficient was calculated, and test- retest reliability analysis was performed. The test-retest method applies a scale to the same subject group twice under the same conditions and within a specific time interval. The correlation coefficient of the measurement values obtained from the two applications is the reliability coefficient of the scale. For the test-retest reliability study, the scale was applied to 34 pre-service teachers twice with an interval of 21 days. The Cronbach alpha for the total STEM-PCK scale was .98. The coefficients for the factors ranged from .89 for Science Pedagogical Knowledge and .97 for STEM Pedagogical Knowledge. Cronbach Alpha reliability analysis results obtained after the application are given in Table 9. Table 7 Distribution of STEM-PCK scale items by factors at the end of EFA analysis Factors Sub-dimensions Scale items after EFA Number of Items F1 STEM Pedagogical Knowledge (STEM Knowledge for Teaching) 8,42,43,44,45,46,47,48,49,50,51,52,53,54, 55,56,57,58,59,60,61,62,63,80 24 F2 Pedagogical Knowledge 64,65,66,67,68,69,70,71,72,73,76 11 F3 Engineering Pedagogical Knowledge 13,17,24,29,32,36,40,20 8 F4 Mathematics Pedagogical Knowledge 15,18,27,34,11,39,31,23 8 F5 Science Pedagogical Knowledge 14,19,10,26,35,38 6 TOTAL 57 Table 8 Fit indices values for STEM-PCK scale Model fit indices Acceptable fit value Perfect fit value Research findings Interpretation NFI =.90 and above =.95 and above 0.94 Acceptable Fit NNFI =.90 and above =.95 and above 0.96 Perfect Fit IFI =.90 and above =.95 and above 0.96 Perfect Fit RFI =.90 and above =.95 and above 0.94 Acceptable Fit CFI =.95 and above =.97and above 0.96 Acceptable Fit RMR =.05 and <.05 =.08 and<.08 0.04 Acceptable Fit RMSEA =.05 and <.05 =.08 and <.08 0.07 Perfect Fit SRMR =.05 and <.05 =.08 and <.08 0.07 Perfect Fit x2/df 2-3 0-2 2.71 Acceptable Fit Journal of Science Learning Article DOI: 10.17509/jsl.v5i1.36293 86 J.Sci.Learn.2022.5(1).79-90 In this study paired sample t-test is used to determine whether the mean difference between two repeated measures is zero. Also, according to the results of paired samples t-test performed to determine whether there is a significant difference between the two applications, it was observed that there was no significant difference between the first application and the last application (t = -0.91, p> 0.05). This result shows that the STEM-PCK scale gives similar results when applied to the same group at different times (Table 10). 3.3. Creating the Final Form of the Scale The Cronbach Alpha (α) internal consistency reliability value of the STEM-Pedagogical Content Knowledge (STEM-PCK) scale was determined as .98. The internal consistency coefficients of the sub-dimensions of the scale were respectively .97 for STEM Pedagogical Knowledge dimension, .92 for Pedagogical Knowledge dimension, .91 for Engineering Pedagogical Knowledge dimension, .90 for Mathematics Pedagogical Knowledge dimension, and .89 for the Science Pedagogical Knowledge dimension. Test-retest reliability founds to be .97 for the scale total. In addition, test-retest reliability of the sub-dimensions of the scale were respectively .97 for the STEM Pedagogical Knowledge dimension, .93 for the Pedagogical Knowledge dimension, .90 for the Engineering, .92 for the Mathematics Pedagogical Knowledge dimension, and .83 for the Science Pedagogical Knowledge dimension. Cronbach Alpha reliability coefficient value of .70 and above is considered sufficient for the reliability of test scores (Buyukozturk, 2007). Based on this, it can be said that the results to be obtained from the scale are highly reliable. Also, correlations between factors were calculated, and the results are given in Table 11 When Table 11 examines, it determined that there is no correlation between F2 (Pedagogical Knowledge) and F3 (Engineering Pedagogical Knowledge) (r = .29, p> .05). Also, it seems that the lowest correlation is between F4 (Mathematics Pedagogical Knowledge) and F2 (Pedagogical Knowledge), and the highest correlation is between F3 (Engineering Pedagogical Knowledge) and F4 (Mathematics Pedagogical Knowledge). 3.4 Discussion In our age, developments in information and technology cause different needs along with it. There- service teachers are expected to have knowledge about science, technology, and engineering, to be able to use the knowledge they have learned and to integrate it with other subjects, as well as to support their technology knowledge Table 9 Internal consistency and test-retest reliability analysis Factors Factors’name The final version of scale items Number of Items Internal consistency reliability Test-retest reliability F1 STEM Pedagogical Knowledge (STEM Knowledge for Teaching) 1,24,25,26,27,28,29,30,31, 32,33,34,35,36,37,38,39,40 ,41,42,43,44,45,57 24 .97 .97 F2 Pedagogical Knowledge 46,47,48,49,50,51,52,53,54 ,55,56 11 .92 .93 F3 Enginering Pedagogical Knowledge 4,7,10,12,15,17, 20,23 8 .91 .90 F4 Mathematics Pedagogical Knowledge 3,6,8,11,14,16,18,22 8 .90 .92 F5 Science Pedagogical Knowledge 2,5,9,13,19,21 6 .89 .83 Total of scale 57 .98 .97 Table 10 Test-retestresults of the STEM-PCK Scale Application N Mean SD Df t p 1. Application 34 3.93 .59 33 -.91 .37 2. Application 34 4.02 .46 33 Table 11 Correlations between factors (Pearson correlation) (N=34) F1 F2 F3. F4 F5 F1 Correlation 1 .63** .56** .64** .44* F2 Correlation .63** 1 .29 .34* .71** F3 Correlation .56** .29 1 .77** .41* F4 Correlation .64** .34* .77** 1 .41* F5 Correlation .44* .71** .41* .41* 1 ** Correlation is significant at the level (2-tailed) *Correlation is significant at the 0.005 level (2-tailed) Journal of Science Learning Article DOI: 10.17509/jsl.v5i1.36293 87 J.Sci.Learn.2022.5(1).79-90 with content knowledge and field-specific pedagogical method knowledge (Mishra & Koehler, 2006). For this purpose, studies focused on developing a scale on STEM knowledge (Corlu, Capraro, & Corlu, 2015; Derin, Aydın & Kirkiç, 2017) and pedagogical content knowledge (Graham et al., 2009; Schmidt et al., 2009; Landry, 2010; Marks, 1990; Sun & Strobel, 2014). Figure 3 T-values of STEM-PCK scale Journal of Science Learning Article DOI: 10.17509/jsl.v5i1.36293 88 J.Sci.Learn.2022.5(1).79-90 For this reason, integrating STEM into the teaching of teacher candidates has become an academically valuable issue. Therefore, the study aimed to develop a valid and reliable scale to determine pre-service teachers' STEM- Pedagogical Content Knowledge (STEM-PCK) levels. Therefore, a validity and reliability study of the draft scale consisting of 85 items carried out with 322 teacher candidates in different grade levels (1., 2., 3. and 4. grades) Figure 4 Standardized solution of STEM- PCK Scale Journal of Science Learning Article DOI: 10.17509/jsl.v5i1.36293 89 J.Sci.Learn.2022.5(1).79-90 from six (6) other teaching departments in seven (7) different universities in Turkey. The Likert-type scale is preferred chiefly among the scale types because it is practical, increases the grading level, and gives measurement results in an equal-interval scale (Tezbasaran, 2008). The items of the STEM-PCK scale prepare as 5-Likert type. As a result of the EFA, it determined that the KMO value for the STEM-PCK scale was .94, and Bartlett's test result was significant. These results showed that the sample size was suitable for factor analysis. The data set provided multivariate normality as a result of the varimax rotation, the implicit structure of the STEM-PCK scale consisting of 57 items and five sub- dimensions reveal. The STEM-PCK scale's sub- dimensions determined are STEM Pedagogical Knowledge, Pedagogical Knowledge, Engineering Pedagogical Knowledge, Mathematics Pedagogical Knowledge, and Science Pedagogical Knowledge. Yildirim & Sahin-Topalcengiz (2019) developed a STEMPCK scale consist of six factors. This scale has only one sub- dimension, which is Pedagogical Knowledge, similar to the STEM-PCK scale. This study aimed to measure pre-service teachers’ pedagogical knowledge in the area of Science, Mathematics, Engineering, and STEM, while Yildirim & Sahin-Topalcengiz’s (2019) STEMPCK Scale only one dimension to measure the pedagogical knowledge level of pre-service teachers. The 5-factor structure of the scale was confirmed by CFA analysis (χ ² /df=2.71, RMSEA = .07, RMR = 0.04, SRMR = .07, NFI = .94, NNFI = .96, CFI = .96, IFI = .96. RFI = .94). Internal consistency and test-retest reliability analyze conducted to determine the reliability of the scale. As a result, the Cronbach's Alpha (α) internal consistency reliability value of the scale was .98, and the test-retest reliability value was .97. Correlation between 5 factors was calculated in the STEM-PCK scale, and it was determined that the highest correlation was between Engineering Pedagogical Knowledge and Mathematics Pedagogical Knowledge. CONCLUSION It can be said that the scale, which is developed as a result of validity and reliability analysis, consists of 57 items and five factors in 5-point Likert type, is a valid and reliable measurement tool for determining the STEM-Pedagogical Content Knowledge (STEM-PCK) levels of pre-service teachers. With the developed scale, it is thought that it will help measure the Pedagogical Content Knowledge toward STEM education, which is spreading rapidly all over the world, of teacher candidates who are educated in national and international fields. REFERENCES Aydin-Gunbatar, S., Boz, Y., & Yerdelen-Damar, S. (2017). A closer examination of TPACK-self-efficacy construct: Modeling elementary pre-service science teachers’ TPACK-self efficacy. Elementary Education Online, 16(3), 917-934. Buyukozturk, S. (2007). Sosyal bilimler için very analizi el kitabı (7. Baskı).[Data analysis handbook for social sciences (7th Edition)] Pegem A Yayıncılık. Canbazoglu-Bilici, S., Yamak, H., Kavak, N., & Guzey, S. S. (2013). Technological Pedagogical Content Knowledge Self-efficacy Scale (TPACK-SES) for pre-service science teachers: Construction, validation, and reliability. Eurasian Journal of Educational Research, 13(52), 37-60. Creswell, J. W., & Clark, V. L. P. (2017). Designing and conducting mixed methods research. Sage publications. Corlu, S., Capraro, R.M., & Çorlu, M. A. (2015). Investigating the mental readiness of pre-service teachers for integrated teaching. International Online Journal of Educational Sciences, 7 (1), 17-28. Derin, G., Aydin, E., & Kirkiç, K. A. (2017). STEM (Fen-Teknoloji- Mühendislik–Matematik) eğitimi tutum ölçeği. [STEM (Science- Technology-Engineering-Mathematics) education attitude scale].El-Cezeri Journal of Science and Engineering, 4(3), 547-559. El-Deghaidy, H., & Mansour, N. (2015). Science teachers’ perceptions of STEM education: Possibilities and challenges. International Journal of Learning and Teaching, 1(1), 51-54. Epstein, D., & Miller, R. T. (2011). Slow off the mark: Elementary school teachers and the crisis in science, technology, engineering, and math education. Education Digest: Essential Readings Condensed for Quick Review, 77(1), 4–10. Graham, R. C., Burgoyne, N., Cantrell, P., Smith, L., St Clair, L., & Harris, R. (2009). Measuring the TPACK confidence of in-service science teachers. TechTrends, 53(5), 70-79. Kleickmann, T., Richter, D., Kunter, M., Jürgen, E., Besser, M., Krauss, S. & Baumert, J. (2013). Teachers’content knowledge and pedagogical content knowledge: The role of structural differences in teacher education. Journal of Teacher Education, 64(1), 90-106. Kaiser, H. F. (1974). An index of factorial simplicity. Psychometrika, 39(1), 31-36. Knezek, G., Christensen, R. & Tyler-Wood, T. (2015). Teacher dispositions toward Science, Technology, Engineering, and Mathematics (STEM). In Liu, L., & Gibson, D.C. (Ed.), Research highlights in technology and teacher education (p.45-53). AACE. Kind, V. (2009) Pedagogical content knowledge in science education: perspectives and potential for progress. Studies in Science Education, 45(2), 169-204. Landry, G. A. (2010). Creating and validating an instrument to measure middle school mathematics teachers’ Technological Pedagogical Content Knowledge (TPACK) (Unpublished Ph.D. thesis). University of Tennessee, Knoxville. Magnusson, S., Krajcik, J., & Borko, H. (1999). Nature, sources, and development of pedagogical content knowledge for science teaching. In J. Gess-Newsome & N. G. Lederman (Eds.), Examining pedagogical content knowledge: The construct and its implications for science education (pp. 95-132). Kluwer Academic Publishers. Marks, R. (1990). Pedagogical content knowledge: From a mathematical case to a modified conception. Journal of Teacher Education, 41(3), 3- 11. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Sage Publishers. Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017-1054. Pamuk, S., Ulken, A., & Dilek, N. (2012). The investigation of pre- service teachers' technology integration competencies from technological pedagogical cont. Mustafa Kemal Üniversitesi Sosyal Bilimler Enstitüsü Dergisi [Journal of Mustafa Kemal University Institute of Social Sciences] , 9(17), 415-438. Park, S., Suh, J. & Seo, K. (2018). Development and validation of measures of secondary science teachers’ PCK for teaching photosynthesis. Research in Science Education, 48, 549-573. Journal of Science Learning Article DOI: 10.17509/jsl.v5i1.36293 90 J.Sci.Learn.2022.5(1).79-90 Rigelman, N. M. (2014). Using rich assessment and the ATEM Pedagogical Content Knowledge Rubric to examine teacher learning. Paper presented at Oregon's Teacher of Teachers of Mathematics (TOTOM) Conference. https://www.slideshare.net/NicoleRigelman/using- rich-assessments-and-the-pck rubric Sarkim, T. (2020). Developing teachers’ PCK about STEM teaching approach through the implementation of design research. Journal of Physics: Conference Series, 1470, 1-8. Schmidt, D. A., Baran, E., Thompson, A. D., Mishra, P., Koehler, M. J., & Shin, T. S. (2009). Technological pedagogical content knowledge (TPACK) the development and validation of an assessment instrument for pre-service teachers. Journal of research on Technology in Education, 42(2), 123-149. Schmidt, M. & Fulton, L. (2015). Lessons learned from the creation of an exemplary STEM unit for elementary pre-service teachers: A case study. In Liu, L., & Gibson, D.C. (Ed.), Research highlights in technology and teacher education(p. 53-60). AACE (Association for the Advancement of Computing in Education). Seker, H & Gencdogan, B. (2014). Psikolojide ve egitimde ölcme aracı gelistirme (2nd ed.). Nobel Yayıncılık. [Developing a measurement tool in psychology and education (2nd ed.). Nobel Publishing.] Srisawasdi, N. (2012). Fostering pre-service STEM teachers’ technological pedagogical content knowledge: A lesson learned from case-based learning approach. Journal of Korea Association Science Education, 32(8), 1356-1366. Srikoom, W., Hanuscin, D. L., & Faikhamta, C. (2017)). Perceptions of in-service teachers toward teaching STEM in Thailand. Asia-Pacific Forum on Science Learning and Teaching,18, (2), 1-23. Srikoom, W., Faikhamta, C. & Hanuscin, D.L. (2018). Dimensions of effective STEM integrated teaching practice. K-12 STEM Education, 4(2), 313-330. Sun, Y., & Strobel, J. (2014). From knowing-about to knowing-to: Development of engineering-pedagogical content knowledge by elementary teachers through perceived learning and implementing difficulties. American Journal of Engineering Education (AJEE), 5(1), 41-60. Tashakkori, A., & Creswell, J.W. (2007). The new era of mixed methods. Journal of Mixed Methods [Editorial]. Journal of Mixed Methods Research, 1(1), 3-7. Tezbasaran, A. (2008). Likert tipi ölcek hazırlama kılavuzu. [Likert type scale preparation guide. Turkish Psychological Association Publications]. Ankara: Turk Psikologlar Dernegi Yayınları Tiryaki, S. H. (2018). Öğretmenlerin Teknolojik Pedagojik Alan Bilgisi (TPAB) ve eğitim bilişim ağı’nı kullanmalarına yönelik özyeterlik algılarının düzeylerinin incelenmesi. [Examining the levels of self-efficacy perceptions of teachers for their use of Technological Pedagogical Content Knowledge (TPACK) and educational information network.] (Unpublished master’s thesis). Balıkesir Üniversitesi Fen Bilimleri Enstitüsü, Balıkesir. Tyler-Wood, T., Knezek, G., & Christensen, R. (2010). Instruments for assessing interest in STEM content and careers. Journal of Technology and Teacher Education, 18(2), 345-368. Wahono, B., Lin, P.L., & Chang, C.Y. (2020). Evidence of STEM enactment effectiveness in Asian student learning outcomes. International Journal of STEM Education, 7(36), 1-18. York, M. K. (2018). STEM content and pedagogy are not integrated. https://grandchallenges.100kin10.org/assets/downloads/stem- content-and-pedagogy-are-not- integrated/GrandChallengesWhitePapers_York.pdf. Yildirim, B., & Sahin-Topalcengiz, E. (2019). STEM pedagogical content knowledge scale (STEMPCK): A validity and reliability study. Journal of STEM Teacher Education, 53(2), 1-20. https://www.slideshare.net/NicoleRigelman/using-rich-assessments-and-the-pck%20rubric https://www.slideshare.net/NicoleRigelman/using-rich-assessments-and-the-pck%20rubric https://grandchallenges.100kin10.org/assets/downloads/stem-content-and-pedagogy-are-not-integrated/GrandChallengesWhitePapers_York.pdf https://grandchallenges.100kin10.org/assets/downloads/stem-content-and-pedagogy-are-not-integrated/GrandChallengesWhitePapers_York.pdf https://grandchallenges.100kin10.org/assets/downloads/stem-content-and-pedagogy-are-not-integrated/GrandChallengesWhitePapers_York.pdf