Benli-Özdemir, E. (2022). Development of Astronomy Literacy Scale: A Study of Validity and Reliability. International Online Journal of Education and Teaching (IOJET), 9(3). 1356-1376. Received : 19.03.2022 Revised version received : 14.06.2022 Accepted : 15.06.2022 DEVELOPMENT OF ASTRONOMY LITERACY SCALE: A STUDY OF VALIDITY AND RELIABILITY (Resarch article) Esra Benli Ozdemir (0000-0002-2246-2420) Ministry of Education, Turkey esrabenli86@hotmail.com Biodata: Esra BENLİ ÖZDEMİR is science teacher and received her master’s and doctoral degrees from Gazi University Graduate School of Educational Sciences Department of Science Education. Her research interests are contemporary teaching methods and techniques, conceptual change, and qualitative and quantitative research methods. Copyright © 2014 by International Online Journal of Education and Teaching (IOJET). ISSN: 2148-225X. Material published and so copyrighted may not be published elsewhere without written permission of IOJET. mailto:esrabenli86@hotmail.com http://orcid.org/xxxx International Online Journal of Education and Teaching (IOJET) 2022, 9(3), 1356-1376. 1357 DEVELOPMENT OF ASTRONOMY LITERACY SCALE: A STUDY OF VALIDITY AND RELIABILITY Esra Benli Özdemir esrabenli86@hotmail.com Abstract The aim of this research is to develop a valid and reliable "Astronomy Literacy Scale" in order to determine the astronomy literacy levels of individuals. Quantitative research method was used in the study. The study group of the research consisted of 5th, 6th, 7th and 8th grade students consists of a total of 519 students studying in 3 different secondary schools in Ankara in the spring term of the 2021-2022 academic year. The data obtained after the application were analyzed with SPSS and Lisrel statistical programs. As a result of the factor analyzes carried out, it was seen that the developed scale was a structure consisting of a total of 15 items with 3 factors. The “Astronomy Literacy Scale (AOS)” consists of four separate parts and 21 judgments in a five-point Likert type. In the scale, part one, self-perception towards astronomy; the second part, the affective dimension to astronomy; the third part includes the behavioral dimension for astronomy and the fourth part includes the cognitive dimension for astronomy. In this research, it has been revealed that the "astronomy literacy scale", which aims to determine the astronomy literacy levels of individuals, is a valid and reliable measurement tool. Keywords: astronomy literacy, astronomical literacy scale, validity, reliability. 1. Introduction The desire to know and explore the sky, the concern for understanding, interest and curiosity have attracted attention to individuals of all ages since ancient times. The importance of astronomy has gradually increased with the fact that astronomy has been a subject of interest, curiosity and research for people of all age groups since ancient times and with the rapidly developing technology. An efficient, effective and successful astronomy curriculum should aim at attitudes, values, interests, motivation, observations and practices towards astronomy as well as cognitive knowledge and skills. In this context, affective and behavioral gains should be included along with cognitive acquisitions related to astronomy at all levels of education. Whether individuals have cognitive, affective and behavioral gains in astronomy subjects and concepts can only be evaluated with "astronomy literacy". The concept of "astronomy literacy" refers to a broad content knowledge about affective and behavioral dimensions as well as basic astronomy concepts. An astronomy literate individual; conceptual knowledge of basic astronomy in terms of cognitive dimension; positive attitude, interest and motivation towards astronomy in terms of affective dimension; In terms of behavioral dimension, it should have observation, participation and action related to astronomy. In a study, the astronomy literacy level of 990 adults over the age of 18 was determined. These individuals were asked six true-false questions. 80% of the participants gave correct answers to 4 of the 6 questions asked. However; The questions that astronomers "found life on Mars" and "can calculate the age of the universe" are among the two questions most frequently answered incorrectly by the participants (Love et al., 2013). When the literature on astronomy is examined, it is emphasized that astronomy literacy is an important part of scientific literacy. The use of different methods and techniques in mailto:esrabenli86@hotmail.com Benli-Özdemir 1358 teaching astronomy concepts and subjects plays a very important role in gaining affective and behavioral goals. In studies carried out to determine attitudes towards astronomy in the literature (Öner Armağan and Demir, 2019; Balbağ and Erdem, 2017; Bektaşlı,2013; Canbazoğlu Bilici, Öner Armağan, Kozcu Çakır, and Yuruk, 2012; Uçar and Demircioğlu, 2011;Wittman, 2009); It is noteworthy that the attitude scales used by Zeilik, Schau, & Mattern, 1999 are adaptation studies of the attitude scale towards astronomy. However, the scales developed are not multidimensional for astronomy; It has been determined that it is aimed at one of the cognitive, affective or behavioral dimensions. However, there are limited studies on developing scales for astronomy (Ertaş Kılıç and Keleş, 2017; Türk and Kalkan, 2017; Zeilik et al., 1999). In this direction, the aim of the study has been shaped based on the absence of a measurement tool for astronomy literacy that can be used by researchers and educators in the literature. In the study, it was aimed to develop a valid and reliable measurement tool for measuring the astronomy literacy levels of individuals. Based on the purpose of the research, the main problem and sub-problems of the research were formed. The main problem of the research: Is the developed "Astronomy Literacy Scale (ALS)" a valid and reliable scale? determined as. The sub-problems that are sought to be answered in the research are: • Does the developed “Astronomy Literacy Scale” have a sufficient level of validity in terms of content validity? • Does the developed “Astronomy Literacy Scale” have a sufficient level of validity in terms of construct validity? • Is the reliability level of the developed “Astronomy Literacy Scale” sufficient? • Are the item properties of the developed “Astronomy Literacy Scale” sufficient? determined as. 2. Method In this part of the research, information about the research model, research process, research sample, data collection tool, data analysis, research ethics and statistical methods and techniques used in data analysis are given. 2.1. Research Model The research is the study of developing a valid and reliable measurement tool for measuring the astronomy literacy levels of individuals. Quantitative research method was used in the study. The quantitative research method can measure the reaction of a large number of individuals on the research topic with a limited number of questions. In this way, it makes it possible to compare and statistically collect data, and to obtain a generalizable set of findings presented in a concise manner (Patton, 2014: 14). 2.2. Study Group The study group of the research consisted of 5th, 6th, 7th and 8th grade students consists of a total of 519 students studying in 3 different secondary schools in Ankara in the spring term of the 2021-2022 academic year. Participation of the students in the study group was ensured on a voluntary basis. In the study, easily accessible case sampling was chosen from purposive sampling methods. In this sampling method, a situation that is easy to access and close is selected (Yıldırım & Şimşek, 2018). In addition, this method is known and preferred as a method that adds practicality and speed to the study (Gök, Turan, & Oyman, 2011). International Online Journal of Education and Teaching (IOJET) 2022, 9(3), 1356-1376. 1359 The study was carried out in 3 different secondary schools in Ankara. The students studying in these secondary schools are capable of reflecting all segments. Therefore, it is possible to say that the studied participants are mixed. In this context, the study is generalizable. It is thought that it is important that the scale to be developed in scale studies is aimed at each individual (Arı & Aslan, 2020). In this respect, it can be said that the scale is generalizable thanks to the fact that the study group is composed of disadvantaged and advantageous student groups. When the literature is examined, it is stated that the number of participants should be at least five times the number of items in the scale in order to carry out factor analysis (Child, 2006; Bryman & Cramer, 2001). In this context, the draft form of the scale to be developed was composed of 45 items. 12 items were removed from the ALS, in which the item pool was created with 45 items, and the scale turned into 33 items in line with expert opinions. In the EFA analysis, it was deemed appropriate to determine the sample size as 519 people (Child, 2006; Bryman & Cramer, 2001). After the EFA analysis, the remaining items were re- administered to a total of 324 students in a different student group. In this context, it was deemed appropriate to determine the sample size as 324 participants to be used in the CFA analysis. 2.3. Research Process The research process of the study; first of all, it was ensured that the literature on the subject was searched, the compositions about astronomy were written to 10 selected students and the item pool was created by consulting the expert opinion. A detailed literature review was conducted in order to form the expressions to be included in the "Astronomy Literacy Scale (ALS)" to be developed within the scope of the study. During the scale development process, it is expected that the literature on the structure to be measured will be scanned and the conceptual framework of the structure will be revealed clearly (Cohen & Swerdlik, 2010). Then, feedback was received on the items of the scale in line with the opinions of 3 academicians, 3 science teachers and 1 language expert. As a result of the evaluations, the items were rearranged by the researcher. From the scale in which the item pool was created with 45 items, 12 items were removed from the ALS in line with expert opinions. The scale was transformed into 33 items. The draft form consisting of 33 items was applied to a total of 247 participants, different from the sample. The main application of the scale, whose pilot implementation was completed, was carried out. At the last stage, the content validity, construct validity and reliability analyzes of the scale were made and the scale was given its final shape. 2.4. Data Collection Tool The draft form of the “Astronomy Literacy Scale (ALS)” consists of four separate parts and 45 judgments in a five-point Likert type. First of all, a short and concise instruction regarding the scale was prepared and added to the beginning of the draft form in order to facilitate the participants. In the scale, part one, self-perception towards astronomy; the second part, the affective dimension to astronomy; the third part includes the behavioral dimension for astronomy and the fourth part includes the cognitive dimension for astronomy. The astronomy literacy levels of the students are numbered from 1 point to 5 points from the lowest to the highest. 2.5. Analysis of Data In the study, data collected from secondary school 5th, 6th, 7th and 8th grade students in the study group were analyzed using quantitative methods. The answers given by the students to the scale form were first transferred to the SPSS statistics program. The most distinctive Benli-Özdemir 1360 feature of the data collection tools used in research is the validity and reliability of the measurements (Ural & Kılıç, 2013). In this context, first of all, necessary analyzes were made for the content and construct validity of the scale. In order to ensure content validity, it was determined that the scale questions covered the features desired to be measured, taking into account the feedback from field experts. In order to ensure construct validity, exploratory factor analyzes (EFA) of the scale data are required. However, before the EFA was conducted, the suitability of the data for factor analysis was checked. For this reason, Kaiser- Mayer-Olkin (KMO) and Barlett's Test analyzes were performed in order to determine that the sample size was sufficient and showed a normal distribution. According to the results of the analysis, it was determined that he was ready for exploratory factor analysis (EFA) (Büyüköztürk, 2018). Then, Exploratory Factor Analysis (EFA) was performed on the data determined to be suitable for exploratory factor analysis. It is stated that a factor load value above 0.45 during EFA is considered a good item, and this factor value can be reduced to 0.30 for a small number of items (Büyüköztürk, 2019). After the exploratory factor analysis, confirmatory factor analysis (CFA) was performed to confirm the items in the scale. Confirmation of the developed scale with confirmatory factor analyzes indicates that that scale is a valid scale (Yaşlıoğlu, 2017). The factor loadings and fit indices obtained by confirmatory factor analysis were validated. In order to determine the reliability of the scale, the Cronbach alpha internal consistency coefficient of the scale was calculated. Reliability; little or no measurement error, and the results are similar or the same when measuring a quality more than once (Sönmez & Alacapınar, 2014). In addition, the total item correlation values of the scale items and the mean independent t-test values of the lower-upper group were also calculated. 2.6. Compliance with Ethical Rules Ethical principles and rules were followed at all stages of this research. In order to determine the compliance of the research with the ethical rules, it was discussed at the meeting of Gazi University Ethics Committee dated 24.05.2022 and numbered 10, and the approval of the Ethics Committee was obtained with the letter dated 01.06.2022 and numbered E. 374376. Approval document related to ethics committee approval is presented in Appendix 2. 3. Findings In this section, the data obtained for the analyzes made within the scope of the research are given. 3.1. Findings Regarding the "Astronomy Self-Perception" Dimension of the Astronomy Literacy Scale In line with the main purpose of the research, astronomy literacy scale development studies were carried out. In the scale, the first part includes the self-perception dimension for astronomy. In this dimension, which is included in the scale, the answers of the participants such as "the level of knowledge about astronomy, doing studies on astronomy, getting information about astronomy, the people they talk to about astronomy, the frequency of talking about astronomy" are given in the table below. International Online Journal of Education and Teaching (IOJET) 2022, 9(3), 1356-1376. 1361 Table 1. Descriptive Analysis of Participants on the Dimension of "Self-Perception of Astronomy" f % Level of knowledge about astronomy Too much 24 4.62 More 54 10.40 Intermediate 302 58.18 Not much 71 13.68 None 68 13.12 Total 519 100 Frequency of studying on astronomy Too much 74 14.25 More 65 12.52 Intermediate 203 39.11 Not much 97 18.69 None 80 15.43 Total 519 100 Way to learn about astronomy Lessons at school 302 58.19 Internet 108 20.81 Tv 90 17.34 Article/book/magazine/newspaper 12 2.31 Family/friends 7 1.35 Total 519 100 People he/she talked to about astronomy Teachers 295 56.85 Friends 79 15.22 Family 16 3.08 People in the virtual environment 37 7.13 No one 92 17.72 Total 519 100 Frequency of speaking about astronomy Too much 61 11.75 More 62 11.95 Intermediate 196 37.76 Not much 113 21,78 None 87 16,76 Total 519 100 Benli-Özdemir 1362 When Table 1 is examined, it is seen that the level of knowledge of secondary school students about astronomy and the frequency of doing studies on astronomy are at a moderate level. However, it is noteworthy that the most common way for students to learn about astronomy is the lessons at school. It is stated that the participants talk about astronomy mostly with their teachers and the frequency of speaking about astronomy is at a medium level. 3.2. Findings Regarding the Validity of the Astronomy Literacy Scale In the study, exploratory and confirmatory factor analysis was performed for the construct validity of the "Astronomy Literacy Scale (ALS)". 3.2.1. Findings on exploratory factor analysis (EFA) Exploratory factor analysis (EFA) was conducted in order to reveal the construct validity of the measurement tool developed within the scope of the research. The suitability of the data obtained from the PPS measurement tool given to 519 participants without factor analysis (EFA) of the Astronomy Literacy Scale (ALS) was examined. The results of the Kaiser Meyer Olkin (KMO) and Barttlett sphericity test were analyzed and results of the test are summarized in Table 1. Table 2. Astronomy Literacy Scale KMO and Barlett Test Results KMO Coefficient .702 Bartlett Test Result <.01 When Table 2 is examined, the KMO value of the study was calculated as .702. Bartlett's test was significant for p<.05. As the KMO coefficient is higher than .60 and the Bartlett sphericity test is significant at the .01 level, it is seen that the data are suitable for factor analysis (Büyüköztürk, 2008; Pallant, 2007). Accordingly, it was decided that the data were suitable for factor analysis. The second step in EFA is to determine the number of factors for the POS measurement tool. In this context, factor eigenvalues and explained variance rates are given in Table 3. Table 3. Factor Eigenvalues and Explained Variance Rates for the Astronomy Literacy Scale Factors Total Explained Variance (%) Cumulative(%) 1 4.81 25.31 25.31 2 3.75 19.75 45.06 3 2.01 10.89 55.95 When Table 3 is examined, the total variance rates explained as a result of factor analysis are seen. It was revealed that 33 items in the scale were grouped under 3 factors, with their International Online Journal of Education and Teaching (IOJET) 2022, 9(3), 1356-1376. 1363 initial eigenvalues greater than 1. It is understood that the items in the scale explained 55.95% of the total variance. The first factor explains 25.31%, the second factor explains 19.75% and the third factor explains 10.89% of the total variance. The factor-eigenvalue line graph (screen plot) of the values that emerged in line with the exploratory factor analyzes performed is presented in Figure 1. Figure 1. Astronomy literacy scale factor scree plot As the third step in EFA, factor loading values obtained by Direct Oblimin rotation technique for the three-factor structure in the measurement tool were examined. Insufficient factor loading or overlapping problematic items were determined. The factor load values of the items were determined as .30. As a result of the item analysis, five items (M17, M11, M15, M28, M32) with a factor load value below .30 were removed from the scale. (+-,10) value is taken as basis for factor load overlapping limit. According to the results of the item analysis, 11 items (M10, M27, M24, M31, M18, M20, M29, M13, M12, M16, M30) contributing to both factors were excluded from the measurement tool. At this stage, a total of 16 items were removed from the measurement tool. Benli-Özdemir 1364 Table 4. EFA results for the astronomy literacy scale Item No Item Factor 1 Affective Dimension Factor 2 Cognitive Dimension Factor 3 Behavioral Dimension 1 I am interested in astronomy. .803 2 I enjoy learning new information about astronomy. .757 3 I can easily learn concepts and subjects related to astronomy. .812 8 I make mistakes when explaining astronomy concepts. .656 9 Astronomy has no contribution to my daily life. .728 16 I don't want to do studies on the sky. .728 28 I can express the geometric shape of the sun. .518 29 I can explain the direction, duration, and consequences of the Sun's rotation. .703 33 I can express the geometric shape of the Earth. .709 35 I can't tell the size of the moon. .580 36 I can explain the moon's age. .687 37 I can express the geometric shape of the moon. .740 39 I cannot explain how stars are formed. .518 19 I observe the sky when natural events such as “Solar Eclipse” or “Lunar Eclipse” occur. .821 20 I observe the sky using sky survey programs (Google sky, sky map, NASA, sky walk etc.). .804 21 I watch the sky with the naked eye (without any observation tool) at night when the weather conditions are favorable. .679 22 I follow astronomer or astronauts working on the sky on social media. .729 Explained Variance (%) Total = 55.95 %25.31 %19.75 %10.89 When Table 4 is examined, as a result of the analyzes made, the number of items of the 3- factor measurement tool is as follows: It was reduced to a total of 17 items, as six items in Factor 1, seven items in Factor 2, and four items in Factor 3. International Online Journal of Education and Teaching (IOJET) 2022, 9(3), 1356-1376. 1365 3.2.2. Findings on confirmatory factor analysis (CFA) It is aimed to determine to what extent a predetermined structure is confirmed by the collected data in confirmatory factor analysis, (Büyüköztürk et al., 2004). After the EFA analysis, the remaining items were reapplied to a total of 324 students in a different student group. In this context, it was deemed appropriate to determine the sample size as 324 participants to be used in the CFA analysis. In the CFA procedures, the three-factor structure obtained from the EFA results was analyzed. According to the results of the fit statistics and modification indices, the suitability of the factor structure was examined. The fit indices of ALS calculated by CFA and the indices accepted in the relevant literature are given in Table 5. Table 5. Fit indices of ALS calculated by CFA DFA Compliance Index Variable Study Findings Good Model Criteria (Çokluk, Şekercioğlu & Büyüköztürk, 2010) Chi-Square/Degree of Freedom χ2/sd 186.77/101=1.85 < 5 Eligibility Index GFI .90 ≥ .90 Non-Normized Compliance Index NNFI .97 Comparative Fitness Index CFI .97 Square Root of Standardized Mean Errors SRMR .072 ≤ .08 Root Mean Square of Approximate Errors RMSEA .071 ≤ .08 When Table 5 is examined, when the findings obtained as a result of CFA and the values accepted in the relevant literature are compared, it shows that all of the fit indices (GFI, CFI, NNFI, RMSEA, SRMR, χ2/sd) are at a good level of the model. At this stage, the significance and error variances of the t values were examined and the non-significant t value was removed from the M19 scale. In the light of the findings, it was concluded that the model was compatible with the data. The path diagram showing the standardized coefficients between the item-implicit variable and the latent variables for POS is given in Figure 2. Benli-Özdemir 1366 Chi-Square= 186.77, df= 101, p-value=.0000, RMSEA=.07 Figure 2. Path Chart Showing the Standardized Coefficients Between Item-Implicit Variables and Implicit Variables for AOS *Factor 1: “Affective Dimension”, Factor 2: “Behavioral Dimension”, Factor 3: “Cognitive Dimension” 3.3. Findings on the reliability of the astronomy literacy scale In order to ensure the reliability of the astronomy literacy scale, Cronbach's alpha internal consistency coefficients were calculated for the whole scale and its factors. In this context, the reliability of the scale was examined and tested. In line with the reliability analyzes of the scale, the calculated Cronbach Alpha values coefficient of the scale is presented in Table 7. International Online Journal of Education and Teaching (IOJET) 2022, 9(3), 1356-1376. 1367 Table 6. Factors and reliability coefficients of the whole scale Factor Number Factors (Sub Dimensions) Number of Items Cronbach Alpha Values Coefficient Factor 1 Affective Dimension 6 .84 Factor 2 Behavioral Dimension 3 .72 Factor 3 Cognitive Dimension 7 .70 Total Full Scale 16 .75 When Table 6 is examined, the data regarding the Cronbach's alpha internal consistency coefficient of the developed scale can be seen. As a result of the analysis, it was calculated as .84 for the "Affective" dimension, .72 for the "Behavioral" dimension and .70 for the "Cognitive" dimension. Cronbach α coefficient for the overall scale. Calculated as 75. As a result of the analysis, it is seen that all of the reliability coefficients are higher than .70. Therefore, it can be said that the scale items are internally consistent with each other. As a result, 29 items were removed from the POS, which was included as 45 items in the item pool. 12 items with expert opinions, 16 item with the AFA application, 1 item with the CFA application were removed from the scale. As a result of the analyzes made; ALS consists of 16 items and three dimensions. The first dimension with six items was named “Affective”, the second dimension with seven items was named as “Cognitive” and the second dimension with three items was named “Behavioral”. The score that can be obtained from the scale varies between 16-80. 3.4. Findings regarding item analysis of the astronomy literacy scale In order to determine the internal consistency of the scale items, the item-total score correlation is also used together with the calculation of the Cronbach's Alpha coefficient. Item-total score correlation is one of the methods used within the scope of item analysis. One of the ways used within the scope of calculating the item analysis was tested using the unrelated t-test for the differences between the item averages of the lower 27% and upper 27% groups, which were formed according to the total scores of the test. In this context, item analysis of the developed astronomy literacy scale was based on the difference between the lower 27% and upper 27% group averages. For this, the total scores of the 324 student group from the scale were ordered from the lowest to the highest, and lower and upper (88 people) groups were formed. The mean scores obtained from the groups formed were analyzed using the independent t-test. After the analysis, the results of the analysis based on the difference between the lower and upper group averages are shown in Table 7. Benli-Özdemir 1368 Table 7. Item analysis results based on the difference between lower and upper group means Item No Group X Sd Item-Total Correlation T-volue p 1 Bottom 27% 3.78 0.872 .42 -6.826 p≤.001 Top 27% 4.41 0.494 2 Bottom 27% 3.55 0.905 .57 11.038 p≤.001 Top 27% 4.60 0.492 3 Bottom 27% 3.33 1.106 .66 -11.550 p≤.001 Top 27% 4.62 0.486 8 Bottom 27% 3.16 1.034 .59 -13.932 p≤.001 Top 27% 4.63 0.484 9 Bottom 27% 3.06 1.109 .63 -13.148 p≤.001 Top 27% 4.54 0.501 16 Bottom 27% 3.70 1.019 .48 -11.68 p≤.001 Top 27% 4.86 0.345 28 Bottom 27% 3.42 1.077 .71 -12.13 p≤.001 Top 27% 4.73 0.448 29 Bottom 27% 2.58 1.019 .70 -15.088 p≤.001 Top 27% 4.01 0.092 33 Bottom 27% 2.74 0.759 .46 -11.275 p≤.001 Top 27% 3.68 0.503 36 Bottom 27% 2.85 1.069 .45 -7.371 p≤.001 Top 27% 3.68 0.554 37 Bottom 27% 2.56 0.687 .36 -11.349 p≤.001 Top 27% 3.62 0.729 39 Bottom 27% 1.89 0.717 .56 -22.254 p≤.001 Top 27% 3.78 0.574 19 Bottom 27% 2.63 0.867 .23 -8.859 p≤.001 Top 27% 3.60 0.799 20 Bottom 27% 2.81 1.370 .18 -3.372 p≤.001 Top 27% 3.38 0.945 21 Bottom 27% 2.50 1.179 .26 -6.497 p≤.001 Top 27% 3.34 0.745 22 Bottom 27% 2.57 1.177 .23 -3.982 p≤.001 Top 27% 3.11 0.869 When Table 7 is examined, item analyzes based on the difference between the bottom 27% and top 27% group averages are seen in order to determine the ability of the scale items to distinguish the participants. According to the table, there is a significant difference between the mean scores of the groups (p<.001). It is seen that the t values of all items in the scale are significant. In this respect, it can be easily said that scale items can easily distinguish between individuals who show or do not display the desired behavior to measure. International Online Journal of Education and Teaching (IOJET) 2022, 9(3), 1356-1376. 1369 When the table is examined, it is seen that the item-total score correlation coefficients calculated for the items in the scale vary between .18 and .71. After the item analysis based on the difference between the bottom and top groups, the Pearson correlation coefficient was used to determine whether the factors in the scale were independent. In this context, the Pearson correlation coefficient values calculated between the factors are presented in Table 8 in order to show that the scale factors are independent from each other. Table 8. Correlation values between factors Factor-1 Factor-2 Factor-3 Factor-1 1 -0.82 0.48* Factor-2 -0.82 1 0.27 Factor-3 0.48* 0.27 1 **p<0.01, r= Pearson Correlation Coefficient When Table 8 is examined, it is seen that there is a statistically significant relationship (p< .01) between the factors in the scale. Between the first factor and the second factor; a high and negative relationship (r = - .82, p< .01), a moderate and positive relationship between the first and third factor (r = .48, p< .01), between the second and third factor; it can be said that there is a low level and positive relationship (r= .27, p< .01). 4. Conclusion, discussion and recommendations In this study, it is aimed to develop a valid and reliable "Astronomy Literacy Scale" in order to determine the astronomy literacy levels of individuals. In this context, on the basis of the stated purpose, the "astronomy literacy scale" has been developed as a valid and reliable measurement tool. In accordance with the purpose of the research, first of all, a literature review on the subject and compositions about astronomy were written to 10 selected students. Then, an item pool was created by taking expert opinion. A detailed literature review was conducted in order to create the expressions to be included in the "Astronomy Literacy Scale (ALS)" to be developed within the scope of the study. Afterwards, feedback was received on the items of the scale in line with the opinions of 3 academicians, 3 science teachers and 1 language expert. As a result of the evaluations, the items were rearranged by the researcher. The item pool was created with 45 items. 12 items were removed from the scale in line with expert opinions, and the scale was transformed into 33 items. The 33-item draft form was applied to a total of 247 participants, different from the sample. The main application of the scale, whose pilot implementation was completed, was carried out. At the last stage, the scale was given its final form after the content validity, construct validity and reliability analyzes of the scale were made. The scale was composed of 33 items and a 5-point Likert type, and construct validity studies were started. In the study, the suitability of the data set for factor analysis was Benli-Özdemir 1370 examined with the Kaiser-Meyer-Olkin (KMO) coefficient and the Barlett Sphericity test. Since the KMO value is greater than 0.60 indicates that the data are suitable for factor analysis (Büyüköztürk, 2019), this can be interpreted as the sample size is sufficient. Bartlett test results were found to be significant and it was observed that the data were suitable for factor analysis (p<0.05). In the first factor analysis, it was revealed that 33 items in the scale were grouped under 3 factors, with their initial eigenvalues greater than 1. It is understood that the items in the scale explained 55.95% of the total variance. 25.31% of the total variance of 55.95% is explained by the first factor, 19.75% by the second factor and 10.89% by the third factor. As the third stage in the EFA of AOS, the factor loading values obtained by the Direct Oblimin rotation technique for the three-factor structure in the measurement tool were examined and items with insufficient factor loading or with overlapping problems were determined. The factor load values of the items were determined as .30. As a result of the item analysis, five items (M17, M11, M15, M28, M32) with a factor load value below .30 were removed from the scale. For factor load overlapping limit, (+-,10) value was taken as basis. In line with the results of the item analysis, 11 items (M10, M27, M24, M31, M18, M20, M29, M13, M12, M16, M30) contributing to both factors were excluded from the measurement tool. At this stage, a total of 16 items were removed from the measurement tool. As a result of the analysis, the number of items in the 3-factor measurement tool was reduced to a total of 17 items, six items in Factor 1, seven items in Factor 2, and four items in Factor 3. The items collected under factor 1 were affective dimension; items collected under factor 2 behavioral dimension; The items collected under factor 3 were named as cognitive dimension. When the findings obtained as a result of CFA are compared with the values accepted in the relevant literature, it shows that all of the fit indices (GFI, CFI, NNFI, RMSEA, SRMR, χ2/sd) of the model are at a good level. At this stage, the significance of the t values and the error variances were examined. Accordingly, a total of 1 item (M19) was excluded from the measurement tool. As a result of the analyzes made, the number of items in the 3-factor measurement tool was reduced to a total of 16 items, six items in Factor 1, seven items in Factor 2, and three items in Factor 3. Each factor should consist of at least 3 items (Özdamar, 2017). Therefore, it can be said that the adequacy of the number of items in the scale factors has been ensured. According to the results of the confirmatory factor analysis study; similarity rate was determined as chi-square statistic χ2/df = 1.85 (p=.000). Root mean square of approximate errors (RMSEA)= 0.071; root-square error of standardized mean (SRMR) = 0.72; goodness of fit index (GFI)=0.90; normed fit index (NFI)=0.97; comparative fit index (CFI)= 0.97. The results obtained have acceptable fit values and confirm the factor structure of the astronomy literacy scale. In addition, the fact that the SRMR value, which gives the model fit regarding the standardized errors of the model, is less than 0.08 (Hu & Bentler, 1999), can be considered as a strong indicator of the data fit with the model. Considering all these values, it can be said that the model of the scale consisting of 16 items and three dimensions has an acceptable model goodness value. In order to ensure the reliability of the astronomy literacy scale, Cronbach's alpha internal consistency coefficients were calculated for the whole scale and its factors. In this context, the reliability of the scale was examined and tested. Data on the Cronbach's alpha internal consistency coefficient of the developed scale can be seen. As a result of the analysis, it was calculated as .84 for the "Affective" dimension, .72 for the "Behavioral" dimension and .70 for the "Cognitive" dimension. Cronbach α coefficient for the overall scale. Calculated as 75. International Online Journal of Education and Teaching (IOJET) 2022, 9(3), 1356-1376. 1371 As a result of the analyzes, it is seen that all of the reliability coefficients are higher than .70 (Büyüköztürk, 2018). In this context, a valid and reliable astronomy literacy scale was developed in the research. This Scale was developed in a five-point Likert type, consisting of a total of 16 items, three dimensional, 5 negative and 11 positive statements (Appendix 1). The scale effectively measures the cognitive, affective and behavioral characteristics of individuals for astronomy. The lowest score that can be obtained from the scale is 16, and the highest score is 80 points. In the first part of the scale, there is also the self-perception dimension for astronomy. In this dimension, which is included in the scale, it measures the perceptions of the participants about astronomy as "the level of knowledge about astronomy, the way of doing studies on astronomy, the way of getting information about astronomy, the people they talk to about astronomy, the frequency of talking about astronomy". The reliability and validity values of the scale prove that the calculated astronomy literacy scale is a measurement tool that can be used by researchers. According to the results obtained from the research, the following suggestions can be made to researchers and practitioners:  The effect of astronomy literacy level on different variables can be examined.  Astronomy literacy scale can be used to determine effective teaching methods for students to acquire cognitive, affective and behavioral characteristics for astronomy education.  The scale can be applied as a pre-test and post-test by providing training on astronomy literacy to students at all levels of education, from pre-school to university. Thus, the effect of the given education on astronomy literacy can be examined.  Instructors can determine students' astronomy literacy levels with this scale and plan their teaching in this direction. Benli-Özdemir 1372 References Balbağ, M. Z., & Erdem, A. (2017). Fen bilgisi öğretmenliği ve fizik bölümü öğrencilerinin astronomiye yönelik tutumlarının bazı değişkenler açısından incelenmesi. Kastamonu Eğitim Dergisi, 25(5), 2007-2018. Bektaşlı, B. (2013). The Effect Of Media On Preservice Science Teachers’ Attitudes Toward Astronomy And Achievement In Astronomy Class. The Turkish Online Journal of Educational Technology, 12(1), 139-146. Bryman, A., & Cramer, D. (2001). Quantitative data analysis with spss release 10 for windows: a guide for social scientists. London: Routledge. Büyüköztürk, Ş., Akgün, Ö., Kahveci, Ö., & Demirel, F. (2004). The validity and reliability study of the Turkish version of the motivated strategies for learning questionnaire. Educational Sciences: Theory & Practice, 4(2), 207-239. Büyüköztürk, Ş. (2018). Sosyal Bilimler İçin Veri Analizi El Kitabı (24. Baskı). Ankara: Pegem A Yayıncılık. Büyüköztürk, Ş. (2019). Sosyal Bilimler İçin Veri Analizi El Kitabı: İstatistik, Araştırma Deseni SPSS Uygulamaları ve Yorum. Ankara: Pegem Akademi Yayıncılık Canbazoğlu Bilici S., Öner Armağan F., Kozcu Çakır, N., & Yürük, N. (2012). Astronomi tutum ölçeğinin Türkçe’ye uyarlanması: Geçerlilik ve güvenilirlik çalışması. Türk Fen Eğitimi Dergisi, (9), 116-127. Child, D. (2006). The essentials of factor analysis. (3rd ed.). London: Continuum. Cohen, R. J., & Swerdlik, M. (2010).psychological testing and assessment: An ıntroduction to tests and measurement. New York: McGraw-Hill Book Co. Çokluk, Ö., Şekercioğlu, G. & Büyüköztürk, Ş. (2010). Sosyal bilimler için çok değişkenli istatistik: SPSS ve LISREL uygulamaları. Ankara: Pegem Akademi. Ertaş Kılıç H., & Keleş, Ö.(2017). Astronomiye yönelik ilgi ölçeği geliştirilmesi: geçerlik ve güvenirlik çalışmaları. Eğitimde Kuram ve Uygulama,13(1), 35-54. Gök, A., Turan, S., & Oyman, N. (2011). Okul Öncesi Öğretmenlerinin Bilişim Teknolojilerini Kullanma Durumlarına İlişkin Görüşleri, Pegem Eğitim ve Öğretim Dergisi, 1 (3), 59-66. Görgülü Arı, A., & Aslan, K. (2020). İklim Okuryazarlığı Ölçeği Geliştirme Çalışması. Jass Studies-The Journal of Academic Social Science Studies, 13(82), 147-174. Hu, L, T., & Bentler, P. M. (1999). Cut‐off criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1‐55. Öner Armağan, F. & Demir, N. (2019). Astronomiye yönelik tutum ölçeği geliştirilmesi: Geçerlik ve güvenirlik çalışması. International Social Sciences Studies Journal, 5(35), 2718-2731. Pallant, J. (2007). SPSS Survival Manual (3rd Edition). McGraw-Hill: New York. Patton, M.Q. (2014). Nitel araştırma ve değerlendirme yöntemleri. 3. Baskıdan Çeviri. Bütün, M & Demir, S. B. (Edt.), Ankara: Pegem Akademi. International Online Journal of Education and Teaching (IOJET) 2022, 9(3), 1356-1376. 1373 Sönmez, V., & Alacapınar, F.G. (2014). Örneklendirilmiş Bilimsel Araştırma Yöntemleri. Ankara: Anı Yayıncılık. Türk, C., & Kalkan, H. (2017). Yükseköğretim öğrencilerine yönelik astronomi tutum ölçeği uyarlama çalışması. Celal Bayar Üniversitesi Sosyal Bilimler Dergisi, 15(3), 69-95. Ucar, S., & Demircioglu, T. (2011). Changes in preservice teacher attitudes toward astronomy within a semester-long astronomy instruction and four-year-long teacher training programme. Journal of Science Education and Technology, 20(1), 65-73. Ural, A., & Kılıç, İ. (2013). Bilimsel Araştırma Süreci ve SPSS İle Veri Analizi. Ankara: Detay Yayıncılık. Yaşlıoğlu, M. M. (2017). Sosyal bilimlerde faktör analizi ve geçerlilik: Keşfedici ve doğrulayıcı faktör analizlerinin kullanılması. İstanbul Üniversitesi İşletme Fakültesi Dergisi, 46, Özel Sayı, 74-85. Yıldırım, A., & Şimşek, H. (2018). Sosyal Bilimlerde Nitel Araştırma Yöntemleri. Ankara: Seçkin Yayıncılık. Zeilik, M. Schau, C., & Mattern, N. (1999). Conceptual Astronomy. II. Replicating Conceptual Gains, Probing Attitude Changes Across Three Semesters. American Journal of Physics, 67(10), 923-927. Wittman, D. (2009). Shaping attitudes toward science in an introductory astronomy course. The Physics Teacher, 47(9), 591-594. Benli-Özdemir 1374 Appendices Appendix 1. ASTRONOMY LITERACY SCALE CHAPTER 1 (Astronomy Literacy Self-Perception) 1. How much do you think you know about astronomy? A. Too much B. More C. Intermediate D. Not much E. None 2. How would you describe yourself about doing studies/research on astronomy? A. I do a lot of study/research. B. I do more work/research. A. I do a moderate amount of study/research. D. I don't do much study/research. E. Never 3. Which of the following contributes the most to learning the subjects and concepts related to astronomy? A. Lessons at school B. Internet C. Television D. Article/book/magazine/newspaper E. Family/Friends 4. Which of the following do you talk about studies, researches or subjects related to astronomy? A. Teachers B. Friends C. Family D. People in the virtual environment E. Nobody 5. How often do you talk to people around you about astronomy-related topics? A. Too much B. More C. Intermediate D. Not much E. None International Online Journal of Education and Teaching (IOJET) 2022, 9(3), 1356-1376. 1375 CHAPTER 2 (The Affective Dimension of Astronomy Literacy) S tr o n g ly a g re e A g re e N e it h e r a g re e n o r d is a g re e D is a g re e S tr o n g ly d is a g re e 1. 1. I am interested in astronomy. 2. 2. I enjoy learning new information about astronomy. 3. 3. I can easily learn the concepts and subjects related to astronomy. 4. 4. I make mistakes when explaining astronomy concepts. 5. 5. Astronomy has no contribution to my daily life. 6. 6. I don't want to work on the sky. CHAPTER 3 (The Behavioral Dimension of Astronomy Literacy) S tr o n g ly a g re e A g re e N e it h e r a g re e n o r d is a g re e D is a g re e S tr o n g ly d is a g re e 7. 1. I observe the sky using sky survey programs (Google sky, sky map, NASA, sky walk etc.). 8. 2. I watch the sky with the naked eye (without any observation tool) at night when the weather conditions are suitable. 9. 3. I follow the astronomer or astronauts who work on the sky on social media. CHAPTER 4 (Cognitive Dimension of Astronomy Literacy) S tr o n g ly a g re e A g re e N e it h e r a g re e n o r d is a g re e D is a g re e S tr o n g ly d is a g re e 10. 1. I can express the geometric shape of the sun. 11. 2. I can explain the direction, duration and consequences of the Sun's rotation. 12. 3. I can express the geometric shape of the Earth. 13. 4. I cannot tell the size of the moon. 14. 5. I can explain the moon's age. 15. 6. I can express the geometric shape of the moon. 16. 7. I cannot explain how stars are formed. Benli-Özdemir 1376 Appendix-2: Ethics Committee Approval