Dilekçi, A. & Çiçek, S. (2022). An analysis of secondary school Turkish language course assessment tools in the sense of PISA reading skill criteria. International Online Journal of Education and Teaching (IOJET), 9(1), 417-431. Received : 03.10.2021 Revised version received : 05.12.2021 Accepted : 09.12.2021 AN ANALYSIS OF SECONDARY SCHOOL TURKISH LANGUAGE COURSE ASSESSMENT TOOLS IN THE SENSE OF PISA READING SKILL CRITERIA (Research article) Atilla Dilekçi 0000-0001-5393-9570 (corresponding author) Ministry of Education, Turkey dilekciatilla@gmail.com Seher Çiçek 0000-0002-1501-2675 Ministry of Education, Turkey sehertabak123@gmail.com Biodatas: Both authors are Turkish language teachers at the Ministry of Education, Turkey. Copyright © 2014 by International Online Journal of Education and Teaching (IOJET). ISSN: 2148-225X. Material published and so copyrighted may not be published elsewhere without written permission of IOJET. mailto:dilekciatilla@gmail.com mailto:sehertabak123@gmail.com http://orcid.org/xxxx http://orcid.org/xxxx Dilekçi & Çiçek 418 AN ANALYSIS OF SECONDARY SCHOOL TURKISH LANGUAGE COURSE ASSESSMENT TOOLS IN THE SENSE OF PISA READING SKILL CRITERIA Atilla Dilekçi dilekciatilla@gmail.com Seher Çiçek sehertabak123@gmail.com Abstract The purpose of this research is to evaluate secondary school Turkish language assessment tools in the sense of PISA reading skills criteria. To this end, document analysis, one of qualitative research methods, was employed in the current study. The research data were collected from 82 assessment tools prepared by 23 teachers. The questions included in these assessment tools were analysed based on cognitive processes defined by PISA reading, proficiency levels and text genres. As a result, the questions used in assessment tools have been found to centre upon the cognitive stages of reaching and understanding information and 1a proficiency level. Besides, it has been seen that the most frequently-used text format used in assessment tools is continuous texts. The most frequently-used text genre in the assessment tools is determined as instructional texts; however, narrative and explanatory texts were found to be included more frequently. The results yielded that the general perspective of the questions used in secondary school Turkish language course assessment tools is not suitable for PISA reading skills criteria. Consequently, it has been recommended that assessment and evaluation activities in secondary school Turkish language course are required to be determinedly and conspiratorially adapted to PISA reading skills criteria. Keywords: assessment tool, PISA, PISA reading skill criteria, Turkish Language Course. 1. Introduction The Programme for International Student Assessment (PISA) was launched by The Organisation for Economic Co-operation and Development, abbreviated as OECD, with the purpose of measuring students’ academic success on an international scale. First administered in 2000 and conducted every 3 years, PISA claims to measure 15 -year- olds’ ability to use their reading, mathematics and science knowledge and skills. The major domain of assessment rotates between reading, mathematics and science in each cycle; however, in 2000, 2009 and 2018, students’ reading literacy was aimed to be measured. Basically, the reading literacy skill is regarded as the ability to understand written language forms. However, it is now generally accepted that our understanding of reading literacy has shifted and the definition of reading literacy has expanded. In this regard, the concept of reading literacy skill is the process of comprehending, reflecting and assessing multidimensional complex structures through effective use of cognitive processes. In other words, it may be defined as effectively reading in line with a certain purpose and task (OECD, 2010). As a notable consequence of rapid spread of information and communication technology among the public, reading has massively shifted from print to digital texts. Students of Generation Z widely use the screen reading tools such as mobile phones, laptops, or computers. This generation often carries out its digital networking acts via chatrooms, mailto:dilekciatilla@gmail.com mailto:sehertabak123@gmail.com International Online Journal of Education and Teaching (IOJET) 2022, 9(1), 417-431. 419 social media and e- newspapers (OECD, 2019). From this point of view, as of 2018, PISA has been performed as a computer- based assessment. With the massive innovations and changes in the nature of reading literacy, the emergence of information, media and technology literacy has been observed. Therefore, readers are, now, required to use the cognitive processes of higher order thinking skills since reaching the information is not enough in case the information is abundant and manipulated by different sources (Allcott, Gentzkow and Yu, 2019); the sources of information should be investigated, credibility of information should be evaluated, the truths and fallacies in a piece of text should be recognized (OECD, 2019) and following all these steps, information should be constructed. The PISA typology of cognitive processes involved in reading literacy was as follows: locating information, understanding, evaluating and reflecting. Students are expected to access and retrieve related information within a piece of text in the process of location information. However, text understanding process refers to the tasks of acquiring a representation of the literal meaning of a text, constructing an integrated text and producing inferences. The third process, which is evaluating and reflecting on, is engaged when students assess the quality and credibility of the information within a piece of text, reflect on the content and form and detect and handle the conflicts (Ministry of National Education, 2019). The process of locating information corresponds to the Knowledge dimension of Bloom Taxonomy, understanding process to the Comprehension and Analysis dimension and evaluating and reflecting process to the Evaluation dimension. In PISA cycles, the reading scale is divided into a range of proficiency levels as Levels 1b, 1a, 2, 3, 4, 5 and 6, in ascending order of proficiency. Each proficiency level poses different levels of challenges arranged in order of difficulty. These competency levels include certain tasks as representing the literal meaning of single or multiple texts in the absence of explicit content or organizational clues, assessing the credibility of the text and constructing a text. The question types, or tasks, in PISA reading literacy may be classified as multiple choices, yes- no and true- false questions. These questions are the ones in which students make a choice among possibilities. Certain questions in Reading Unit may also entail long or short constructed responses. Through those tasks, students are required to not only choose the right option among the possibilities but also to construct and interpret the truth by themselves. As PISA is a reading assessment, not a writing assessment, writing skills (spelling, grammar, organization) are not evaluated. Indeed, the main purpose is to exclude other factors except the acquisitions aimed to be measured from assessment process. In designing PISA reading literacy assessment, there is a broad coverage of what students read and for what purposes students read. Thus, the PISA reading literacy aims to measure students’ mastery of reading and managing those texts by representing them a natural range of texts and tasks. Therefore, six major text types have been identified: descriptions, narrations, expositions, argumentations, instructions and transactions (OECD, 2021). Expositions, argumentations, narrations and descriptions are the most well- known and commonly- used text types. On the contrary, instructions and transactions may be said not to be used widely in education. An instruction is the act, process or art of imparting knowledge or skill. Samples of instruction may be food recipes, a device’s installation steps or a manual. Transactions are characterized by their purpose of presenting the information to the reader through an interaction. Samples of transactions may be letters or messages between two interlocutors, invitations, questionnaires or interview forms. The texts in PISA reading literacy assessment are evaluated in three ways as continuous, non- continuous and mixed texts. The difference among these texts is the representation of information (Bozkurt, 2016). Continuous texts are formed by sentences organised into Dilekçi & Çiçek 420 paragraphs. Non- continuous texts are composed of a number of lists. Mixed texts, however, consists of a set of elements in continuous and non- continuous formats (OECD, 2021). Accordingly, essays, novels, short stories, reviews are accepted as continuous texts; lists, tables, graphs, diagrams and schedules as non- continuous texts and a paragraph together with a picture, or a graph as mixed texts. The inclusion of different text formats in reading literacy assessment is due to the fact that students are required to encounter with a wide range of texts in daily life (Bozkurt, 2016). PISA is a monitoring research based mainly on reading literacy skills. Reading literacy is defined by PISA as a life- long interactive process, not merely a skill acquired during the first years of education (Aşıcı et al., 2019; MoNE, 2018; OECD, 2019). It is acknowledged that language courses mostly centre upon reading skills. Through texts, teachers focus on students’ reading comprehension skills. Yet, Turkish students have been found to be unable to obtain high scores from PISA reading literacy assessment since reading skills entail various skills apart from reading comprehension skill. In consequence, PISA results have an impact on participating countries’ education policies, curricula, course books and teachers’ competencies (Benzer, 2000). A number of factors such as curricula, teacher training systems, teaching technologies, physical conditions of schools and assessment- evaluation processes have been determined on the basis of countries’ global objectives (Koç, 2021). In order to enhance their success in PISA, such countries as Turkey, Ireland and Austria have improved their curricula and made certain amendments in their teacher guides (Benzer, 2020). Previous studies in the literature investigated secondary school Turkish Language curricula (Batur and Ulutaş, 2013; İnce, 2016; İşeri, 2019; Karabulut, 2017; Koç, 2021; Sefer et al., 2017; Temizyürek and İnce), Turkish Language course books (Benzer, 2019; Bozkurt, Uzun and Lee, 2015; Coşkun, 2013; Yağmur, 2009) and transition to secondary education exams(Aşıcı, Baysal and Şahenk, 2012; Batur, Ulutaş and Beyret, 2019; Savran, 2004; Tuzlukaya, 2019) within the context of PISA reading literacy criteria. The above- mentioned studies have yielded a common result that curricula, course books and national exams do not comply with PISA assessment. Besides, no study which examines secondary school Turkish language course assessment instruments based in PISA reading literacy criteria has been found. 1.1. Objective and Research Questions Apart from previous studies, the main purpose of this research is to evaluate secondary school Turkish language assessment tools in the sense of PISA reading skills criteria. To this end, this study, in fact, was guided by the following questions: 1. What is the general perspective on cognitive processes and proficiency levels of the questions used in secondary school Turkish language course assessment instruments? 2. What is the general perspective on text formats of the questions used in secondary school Turkish language course assessment instruments? 3. What is the general perspective on text types of the questions used in secondary school Turkish language course assessment instruments? 2. Method 2.1. Research Design In the current study aiming to evaluate secondary school Turkish language assessment tools in the sense of PISA reading skills criteria, document analysis, one of qualitative research methods, was employed. The main prerequisite of this method is to access to appropriate documents related to the research purpose (Karasar, 2012) and to rigorously and International Online Journal of Education and Teaching (IOJET) 2022, 9(1), 417-431. 421 systematically analyse the contents of written, published or oral documents concerning the research aim (Yıldırım and Şimşek, 2013). Accordingly, related research documents were obtained in order to submit a general perspective on the questions used in secondary school Turkish language course assessment instruments based on PISA reading literacy criteria. The descriptive analysis of these assessment instruments were conducted with respect to PISA reading literacy criteria. 2.2. Data Collection Procedure The research data were collected from written exam papers used in Turkish language course in 2020-2021 academic year. A total of four written exams, as two exams in each term, are conducted in an academic year. 82 written exam papers constitute the research data. Of all papers, 13 papers belong to 5th grades, 22 to 6th grades, 22 to 7th grades and 22 to 8th grades. The written exam papers used in the current study were obtained from 23 teachers. Demographic information concerning teachers is presented in Table 1. Table 1. Demographic information of teachers from which assessment instruments were gathered Demographic Information  Gender Female 14 Male 9 Age 21-30 years 5 31-40 years 14 41-50 years 4 Professional Seniority 1-10 years 7 11-20 years 14 21-30 years 2 Level of Education Bachelor’s Degree 18 Master’s Degree 4 Doctor’s Degree 1 As seen in Table 1, out of the total participants, 14 of Turkish teachers were female and 9 were male. Of all Turkish teachers, 5 of them are 21- 30, 14 are 31- 40 and 4 are 41- 50 years old. As for professional seniority, 7 of Turkish teachers have 1-10, 14 have 11-20 and 2 have 21- 30 years of experience. In addition, 18 of Turkish teachers have bachelor’s degree, 4 have master’s degree and 1 have doctor’s degree. 2.3. Data Analysis In the current study, descriptive analysis, one of qualitative research methods, was used to analyse research documents. In this type of analysis, firstly, themes are generated; data obtained from the analysis of documents are coded under relevant themes (Yıldırım and Şimşek, 2013). Accordingly, the data were examined based on PISA reading literacy criteria Dilekçi & Çiçek 422 which are locating information, understanding, evaluating and reflecting in order to analyse the first research question. The data whose cognitive processes were determined were placed under appropriate one among 1c, 1b, 1a, 2, 3, 4, 5 and 6 proficiency levels. In the present study, cognitive process and proficiency level of 770 questions were attempted to be placed under appropriate category. With the purpose of analysing the second research question, texts were classified as continuous, non- continuous and mixed based on text formats. In order to analyse the third research question, texts used in assessment tools were analysed according to text types that were description, narration, exposition, argumentation, poetry, instruction and transaction identified in PISA. Text formats and types of 618 questions were classified under appropriate category in the present study. Since more than one question was asked in the same text, 618 questions were analysed according to text format and type although 770 questions were analysed based on cognitive processes and proficiency levels. Besides, grammar questions in written exam papers were excluded from the study as those questions are not included in PISA. Following descriptive analysis of the data obtained, frequency of themes were computed and tabulated. 2.4. Validity and Reliability In order to ensure reliability in the current research, the data were analysed independently and coded under categories by researchers. The researchers, then, compared the results obtained. As a result of comparison, inter-rater reliability between researchers was 95% (Miles and Huberman, 1994). The data on which researchers disagreed were reconsidered again and coded under appropriate themes. Researchers, therefore, achieved a consensus. 3. Results This section covers the findings concerning cognitive processes, proficiency levels, text formats and text types of the questions used in secondary school Turkish language course assessment tools. The results were presented in accordance with sub- statements of the study together with tables. 3.1. Findings concerning the First Sub-Research Question Findings concerning PISA cognitive processes and proficiency levels of questions used in secondary school Turkish language course assessment tools were shown in Table 2. International Online Journal of Education and Teaching (IOJET) 2022, 9(1), 417-431. 423 Table 2. Cognitive processes and proficiency levels of the questions used in secondary school Turkish language course assessment instruments Class Level The Number of Assessment Instruments The Number of Questions Proficiency Levels Locating Information Understanding Evaluating and Reflecting     5 13 177 1c 30 23 6 1 1b 33 30 3 - 1a 94 38 56 - 2 15 3 12 - 3 5 - 5 - 4 - - - - 5 - - - - 6 - - - - 6 22 218 1c 33 24 8 1 1b 43 38 5 - 1a 110 46 61 3 2 28 - 28 - 3 4 - 4 - 4 - - - - 5 - - - - 6 - - - - 7 25 195 1c 17 14 3 - 1b 42 40 2 - 1a 88 36 50 2 2 36 2 32 2 3 10 - 3 7 4 - - - - 5 2 - 2 - 6 - - - - 8 22 180 1c 22 16 5 1 1b 13 10 3 - 1a 78 36 39 3 2 42 1 39 2 3 19 - 9 10 4 - - - - 5 6 - 6 - 6 - - - - According to Table 2, as for cognitive processes of questions used in secondary school Turkish language course assessment tools, in 5th grade assessment tools, 94 questions were found to be at locating information level, 82 at understanding level and 1 at evaluating and reflecting level. In 6th grade assessment tools, 108 questions were indicated to be at locating information level, 106 at understanding level and 4 at evaluating and reflecting level. As for 7th grade assessment tools, 92 questions were revealed to be at locating information level, 92 at understanding level and 11 at evaluating and reflecting level. Finally, in 8th grade Dilekçi & Çiçek 424 assessment tools, 63 questions were shown to be at locating information level, 101 at understanding level and 16 at evaluating and reflecting level. Table 2 also shows proficiency levels of questions used in secondary school Turkish language course assessment tools. Accordingly, in 5th grade assessment tools, 30 questions were placed under 1c, 33 questions under 1b, 94 questions under 1a, 15 questions under 2nd and 5 questions under 3rd proficiency levels. As for 6th grade assessment tools, 33 questions were placed under 1c, 43 questions under 1b, 110 questions under 1a, 28 questions under 2nd and 4 questions under 3rd proficiency levels. However, in 7th grade assessment tools, 17 questions were placed under 1c, 42 questions under 1b, 88 questions under 1a, 36 questions under 2nd, 10 questions under 3rd and 2 questions under 5th proficiency levels. When looking at 8th grade assessment tools, 22 questions were placed under 1c, 13 questions under 1b, 78 questions under 1a, 42 questions under 2nd, 19 questions under 3rd and 6 questions under 5th proficiency levels. 3.2. Findings concerning the Second Sub-Research Question Findings concerning PISA text formats of questions used in secondary school Turkish language course assessment tools were shown in Table 3. Table 3. Text formats of questions used in secondary school Turkish language course assessment instruments Class Level The Number of Examined Assessment Instruments The Number of Examined Texts Text Format  5 13 143 Continuous Text 140 Non- continuous Text 0 Mixed Text 3 6 22 174 Continuous Text 171 Non- continuous Text 0 Mixed Text 3 7 25 154 Continuous Text 149 Non- continuous Text 2 Mixed Text 3 8 22 147 Continuous Text 146 Non- continuous Text 0 Mixed Text 1 As seen in Table 3, in 5th grade assessment tools, 140 texts were determined as continuous and 3 as mixed texts. As for 6th grade assessment tools, 171 texts were found as continuous and 3 as mixed texts. However, in 7th grade assessment tools, 149 texts were determined as continuous, 2 as non- continuous and 3 as mixed texts. When looking at 8th grade assessment tools, 146 texts were determined as continuous and 1 as mixed texts. It was indicated that no mixed text was included in 5, 6 and 8th grade assessment tools. International Online Journal of Education and Teaching (IOJET) 2022, 9(1), 417-431. 425 3.3. Findings concerning the Third Sub-Research Question Findings concerning PISA text types of questions used in secondary school Turkish language course assessment tools were presented in Table 4. Table 4. Text types of questions used in secondary school Turkish language course assessment instruments Class Level The Number of Examined Assessment Instruments The Number of Examined Questions Text Type  5 13 143 Description 5 Narration 19 Exposition 12 Argumentation 1 Poetry 5 Instruction 101 Transaction 0 6 22 174 Description 3 Narration 23 Exposition 22 Argumentation 7 Poetry 10 Instruction 108 Transaction 1 7 25 154 Description 5 Narration 20 Exposition 27 Argumentation 11 Poetry 6 Instruction 85 Transaction 0 8 22 147 Description 6 Narration 14 Exposition 46 Argumentation 4 Poetry 6 Instruction 71 Transaction 0 According to Table 4, as for text types used in secondary school Turkish language course assessment tools, in 143 questions used in 5th grade assessment tools, 101 texts were determined as instruction, 19 as narration, 12 as exposition, 5 as poetry, 5 as description and 1 as argumentation. As for 174 questions used in 6th grade assessment tools, 108 texts were determined as instruction, 23 as narration, 22 as exposition, 10 as poetry, 7 as argumentation, 3 as description and 1 as transaction. In 154 questions used in 7th grade assessment tools, however, 85 texts were determined as instruction, 27 as exposition, 20 as narration, 11 as argumentation, 6 as poetry and 5 as description. Finally, in 147 questions used in 8th grade assessment tools, 71 texts were determined as instruction, 46 as exposition, 14 as narration, 6 Dilekçi & Çiçek 426 as poetry, 6 as description and 4 as argumentation. It was demonstrated that no transaction text type was included in 5, 7 and 8th grade assessment tools. 4. Discussion and Conclusion The findings of the current research whose purpose is to evaluate secondary school Turkish language assessment tools in the sense of PISA reading skills criteria were elaborated on the basis of relevant studies in the literature. Previous studies in the literature posit that when examining the questions in transition to secondary education exam in terms of PISA reading literacy cognitive processes, they mostly address to ‘understanding’ proficiency level (Aşıcı, Baysal and Şahenk, 2012; Tuzlukaya, 2019). Tuzlukaya (2019) stated that Turkish language questions in 2017 transition to secondary education exam centred upon ‘understanding’ level. Similarly, Aşıcı, Baysal ve Şahenk (2012) noted that Turkish language questions in 2009 transition to secondary education exam centred upon ‘understanding’ level. On the contrary, studies investigating questions in Turkish language course books based on PISA reading literacy cognitive processes showed that questions were at ‘locating information’ level to a great extent. In this regard, Uzun and Lee (2015) determined that in Korean and Turkish language course books, 50% of the questions were at ‘locating information’ level, 27% at ‘understanding’ level and 23% at ‘evaluating and reflecting’ level. Likewise, Benzer (2019) found out that questions used in Turkish language course books mostly focused on ‘locating information’ level. Accordingly, it may be concluded that questions in Turkish language course books are at ‘locating information’ in terms of cognitive process. Besides, it has been understood that cognitive processes of questions used in Turkish language course books are below the cognitive levels of questions used in central examinations. Indeed, Yağmur (2009) observed that no current approach to reading was included in Turkish language course books. In the present study, it was seen that questions used in 5, 6, 7, and 8th grades Turkish language course assessment tools centre upon ‘locating information’ and ‘understanding’ cognitive processes. Thus, the fact that questions used in secondary school Turkish language course assessment tools focus on both cognitive processes is in accordance with central examination system held for transition to secondary education. However, it has been revealed that questions used in 8th grade assessment tools were mostly at ‘understanding’ level. The number of questions that are ‘evaluating and reflecting’ level increases from 5th grade to 8th grade; yet, the number of those questions are still highly low. Similarly, Demiral and Menşan (2017) stated that questions which were prepared by teachers and used to evaluate 8th grade Turkish language course aimed to measure knowledge and basic language skills. Therefore, the fact that a limited number of questions regarding ‘evaluating and reflecting’ level was included in secondary school Turkish language course assessment tools shows that students are not provided opportunities of reflecting and evaluating the quality and reliability of a given text. For this reason, as for PISA 2003- 2006 results, reading skills of Turkish students were indicated to be lower compared to the students from top five OECD countries and reading skills of only a minority of students were at higher levels (Aydın, Erdağ and Taş, 2011). As for the studies investigating Turkish language course curriculum based on PISA reading literacy skills, acquisitions were mostly observed to be lower levels (Batur and Ulutaş, 2013; Demiral and Menşan, 2017; İnce and Gözütok, 2016; İşeri, 2019; Karabulut, 2017; Koç, 2021). Batur and Ulutaş (2013), İnce and Gözütok (2016) and Koç (2021) examined 2016 and 2019 5-8th grades Turkish language course curriculum and found out that, in both curricula, acquisitions were mostly at 1a, 1b, 2 and 3rd competency levels. Additionally, it was also revealed that no acquisition at 5th and 6th competency levels was included in the curricula. İşeri (2019) determined that 9th grade acquisitions in Literature course curriculum hardly reached to PISA 3rd competency level. Nevertheless, Karabulut International Online Journal of Education and Teaching (IOJET) 2022, 9(1), 417-431. 427 (2017) stated that the majority of acquisitions in Turkish language curse curriculum were not in accordance with PISA reading literacy competencies. Demiral and Menşan (2017) noted that a great number of acquisitions in 8th grade Turkish language course curriculum were at moderate levels and no higher- level reading competency was included. According to the results of afore-mentioned studies, acquisitions of Turkish language course curriculum are in compliance with PISA reading literacy varying up to 3rd competency skills. In the current study, it was shown that questions used in 5, 6, 7, and 8th grade assessment tools centred upon 1, 2 and 3rd competency levels from more to less. Contrary to this, in 7th grade, 2 questions were determined to be at 5th competency level and, in 8th grade, 6 questions were found to be at 5th competency level. The results show that levels of acquisition in the curriculum and question in Turkish language course assessment tools may reach to 3rd competency level; therefore, acquisitions in the curriculum and questions used in assessment tools were found be consistent. Nonetheless, it may be said that questions in assessment tools are able to reach to 3rd competency level in PISA reading literacy assessment and questions towards higher competency levels are not included in assessment tools. Among the reasons of this situation, Demiral and Menşan (2017) highlight teachers’ incompetencies of writing questions. In fact, researchers stated that the questions examined were parallel to one another and teachers made use of similar educational websites while preparing questions. Yıldız (2021) asserted that teachers used the questions in course book for assessment and evaluation process and also added that they referred to lack of time and students’ low academic levels as a reason for this situation. Likewise, Karatay and Dilekçi (2019) revealed that teachers were unable to write appropriate questions for cognitive levels by emphasizing that questions were limited to knowledge and comprehension levels. Besides, similarities between the content and types of questions used in assessment tools may cause the levels of questions to remain low. Indeed, Karatay and Dilekçi (2019) observed that the most- frequently used questions by teachers in exams were multiple choices and determined that as class level increased the rate of the use of those questions rose. Demiral and Menşan (2017) also noted that almost all questions in transition to secondary education exam were at moderate level based on Bloom taxonomy and no open-ended question intended to measure higher order thinking levels was included in these examinations. For this reason, the fact that multiple choices questions facilitate assessment and evaluation processes in crowded classes and are preferred in central examinations may lead teachers to use these type of questions to a greater extent. When investigated the questions in transition to secondary education based on PISA text format, continuous texts were determined to be used in questions (Savran, 2004; Tuzlukaya, 2019). Savran (2004) observed no compliance between question types and topics in transition to secondary education exams and those of PISA. Similarly, Tuzlukaya (2019) also reported that continuous texts were used in all Turkish language course questions in transition to secondary education exams. In the present study, however, as for text formats used in secondary school Turkish language course assessment tools, almost all texts were revealed as continuous texts. It was found out that 3 mixed texts were used in 5 and 6th grades; 3 mixed and 2 non- continuous texts in 7th grade and 1 mixed text in 8th grade. The results indicated that text formats used in secondary school Turkish language course assessment tools were not in accordance with those used in PISA assessment since such non- continuous and mixed texts as tables, diagrams and charts were employed in PISA reading literacy assessment. In line with the results of the present study, İnce (2016) reported that texts used in Turkish language course books did not vary to a great extent and observed that certain non- continuous texts as forms, graphs, lists, schemes, plans, catalogues or tables were not included. Moreover, Benzer and Evci (2021) determined that the most- frequently used text format used in course books were continuous texts. Coşkun (2013) found that assessment and evaluation questions in the course books addressed to lower-level reading skills due to Dilekçi & Çiçek 428 frequent use of continuous texts. In a study conducted by Yıldız (2021) with teachers, it was posited that non- continuous texts were not used in classes due to the presence of continuous texts in course books and teachers were not well- informed regarding types and contents of texts in PISA assessment. Tuzlukaya (2019) investigated questions used in transition to secondary education exams based on PISA text types and found that 11 argumentations, 4 narrations, 2 descriptions, 2 expositions and 1 poetry were included in 2017 transition to secondary education exam and stated that such text types as instruction and transaction were included in the exam. In parallel, İnce (2016) emphasized that narrations were mostly used in Turkish language course books. Sefer et al. (2017) examined text types in Turkish language course curriculum on the basis of PISA text types criteria and reported that no non- continuous text format as bulletin boards, charts, graphs, tables, diagrams, maps, certificates, receipts and forms were included in the curriculum. In the current study, text types used in questions in secondary school Turkish language course assessment tools were determined as instructions. The questions with this text type have been found to be built on words and short sentences. In accordance with the results of a study conducted on course books, in this study, the most- frequently used text types were indicated as expositions and narrations. Poetry, argumentation and description were revealed to be used in assessment tools at similar rates. Previous studies in the literature have shown that students engaged in reading tent to achieve higher scores in such exams built on reading and reading comprehension skills. Furthermore, Jerrım and Yosunu (2018) examined the association between the frequency teenagers read five different types of text (magazines, non- fiction, fiction newspapers and comics) and their PISA reading scores and found evidence of a sizeable ‘fiction effect’; that is, young people who were engaged in this type of text frequently had significantly stronger reading skills compared to their peers who did not. Therefore, it was concluded that the inclusion of narrations in course books and assessment tools was in students’ benefits. Besides, Evci (2021) developed a ‘Textuality Rubric in PISA Criterion’ and found that the majority of 99 continuous texts in secondary school Turkish language course books were at sufficient levels. On the contrary, Kemiksiz (2018) observed that these texts ensued one another in each year and class level. In sum, course books as the most- frequently used teaching tools should include different and original texts. In the light of these explanations, it may be concluded that the text types used in curricula, course books, central examinations and in- class assessment tools are traditional and limited compared to the texts used in PISA assessment. 5. Suggestions In the light of current research findings, it may be posited that open-ended questions are required to be included to a greater extent in secondary school Turkish language course assessment tools since this question type provides a measurement of reading comprehension and higher order cognitive skills based on reading literacy skills. In addition, non- continuous and mixed texts can be used more in assessment and evaluation questions in line with PISA text formats. In accordance with PISA text types, teachers can make use of original fictions both during lectures and in assessment activities. The fact that certain amendments towards cognitive processes, proficiency levels, text types and text formats within the context of PISA reading literacy skills have been represented in course books is inevitable for the coherence and consistency of assessment and teaching processes. Lastly, certain in- service training regarding writing questions in accordance with PISA reading literacy skills criteria for secondary school Turkish language teachers may be organized in order to foster their assessment competencies. International Online Journal of Education and Teaching (IOJET) 2022, 9(1), 417-431. 429 References Allcott, H., Gentzkow M., & C. Yu (2019). Trends in the diffusion of misinformation on social media. Research & Politics. 6(2), 1-8. 30 March 2021 retrieved from http://dx.doi.org/10.1177/2053168019848554 Aşıcı, M., Baysal, Z. N., & Şahenk, S. (2012). Türkiye’de yapılan 2009 PISA ve seviye belirleme sınavındaki (SBS) okuma becerileri sorularının karşılaştırılması. Journal of Research in Education and Teaching, 1(2), 2146-9199. Aydın, A., Erdağ, C., & Taş, N. (2011). A comparative evaluation of PISA 2003–2006 results in reading literacy skills: an example of top-five OECD countries and Turkey. Educational Sciences: Theory & Practice, 11(2), 665-673. Batur, Z., & Ulutaş M. (2013). PISA ile Türkçe öğretim programındaki okuduğunu anlama kazanımlarının örtüşme düzeyinin incelenmesi. Journal Of Academic Social Studies Jass, 6(2), 1549-1562. 15 January 2021 retrieved from http://dx.doi.org/10.9761/jasss_651 Batur, Z., Ulutaş, M., & Beyret, T. (2019). 2018 LGS Türkçe sorularının PISA okuma becerileri hedefleri açısından incelenmesi. Milli Eğitim Journal, Primary Education Special Issue, 48(1), 595-615. Benzer A., & Evci, B. (2021). PISA ölçütünde metinsellik rubriği geliştirme. Turkish Journal of Primary Education, 6(1), 18-35. 3 October 2021 retrieved from http://dx.doi.org/10.52797/tujped.796710 Benzer, A. (2019). Türkçe Ders kitaplarının PISA okuma becerileri yeterlik düzeyleri ile imtihanı. Okuma Yazma Eğitimi Araştırmaları, 7(2), 96-109. 11 February 2021 retrieved from http://dx.doi.org/10.35233/oyea.659740 Benzer, A. (2020). Türk eğitim sisteminden PISA ne istiyor?. Neşîde Magazine, 5(5), 66-76. Bozkurt, B. Ü. (2016). Türkiye’de okuma eğitiminin karnesi: PISA ölçeğinde çıkarımlar. Abant İzzet Baysal University Journal of Education Faculty, 16(4), 1673-1686. Bozkurt, B. Ü., Uzun, G.L., & Lee, Y. H. (2015). Korece ve Türkçe ders kitaplarındaki metin sonu sorularının karşılaştırılması: PISA 2009 sonuçlarına dönük bir tartışma. International Journal of Language Academy, 3(9), 295-313. Coşkun, Y. D. (2013). Türkçe ders kitaplarının PISA sınavı okuma ölçütleri açısından incelenmesi. Mehmet Akif Ersoy University Journal of Education Faculty, 13(26), 22 - 43. Demiral, H., & Menşan, N. (2017). Sekizinci sınıf Türkçe dersinin PISA okuma becerilerine göre değerlendirilmesi. Küreselleşen dünyada eğitim (Edt: Özcan Demirel, Serkan Dinçer). Ankara: Pegem Publishing. Dilekçi, A. (2021). 21. yüzyıl becerilerine göre tasarlanan öğretim etkinliklerinin eleştirel ve yaratıcı düşünme becerilerini geliştirmeye etkisi (Unpublished Doctoral Dissertation). Bolu Abant İzzet Baysal University, Bolu. Evci, B. (2021). Ortaokul Türkçe ders kitaplarındaki metinlerin PISA ölçütünde metinsellik özelliklerine göre incelenmesi (Unpublished Master Thesis). Marmara University, İstanbul. İnce, M. (2016). Türkçe 6, 7, 8. sınıf öğretim programının uluslararası öğrenci değerlendirme programı’nda (PISA) yoklanan "okuma becerileri" açısından analizi (Zonguldak örneği) (Unpublished Doctoral Dissertation). Ankara University, Ankara. http://dx.doi.org/10.1177/2053168019848554 http://dx.doi.org/10.9761/jasss_651 Dilekçi & Çiçek 430 İnce, M., & Gözütok, F. D. (2016). Türkçe 6, 7, 8. sınıf öğretim programının uluslararası öğrenci değerlendirme programında (PISA) yoklanan okuma becerileri açısından analizi (Zonguldak örneği). Kastamonu Education Journal, 26(5), 1613-1621. İşeri, A. (2019). Uluslararası PISA yeterlikleri ve Türkiye öğretim programları kazanımları. Mersin University Journal of Education Faculty, 15(2), 392-418. Jerrim, J., & Yosunu, G. (2018). The link between fiction and teenagers’ reading skills: International evidence from the OECD PISA study. British Educational research Journal, 45(1). 181-200. Karabulut, A. (2017). MEB’in yayınlamış olduğu örnek PISA sorularının 2015 Türkçe öğretim programı okuma anlama kazanımları çerçevesinde analizi. International Journal of Education Science and Technology, 3(3), 166-174. Karasar, N. (2012). Bilimsel araştırma yöntemi. Ankara: Nobel Publishing. Karatay, H. ve Dilekçi, A. (2019). Türkçe öğretmenlerinin dil becerilerini ölçme ve değerlendirme yeterlikleri. Milli Eğitim Journal, Primary Education Special Issue, 685-716. 14 April 2021 retrieved from https://dergipark.org.tr/tr/pub/milliegitim/issue/51765/674598 Koç, E. S. (2021). Türkçe dersi öğretim programları ile okuma becerileri öğretim programının uluslararası okuma yeterlikleri bağlamında incelenmesi. Milli Eğitim Journal, 50(230), 169-193. 18 May 2021 retrieved from http://dx.doi.org/10.37669/milliegitim.702775 MEB. (2019). Uluslararası öğrenci değerlendirme programı PISA 2018 Türkiye ön raporu. 28 June 2021 retrieved from https://www.meb.gov.tr/meb_iys_dosyalar/2019_12/03105347_PISA_2018_Turkiye_ On_Raporu.pdf Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd edition). Thousand Oaks, CA: Sage. OECD (2019). PISA 2018 results volume I: What students know and can do. Paris: OECD Publishing. 21 July 2021 retrieved from https://doi.org/10.1787/5f07c754-en. OECD (2021). 21st-century readers: Developing literacy skills in a digital World, PISA. Paris: OECD Publishing. 28 October 2021 retrieved from https://doi.org/10.1787/a83d84cb-en. Savran, N. (2004). PISA-projesi'nin Türk eğitim sistemi açısından değerlendirilmesi. Journal of Turkish Educational Sciences, 2(4), 397-412. Sefer, A., Ören, Z., Sarıtaş, H., & Konuk, S. (2017). Ortaokul Türkçe öğretim programlarındaki metin türleri ile PISA’daki metin türlerinin karşılaştırılması. International EJER 2017 Congress. Ankara: Anı Publishing 28. 10. 2021 retrieved from https://ejercongress.org/pdf/bildiri_kitabi_2017.pdf Temizyürek, İ. & İnce, V. (2019). Okuma becerileri dersi kazanımları ile PISA okuma becerilerine ait düzeylerin içerik açısından incelenmesi. International Journal of Language Academy, 7(3), 157-166. Tuzlukaya, S. (2019). 8. Sınıf Türkçe dersi merkezî sınav sorularının PISA okuma becerileri yeterlilikleri açısından incelenmesi. The Journal of International Lingual, Social and Educational Sciences. 5(1). 9 September 2021 retrieved from http://dx.doi.org/1034137/jilses.505073 https://bera-journals.onlinelibrary.wiley.com/action/doSearch?ContribAuthorRaw=Jerrim%2C+John https://bera-journals.onlinelibrary.wiley.com/action/doSearch?ContribAuthorRaw=Moss%2C+Gemma https://doi.org/10.1787/a83d84cb-en International Online Journal of Education and Teaching (IOJET) 2022, 9(1), 417-431. 431 Yağmur, K. (2009). Türkçe ders kitapları üst düzey bilişsel becerilerin gelişimini ne oranda sağlamaktadır? Cito Education: Theory and Practice, 5, 20-34. Yıldırım, A. ve Şimşek, H. (2013). Sosyal bilimlerde nitel araştırma yöntemleri (9th edition). Ankara: Seçkin Publishing. Yıldız. D. (2021). Turkish and Turkish language and literature teachers’ views’ on the reading skills and Turkey's performance in PISA: A focus group interview. Journal of Qualitative Research in Education, 27, 208-231. 10 October 2021 retrieved from http://dx.doi.org/10.14689/enad.27.10