142 © 2023 by the authors; licensee Asian Online Journal Publishing Group Asian Journal of Education and Training Vol. 9, No. 4, 142-150, 2023 ISSN(E) 2519-5387 DOI: 10.20448/edu.v9i4.5263 © 2023 by the authors; licensee Asian Online Journal Publishing Group An analysis of the learning outcomes of the 2018 geography curriculum for 9th and 10th grades according to Webb’s depth of knowledge Nurcan Demiralp1 Hulya Yigit Ozudogru2 ( Corresponding Author) 1Department of Turkish and Social Sciences Education, Gazi University, Gazi faculty of education, Division of Geography Education, Ankara, Turkey. Email: nurcan@gazi.edu.tr 2Faculty of Letters, Department of Geography, Department of Human and Economic Geography, Ankara Hacı Bayram Veli University, Ankara, Turkey. Email: hulya.yigit@hbv.edu.tr Abstract There are many taxonomies in education related to the cognitive learning domain. This study aimed to analyze the learning outcomes in the 9th and 10th grade 2018 geography curriculum according to Webb's depth of knowledge. Document analysis was used in the study. The geography curriculum was analyzed by considering the four levels of Webb’s depth of knowledge and the scope of the acts and learning outcomes identified for social sciences. In this context, the levels of 56 learning outcomes in the 2018 geography curriculum were analyzed by grade and unit. It was found that the learning outcomes in Webb's depth of knowledge were not distributed evenly across grades, units and levels. Half of the 9th and 10th grade outcomes were at Level 2, while the other half were distributed between Level 1 and Level 3. At Level 4, there was only one outcome in Grade 9. Level 2 was predominant in the learning outcomes of the Natural Systems unit in Grade 9 and of the Human Systems unit in Grade 10. Tasks with a high level of complexity were more common in Grade 10. "The Environment and Society" and "The Global Environment: Regions and Countries" units did not have learning outcomes at all levels. The characteristics of the course content were effective in determining the level of the learning outcomes. For a holistic evaluation of the 2018 geography curriculum in terms of Webb's Depth of Knowledge, it is recommended to examine how the 11th and 12th-grade outcomes were distributed. Keywords: Achievement, Education, Geography curriculum, Learning outcome, Taxonomy, Turkey, Webb depth of knowledge. Citation | Demiralp, N., & Ozudogru, H. Y. (2023). An analy sis of the learning outcomes of the 2018 geography curriculum for 9th and 10th grades according to Webb’s depth of knowledge. Asian Journal of Education and Training, 9(4), 142–153. 10.20448/edu.v9i4.5263 History: Received: 20 October 2023 Revised: 28 November 2023 Accepted: 12 December 2023 Published: 22 December 2023 Licensed: This work is licensed under a Creative Commons Attribution 4.0 License Publisher: Asian Online Journal Publishing Group Funding: This study received no specific financial support. Institutional Review Board Statement: Not applicable. Transparency: The authors confirm that the manuscript is an honest, accurate, and transparent account of the study; that no vital features of the study have been omitted; and that any discrepancies from the study as planned have been explained. This study followed all ethical practices during writing. Competing Interests: The authors declare that they have no competing interests. Authors’ Contributions: Both authors contributed equally to the concept ion and design of the study. Both authors have read and agreed to the p ublished version of the manuscript. Contents 1. Introduction .................................................................................................................................................................................................143 2. Method...........................................................................................................................................................................................................145 3. Findings and Implications .......................................................................................................................................................................146 4. Discussion, Conclusion, and Recommendations...............................................................................................................................148 References .........................................................................................................................................................................................................149 mailto:nurcan@gazi.edu.tr mailto:hulya.yigit@hbv.edu.tr https://creativecommons.org/licenses/by/4.0/ https://creativecommons.org/licenses/by/4.0/ https://www.doi.org/10.20448/edu.v9i4.5263 https://orcid.org/0000-0003-4102-1409 https://orcid.org/0000-0001-5504-1641 Asian Journal of Education and Training, 2023, 9(4): 142-150 143 © 2023 by the authors; licensee Asian Online Journal Publishing Group Contribution of this paper to the literature This study was deemed important to conduct this study since it was thought that analyzing the 9th and 10th-grade learning outcomes in the geography curriculum 2018 according to Webb's depth of knowledge levels would contribute to the literature and curriculum developers. 1. Introduction Education is defined as the process of bringing positive and desired changes in the behaviors of an individual in line with predetermined goals (Ertürk, 2016), as well as a lifelong process in which an individual acquires the behaviors that he/she should have for life and takes an active role in the formation of these behaviors with his/her unique qualities (Taşpınar, 2014). This process can be carried out properly through systematically well-planned education and training programs. An education program includes all educational activities planned to reach determined objectives, and a curriculum includes all teaching activities related to subjects to be addressed in various classes and courses at an educational level (Demirel, 2012). Curricula consist of four basic elements: objectives (learning outcomes), content (subject area), learning-teaching process (educational situations) and assessment and evaluation (Demirel, 2012; Hesapçıoğlu, 2010). Learning outcomes consist of knowledge, skills, values and attitudes that are decided to be gained by students through planned and organized experiences and those learning outcomes constitute the objective component of a curriculum (MEB, 2005). They are also expressed as a systematic arrangement of the subjects that will be a source for students to have the specified behaviors (Taşpınar, 2014). The geography curriculum is one of the curricula, that are created by limiting the objectives planned to be achieved by students. The 2018 geography curriculum is unit-based and consists of four units at the 9-12th grade level. These are the Natural Systems, Human Systems, Global Environment: Regions and Countries, and Environment and Society. In the geography curriculum 2018, there are a total of 125 learning outcomes, 22 of which are taught to students in 9th grade and 34 in 10th grade. Learning outcomes, one of the basic elements of a curriculum, should be arranged in a way that reflect the high-level skills envisaged in the general objectives of the curriculum and enable students to acquire these skills (Bümen, 2006; Demiralp, 2017; Şengül & Isik, 2014). Learning outcomes (objectives) in curricula should be prepared according to the cognitive levels of students to achieve their objectives. The stage of the cognitive level in the learning outcomes should be in a spiral structure so that the upper cognitive level covers the lower cognitive levels. This cognitive inclusiveness will lead students to use metacognitive skills as well as provide meaningful and permanent learning, thus enabling them to transform their knowledge into skills (Demiralp, 2006; Göçer & Kurt, 2016; Gülersoy & İlhan, 2020; Öner, 2022). During the preparation of the curricula in Turkey in 2005, the constructivist learning approach was adopted as the basis of education. Students’ gaining problem-solving ability using their own experiences and producing their subjective knowledge during the process formed the basis of education according to the Ministry of National Education of Turkey (MEB, 2005). In the curricula, students are defined as individuals who produce knowledge, use it functionally in life, solve problems, think critically, are entrepreneurial, determined, have communication skills, empathize, and contribute to society and culture. Geography course teaching aims to raise students as individuals who have geographical awareness, can use the knowledge they produce in life, find solutions to problems, think critically and have the ability to question (Akınoğlu, 2004; Karabağ & Şahin, 2021; Kızılçaoğlu, 2006; Koç & Aksoy, 2010; MEB, 2018; Şahin, 2019). In Turkey, the 2018 Geography Curriculum, which was prepared by the Ministry of National Education (MEB) in 2018 and is currently in use, guides the use of metacognitive skills, provides meaningful and permanent learning, is robust and associated with prior learning, as well as integrated with other disciplines and daily life within the framework of values, skills and competencies (MEB, 2018). Especially since the 1950s until today, various taxonomy studies have been carried out to facilitate the correct determination of educational objectives or to guide the determination of educational objectives. Some of these are Bloom's Taxonomy, MATH Taxonomy and Solo Taxonomy. Bloom's taxonomy includes a six-stage description of the objectives related to the cognitive domain, while the Solo taxonomy has a five-stage and hierarchical structure according to the complexity of students' responses (Özkaya, 2022). In Webb's Depth of Knowledge, an important point is that each level builds on previous one. At Level 1, students should recall basic information and at Level 2 they should demonstrate understanding of concepts and skills. At Level 3, they must apply their knowledge in different contexts before engaging in critical thinking and synthesis. At Level 4, they should succeed in critical thinking and synthesis. For example, at the recall and reproduction level (level 1), students might be asked to define climate change or name greenhouse gases. At the skills and concepts level (level 2), they might be asked to explain the causes of climate change or interpret data. At the strategic thinking level (level 3), students could be asked to apply their knowledge by creating a plan to reduce carbon emissions. At the extended thinking level (level 4), which is the highest level, students can synthesize information from multiple sources to develop a solution to reduce the negative effects of climate change (Webb, 2002). The literature on the taxonomy of learning shows that Bloom, Solo and Webb's depth of knowledge models have been analyzed in terms of curricula and exam questions. One of them is a study conducted to determine the role of students' metacognitive skills in solving mathematics problems according to Webb's depth of knowledge levels (Şengül & Isik, 2014). In the literature, there are also studies in which standardized test questions and curriculum outcomes of different courses are analyzed according to Webb's depth of knowledge levels. There are also studies conducted to analyze the mathematics (Birinci, 2014) and science questions (Özden et al., 2014) in standardized tests, as well as studies analyzing the outcomes of science (Eke, 2018) and logic curricula (Arslan, 2019). In addition, there are also studies in the literature that evaluate the 10th, İlhan and Gülersoy (2019) 11th and 12th-grade outcomes of the geography curriculum according to the revised Bloom's Taxonomy (Gülersoy & İlhan, 2020); 9th and 10th grade (Öner, 2021) and 11th and 12th grade (Öner, 2022) outcomes according to the Solo Taxonomy. Provided that it is limited to the resources accessed, no study that analyzes the learning outcomes in Asian Journal of Education and Training, 2023, 9(4): 142-150 144 © 2023 by the authors; licensee Asian Online Journal Publishing Group the 2018 geography curriculum according to Webb's Depth of Knowledge Levels was found in the literature. For this reason, it was deemed important to conduct this study since analyzing the 9th and 10th-grade learning outcomes in the 2018 geography curriculum according to Webb's depth of knowledge levels would contribute to the literature and curriculum developers. 1.1. Depth of Knowledge (DOK) Levels Webb (2002) and Webb (2009) developed the depth of knowledge model to classify educational activities according to the level of complexity in thinking. Depth of Knowledge (DOK), developed by Webb (2002) and Webb (2009), is more than a categorization, it is a model that focuses on the depth of knowledge, revealing how deeply students need to know the content to complete a task. Webb's Depth of Knowledge model is a framework for analyzing learners' cognitive requirements and examining the consistency between outcomes and assessment. This model, which is used in four important content areas: mathematics, science, language arts and social studies, aims to reach the highest level that cognitive development can reach. The model is based on analyzing cognitive complexities at various levels in academic standards and curricula (Birinci, 2014; Webb, 2009) . Webb (2009) states that the depth of knowledge model aims to determine the level of depth that a piece of information a student has learned gives him/her at the cognitive level. The model is based on the assumption that it can be categorized based on cognitive demands in a way that covers all the elements that make up the program. For this reason, in educational environments, Webb divided the problems into levels covering all objectives in terms of level and content area (Şengül & Isik, 2014; Webb, 2009). Task groups at each level reflect a different level of cognitive expectation required to complete that task. Each depth of knowledge level defines the type of thinking required by the task, not whether the task is "difficult" or not (Tezci, 2022). These levels, which Webb divides into four, are recall and reproduction, skills/concepts, strategic thinking and extended thinking. The characteristics of Webb's Depth of Knowledge levels (Webb, 2002) in social studies are as follows; 1.1.1. Recall and Reproduction (Level 1) Level 1 involves students recalling facts, terms, concepts, trends, generalizations and theories, recognizing or identifying specific information contained in graphs. This level usually requires students to identify, list or define. Tasks ask students to recall who, what, when and where. "Identify or explain" means to recall (remind), recite or reproduce the task. Tasks that require students to "define" and "explain" can be categorized at level 1 or level 2, depending on what is being explained and what is being defined. Tasks that require students to recognize or identify specific information contained in maps, charts, tables, graphs, graphs, and drawings are mostly at this level (Webb, 2002). Students are expected to perform simple types of thinking such as "describe, explain, identify" and routine processes required by the level. For example, students are expected to determine in which month the most precipitation falls by looking at the precipitation graph of a place. Students may be given tasks such as recognizing, defining and explaining the scale of a map, measuring the length of a line scale, reading information about a temperature graph, recognizing and defining the information in a diagram or drawing showing the stages of valley formation, and identifying the phases of valley formation. 1.1.2. Skills / Concepts (Level 2) Level 2 involves engaging in some mental processing beyond recall. This level usually requires students to compare or contrast people, places, events and concepts. Converting information from one form to another, giving examples, classifying/sorting patterns into meaningful categories identify, interpret or explain problems (questions) and cases, patterns, reasons, cause and effect, significance or impact, relationships, perspectives or processes. This level requires students to interpret a result by going beyond the explanation or description of "how" or "why". Tasks that require students to use maps, models, or puzzles related to a topic, to make definitions, interpretations and explanations about them, and to make inferences are mostly at level 2. At this level, students are expected to understand the pattern on a map related to a topic and put the pieces together (Webb, 2002). Students are asked to predict how and why migration occurs and what its consequences or possible consequences might be. Students are expected to classify the climate and plant communities (plant formation) on Earth, use topography maps to learn the properties of contour curves (to show them on the map), use various models to understand the shape and movements of the Earth, predict possible situations and changes in the climate in the world and in the place where they live in the next fifty or one hundred years based on scientific data, distinguish the causes of natural and unnatural environmental problems (drought, desertification, floods and inundations, etc.) occurring on earth, explain the relationships between the causes and consequences of migration, explain the cases that are and are not examples of internal migration and external migration. With the tasks given at this level, students are expected to use information in a different context than they have learned, apply skills or concepts, take photos or videos, create mind maps, interview or make presentations. 1.1.3. Strategic Thinking (Level 3) The tasks at Level 3 involve drawing conclusions using evidence, applying concepts to new situations to solve problems, proposing solutions by analyzing similarities and differences in events and phenomena, and evaluating the results. It also requires recognizing misconceptions and understanding the relationship between time and space. Level 3 requires complex reasoning, evidence use and thinking skills, encompassing the previous two levels. Cognitive processes at level 3 are more complex and more abstract than those at levels 1 and 2. Students are expected to justify answers to questions of how and why based on evidence and make concrete applications. Tasks at Level 3 require concluding; using evidence, citing evidence, applying concepts to new cases, using concepts to solve problems; analyzing similarities and differences in issues and problems; proposing solutions to problems and Asian Journal of Education and Training, 2023, 9(4): 142-150 145 © 2023 by the authors; licensee Asian Online Journal Publishing Group evaluating; recognizing and explaining misconceptions or establishing connections across time and space to explain a concept or an idea (Webb, 2002). Tasks at this level involve short-term use of higher-order thinking processes such as analysis and evaluation to solve real-world problems with predictable outcomes. Reasoning is an important indicator of tasks at this level. Tasks require the coordination of knowledge and skills across multiple subject areas to arrive at a solution. Mental processes at this level consist of "analyzing, explaining by supporting with evidence, generalizing and creating" (Tezci, 2022). Students can be expected to examine the change in land use of a place in certain periods, predict the possible change in the future in various aspects, and prove this prediction. They may be asked to show the similarities and differences between two situations in this place by using graphic designs such as Venn diagrams. Students are given data about the soil, climate, hydrographic and topographical characteristics of the place where they live and the conditions suitable for the growth of various agricultural products and are expected to explain which agricultural products can be produced where with their justifications. Studies that are conducted to answer questions such as "What are village and sub-village settlements? What are the similarities and differences between the general characteristics of these settlements and their agricultural and animal production?" can be given as examples. Students are expected to create tables and graphs, produce various maps using geographical information systems, make short movies, create checklists, and write summaries or reports as the evidence and justification for these tasks. One of the important expectations of this level may be to organize a panel discussion as an indispensable part of discussion and conclusion skills. 1.1.4. Extended Thinking (Level 4) Level 4 is achieved by adding planning, research and development tasks on top of the previous level. It is mostly time-consuming, but often time and duration spent on the tasks do not have a distinguishing characteristic. Students have to undertake more complex tasks. Students are expected to make associations between knowledge, skills, ideas and concepts related to the subject matter. The most important feature of this level is that students are required to develop a product and prove it with evidence. Students analyze and synthesize information obtained from various sources and examine it critically. In addition, students are expected to make predictions, plan by putting forward an opinion and propose a solution. Level 4 requires the addition of planning, research, development, and reflection on top of the abstract and complex reasoning of level 3, often over an extended period. The length of time taken to carry out the study is not a distinguishing factor unless the required study is merely repetitive and requires significant conceptual understanding and the application of higher-level thinking. At this level, cognitive demands should be high and mental operations much more complex. To be at this highest level, students should be asked to relate knowledge, skills, ideas, and concepts within or across content areas, and the contexts of events, facts, and cases. The distinguishing factor for this level will be proving that the cognitive demands have been met through a task or product. Level 4 requires students to analyze the information from multiple sources, synthesize it, examine alternative perspectives from a variety of sources, and explain and demonstrate how common themes and concepts are found across time and space. In some tasks, students are expected to make predictions supported by evidence and plan and develop solutions to problems by developing a logical argument. This level involves active and authentic learning experiences, such as project-based or problem-based learning, as students conduct investigations to solve real-world problems (Webb, 2002). For example, students are expected to analyze how the factors that are effective in the drying of Lake Amik have changed over time depending on the characteristics specific to that place, what needs this was caused by, and what new problems it caused, and to bring solutions. In addition, students are expected to transform their studies into a product and prove it through the product. At this level, students are asked to conduct problem-based or project-based spatial analysis studies based on accurate and reliable data. Students make spatial plans and develop suggestions on how to prevent frequent floods and inundations. Students may be given tasks such as revealing the earthquake risk in all its aspects through risk analysis and determining the measures that can be taken for the earthquake risk in that area and deciding where to install a water treatment plant in a city. At Level 4, students begin to use their knowledge and skills that are related to the content in new environments. In this way, a learning environment is created in which their thinking processes are fully engaged , and the process is entirely shaped by the students themselves. As a result, especially in this process, where new products are designed and developed, there are outputs in which students take on tasks that develop their cognitive, affective and psychomotor skills. Students can design a new story, a new plan or project, or a new digital content. Whatever product students develop at the end of this level, it must be new, created by the student and the student must manage the process as the sole leader. 1.2. Purpose of the Study and Problem Statement Since Webb's Depth of Knowledge is a model in which students need to know the content in depth to complete a task, this study aims to analyze the learning outcomes for the 9th and 10th grades in the 2018 geography curriculum according to this model. In this respect, the study sought to answer the question: How are the learning outcomes in the 2018 geography curriculum distributed according to Webb's Depth of Knowledge levels in the "natural systems, human systems, global environment: regions and countries, and environment society" units in the 9th and 10th grades? 2. Method A basic qualitative research design, which is widely used in educational studies, was used as the methodology of this study. In basic qualitative research, data are obtained from documents, interviews or observations. The data used in this study were obtained from documents. Document analysis can be used as a complementary analysis or as an independent method and takes less time (Kiral, 2020). It requires data selection rather than data collection (Bowen, 2009). In this method, which is used to identify the recurrent pattern in the data, the findings are presented as themes. Interpretations are the researcher's understanding of the phenomenon of interest (Merriam, Asian Journal of Education and Training, 2023, 9(4): 142-150 146 © 2023 by the authors; licensee Asian Online Journal Publishing Group 2018). In this study, the learning outcomes in the 2018 geography curriculum were analyzed by using the document analysis method. The learning outcomes were analyzed according to Webb's depth of knowledge levels according to units and grades and the findings were compared. A basic qualitative research method was used in this study. Basic qualitative research is the most widely used research in education. Data are collected through interviews, observation or document analysis (Merriam, 2018). Although document analysis is mostly complementary to other research methods, it can also be used as an independent method (Kiral, 2020). Document analysis, which means examining the documents related to the researched subject in accordance with scientific principles, takes less time than other research methods. It requires data selection instead of data collection (Bowen, 2009). Data analysis involves identifying recurring patterns by characterizing the data. Findings are themes. Interpretations are the researcher's understanding of the phenomenon of interest (Merriam, 2018). In this study, the learning outcomes in the 2018 geography curriculum were analyzed by document analysis method. The learning outcomes were analyzed according to Webb's depth of knowledge levels by unit and grade and the findings were compared. 2.1. Data Collection Tool, Data Collection and Analysis In the study, the 2018 geography curriculum was used as the data collection tool. The 2018 geography curriculum was accessed at the website of the Presidency of the Board of Education (MEB, 2018). The curriculum is open to public use and review. This curriculum was selected as it is the current curriculum that is still in use in schools. The learning outcomes for the 9th and 10th grades and the distribution of these learning outcomes to the units were obtained from this document. After that, contents of the geography curriculum were analyzed. In the analysis process, verbs that are suitable for Webb's Depth of Knowledge levels were first determined (MEB, 2018). While determining these verbs, studies in the literature were reviewed. Among these studies, Social Studies Levels of Depth of Knowledge for Social Studies (Webb, 2002) was utilized. At this stage, the characteristics of the levels were tabulated. In addition, the learning outcomes for 9th and 10th grades and the list of verbs or verb phrases at the end of these outcomes were determined separately for each grade. These verbs were compared with the verbs identified in Webb's Depth of Knowledge circle, and it was decided at which level of Webb's Depth of Knowledge the verbs and verb phrases in the outcomes should be. Then, the content of the outcomes was focused on. With these contents, the outcomes were re-examined, and for each outcome, it was re-decided at which level the outcomes should be located by taking into account the task and thought indicators expected from students according to Webb's Depth of Knowledge levels. To determine which level the verbs of the outcomes belonged to, the researchers first coded them separately, and then the two separate lists were reduced to a single list based on a joint decision. This was done to avoid any deviation in Webb's Depth of Knowledge levels. Then, the opinions of two field experts were consulted to make a preliminary evaluation of this list in terms of suitability. To obtain the opinions of the field experts, explanations of Webb's Depth of Knowledge levels, verbs determined in accordance with these explanations, and examples prepared for some of the topics within the scope of the geography curriculum were placed in this list. A table was created in which the relevant outcomes were written next to each Webb's Depth of Knowledge level. This table was presented to two geography educator field experts as a list with the headings "appropriate, not appropriate, should be changed". The outcomes that both of the experts were in agreement with each other were determined, and the outcomes that were evaluated as different from each other were re-examined and analyzed by the researchers and the final table was created. In this way, it was attempted to reduce errors in the analysis process. After the final assessment tool was created, the data were analyzed descriptively. The frequencies and percentages of each of the 9th and 10th-grade learning outcomes of natural systems, human systems, global environment: regions and countries, and environment society units in the geography curriculum were calculated separately according to Webb's Depth of Knowledge levels and grades. The models and patterns between the grades were tried to be revealed. In this study, Webb's Depth of Knowledge levels were named as Level 1, Level 2 , Level 3 and Level 4 in order from the simplest to the most complex task. The most discussed verb by the researchers was "explain". However, since the complexity of tasks rather than verbs is an important classification tool in Webb's depth of knowledge levels, the verb "explain" was re-evaluated together with the outcome. The outcomes defined as "Explain" in the geography curriculum are used in the sense of "tell" and are accepted as level 1 because they involve defining what students have learned or expressing basic knowledge, that is, recalling. The verb "Explain with examples", on the other hand, has more than one level of complexity, so the difficulty level of the task has increased, and for this reason, it has been determined as level 2. 3. Findings and Implications In this section, the findings on the learning outcomes of 9th and 10th grades were analyzed respectively according to the units and Webb's Depth of Knowledge levels. a) The results of the analysis conducted to answer the question "How are the 9th-grade learning outcomes of the Natural Systems, Human Systems, Global Environment: Regions and Countries, and Environment and Society units in the 2018 geography curriculum distributed according to Webb's Depth of Knowledge levels?" are given below. First, the numbers, durations and percentages of the 9th-grade learning outcomes and the verbs at the end of these outcomes are given in Table 1. According to the table, there are a total of 22 learning outcomes in the 9th grade. There are 13 outcomes in natural systems, 4 in human systems, 3 in global environment: regions and countries, and 2 in environment and society unit. The time allocated to the learning outcomes from maximum to minimum is as follows: Natural Systems, Human Systems, Global Environment: Regions and Countries, and Environment and Society units. 11 verbs were identified for a total of 22 outcomes. Among these, the most repeated verbs are "explain, explain with examples, make inferences and evaluate" (Table 1). Asian Journal of Education and Training, 2023, 9(4): 142-150 147 © 2023 by the authors; licensee Asian Online Journal Publishing Group Table 1. The numbers, durations and percentages of the 9th-grade learning outcomes and the verbs included in the learning outcomes. Unit N u m b er o f o u tc o m es D u ra ti o n o f o u tc o m es P er ce n ta g e o f o u tc o m es E x p la in G iv e ex am p le s M ak e in fe re n ce s R el at e C o m p ar e D is ti n g u is h E x p la in w it h e x am p le s C la ss if y E v al u at e A n al y ze s U se T o ta l Natural systems 13 47 65 5 3 1 1 1 1 1 13 Human systems 4 15 21 1 2 1 4 Global environment: Regions and countries 3 5 7 1 1 1 3 Environment and society 2 5 7 1 1 2 Total 22 72 100 5 1 3 1 1 1 4 1 3 1 1 22 These verbs were analyzed according to Webb's Depth of Knowledge levels considering the content of the learning outcomes and the descriptive statistics results are given in Table 2. Table 2. Frequency and percentage distributions of 9th-grade learning outcomes according to Webb's Depth of Knowledge levels. Units/ Webb’s depth of knowledge levels Recall/Reproduction Skills/Concepts Strategic thinking Extended thinking Total f % f % f % f % Natural systems 5 38.5 6 46 1 7.7 1 7.7 13 Human systems - - 3 75 1 25 - 4 Global environment: Regions and countries - - 2 66 1 33 - 3 Environment and society 1 50 1 50 2 Total 6 27 11 50 4 18 1 4.5 22 9th-grade learning outcomes were analyzed according to the units. It was found that 5 of the 13 outcomes in the Natural Systems unit are at Level 1, six at Level 2, 1 at Level 3 and 1 at Level 4. Three of the 4 objectives in the Human Systems unit are at Level 2 and 1 is at Level 3. Two of the three outcomes in the Global Environment: Regions and Countries unit are at Level 2 and 1 is at Level 3. One of the two outcomes in the Environment and Society unit is at Level 1 and 1 is at Level 3. In the Natural Systems unit, which has the highest number of outcomes, the numbers of outcomes allocated to Level 1 and Level 2 are quite close to each other (Table 2). Level 4 outcome is only found in the Natural Systems unit. Based on these results, it can be said that the only unit covering all levels is the Natural Systems unit. Therefore, the Natural Systems unit is the only unit where students have the opportunity to study at all levels. The 9th grade outcomes were analyzed according to their levels, and it was found that six of the 22 outcomes are at Level 1, eleven are at Level 2, four are at Level 3 and one is at Level 4. Accordingly, the learning outcomes are distributed as Level 2, Level 1, Level 3 and Level 4 respectively. b) The results of the analysis conducted to answer the question of "How are the learning outcomes of the 10th- grade Natural Systems, Human Systems, Global Environment: Regions and Countries, and Environment and Society units in the 2018 geography curriculum distributed according to Webb's Depth of Knowledge levels?" are given below. First, the number, duration and percentages of the 10th-grade outcomes and the verbs at the end of these outcomes are given in Table 3. Table 3. The numbers, durations and percentages of the 10th-grade learning outcomes and the verbs in these learning outcomes. Unit N u m b er o f o u tc o m es D u ra ti o n o f o u tc o m es P er ce n ta g e o f o u tc o m es E x p la in E x p la in b y a ss o ci at in g E v al u at e M ak e in fe re n ce s U se R el at e C o m p ar e A n al y ze D is ti n g u is h C la ss if y G iv e ex am p le s E x p la in b y e x am p le s Natural systems 17 36 50 6 1 3 - 4 - 1 - 2 - Human systems 12 24 33 3 5 - 1 - 1 1 - 1 Global environment: Regions and countries 1 4 6 - - 1 - - Environment and society 4 8 11 2 - 2 - - - Total 34 72 100 8 1 6 5 - 7 - 3 1 2 - 1 There are a total of 34 learning outcomes in the 10th grade. There are 17 outcomes in the Natural Systems, 12 outcomes in the Human Systems, 1 outcome in the Global Environment: Regions and Countries, and 4 outcomes in the Environment and Society unit. For a total of 34 outcomes, 10 verbs were identified. Among these, the most recurrent verbs are "explain, relate, evaluate and make inferences". These verbs were analyzed according to their levels considering the content of the learning outcomes and the descriptive statistics results are given in Table 4. Asian Journal of Education and Training, 2023, 9(4): 142-150 148 © 2023 by the authors; licensee Asian Online Journal Publishing Group Table 4. Frequency and percentage distributions of the 10th-grade learning outcomes according to Webb's depth of knowledge levels. Units Webb’s Depth of knowledge levels Recall/Reproduction Skills/Concepts Strategic thinking Extended thinking Total f % f % f % f % Natural systems 6 35 7 41 4 23.5 - - 17 Human systems - 8 67 4 33 - - 12 Global environment: Regions and countries - - - - 1 100 - - 1 Environment and society 2 50 2 50 - - - 4 Total 8 23.5 17 50 9 26.5 - - 34 The learning outcomes of the 10th grade were analyzed according to the units. Six of the 17 outcomes in the Natural Systems unit are at Level 1, seven are at Level 2 and four are at Level 3. While eight of the 12 outcomes in the Human Systems unit are at Level 2, four of them are at Level 3. Only one outcome in the Global Environment: Regions and Countries unit is at Level 3. Two of the four outcomes in the Environment and Society unit are at Level 1 and the other two are at Level 2. The numbers of outcomes allocated to Level 1 and Level 2 in the Natural Systems unit are quite close to each other. In the Human Systems unit, there are more outcomes at Level 2 and Level 3. In the Environment and Society unit, Level 1 and Level 2 outcomes are equal to each other (Table 4). When analyzed according to their levels, eight of the 34 outcomes are at Level 1, seventeen are at Level 2 , and nine are at Level 3. According to this information, it is possible to say that the learning outcomes are respectively at Level 2, Level 1 and Level 3. In the 10th grade, there are no outcomes at Level 4. Therefore, there are no tasks that students can be assigned. 4. Discussion, Conclusion, and Recommendations This study aims to analyze the learning outcomes for the 9th and 10th grades in the 2018 geography curriculum according to Webb's Depth of Knowledge levels by unit and grade. In this context, a total of 56 outcomes were evaluated. Half of all outcomes (50%) are at Level 2, while the other half is shared between Level 1 and Level 3. There is only one learning outcome at Level 4 in the 9th grade, and there is no learning outcome at Level 4 in the 10th grade. In the 9th and 10th grades, the outcomes in the Natural Systems unit are evenly distributed between Level 1 and Level 2. In both 9th and 10th grades, there are more outcomes in the Human Systems unit at Level 2. The Environment and Society unit and the Global Environment: Regions and Countries unit do not have a learning outcome representing each level due to the low number of learning outcomes. Considering the content of these units, the lack of outcomes at Level 4 and the absence of more in-depth, long-term and problem-based studies can be considered as a deficiency. ‘The Natural Systems’ unit at 9th grade include all levels. The same unit at 10th grade includes Level 1, Level 2 , and Level 3. Level 4 is only included in one learning outcome in the Natural Systems unit in Grade 9. According to Webb's Depth of Knowledge levels, a good balance of cognitive demand was not achieved at both grade levels in the 9th and 10th grade outcomes. It was concluded that the outcomes of strategic thinking and extended thinking levels should be included in all grade levels. Based on these results, it is seen that there is no systematic and balanced distribution of the 9th and 10th-grade outcomes according to Webb's depth of knowledge levels and that the outcomes are concentrated at the level of recall and reproduction and skills/concepts. It is understood that strategic thinking and extended thinking outcomes are given very little space. The reason for this may be that the outcomes were created by prioritizing the characteristics and qualities of the content within the scope of the outcomes. In addition to this, it can be said that most of the outcomes are concentrated at level 2 because the geographical knowledge that will form the basis for the students in the 9th and 10th grades has intensive content, also consists of basic concepts and skills and their teaching requires a long time. According to Webb's Depth of Knowledge level, considering the age and prior learning of the students, it would have been expected that 10th grade would have learning outcomes belonging to Level 4, even more than the others. However, when the learning outcomes of both 9th and 10th grades are taken into consideration, it is seen that this expectation is not realized. While a 10th-grade student can take on more abstract and complex tasks than a 9th-grade student, the outcomes do not serve the development of students' thinking skills and their ability to create products. A similar finding is found in Öner (2021) and Öner (2022) study. Öner, who analyzed the 2018 geography curriculum according to the SOLO taxonomy, stated that while there should be an increase in the number of outcomes belonging to the extended abstract stage as the grade level increases, this did not happen. İlhan and Gülersoy (2019) and Gülersoy and İlhan (2020) reached a similar conclusion in their studies in which they analyzed the 10th, 11th and 12th-grade geography course outcomes according to Bloom's taxonomy and revealed that the cognitive demands of the outcomes were insufficient and some cognitive stages were not included at all. In the same study, they determined that the learning outcomes in the 2018 geography curriculum do not have a homogeneous distribution and that the learning outcomes related to high-level knowledge and cognitive process dimensions that ensure effective learning are insufficient. It is seen that this situation is not specific to geography course only. Similar results were found in the studies conducted in the fields of Turkish, logic, mathematics, and science in analyzing the outcomes and the questions asked to students in exams conducted by different institutions (Eke, 2016, 2018; Karabulut & Tunagür, 2021; Yegen, 2022). According to Eke (2016), the outcomes in the secondary school physics curriculum (2013) are distributed at the first three levels according to Webb's Depth of Knowledge level, and there are no outcomes at the 4th level. Eke (2018) reached the same conclusion in his similar study for science course outcomes. He found that most of the outcomes were at the 2nd level of Webb's Depth of Knowledge. He also found that the 4th-level outcomes were not included in the 7th and 8th grades. Karabulut and Tunagür (2021) concluded that Webb's Depth of Knowledge level in the 2019 Turkish Course Curriculum is not evenly distributed to learning outcomes according to grades and levels . It was determined that almost half of the outcomes in the Turkish curriculum were allocated to Level 2, followed by Asian Journal of Education and Training, 2023, 9(4): 142-150 149 © 2023 by the authors; licensee Asian Online Journal Publishing Group Level 3, and finally Level 4. Özden et al. (2014), in their study analyzing the TEOG (The standardized test for transitioning from primary to secondary education in Turkey) questions of the 2013-2014 academic year, found that the questions were concentrated at the first two levels, Level 3 questions, which measure higher cognitive skills, were included less and there were no questions at Level 4. When the results of the study are evaluated together with the results in the literature, it is understood that this situation is not specific to the geography curriculum, but similar results are also found in the curricula that were prepared in previous periods and are still current. Considering all the studies together, it is seen that similar results are not limited to the curricula and that the standardized test questions are also compatible with the results that were obtained about the learning outcomes. For this reason, it has been revealed that the issue should be handled with a supracurricular approach. The following recommendations are presented based on these results: Training should be provided on taxonomy, classification and cognitive processes to those involved in curriculum development, Learning outcomes should be organized in a way that cover all four levels of Webb's depth of knowledge, Considering the developmental characteristics of students, more emphasis should be placed on level 3 outcomes to develop skills such as critical thinking, analyzing, and interpreting, Level 4 tasks should be given more place to raise geography-literate individuals who can produce solutions to today's problems with an inquiring mindset and who can produce products by using their creativity, To make a holistic evaluation of the 2018 geography curriculum in terms of Webb's Depth of Knowledge levels, how the 11th and 12th-grade learning outcomes are distributed should be examined, Preparation and assessment questions and activities of geography textbooks should be analyzed according to Webb's Depth of Knowledge levels, Geography questions prepared by MEB and ÖSYM (Student Selection and Placement Center in Turkey) should be analyzed according to Webb's Depth of Knowledge levels, Webb's Depth of Knowledge levels should be taught to prospective geography teachers, and they should be encouraged to make sample applications using these levels, Curriculum developers should take Webb's depth of knowledge levels into account when setting learning outcomes, How the outcomes are realized in learning environments should be investigated to test the results of this study. References Akınoğlu, O. (2004). Constructivist learning and geography teaching. Marmara Geography Journal, 10, 73-94. Arslan, A. (2019). Evaluation of 2009 logic course curriculum achievements in terms of Webb's depth of knowledge levels . Ankara: Pegem Academy Publications. Birinci, D. K. (2014). First experience in central system common exams: Mathematics lesson. Journal of Research in Education and Teaching, 3(2), 8-16. Bowen, G. A. (2009). Document analysis as a qualitative research method. Qualitative Research Journal, 9(2), 27-40. Bümen, N. T. (2006). A milestone in program development: Revised bloom's taxonomy. Education and Science, 31(42), 3-14. Demiralp, N. (2006). Map and globe usage skills in geography education. Turkish Journal of Educational Sciences, 4(3), 323-343. Demiralp, N. (2017). Examination of programs in geography teaching in terms of design and program elements and the 2017 curriculum. Education and Society in the 21st Century, 6(17), 521-545. Demirel, Ö. (2012). Curriculum development in education from theory to practice. Ankara: Pegem Academy. Eke, C. (2016). Analysis of secondary school physics course curriculum outcomes according to Webb's depth of knowledge levels . Journal of Education and Training Research, 5(3), 35-40. Eke, C. (2018). Analysis of the achievements in the science course curriculum according to Webb's depth of knowledge levels. Journal of Social Research and Behavioral Sciences, 4(6), 174-190. Ertürk, S. (2016). Program development in education. Ankara: Edge Academy Publishing. Göçer, A., & Kurt, A. (2016). Examination of 6th, 7th and 8th grade oral communication achievements of the Turkish language curriculum according to the SOLO taxonomy. Bitlis Eren University Social Sciences Institute Journal, 5(3), 215-228. Gülersoy, A. E., & İlhan, A. (2020). Evaluation of 11th grade geography course curriculum achievements according to the renewed Bloom taxonomy. Paper presented at the Conference II. International Congress on Geographical Education. Full-Text Proceedings Book. Hesapçıoğlu, M. (2010). Teaching principles and methods, educational programs and teaching, Nobel Academic Publishing . Retrieved from https://www.structural-learning.com/post/webbs-depth-of-knowledge İlhan, A., & Gülersoy, A. E. (2019). Evaluation of 10th grade geography course curriculum achievements according to the renewed Bloom taxonomy. International Journal of Geography and Geography Education, 39 , 10-28. Karabağ, S., & Şahin, S. (2021). 2005 Geography course curriculum. Karabağ, S. & Şahin, S. & Şahin, B. & (Ed.). Geography education geography course curriculum. In (pp. 95-110). Ankara: Pegem Academy. Karabulut, A., & Tunagür, M. (2021). Analysis of the achievements in the Turkish course curriculum according to webb's knowledge depth levels. Rumelia Journal of Language and Literature Research , 23, 15-29. Kiral, B. (2020). Document analysis as a qualitative data analysis method. Siirt University Social Sciences Institute Journal, 8(15), 170-189. Kızılçaoğlu, A. (2006). Thoughts about the geography course curriculum. Balıkesir University Social Sciences Institute Journal, 9(16), 1-19. Koç, H., & Aksoy, B. (2010). Evaluation of teachers' opinions on the 2005 geography course curriculum. Black Sea Journal of Social Sciences, 2(2), 17-52. MEB. (2005). Geography course curriculum. Ankara: Board of Education and Discipline. MEB. (2018). geography course curriculum. Retrieved from http://mufredat.meb.gov.tr/Dosyalar/2018120203724482- Cografya%20dop%20pdf.pdf Merriam, S. B. (2018). A guide to qualitative research, design, and practice: (Tr. ed. S. Turan) . Ankara: Nobel. Öner, S. (2021). Examination of 9th and 10th grade achievements of the geography course curriculum according to SOLO taxonomy . Paper presented at the International Geography Education Congress, October, Sivas, Abstracts. Öner, S. (2022). Analysis of 11th and 12th grade acquisitions in the geography course curriculum according to Solo Taxonomy. International Social Sciences Studies Journal, 93(8), 228-235. Özden, M., Akgün, A., Çinici, A., Sezer, B., YILDIZ, S., & Taş, M. M. (2014). Analysis of central system common exam science questions according to webb's knowledge depth levels. Adıyaman University Journal of Science, 4(2), 91-108. Özkaya, M. (2022). Solo taxonomy E. In (pp. 73-90). Yeşilyurt: Educational Taxonomies Vizetek. Şahin, B. (2019). A comparative evaluation of the 2005 and 2018 geography course curriculums. Turkish Journal of Educational Sciences, 17(1), 81-102. Şengül, S., & Isik, S. C. (2014). The role of 1st grade students' metacognitive skills in solving problems related to "Webb's depth of knowledge levels". The Journal of Academic Social Science Studies, 24(1), 93-127. Taşpınar, M. (2014). Teaching principles and methods from theory to practice. Ankara: Edge Academy Publishing. https://www.structural-learning.com/post/webbs-depth-of-knowledge http://mufredat.meb.gov.tr/Dosyalar/2018120203724482-Cografya%20dop%20pdf.pdf http://mufredat.meb.gov.tr/Dosyalar/2018120203724482-Cografya%20dop%20pdf.pdf Asian Journal of Education and Training, 2023, 9(4): 142-150 150 © 2023 by the authors; licensee Asian Online Journal Publishing Group Tezci, İ. H. (2022). Webb's depth of knowledge taxonomy. E. Yesilyurt (Ed.). In Educational Taxonomies. In (pp. 183-202). Ankara: Vizetek Publishing. Webb, N. L. (2002). Depth of knowledge levels for four content areas unpublished manuscript wisconsin center for education research . Madison, WI: University of Wisconsin-Madison. Webb, N. L. (2009). Webb's depth of knowledge guide-career and technical education definitions. Retrieved from http://www.aps.edu/sapr/documents/resources/Webbs_DOK_Guide.pdf Yegen, Ü. (2022). Comparative analysis of the questions in the Turkish subtest of the central exam held in 2019 and 2021 according to Webb's knowledge levels. Journal of Education and Humanities: Theory and Practice, 13(25), 123-142. Asian Online Journal Publishing Group is not responsible or answerable for any loss, damage or liability, etc. caused in relation to/arising out of the use of the content. Any queries should be directed to the corresponding author of the article. http://www.aps.edu/sapr/documents/resources/Webbs_DOK_Guide.pdf