131 | IJET| Volume. 7, Issue 2. December 2018 LEARNERS’ COGNITIVE LEVEL: LEARNING OUTCOME-BASED ANALYSIS Muhammad Saifuddin muhammadsaifuddin@fbs.unipdu.ac.id Pesantren Tinggi Darul Ulum University, Indonesia Dwi Nurcahyani dwinurcahyani@gmail.com Pesantren Tinggi Darul Ulum University, Indonesisa Abstract. This study explores students’ level of cognitive domain depicted in learning objectives of the TOEFL subject matter in an Indonesian higher education institution. Using learning outcomes based analysis, this study further investigates the attainable learning outcomes. The investigation covered an identification of learners’ cognitive skill level based on learning taxonomy and evaluation of the attainment of learning outcomes. Data were collected through textual analysis of the current syllabus used in TOEFL teaching and learning process and through observations of the learning process to find out the suitability between stated learning objectives and the learning activities carried out in the class as well as their relationship to the application of learning taxonomy. The results show that the designed syllabus was not well formulated regarding performance objectives. About cognitive domain, the objectives are not formulated using proper action verbs and result in unreachable learning outcome. The students’ cognitive level was at ‘understanding’ level that it could not achieve the outcome set in ‘applying’ level. Key Words: Learning Outcome, Learning Taxonomy, Cognitive Domain INTRODUCTION An educational purpose emphasizes how to create an effective and purposeful learning process. There must be good communication among education aspects, for instance, academic aspect. This is not about policy driven curriculum, but rather on sequencing the curriculum design. Factors considering the design are derived from learners’ previous knowledge and weaknesses, teachers’ teaching skill and strategies, and available resources (Nation & Macalister, 2010). Regarding the implementation of teaching activities, teachers take major responsibility. The responsibility invites teachers to transform the written document into the learning process. When it comes to the teaching-learning process, there are considerations to be assessed in which they determine the development of learning result (Biggs, 2014). He further classified the responsibilities as skills, knowledge, and students’ attitude. Learners’ Cognitive Level 132 | IJET| Volume. 7, Issue 2. December 2018 This study aimed at revealing students’ level of cognitive domain depicted in learning objectives of the TOEFL subject matter. Using learning outcomes based analysis, this study further sought to investigate the attainable learning outcomes. The investigation covered an identification of learners’ cognitive level, in what level the learners’ cognitive skill is about learning taxonomy and whether the learning outcomes was attainable or not. REVIEW OF LITERATURE Learning Outcome and the success of Education Process Shishkovskaya et al. (2015) argue that the measurement of the success of the educational process is based on competent and a combination of adequate ways of teaching. In that case, it implies on systematic values of the implementation of the learning process. An instructional design gives a comprehensive procedure of how the learning process takes place. Based on the theory of systematic design proposed by Dick and Carey (1996), one of the characteristics of instructional design is goal-directed. The goal as the learning outcomes becomes the most defining aspect of developing the teaching and learning process. Thus, it is very crucial to determine the learning outcomes. The learning outcomes should be specific to explain learners’ competences. The notion of specifying the learning outcomes is expected to be able to determine how the content of the materials will be given and how to evaluate or assess the instructional goals (Khalil & Elkhider, 2016). On the other hand, when formulating the objectives, there are things need to have attention, in what criteria the goals will be. Referring to learning competence, it should cover some abilities, like knowledge, skills, and attitude. Biggs and Tang (2011) who define that to gain learning development as a result of learners’ learning, the skills, knowledge, and students’ attitudes should come together into a learning outcome. By this learning outcomes, the learning activities at least challenge the students to liven their competences, to make their competences measurable. It means that the expectation of learning result should focus not only on knowledge but also skills which are more than their level of mastery to keep motivating students to be engaged and excited to learn (Brophy, 2013). Thus, the implementation of the learning objectives implants learners with enriched skills and knowledge become successful learners with a good mental state (Kleebbua & Siriparp, 2016). Cognitive Domain of Learning Taxonomy In developing learning objectives, cognitive domain of learning taxonomy must put much attention on its stages. Bloom taxonomy (Anderson & Kathwrol) gives systematic arrangement of the learning objectives which will determine learning methods, contents, or assessment. The following is the figure describing the relationship among learning taxonomy, learning objective, learning method, and evaluation. Saifuddin and Nurcahyani 133 | IJET| Volume. 7, Issue 2. December 2018 Figure 1. A Rationale of Instructional Design proposed by Khalil and Elkhider (2016) This development must follow the ideas that learners should master the first level before going through the next level. Based on figure 1, it implies that the learning taxonomy based cognitive domain should be well developed by Bloom taxonomy in which each stage is initialized by the certain action verbs. The development of learning objectives determines types of assessment as well as a teaching method. METHOD Qualitative research provided this study to investigate the organization of the learning outcomes and its implementation in the learning process. More specifically, this study aimed at figuring out the learners’ level of cognitive-based on the syllabus and learning process. When it came to the identification of cognition in which it can be observed and measure, this referred to the use of a qualitative method. As it is in line with Mack et al. (2005) that an effective way of using qualitative research is that it seeks culturally specific information of opinions, social context, behaviors, and opinions. The purpose of this qualitative research used here was that it sought an understanding of a certain phenomenon under the investigation to give a brief picture of the phenomenon (Ary, Jacobs, & Sorensen, 2010). This study proceeded to gain the data from current syllabus used as the primary source about learning objectives formulation. This included a textual analysis of the document using the theory of Bloom’s taxonomy (Anderson & Krathwohl, 2001). The textual analysis used in this study involved an analysis using instructional analysis. Learning objectives written in a syllabus were the sources of the analysis while another source was from the observation. 1. Participants In association to investigating learners’ cognitive level, students of university level were the participants of this study. They were students in the first year of academic learning. They had to take a subject called ‘English Consortium’ as a prerequisite to continue to the next level of learning semester. Students involving this study were from a different department. They were from 4 different departments; engineering, administration and business, mathematics, and Islamic studies department. The ideas under the investigation intended that they were from the non-English department. Learners’ Cognitive Level 134 | IJET| Volume. 7, Issue 2. December 2018 2. Instruments Observation and textual analysis on current syllabus used were done qualitatively. Both instruments sought the indications of the specific learning activities about learning cognitive aspect. Through the study of the current syllabus described the criteria of students’ cognitive level. It tried to find a relation between the formulation of the learning objectives regarding the cognitive level and how the activities carried out to present their attainment of the targeted cognition. The followings were the indicators of the observation: 1. Learning Achievement a. Students indicate their learning achievement for each activities b. Learning activities measure students’ cognitive skill c. Students learning results represent the learning outcome 2. Learning Activities a. Teacher applies appropriate activities based on the syllabus b. Teacher applies to learn stages; Cognitively, their learning abilities were visualized through the seen-activities performed by the learners. The observation emphasized figuring out the clues or proofs representing their cognitive level during the learning process. Also, this observation was conducted at some meetings. It was due to different objectives implemented for different meetings. On the other hand, the textual analysis which used instructional analysis covered hierarchical, cluster, and procedural approach. They further explained the relationship among one objective to other objectives. Through it, it also measured how learning outcome could be achieved by existing learning objectives. 3. Data Analysis As it is defined as qualitative research, the collected data were analyzed qualitatively. There were data found from the result of observation and syllabus analysis. Those data from two instruments were analyzed procedurally and systematically to figure out learners’ cognitive aspect. Tabulation was used to analyze teacher and learners’ activities. The description findings from this observation were also counted to find its relationship to syllabus design of TOEFL learning activities. While the analysis of the syllabus resulted in the description of learner’s learning level in term of cognition. The analysis covered the suitability of learning goals formulation toward the principles of learning taxonomy, referring to Bloom taxonomy revision (Anderson&Krathwohl, 2001). Also, to find its implication, the data were measured upon the activities done during the learning process. FINDINGS A major aim of this study relied on the activities conducted during the learning process which represented the focus of the learning objectives. Briefly, activities brought in the class described how students learned of which their way of learning was the evidence of these activities could interpret the learning levels based on the cognitive domain. The interpretation of the cognitive domain stated in learning taxonomy assessed in what level of the activities looked like. The first step of the analysis involved textual analysis on the syllabus used. It dealt with the systematic and procedural formulation of the learning outcome the analysis used an instructional analysis approach. Though this study limited to figuring out students’ level of cognitive skill, the analysis still used three different approaches to instructional analysis, Saifuddin and Nurcahyani 135 | IJET| Volume. 7, Issue 2. December 2018 covering hierarchical, cluster, and procedural approach. Moreover, the analysis was carried out to determine what level of expected learning goals achieved. There were different learning domain could be explicitly seen, there was a cognitive, psychomotor, and affective domain. Based on the analysis of the current syllabus, there were cognitive skills formulation stated. It derived from the action verbs used for each learning objective. Regarding syllabus analysis, this study rewrote the learning objectives into a graph. It described the procedure of formulation and its learning route. There were 14 learning objectives under a learning outcome. It implied that to achieve the learning outcomes, the fourteen-learning objectives should systematically support one another. In the analysis, both the learning outcome and learning objectives were analyzed based on the learning taxonomy, focusing on the cognitive domain. It aimed to identify whether the learning objectives had achieved the learning outcome or not. Table 1. Analysis on Cognitive Doman in Syllabus Cognitive Taxonomy Objectives in Syllabus Learning Outcome Creating - - Evaluating - - Analyzing - - Applying - 2) Mampu menerapkan pemahaman tersebut untuk mengerjakan soal- soal sederhana TOEFL dengan baik dan benar. Understanding 2) Mahasiswa mampu memahami tentang materi Listening part A, skill 2 dan 3 dengan baik dan benar. 1) Mahasiswa mampu memberikan penjelasan tentang materi TOEFL: Listening part B, part C, Structure and Written Expression, Reading Comprehension dan Remembering 1) Mahasiswa dapat mengetahui dan memahami ruang lingkup TOEFL dan materi Listening part A, skill 1 Through the analysis on table 1, it was identified that one learning objective represented remembering while the rest of the learning objectives represented understanding. The identification of the learning objectives relied on the interpretation of them which should reflect based on the cognitive domain. About classification of the objectives, action verbs identified their level of learning taxonomy. The learning objectives used in the syllabus showed that it was expected for the students to be able to understand/ identify the TOEFL materials covering listening comprehension, structure and written expressions, and reading comprehension, that these objectives reflected the understanding stage of learning taxonomy. Furthermore, based on table 1 about the learning outcome, it indicated that the outcomes focused on understanding and applying. Here, to be able to reach the outcomes, the Learners’ Cognitive Level 136 | IJET| Volume. 7, Issue 2. December 2018 formulation and the arrangement of the learning objectives should be able to reach the outcome which meant that the outcomes were reachable because the learning objectives were formulated staging and the last objective should reflect the outcomes. There was a missing objective about achieving the second outcome. The outcomes stated in the syllabus meant learners should have an understanding ability and an ability to apply using their understanding. The interpretation of these outcomes was that at the end of the course, learners were able to apply their existing knowledge of TOEFL material understanding in answering the questions. Since the outcomes stated applying then learners’ activities should also be dealt with applying stage. On the other hand, to see the suitability of the action toward the learning outcomes stated before, the observation was carried out. The observation covered three main aspects; they were students’ learning competence, students’ activities, and the attainment of the learning objectives. Chart 1. Percentage of Learning Activities Based on Taxonomy During the observation, the identification of learners’ learning activities can be described in chart 1. It was revealed that most activities used during the learning process represented understanding taxonomy. There was 93% of the total activities applied. Learners were required to do some activities which asked them to understand the material. In this case, the activities were done through the explanation given by the lecturer. Learners’ activities during the explanation were only to listen, which meant that they were in remembering stage, they defined and remembered a lot upon the lesson in which there was notified 7% of the remembering activities. Regarding the above explanation, learners were also required to be able to explain or retelling based on their understanding of the lesson. It was identified that learners monotonously listened to lecturers explanation while they tried some exercises to have practices to answer the TOEFL questions provided by the lecturer. At the same time, these findings also described that learners, when answering the questions, tended to apply their understanding. They got the explanation of the lesson and did some practices. However, it was not categorized as applying, but rather remembering and understanding. Based on students’ activities has resulted in the observation phase, in line to the findings in chart 1, most activities done by learners were that they listened to the materials a lot. Cognitively, the activities seemed to be passive and it activated their remembering and understanding ability by identifying the concept, principles, and the definition of the lesson. Also, something quite different regarding applying, a few learners performed making sentences in front of the class explaining the sentence based on the topic they discussed. Another important thing for measuring learning success is that looking at how far learners attain their best achievement. During the learning process, it was quite different from the syllabus. Learners still found their learning difficulties to understand the lesson. Their activities were always preceded and interfered by the lecturer. Learners’ were not an independent learners as they should be. It influenced learners’ learning quality. Saifuddin and Nurcahyani 137 | IJET| Volume. 7, Issue 2. December 2018 Table 2. Learning Quality Indicators Very good Good Poor Very Poor a. Conducting evaluation and review to measure learners’ achievement of the learning outcome √ b. Learners show their abilities representing their achievement of every learning objectives stated in syllabus. √ c. Every activity measures the achieveable learning objectives √ The objectives should give learning activities more references how the lecturer and learners achieve the learning goals. However, since there was no effort to identify the needs of achieving learning objectives, it was hard for learning success. Moreover, table 2 showed how learning activities could not support their learning ways. The lecturer did not see whether individual learners had already attained the objectives or not. Although the objectives were clearly stated, learners were not led well regarding giving them feedback for their problem solving of their difficulties. Thus, learners did not show their achievement of individual learning objectives. The learning process was also based on the handbook provided by the team of the writers. This book was not specifically addressed for beginners learners, especially those who were non-English students. However, regarding cognitive domain, this book entailed learners’ understanding or subject mastery of TOEFL learning materials in which the activities provided in it. Nevertheless, it was further found that there were no systematic and procedural activities carried out during the process that the activities could benefit the learners in stepping forward from a different level of difficulties and attained the goals. DISCUSSION Today’s challenging ideas of teaching is that bringing learners into real learners who are experiencing the life of the real world. This implies teaching and learning process that it should be advantageous and meaningful which it supports the learning success. The ideas of learning taxonomy are that to ensure the learners to transforms their knowledge into a higher level of critical thinking rather than just a merely of memorizing (Hyder & Bhamani, 2016). Thus, the demand of the teaching is to give students build their ability to maintain their higher thinking order and to liven their learning skills through learning stages as written in learning taxonomy. The analysis of the outcomes brought an understanding of how the activities brought in the class. Besides, it was a merit to figure out the strength and the weakness of the stated objectives. The analysis was based on the learning taxonomy, bloom taxonomy in term of the cognitive level (Anderson & Krathwohl, 2001). Every stage from the learning taxonomy had their learning description of the achievement. Based on the learning outcomes analysis, although there were 14 learning objectives stated in the previous syllabus, the stated objectives did not show the procedurally learning stage. The objectives showed not more than understanding. What should be there was that the Learners’ Cognitive Level 138 | IJET| Volume. 7, Issue 2. December 2018 14 learning objectives should be able to express the systematic procedure of learning route. Learners began learning from very easy task to a more difficult one. The analysis could not be brought into a pyramid of learning taxonomy which described the complexity of the learning activities (Rupani, 2011). It was also revealed that most objectives used action verb representing understanding, that was understood/comprehend. In relation to the stated learning outcome, one of them was an application. This stage required learners to use their knowledge got from the level of understanding which meant learners could not proceed to go higher unless they mastered the previous one. Looking at the result of the analysis, there was an expectation of the learning outcome that students were able to apply their understanding to answer the TOEFL questions. This learning outcome did not find any supportive learning objective. In other words, it was not found any evidence showing learning objective which implemented the concept of applying. It means that students should not be able to maintain applying stage because they had not achieved the applying objectives. If the outcome was about applying, then there must be certain procedural activities (Anderson & Krathwohl, 2001). Cognitively, it stated that students should be able to apply of their knowledge, but it implied procedural activities which lead them doing the activities from the beginning to the next step. This procedure entailed physical movement, more specifically psychomotor aspect. Learners performed their ability of how to do something which became the idea of applying level. This procedure should be well developed in learning objectives. Besides, related to the analysis, instructional analysis using a procedural approach gave more details that the objectives were not well formulated regarding the achievement of the learning outcomes. CONCLUSION The analysis of learning outcomes is essentially done to measure how far learners attain their skills and more importantly learning achievement. From the result of the analysis, teachers are given more information on how conducting valuable and meaningful learning activities which represented the objectives that systematic and procedural learning is easily carried out. Furthermore, the learning process should be able to provide and build learners’ higher thinking in which they maintain it through learning stages. It shows that the activities of learning lead learners form the easy one to more difficult. In term of cognitive level, learning activities should picture the pyramid of learning taxonomy that there are remembering, understanding, applying, analyzing, evaluating, and creating. In conclusion, using this analysis, it enables teachers to determine what level of learning stage they were in. Based on the findings and discussion, it can derive a conclusion that learners’ level of cognition was at understanding. The conclusions of the analyses found no activities to support the outcomes though they were formulated as in applying stage. Acknowledgement Appreciation and gratitude to Direktorat Riset dan Pengabdian Masyarakat, Dirjen Pengusulan Riset dan Pengembangan, Kementrian Riset, Teknologi dan Pendidikan Tinggi, who has financed Penelitian Dosen Pemula 2018 under the title “Pengembangan Bahan Ajar Bahasa Inggris Konsorsium untuk Mahasiswa Non Bahasa Inggris di Lingkungan Pesantren Berdasarkan Analisis Instruksional”. Saifuddin and Nurcahyani 139 | IJET| Volume. 7, Issue 2. December 2018 REFERENCES Anderson, L. W., & Krathwohl, D. R. (2001). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives (Abridge Edition). Boston: Allyn & Bacon. Ary, D., Jacobs, L. C., & Sorensen, C. (2010). Introduction to Research in Education. California: Wadsworth Cengage Learning. Biggs, J. (2014). Constructive Alignment in University Teaching. HERDSA Review of Higher Education, 1(5), 5-22. Biggs, J., & Tang, C. (2011). Teaching for Quality Learning at University: What the students does (Fourth Edition ed.). London: McGraw-Hill Education. Brophy, J., E. (2013). Motivating Students to Learn. New York: Routledge. Dick, W., & Carey, L. (1996). The Systematic Design of Instruction. New York: Harper Collin. Hyder, I., & Bhamani, S. (2016). Bloom's Taxonomy (Cognitive Domain) in Higher Education: Reflection Brief. Journal of Education and Educational Development, 3(2), 288- 300. Khalil, M. K., & Elkhider, I., A. (2016). Applying Learning Theories and Instructional Design Models for Effective Instruction. Advance in Physiology Education, 40, 147-156. Kleebbua, C., & Siriparp, T. (2016). Effects of Education and Attitude on Essential Learning Outcomes. Procedia Social Behavioral Sciences, 941-949. Mack, N., Wodosong, C., MacQueen, K. M., Guest, G., & Namey, E. (2005). Qualitative Research Method: A Data Collector's Field Guide. North Carolina: Family Health International. Nation, I. S. P., & Macalister, J. (2010). Language Curriculum Design. New York: Routledge. Rupani, C. M. (2011). Evaluation of Existing Teaching Learning Process on Bloom's Taxonomy. International Journal of Academic Research in Business and Social Sciences, 1, 119-126. Sishkovskaya, J., Bakalo, D., & Grigoryev, A. (2015). EFL Teaching in the E-Learning Environment: Updated Principles and Methods. Procedia Social Behavioral Sciences, 199- 2014.