EEJ 9 (4) (2019) 428 - 436 English Education Journal http://journal.unnes.ac.id/sju/index.php/eej Using Revised Bloom’s Taxonomy to Evaluate Higher Order Thinking Skills (Hots) in Reading Comprehension Questions of English Textbook for Year X of High School Kurnia Atiullah1, Sri Wuli Fitriati,2 Dwi Rukmini2 1. MAN Temanggung, Indonesia 2. Universitas Negeri Semarang, Indonesia Article Info ________________ Article History: Recived 03 February 2019 Accepted 25 July 2019 Published 23 December 2019 ________________ Keywords: Higher Order Thinking Skills (HOTS), Reading Comprehension Questions ____________________ Abstract ____________________________________________________________ Higher Order Thinking Skills (HOTS) has been widely recognized as a set of important devices of cognitive development. The research was conducted to evaluate the availability of higher order thinking skills in reading comprehension questions of the English textbook for Year X of high school published by ministry of education. There are 158 reading comprehension questions from 15 reading texts which are analyzed using revised Bloom’s taxonomy. An intuitive approach to the analysis of qualitative data is taken for the study in which an individual researcher intuitively relates data from various instruments to each other (Smaling, 1987) in Meijer, et al (2002, p. 146). The data were analyzed qualitatively to determine the cognitive level of each questions according to revised Bloom’s taxonomy, Question types based on which cognitive process is required to answer and forms of questions. The result shows that majority of reading comprehension questions in the textbook being studied is in the lowest level of revised Bloom’s taxonomy; remembering with 134 items whereas higher order thinking skills are only found in 24 out of 158 items. It was concluded that the reading comprehension questions of the English textbook for Year X of high school is lack of higher order thinking skills. The result of the study is expected to benefit the English teachers, textbook writers and further researchers to elaborate the implementation of higher order thinking skills in English teaching and learning. © 2019 Universitas Negeri Semarang Correspondence Address: Jl. Jenderal Sudirman 184, Temanggung, Indonesia E-mail: kenhatchiko@gmail.com p-ISSN 2087-0108 e-ISSN 2502-4566 mailto:roniwahyudi@students.unnes.ac.id Kurnia Atiullah, Sri Wuli Fitriati, Dwi Rukmini/ EEJ 9 (4) 2019 428 - 436 429 INTRODUCTION The conduct of the study is mainly intended to examine to what extent higher order thinking skills is integrated in reading activities of Year X English textbook published by Indonesian ministry of education in 2017. There have been some studies which evaluate the English textbooks in the light of higher order thinking skills in Indonesia. However, a study which aims at examining higher order thinking skills in the current textbook published by the ministry of education has not been found. Therefore, the result of this study is hoped to contribute in the betterment of the textbook content with the emphasis on the accommodation of higher order thinking skills. Higher order and lower order thinking skills are two sets of thinking skills compared in terms of the complexity degree of a thinking process. The two thinking processes are defined by experts in different ways yet agree on some common grounds. In Lewis and Smith (2009, p. 132), Meier (1993) described higher order thinking skills as reasoning or productive behavior. On the other hand, lower order thinking skills are categorized as learned behavior or reproductive thinking. As reasoning or productive behavior, higher order thinking skills enable a learner to work on a problem which are not immediately recognized by his/her learned behavior. In other words, higher order thinking skills are used to solve a problem where learned behavior is not applicable without reasoning. An example given is a learner’s ability to manipulate the formula of parallelogram by converting a parallelogram to a rectangle of the same area. Reasoning is thus used to solve the problem. According to Maier (1993), a problem occurs when a target or a solution is not achieved by the first effort and therefore needs reasoning to work it out. Bloom’s Taxonomy was named after Benjamin Bloom, then an Associate Director of the Board of Examinations of the University of Chicago, who initiated a discussion among a group of evaluation experts from across United States to outline some educational objectives and assessments for institutions to use (Krathwol, 2002, p. 212). Revised Bloom’s taxonomy introduces six grades of cognitive process namely the cognitive domain of remembering, understanding, applying, analysing, evaluating and creating in which the first three stages are classified as lower-order thinking skills and the other three upper stage belong to higher- order thinking skills. Higher-order thinking skills and reading are two interconnected skills. In reading, students not only have to understand the meaning of the words written down (Yu-hui, 2010, p.60). She further emphasizes that reading involves a complicated, actively thinking mental activity which requires the students to experience, predict, verify and acknowledge information based on reader’s background knowledge and experience. Doing a reading task will need the students’ effort to recall and apply what they have learnt (this process resembles transferring in higher-order thinking skills) to perform some critical thinking about the reading text they have to deal with to make meaning. When the students find difficulties in achieving the objectives of a reading task, either finding main ideas, supporting ideas, detailed information, or implicit information, their problem-solving skills is urgent to be used. Providing that reading assignment is a complicated task, it will be harder for students to perform well without equipping themselves with higher order thinking skills. Some studies on the Indonesian students’ reading difficulties revealed that some of the issues which hamper Indonesian students to achieve good result in reading test are students’ passive attitude and students’ lack of reading strategies (Suryanto, 2017, p. 201-202). Some reading difficulties also found in the students’ insufficient vocabularies, lexical inefficiency, structural complexity, language inaccessibility, poor reading skills, lack of schemata, and students’ low motivation (Rahman, 2007, p. 153). To relate these issues with higher order thinking skills, students’ incapability of applying their linguistic knowledge to assist them in reading shows students’ lack in transferring skills Kurnia Atiullah, Sri Wuli Fitriati, Dwi Rukmini/ EEJ 9 (4) 2019 428 - 436 430 while their passivity in classroom might indicate the deactivation of critical thinking and their sufficiency in reading strategies is an evidence of the absence of problem solving process. Should there be a solution in increasing students’ reading skills, it must be a qualified textbook which facilitates the development of both reading skills and higher order thinking skills. The latest English textbook published by the Indonesian ministry of education was revised in 2017 in is one of the teaching resources widely used by English teachers of Secondary high schools. The textbook was published before Buku Pegangan Pembelajaran Berorientasi pada Ketrampilan Berpikir Tingkat Tinggi (Handbook of Higher Order Thinking Skills-Oriented Teaching) issued by Indonesian Ministry of Education in 2018. However, it is supposed to response the need of implementing higher order thinking skills in English classrooms as stated in the background of the textbook that the practical framework of the 2013 curriculum implementation in classroom put one of the emphasizes on the reinforcement of students’ critical thinking which is part of higher order thinking skills. Regarding reading activity, a good textbook should provide enough reading materials along with recommended teaching strategies and samples of reading comprehension questions with which the teachers equip themselves for teaching and developing higher order thinking skills at the same time. Textbook remains to have an essential part in the english teaching and learning activities in classroom. Most teachers use textbook as their main teaching reference. Especially in reading activities, many teachers still depend on the reading material and exercises provided by the textbook. Some teachers even do not elaborate further to decide what strategies to be best used in a certain reading activity. Thus, developing a qualified textbook which facilitate teachers whose time and ideas are limited will need a careful analysis on its content not only to improve its quality but also to adjust with the current objectives of english education in general. Since indonesian education has been encouraged to innovate with higher order thinking skills, promoting students’ higher order thinking skills through textbook is an important decision to take. Thus, evaluating to what extent higher order thinking skills has been adopted in the reading activities of the textbook, especially reading comprehension questions is vital. The evaluation will yield in insightful feedbacks for the betterment of the textbook. A thorough analysis of the content of the textbook which specifies in reading comprehension questions will benefit in the improvement of the reading comprehension exercises to be used by teachers in classrooms. There are also some studies which shed light on the inclusion and implementation of higher order thinking skills in reading activity. Keshta and Seif (2013, p. 47-69) conducted a study to evaluate the treatment and availability of higher order thinking skills in English for Palestine Grade 8 in reading comprehension. Using content analysis card, his study sought to collect data about to what extent the reading exercises include analysis skills, synthesis skills, and evaluation skills. The findings suggested that higher order thinking skills in reading exercise are not well covered, not well implemented nor well distributed. An interesting study on how higher order thinking skills affect students’ reading ability is done by Samelian (2017). Through series of guided-reading sessions and independent work time, she collected and evaluated students’ responses and analyzed them in terms of the level of understanding and comprehension of the text, the use of higher order thinking skills and justification and evidence-based report. In the end of the study she found that guided reading strategy has led into a deeper understanding, more questions, and secure and specific comprehension. The critical thinking approach has helped the students to get engaged in the Kurnia Atiullah, Sri Wuli Fitriati, Dwi Rukmini/ EEJ 9 (4) 2019 428 - 436 431 activity. The impact of the study to herself as a teacher was that she improved her higher-level questioning skills and realized that questioning and students’ thinking are significantly interconnected. METHOD This study is a descriptive qualitative study which uses content analysis as the method of the study. The method is used to describe the content of the textbook. One of the purposes of this method is to identify the higher-order thinking skills in reading questions presented in the English textbook for Year X students of high school by using the revised Bloom’s taxonomy. The source of the data in this study is the reading questions available in Year X English textbook for students. The reading questions to be used as the data of the study are the ones given after some functional and short functional texts. Unit of analysis in this study is reading questions which aim at advancing students’ comprehension. The smaller unit analysis of the reading questions will include the grammatical form of the questions and the action verbs used in the questions. In qualitative studies, the human investigator is the primary instrument for the gathering and analyzing of data (Ary et al, 2010, p. 241). Therefore, in this study, as the researcher I play the roles of a data collector and analyst. As a data collector, I collect the data utilizing a number of instruments and analyze the data by referring Bloom (1956, p. 70), Anderson, L. W., & Krathwohl,D.R. (2001, p. 40), Nuttall (1996, p. 186) and Pearson and Johnson (1972) as presented by Day and Park (2005, p. 62-65) in relation with reading questions. Two forms of checklist will be used to assess the availability of higher order thinking skills I the questions. The first one is a checklist classifying the questions according to revised Bloom’s taxonomy. The result will then be triangulated using the types and comprehension forms of questions. The researcher herself will be the reviewer who performs analysis of the questions based on the criteria provided in the checklist form. RESULTS AND DISCUSSIONS Using revised Bloom’s taxonomy (2001), this study finds that among 158 items comprising from 26 items of Yes/No reading questions and 132 items of W/H questions of the Year X English textbook, 119 items (75.31%) of them belong to this cognitive domain of remembering. Most of reading comprehension questions in the textbook contains questions whose answers can be easily located in the texts. The first reading text given to the readers is an email sent by Hannah to her penpal Alia contains eight reading questions which all of them entail the readers to recall the previous information learnt from the text such as “Does Hannah want to be Alia’s friend?”. To respond to this question, students can simply refer to the excerpt of the text which stated, “ I’d really like to be your E-pal”. It is also revealed that there are only 10 reading comprehension questions or 6.32 % of them which meet the qualification of understanding domain. Surprisingly, out of 158 reading comprehension questions being studied, there are found only four questions which fulfil the criteria of applying level. Like applying level, it was found there are only four questions which include analysing level. Next, the study found there are 14 items or 8.86 % which reflects the incorporation of this evaluating level of higher order thinking skills. Regarding the highest level of cognitive domain in revised Bloom’s taxonomy, there are only six questions which meet the criteria of the level, or 3.80% of the total number. Answering the main problem of the present study about the availability of higher order thinking skills in the reading comprehension questions of Year X English textbook, the present study concluded that Year X English textbook entitled Bahasa Inggris Kelas X by Ministry of culture and education does not adequately provide reading Kurnia Atiullah, Sri Wuli Fitriati, Dwi Rukmini/ EEJ 9 (4) 2019 428 - 436 432 questions which facilitate the students in developing their higher-order thinking skills. This is proved by the findings which indicated that there are only 24 reading questions fall into higher order thinking questions while questions which are considered as lower thinking order outnumbered by 134 items out of 158 items in total. This means that the availability of higher order thinking skills in the textbook is only 15.18% compared to that of the reading questions which belong to lower order thinking skills which are 84.81%. This is quite surprising and disappointing fact considering that reading questions are supposed to be fostering students’ higher order thinking skills rather than lower order thinking skills. Having analysed the reading comprehension questions of the textbook being studied and classified them into each category of the revised Bloom’s taxonomy does not necessarily justified the findings. In the light of Bloom’s taxonomy, there are some distinguishing opinions among experts. Despite the fact that both original and revised Bloom’s taxonomy are being referred by many as indicators for pedagogical assessment and learning objectives, criticisms have been made pointing out the taxonomy to be oversimplifying the nature of thinking and its relationship to learning (Furst, 1994) in Rahman and Manaf (2017, p. 246). Widodo, an associate professor in Shantoung University and a prominent researcher in the field of English education sparked a debateable opinion on the application of Bloom’s taxonomy which discreetly divide thinking processes into six hierarchical cognitive domains. On one of his Facebook’s feed (April 16, 20119) he wrote that grouping levels of thinking into lower order thinking skills and higher order thinking skills are questionable as each level contains skills which are not deliberately hierarchical. He further argued that all levels of the cognitive domains are in fact, interdependent. Besides, he highlighted the use of operational words in Bloom’s taxonomy which are widely agreed as indicators for the attainment of a certain level. He disapproved the use of those operational words as the only indicator for the accomplishment of each thinking level, saying that those operational words are too general and cannot function as a valid and reliable assessment tool. Taking the above criticisms into consideration, I believe it is important to compare the analysis based on revised Bloom’s taxonomy with questions analysis based on the grammatical features; types and forms to figure out the relevance of the questions with the cognitive skills required in responding the questions. As discussed in chapter 2, questions are distinguished grammatically into five types of questions namely yes/no questions, alternative questions, true/false questions, w/h- questions (who, what, when, where, which, how, why), and multiple-choice questions. Meanwhile, in relation to the cognitive aspects employed by the students to respond, questions are classified into six hierarchical categories namely questions of literal comprehension, questions involving reorganization or reinterpretation, questions of inference, questions of evaluation, questions of personal response and questions concerned with the reader’s prediction as proposed by Nutall (1996, p. 186) and Pearson and Johnson (1972) in Day and Park (2005, p. 62-65). Reading comprehension questions contained by the English textbook for Year X students are dominated by open ended questions (W/H Questions) by 87.97% while only 16.45% of the total items are in the form of closed ended questions (Yes/No questions). This is very interesting as open ended questions are regarded as helpful for teachers to engage students in higher-level questions (Roth, 1996) as cited by Cakir and Cengiz,( 2016, p. 61). It is believed that open ended questions emphasizes on the process of working together and stimulates students’ ability to connect new information with their background knowledge and interest Kurnia Atiullah, Sri Wuli Fitriati, Dwi Rukmini/ EEJ 9 (4) 2019 428 - 436 433 (Panitz, 1999) as quoted further by Cakir and Cengiz (2016, p.61). In addition, according to Lee, Kinzie, and Whittaker (2012) in Cakir and Cengiz (2016, p.62), open ended questions facilitates students in exploring and expanding the answers by elaborating their thinking and rationale. As the result, the data above likely shows a larger number of reading comprehension questions in the English textbook for year X which promote students’ higher order thinking skills. However, assumption made by merely classifying the questions into open and closed- ended questions seems to be inadequate. Having a deeper look into the questions by analysing the cognitive aspects called for answering the questions is needed to result in a thorough conclusion. The data below presents the review on the reading comprehension questions based on the cognitive skills involved in solving the problems. It is important to note that six divisions of the questions based on the cognitive skills required are graded in terms of thinking complexity from the least complex to the most complex one. In other words, these six categories suggest that literal level is the lowest order thinking skills, reorganization or reinterpretation is the second lowest, and inference is the third lowest. Next, evaluation level is the fourth level higher than the previous three, personal response is the fourth higher order thinking skills, and the top one is prediction level. Literal comprehension is when the reader might obtain information which is explicitly presented in the text. This is like that of remembering in revised Bloom’s taxonomy. Next, reorganization or reinterpretation is when the readers need to combine several literal information to generate a correct interpretation. This is equal to understanding. The third one is inference. It entails the readers to dig the implied information and to put together information to result in a solution to be applied in new situation or to answer the question. This level is connected to applying level in Bloom’s taxonomy. Then, evaluation level. This form of question demands reader’s evaluative view and sometimes justify it based on grounded reasons and true evidences but not personal ones. This level is the same as evaluating which takes the fifth place in revised Bloom’s taxonomy. Almost similar to evaluation level is the following type which is called personal response question. This type of question enquires the reader’s opinion where objective evidences and data are not always necessary. This level falls into evaluating in revised Bloom’s taxonomy. The highest one is this cognitive level according to the type of the question is prediction level. In this stage, the readers are challenged with providing their prediction by imagining and estimating the result based on their learned information and background knowledge. This final level is equal with the top rank of the cognitive domain in revised Bloom’s taxonomy, creating. From the comparison above, it is observable that each level in both categories correspond to each other. Even though the definition carried by each level of thinking in both groups differs in some extent, the values contained by each division is equal respectively. It is also recognizable that the three-lower level of both categories; revised Bloom’s taxonomy and Question type agreed the definition of lower order thinking skills while the upper three levels confirm the requirement of higher order thinking skills. Thus, discussing reading comprehension questions based on the question types will help validate the classification made based on revised Bloom’s taxonomy in the findings. As expected, triangulation by theory using questions analysis based on the type and comprehension forms indicated the same number of 134 items for lower order thinking skills and 24 items for higher order thinking skills. However, there is a slightly different in the number of questions in each category. Kurnia Atiullah, Sri Wuli Fitriati, Dwi Rukmini/ EEJ 9 (4) 2019 428 - 436 434 Table 1. Summary of Questions according to revised Bloom’s taxonomy Basic Competences Number of Questions The revised Bloom’s taxonomy Total Remembering Understanding Applying Analyzing Evaluating Creating Hannah’s email 8 8 0 0 0 0 0 8 Saidah’s email 10 10 0 0 0 0 0 10 Congratulating and complimenting Alif 6 4 2 0 0 0 0 6 Congratulating and complimenting Dita 6 3 2 0 0 0 1 6 Tanjung Puting 13 7 2 2 0 2 0 13 Taj Mahal 14 6 1 2 2 3 0 14 Niagara Falls 12 10 0 0 0 0 2 12 Concert cancellation 8 7 1 0 0 0 0 8 Course registration 7 6 1 0 0 0 0 7 Meeting My Idol 13 9 1 0 1 2 0 13 The Battle of Surabaya 13 5 1 0 1 5 1 13 BJ. Habibie 12 12 0 0 0 0 0 12 Cut Nyak Dien 16 15 0 0 0 0 1 16 Malin Kundang 10 8 0 0 0 1 1 10 Strong Wind 10 9 0 0 0 1 0 10 Total 158 119 items = 75.31% 11 items = 6.96% 4 items = 2.53% 4 items = 2.53 % 14 items = 8.86% 6 items = 3.79% 158 items = 100% Kurnia Atiullah, Sri Wuli Fitriati, Dwi Rukmini/ EEJ 9 (4) 2019 428 - 436 435 Table 2.Summary of Types and Comprehension Forms of Questions Reading Texts Number of Questions Types and Comprehension Forms TTotal Literal Reorganizational Inferential Evaluative Personal Response Predicting Hannah’s email 8 8 0 0 0 0 0 8 Saidah’s email 10 10 0 0 0 0 0 10 Congratulating and complimenting Alif 6 4 0 2 0 0 0 6 Congratulating and complimenting Dita 6 2 3 0 1 0 0 6 Tanjung Puting 13 7 2 2 0 1 1 13 Taj Mahal 14 6 0 3 2 3 0 14 Niagara Falls 12 10 0 0 0 1 1 12 Concert cancellation 8 7 1 0 0 0 0 8 Course registration 7 6 1 0 0 0 0 7 Meeting My Idol 13 10 1 0 1 1 0 13 The Battle of Surabaya 13 5 1 0 2 4 1 13 BJ. Habibie 12 12 0 0 0 0 0 12 Cut Nyak Dien Malin Kundang Strong Wind 16 10 10 15 8 9 0 0 0 0 0 0 0 1 0 1 1 1 0 0 0 16 10 10 Total 158 119 items = 74.05% 10 items = 6.322% 5 items = 3.16% 7 7 items = 43.43% 13 items = 8.22% 4 items = 2.53% 158 = 100% CONCLUSION AND SUGGESTION In conclusion, this present study conveyed that the English textbook for Year X of high school published by ministry of education revised edition 2017 does not provide enough reading comprehension questions which stimulate students’ higher order thinking skills. The comparison between reading comprehension questions which require students’ lower order thinking skills and the ones which foster students’ higher order thinking skills shows a significant gap by 134:24 in totals, or 84.81% of the total number is categorized as lower-order thinking skills questions and leaving only 24 (15.18%) reading comprehension questions regarded as promoting higher order thinking skills. Findings of this study are evident that teachers should not depend solely on textbook in developing teaching material. When higher order thinking skills become English teacher’s concern and is are believed to be important to be fostered, it is better for English teachers to enrich their teaching materials with various resources other than English textbook provided by the Kurnia Atiullah, Sri Wuli Fitriati, Dwi Rukmini/ EEJ 9 (4) 2019 428 - 436 436 government. Textbook writers also should carefully consider choice of texts and reading comprehension exercises as to stimulate the growth of students’ higher order thinking skills. The data gained from this study is an important reference for further studies and hopefully benefit the researcher in seeing through the problems of English teaching in Indonesia in the light of the development of higher order thinking skills. REFERENCES Ary, D., Jacobs, L. C., Irvine, C. K. S., & Walker, D. (2010). Introduction to research in education. Cengage Learning. Anderson, L. W., Krathwohl, D. R., & Bloom, B. S. (2001). A taxonomy for learning, teaching,and assessing : A revision of bloom's taxonomy of educational objectives. New York; London: Longman. Bloom, B. S. (1956). Taxonomy of educational objectives. Boston, MA: Allyn & Bacon. Cakir, H., & Cengiz, O. (2006). The use of open ended and closed ended questions in Turkish Classrooms. Open Journal of Modern Linguistics. 6, pp. 60-70. Day, R.R., & Park, J.S. (2005). Developing reading comprehension questions. ERIC. 17 (1), pp.60-73. Nuttall, C. (1996). Teaching reading skills in a foreign language (2nd ed.) with chapter on testing reading by J. Charles Alderson. Oxford: Heinemann English Language Teaching Meijer, P.C., Verloop, N., & Beijard, D. (2002). multi-method triangulation in a qualitative study on teachers’ practical knowledge: an attempt to increase internal validity. Quality & Quantity, 36, 145-147 Lewis, A., & Smith, D. (1993). Defining higher order thinking. Theory into practice, 32(3), 131-137. Keshta, A.S., & Seif, A. (2013). Evaluating higher order thinking skills in reading of English for Palestine grade eight. Asian Journal of Education and E-learning. 01 (01), pp. 47-69. Krathwohl, D. R. (2012) A revision of Bloom’s taxonomy: An overview. Theory into Practice College of Education, The Ohio State University. 41(4), 213-218. Samelian, L. A. (2017). How higher order questioning and critical thinking affects reading comprehension. Retrieved from https://digitalcommons.hamline.edu/h se_all/4345 Rahman, M. H. (2007). An evaluation of the teaching of reading skills of English in Bangladesh. (Master’s thesis). https://www.asian-efl-journal.com Rahman, Syukron A., Manaf Nor F. A. (2017). A critical analysis of Bloom’s taxonomy in teaching creative and critical thinking skills in Malaysia through English literature. English Language Teaching. 10 (9), 245-256. Seif, A. A. A. Q. A. (2013). Evaluating the higher order thinking skills in reading exercises of English for Palestine grade 8. Asian Journal of Education and e-Learning. 1(1), 47-69 Suryanto. (2017). An investigation on English reading comprehension problems in Indonesian cultural contexts. The 1st International Conference on Education, Science, Art and Technology (the 1st ICESAT. 200-205 Widiati, U., Rohmah, Z., & Furaidah. (2017). Buku Siswa Bahasa Inggris Kelas X Sma/Ma/Smk/Mk Edisi Revisi 2017. Kementerian Pendidikan dan Kebudayaan Republik Indonesia Yu-hui, L., Li-rong, Z., Yue, N., (2010) Application of schema theory in teaching college students reading. Canadian Social Science 6(1), 59-65. https://digitalcommons.hamline.edu/hse_all/4345 https://digitalcommons.hamline.edu/hse_all/4345 https://www.asian-efl-journal.com/