ENGLISH REVIEW: Journal of English Education p-ISSN 2301-7554, e-ISSN 2541-3643 Volume 9, Issue 1, December 2020 https://journal.uniku.ac.id/index.php/ERJEE 145 DEVELOPING QUESTIONS OF HIGH ORDER THINKING SKILL USING ADVENTURE STORY NARRATIVE TEXT FOR JUNIOR HIGH SCHOOL STUDENTS IN PALEMBANG Octa Risfa Ria Ringkisa English Language Education, Faculty of Teacher Training and Education, Sriwijaya University, Indonesia Email: octarisfa95@gmail.com Machdalena Vianty (Corresponding author) English Language Education, Faculty of Teacher Training and Education, Sriwijaya University, Indonesia Email: vianty.unsri@gmail.com Bambang A. Loeneto English Language Education, Faculty of Teacher Training and Education, Sriwijaya University, Indonesia Email: loenetobambang@gmail.com APA Citation: Ringkisa, O. R. R., Vianty, M., & Loento, B. A. (2020). Developing questions of high order thinking skill using adventure story narrative text for junior high school students in Palembang. English Review: Journal of English Education,9(1), 145-156. https://doi.org/10.25134/erjee.v9i1.3787 Received: 29-06-2020 Accepted: 27-09-2020 Published:11-12-2020 INTRODUCTION Education plays an important role in the 4.0 industrial revolution. It helps students faced the challenges for the industry 4.0 workforce that requires such skills as critical thinking, creativity, and problem solving. These skills are crucial for students’ success in this globalized world. The 21st century learning skill has an important role in preparing students to face the 4.0 industrial revolution era since 21st century learning emphasizes on students’ ability of critical thinking, connecting their knowledge to real world, using Abstract: The aim of this study was to develop questions of high order thinking skill using adventure story narrative text based on Bloom’s Taxonomy framework. The research method applied was the developmental research developed by Akker (1999). The procedure of the study consisted of three stages, analysis, design, and evaluation. Formative evaluation by Tessmer was conducted to see the validity, practicality, and whether or not the developed product had potential effect to the students. The subject of this study was eighth grade students at SMP N 17 Palembang. The sample of the study was selected purposively based on the category of their English proficiency i.e: low, medium, and high English proficiency. Document related to Higher Order Thinking questions, students’ need, and students’ reading level were analyzed. The results of the analysis became the basis for designing the developed questions. The prototype I, the developed question as the beginning design was validated by 2 experts (expert review) and stated that the prototype 1 consists of 60 questions are valid with average score 4.3, and appropriate to be used. In one-to-one stage, prototype II was tested on 3 students, there were 10 questions that needed to be revised because the students had difficulty understanding what was being asked by the questions. The result of revised questions was called Prototype II. In a small group stage, Prototype II was tested on 6 students, there were 11 questions need to revised, the revised question then produced prototype III. Field test stage aimed to evaluate potential effect of the developed questions. The judgment of potential effect was seen from the mean score of the students in the field test. The results showed that students’ average score was 73. It means that the developed questions of high order thinking skill was categorized as very had high potential effect to the students. Keywords: Developmental research; developing questions; higher order thinking skill; higher-order questions; adventure story; narrative text https://doi.org/10.25134/erjee.v9i1.3787 Octa Risfa Ria Ringkisa, Machdalena Vianty, & Bambang A. Loeneto Developing questions of high order thinking skill using adventure story narrative text for junior high school students in Palembang 146 technology of information and collaborating. Partnership for 21st Century Skills (2009) proposed the main subject of 21st century learning, they are: information media and technology skills, learning and innovation skill, life and career skills. In addition, in the 21st century, people are required to have the following qualities: (1) willingness to ask question (2) strategic thinking skill (3) logical reasoning (4) ability to make inferences (5) ability to problem-solve (6) innovation and creativity (7) emotional intelligence and effective communication skills (Conklin, 2011). Therefore, it is important for students to have those qualities in order to succeed in both work and life in this 21st century. Schools are expected to be the party that can provide students with the required skills of 21st century learning. Curriculum is one element that provides a significant contribution to realize the development process of the potential quality of students. The education system requires a curriculum that has a large impact on the quality and competitiveness of the future workforce. In relation to this, on July 15th 2013, Indonesian government enacted the 2013 curriculum although the KTSP curriculum is still applied. It is one of the Indonesian government's efforts to reform the quality of the education system. The 2013 curriculum is expected to help developing Indonesian students' ability to think critically, rationally and to solve problem. The demand for the 2013 curriculum is to make students more critical and creative. It is in line with the framework of Partnership for 21st Century Skills that identifies the competencies needed in the 21st century namely "the Four Cs". (Communication, Collaboration, Critical Thinking, and Creativity). Those are the essential competencies of a 21st century students, and students nowadays should master those competencies, they have to think critically and creatively. According to Conklin (2011), think critically and creatively are the characteristic of higher order thinking skill (HOTS). HOTS is a concept of education reform based on learning taxonomies (Bloom’s taxonomy). Bloom’s taxonomy is a concept in education that deals with classifying educational objectives in order to promote Higher Order Thinking Skills or HOTS and progress from Lower Order Thinking Skills or LOTS (Llewellyn, 2013). Bloom Taxonomy system is divided into six levels or categories, which are divided into two main areas: LOTS and HOTS. HOTS makes up knowledge and comprehension, while LOTS makes up application, analysis, synthesis, and evaluation. However, HOTS is very challenging for Indonesian students. The result of PISA (Programme for International Student Assessment) which measure students’ performance in Mathematics, Reading, and Science literacies show that Indonesian student performance in Reading literacies from 2000 to 2018 the result is far from satisfactory which is statistically significantly below the OECD average. Indonesian students' reading literacy skills was ranked 71 out of 78 PISA-participating countries, and ironically the results of PISA 2018 show that the average reading score of Indonesian students' reading literacy skill decreased from 397 to 371 (PISA, 2018). The PISA results of Indonesian Students’ suggest that there is a serious problem in the quality of education services in Indonesia, particularly in Reading Literacy. There are several factors that contributed to the low score of Indonesia students’ reading literacy. According to OECD (2003), Indonesian Students only able to recall facts, and have a low ability when find a contextual problem that require problem-solving abilities. Indonesian students are two levels behind the OECD country average in terms of reading literacies, and most of Indonesian students can do the simplest reading problem solving, but fail to solve more complex problems, particularly those that require a higher level of reasoning as well using and integrating more concepts in dealing with a phenomenon (OECD, 2012). According to Carver and Orth (2017), questioning is a powerful tool for helping students to better understand concepts while setting high expectations and promoting critical and creative thinking. In addition, Jo and Bednarz (2011), believe that learners should be asked high order thinking skill questions to develop his or her thinking skills. The whole point of asking high order questions is to stimulate students’ thinking, encourage them to reason through their answers and to engage them in enquiry (Butt, 2010). Hence, question is one of the best ways in training Higher Order Thinking Skill. ENGLISH REVIEW: Journal of English Education p-ISSN 2301-7554, e-ISSN 2541-3643 Volume 9, Issue 1, December 2020 https://journal.uniku.ac.id/index.php/ERJEE 147 The study conducted by Purwani, Rochsantiningsih and Kristina (2017) who analyze the content of Bright 1: A 2013 curriculum-based textbook for VII grade students found that most of the questions in the textbook for Junior High School focus on LOTS questions than HOTS questions. It can be implied that students are poorly exposed in working on HOTS questions due to lack of the provision of HOTS questions in the textbook. This is in line with Ilma (2018) who investigated higher order thinking skill questions in reading exercises of Bright Course Book for the Seventh Grade of Junior High School found that reading Exercises in Bright Course Book for Junior High School are focused on LOTS questions than HOTS questions. There are 70 reading questions from 33 reading exercises. From those reading exercises, there are 11 reading questions that included in HOTS meanwhile 59 reading questions included in LOTS. In addition, Mahzum (2018) who analyzed Higher Order Thinking-Based Objective Test at State Junior High School 5 Banda Aceh. He found that only 14 out of 25 question items which fulfilled the criteria of the development of HOTS and 11 question items did not meet the criteria of HOTS development. In can be concluded that the number of HOTS question were not enough to improve their HOT skill, whereas HOTS is important to be mastered by the students in facing the challenge of 21st century skill. In relation to the explanation above, the aim of this study was to develop questions of High Order Thinking skill using adventure story narrative text for Junior High School students in Palembang. The objectives of this study were to find out whether or not the developed develop questions of High Order Thinking skill using adventure story narrative text for Junior High School and practical, and whether or not the developed develop questions of High Order Thinking skill using adventure story narrative text for Junior High School had potential effect. METHOD The research method that applied in this research was the developmental research method that was developed by Akker (1999) since the aim was to develop questions of High Order Thinking skill using adventure story narrative text for Junior High School students in Palembang. The population of this study was the eighth-grade students in SMP N 17 Palembang. The sample of the study was selected purposively based on the category of their English proficiency i.e: low, medium, and high English proficiency. In one to one evaluation, there were three students chosen, each student has low, medium, and high English proficiency. In small group evaluation, there were six students chosen, two students for each category, i.e: low, medium, and high English Proficiency. In the field test, there were thirty students chosen and students in the one-to-one and small group evaluation were excluded. This research development was carried out in three stages: analysis, design and evaluation stages (Akker, 1999). The first stage was analysis, which was the process of identifying the needs and goals of a system and determining the process among them. There were three analyses were conducted. The first was document analysis. In this stage, the writer analyzed 2013 curriculum and syllabus. The developed questions were analyzed in terms of basic competence, learning indicators, and learning objectives which related to the developed questions. After that, students’ need analysis, this analysis was conducted to find out the information about students need analysis, the writer gave the questionnaire adapted from Tessmer (1993) to the students dealing with students’ opinion towards reading, students’ knowledge about Higher Order Thinking Skill, students’ opinion about the importance of Higher Order Thinking Skill, and students’ opinion about Higher Order Thinking Skill question. The information from questionnaire was useful for the writer in designing the product. The last, students reading level analysis was applied to identify the appropriateness between students’ reading ability with the readability of the text that used in developing the questions. In this study, Jennings Informal Reading Assessment developed by Dr. Joyce used to measure students’ reading level. The next stage was design. Design was the process by which objectives, strategies, techniques, and media for achieving the instructional goals are determined and specified. In this stage, the writer designed the questions of High Order Thinking skill using adventure story narrative text, there were 5 adventure stories Octa Risfa Ria Ringkisa, Machdalena Vianty, & Bambang A. Loeneto Developing questions of high order thinking skill using adventure story narrative text for junior high school students in Palembang 148 narrative texts with different title used. The adventure story narrative text used in developing the questions were the adventure of Tom Sawyer, Peter Pan, Call of the Wild, Journery to the Center of the Earth, and King Solomon’s Mines. In determining the readability levels of text used in developing the product, which were matched students’ reading proficiency, those adventure story narrative text was analyzed by using Flesh Kincaid. In designing the questions, the writer used the framework of Bloom Taxonomy. After that, the writer produced the product which is called prototype I. The product was constructed in paper- based design. The product covered the skill of Higher Order Thinking (analyze, evaluate, and create). The last was the evaluation, at this evaluation stage, the formative evaluation developed by Tessmer (1993) was used. Formative evaluation consists of self-evaluation, expert review, one to one, small group, and field test. The first step was self-evaluation. In the self-evaluation stage, the writer checked the product in terms of language, content, and instructional design before the product was validated by the experts in the expert review stage. Next, in expert review stage, the experts assessed, evaluated, and validated questions that have been designed by the writer so that the strengths and weaknesses of the questions itself can be identified. The experts were asked for giving suggestions and comments on the validity sheet as the material to revise prototype 1 and stated that prototype 1 was valid or not valid. Revision from experts was used as the material to test one to one. This stage aimed to test the validity of the questions being developed. The third stage was one-to-one. At this stage, the writer tested prototype 1 to three students from the eighth-grade students as a tester. The comments of the students were used to revise the question items. One-to-one evaluation was applied to know the students’ ability in understanding the language used in HOTS questions and to know the clearness of the developed questions, the weaknesses of the items then were revised. Next step was small group, the result of the revised decision on the prototype I has resulted in prototype II. Then, prototype II has been tested to 6 students from the eighth-grade students. At this stage, 6 students were asked for answering the questions. Students were asked for comments and suggestions about the questions they have worked on. The suggestions and comments from the students then become the background to revise prototype II. The last stage of the evaluation was the field test. This stage was aimed at finding out whether or not develop questions of high order thinking skill using adventure story narrative text had potential effect to the students. In collecting the data, there were three instruments used in developing higher order thinking skill questions. Those were questionnaire, expert validation, and interview. In analyzing the data, the writer analyzed the validity, practicality, and whether or not developed questions of high order thinking skill using adventure story narrative text had potential effect. The validity of the product was validated based on experts' evaluation. After applying the developed questions of high order thinking skill using adventure story narrative text, the students who included in one-to-one and small group evaluation gave their comments on the questionnaire related to the develop product. After that, to determine the practicality of the product, the data from the questionnaire were administered. In one-to-one and small group evaluation: Google Classroom, Google Form and WhatsApp Video Call were used. In this stage, the students were given HOTS questions (Prototype 1) that were developed, the writer has previously uploaded the developed questions to Google Classroom. Then students were invited to take part in Google Classroom and then students used, reviewed, evaluated and answer the developed questions. After the students use, review and evaluate and answer the developed questions, each student was given a questionnaire that has been translated into Bahasa Indonesia. It was intended to determine the practicality of the developed questions, the questionnaire was given to students in the form of a google form. Each student was also interviewed by using a list of guiding questions to obtain information about their comments on the developed questions. The interview was done by using WhatsApp Video Call. The revised questions from students' comments and suggestions in one-to-one evaluation then became prototype II, and the revised questions from students' comments and ENGLISH REVIEW: Journal of English Education p-ISSN 2301-7554, e-ISSN 2541-3643 Volume 9, Issue 1, December 2020 https://journal.uniku.ac.id/index.php/ERJEE 149 suggestion in small group evaluation then became prototype III. The last was field test. In the field test, the HOTS questions developed (prototype III) which had been declared valid and practical were used, then prototype 3 was tested to the 8th grade students of SMP N 17 Palembang, the field test involved the eighth-grade students consisting 30 students having high, medium and low English proficiency. The students participating in one-to- one and small group evaluation were not involved anymore in this field. the writer used Google Form as a media used for the students in answering the questions. Field test was intended to know the potential effect of the developed Higher Order Thinking Skill questions using adventure story narrative text. The results of the student answer sheets in the field test stage were used to determine what potential effects arose from the questions developed by the writer. To find out whether or not developed Higher Order Thinking Skill questions using adventure story narrative text had potential effect, the resulting score of students were calculated to find out the percentage of the students who passed the score criteria which is 75. The test can be considered has potential effect if 70% or more students reach minimum criteria which is 75 in answering the developed product. The percentage were calculated as follows: Mastery percentage (%) = RESULTS AND DISCUSSION Analysis Stage In this stage, there were three main activities conducted to get information about the eighth graders students which was document analysis, students’ need analysis and students’ reading level analysis. In the document analysis, the writer analyzed the questions related to Higher Order Thinking Skill (HOTS) available in the students’ textbook. From the textbook, the writer found that the number of HOTS questions available in the students’ textbook were still insufficient. Beside analyzing HOTS questions in the textbook, the writer also analyzed several documents, the documents included are 2013 curriculum and syllabus. The developed questions were analyzed in terms of basic competence, learning indicators, and learning objectives which related to the developed questions by checking the syllabus of the eighth graders students. From the 2013 curriculum analysis, it is then associated with indicators of HOTS questions as stated in the Revised Bloom Taxonomy, while the results of the 2013 curriculum analysis obtained Basic competence in accordance with the HOTS category. The indicators and learning objectives of developed questions were derived from basic competence. The core competence, basic competence, indicators, and learning objectives were described as follows: Core competence Core Competence 3. To understand and to apply the knowledge (fact, concept, and procedures) on the basis of student’s curiosity about the knowledge, technology, art, and culture related to observable phenomena and events. Core Competence 4. To process, present, and associate in concrete domain (applying, explaining, composing, modifying, and creating) and abstract domain (writing, reading, counting, drawing and composing) suitable with school’s materials and other sources which have the same views or theories. Basic Competence To understand social function, text structure, and language feature of narrative text in the form of fable, in accordance with its contextual use. To understand social function, text structure, and language feature of narrative text in the form of Folklore , in accordance with its contextual use. Indicators 1. Making the right conclusion based on the text 2. Giving a judgement about a certain situation in the text 3. Assessing the event that was experienced by the characters in the text Learning Objectives Based on the learning indicators, the writer formulated the learning objectives as follows: by using developed of HOTS question adventure story narrative text, the students are able to (1) give a Octa Risfa Ria Ringkisa, Machdalena Vianty, & Bambang A. Loeneto Developing questions of high order thinking skill using adventure story narrative text for junior high school students in Palembang 150 judgement about the situation and conditions in the reading text (2) make the right conclusion (3) assess the event that occurred with the characters in the reading text. In this study, the HOTS questions asked some aspects of reading such as: main idea, details, cause and effect, inference, vocabulary and sequence questions. Next, students’ need analysis was conducted to obtain information about the students’ needs in terms of Higher Order Thinking Skill Questions. Aspects which were included into consideration to obtain information about students’ need were (1) Students opinion towards reading, (2) Students knowledge about Higher Order Thinking Skill, (3) Students opinion about the importance of Higher Order Thinking Skill, and (4) Students opinion about Higher Order Thinking Skill question. To get the information about students’ opinion about reading, students’ knowledge about Higher Order Thinking Skill, Students opinion about the importance of Higher Order Thinking Skill, and Students opinion about Higher Order Thinking Skill question, a questionnaire consisting of 14 questions was administered to 51 eighth grade students consisting of 6-7 representative students from each class (from class VIII-1 to class VIII-8). The results are as follows: Table 1. The summary of students need analysis Aspects of Questionnaire Percentage Student’ ability in learning English was good 72% The difficulty level of English reading text in the English textbooks used in schools is easy for the students 79% Students often answer questions on the reading text 45% The questions given by the English teacher are usually essay questions 63% The students difficulties when answering the questions are concluding the cause or effect 25% The students who know about HOTS 29% English teachers sometimes give reading questions in the form of High Order Thinking Questions (HOTS) 51% Inserting Higher Order Thinking skill questions into the question 69% in the English reading text was important Including questions that hone the ability to analyze, evaluate and create into the questions in the English reading text was important 59% Having the ability to analyze, evaluate and create was important Solving Higher Order Thinking questions often can help students improve their Higher Order Thinking skill. 72% Higher Order Thinking questions was important in improving their Higher Order Thinking skill 69% Based on the results of students needs analysis above students realized that higher order thinking skill was important, but their English teachers sometimes give reading questions in the form of High Order Thinking Questions (HOTS). Furthermore, the students were agree that solving Higher Order Thinking questions often can help them improve their Higher Order Thinking skill and Higher Order Thinking questions was important in improving their Higher Order Thinking skill, It can be concluded that developing HOTS questions were needed to be developed. Students’ reading level analysis was conducted to determine the subject of the study and in matching the students reading ability with the readability text used in developing higher order thinking questions. With hopes of the readability levels of the text used in developing questions were appropriate to students' reading level. In this analysis, the writer used Jennings Informal Reading Assessment developed by Dr. Joyce. The level of reading text in the test included level 1, 2, 3, 4, and 5. The reason of giving these reading levels was the eighth-grade students' reading achievement was at low level and based on the previous study done by . . . (..) that reading level of the eighth-grade students was in level 2. The results of students’ reading analysis were described as follows: ENGLISH REVIEW: Journal of English Education p-ISSN 2301-7554, e-ISSN 2541-3643 Volume 9, Issue 1, December 2020 https://journal.uniku.ac.id/index.php/ERJEE 151 Table 2. The distribution of students’ reading level Text Level Reading Stage Frustrational (Correct number ≤ 4) Instructional (Correct number ≤ 5-6) Independent (Correct number ≤ 7-8) NOS* %* NOS* %* NOS* %* Level 1 33 64.7 % 16 31. 3 % 2 3. 9 % Level 2 17 33.3 % 22 43.1 % 12 23. 5 % Level 3 24 47. 05 % 12 23. 5 % 15 29. 4 % Text Level Reading Stage Frustrational (Correct number ≤ 4) Instructional (Correct number ≤ 5-6) Independent (Correct number ≤ 7-8) NOS* %* NOS* %* NOS* %* Level 4 44 86. 2 % 6 11. 7 % 1 1.9 % Level 5 49 96. 07 % 1 1.9 % 0 0 *Nos : Number of Student *% : Percentage of Student From the distribution of the reading level above, it can be seen that students’ reading level were at the frustational (64.7%), (47.05%), (86.2%), (96.07%) for text level 1, 3, 4 and 5. While for text level in 2, students were at the instructional stage (43.1%). Therefore, it can be concluded that students reading level was at level 2 since the highest percentage of the instructional stage was in level 2. Design stage In this stage, the writer designed developed questions of Higher Order Thinking Skill questions. First of all, the writer was collecting adventure story narrative text from the internet. These adventure stories were then adapted. To know the readability of the text used in developing the questions, these adventure stories were then calculated automatically by using online tool named Flesch Kincaid (i.e.: https://www.webfx.com/tools/read-able/). There were 5 texts adventure stories. The illustrated of the results were described as follow: Table 3. The readability level of the text used in developing higher order thinking skill questions Reading Text for developing HOTS questions NO Tittle Readability Level Remarks 1. The adventure of Tom Sawyer 1 Adapated 2. Peter Pan 2 3. Call of the Wild 3 4. Journey to the Center of the Earth 4 5. King Solomo’s Mines 5 Evaluation In this stage, the evaluation and revision were conducted together because those two stages were related to each other. Before being processed the next evaluation, the developed Higher Order Thinking Skill questions might be revised. In evaluation stage, there were five stages of formative evaluation proposed by Tessmer (1993). It consisted of experts' review, and one-to-one evaluation in prototype 1, small group evaluation in prototype 2, and the field test was conducted in prototype 3. The evaluation was needed to improve the quality of the developed products. Thus, to know Octa Risfa Ria Ringkisa, Machdalena Vianty, & Bambang A. Loeneto Developing questions of high order thinking skill using adventure story narrative text for junior high school students in Palembang 152 the validity of the product, firstly it was evaluated by two experts in experts’ review stage. After that, the practicality of the product was determined based on the students’ comments in one-to-one and small group evaluation stage. Afterward, to see whether or not the developed Higher Order Thinking Questions had potential effect there was calculating the percentage of the students who passed the minimum score criteria (KKM). Self-evaluation In the self-evaluation stage, the writer checked the developed questions before the developed questions were validated by the experts in the expert review stage. It was considered that there were ungrammatical sentences and inappropriate of word used were found in the developed questions that should be revised by the writer. Self- evaluation conducted to find out the weaknesses of the developed questions so that the writer could revise the developed questions to be valid, practical, and has potential effect. Expert review The next stage of evaluating the developed product was experts' review. There were two experts in this stage. First was the expert of instructional design, and the expert of language and content. The experts had evaluated the products and given their comments and suggestions on the questionnaire. All of the experts said that the products were valid with revision. It means that the writer should revise the products which related to the content, instructional design, and language. The summary of the results of questionnaires is shown in the table below. Table 4. Summary of Expert Review Aspects Average Score Remarks Language and Content 4.7 Very Highly Valid Instructional Design 4 Highly Valid Total Average Score 4.3 Very Highly Valid Moreover, based on the expert's response on the questionnaire, the data indicate that the average score (4.3) of two aspects of instructional design, language and content which were categorized as very highly valid. One-to-one evaluation One-to-one Evaluation was conducted to find out the practicality level of developed Higher Order Thinking Skill (HOTS) questions using adventure story narrative text. In this stage, there were three students representing frustration, instructional, and independent students’ English proficiency. Students were asked to read and review the developed HOTS questions. The media used by the students in reading, reviewing the developed product were Google classroom. A questionnaire was administered to students aiming to evaluate the practicality of product after they have read and review the developed questions. The questionnaire was given in the form of google form. The average score of questionnaires were calculated the practicality was interpreted based on the average score calculated from the questionnaire. The summary of the results of questionnaires is shown in the table below: Table 5. Summary of practicality in one-to-one stage Aspects Average Score Remarks The clarity of questions developed 3.11 High Practicality The clarity of the instructions 3.33 Very High Practicality The effectiveness and efficiency of the questions developed 3.39 Very High Practicality The display quality of the developed questions 3.5 Very High Practicality Total Average Score 3.33 Very High Practicality The next activity was a discussion session. Students were asked their opinion of the developed questions. The discussion was conducted by using WhatsApp video call. All of them marked some ENGLISH REVIEW: Journal of English Education p-ISSN 2301-7554, e-ISSN 2541-3643 Volume 9, Issue 1, December 2020 https://journal.uniku.ac.id/index.php/ERJEE 153 difficult words in questions. All of the students had more than 3 difficult words in the questions. In addition, the students who categorized as independent student have already understood the purpose of the question and what being asked. Meanwhile, frustration and instructional students have difficulty understanding the meaning of the questions because they did not know the meaning of the word and they said that the words were new for them. However, the students mentioned that the developed questions were quite difficult but they can handle it. In this stage there were 10 questions that need to be revised, the writer only changed the vocabulary which the students did not familiar with the word used in developing questions. Referring to the average score of the total scores above, it means that the product had very high practicality level. However, the writer still needs to revise some of the questions based on students’ comments and suggestions so that the writer could continue the next evaluation stage. Small group evaluation The practicality of developed product was also assessed in small group evaluation stage. The developed questions in this stage called prototype 2 which was the revision of the product in one-to- one stage. In this stage involved 6 students in which every two students categorized to frustration, instructional, and independent student. The students were not the same as those who were in one-to-one evaluation. Small group evaluation was conducted in one meeting. Allocation time was 120 minutes. Students asked to read, review and answer the prototype 2, after that, they filled the questionnaires. The questionnaires were given to the students as the basis of practicality evaluation. In the small group evaluation, the media used were the same as in the one-to-one evaluation, Google Classroom was also used in this stage, students read and reviewed the developed questions by using Google Classroom, while Google Forms was used by the students in answering questions and filled the questionnaire. In this stage, the writer also interviewed students to find out students’ feedback, comments, suggestions and to find out student difficulties. After the writer asked the students in the small group to work on the developed questions, the writer ask students’ comment by using WhatApp video call. Students' comments were obtained regarding the questions they were working on and there were several questions that students were still a little confused about, some students mentioned that there were still some complicated questions and they did not understand what is being asked by the question. From the of students' comments in the small group, it was concluded that there were still a few students who have difficulty understanding the purpose of the questions and what being asked is, because they did not know the meaning of the word and they said that the words were new for them. In this stage, there were 11 questions that need to be revised, the writer only changed the vocabulary which the students did not familiar with the word used in developing questions. After that the writer revised the questions, the writer asked the students again to read, review and evaluated the developed questions and all of the students mention that students have already understood the purpose of the questions and what being asked was. So that the writer did not need to revised the question anymore, the revised question in this stage then become Prototype III. The calculation of practicality in small group evaluation is shown in the table below: Table 6. Summary of practicality in small group evaluation Aspects Average Score Remarks The clarity of questions developed 3. 21 High Practicality The clarity of the instructions 3.33 Very High Practicality The effectiveness and efficiency of the questions developed 3.45 Very High Practicality The display quality of the developed questions 3.24 High Practicality Total Average Score 3.30 Very High Practicality The results of questionnaires showed that practicality of the developed questions was very high. The first aspect of practicality, The clarity of questions developed, had been evaluated as high. It Octa Risfa Ria Ringkisa, Machdalena Vianty, & Bambang A. Loeneto Developing questions of high order thinking skill using adventure story narrative text for junior high school students in Palembang 154 can be said that the developed questions was clear, the sentences contained in the developed higher order thinking skills questions (HOTS) were clear and the vocabulary used was easy to understand. The second one was the clarity of the instructions was assumed very highly practical. It was affirmed that the instructions were easy to be understood and the students can carry out any instructions contained in the developed of higher order thinking skills. The third, the effectiveness and efficiency of the questions developed was assumed very high practicality. It was affirmed that the grammar used in the developed of higher order thinking skills (HOTS) was clear, the developed high-order thinking skills (HOTS) questions was easy to apply and use in the learning process and students can use it flexibly anywhere. Furthermore, the developed higher order thinking skills (HOTS) questions can improve students' higher order thinking skill and help them practice higher order thinking skill. The last one was the display quality of the developed questions. This aspect was evaluated high practical. The display type and font size in the developed of high-order thinking skills (HOTS) has a good quality and the images used in the developed of high-order thinking skills (HOTS) were interesting for the students. Furthermore, all aspects of practicality have been examined, and the results showed that the developed product was very highly practical. In discussion session, students mentioned that the developed HOTS questions were quite difficult for them however, they still can work on the questions. In conclusion, the developed questions were ready to be evaluated whether or not the developed product has the potential effect on the students in field test without revision anymore. Field test The last stage of the evaluation was field test. The HOTS questions developed (prototype 3) which had been declared valid and practical were used, then prototype 3 was tested to the 8th grade students of SMP N 17 Palembang, the field test involved the eighth grade students consisting 30 students consisting of 3/4 representative students from each class (from class VIII-1 to class VIII-8) having high, medium and low English proficiency. The students participating in one-to-one and small group evaluation were not involved anymore in this field. The field test in this study hold in 12nd August 2020, with time allocation 120 minutes and in this stage, the writer used Google Form as a media used for the students in answering the questions. Field test was intended to know the potential effect of the developed Higher Order Thinking Skill questions using adventure story narrative text. The results showed that all of the students passed the minimum score which was 70. With the average score of the English version was 78 and the average score of Bahasa Indonesia version was 83. It means that based on both average scores, PISA-based reading materials in Indonesian context in the form of continuous text was categorized as very had high potential effect on the students. RESULTS AND DISCUSSION The developed product was High Order Thinking Skill questions using adventure story narrative text for Junior High School students in Palembang. The research method that applied in this research was the developmental research method that was developed by Akker (1999). This research development was carried out in three stages: analysis, design and evaluation stages (Akker, 1999). In this study, formative evaluation proposed by Tessmer (1993) was applied. Formative evaluation was used to improve the quality of the intervention. There are three main criteria for quality of the intervention, i.e.: validity, practicality, and potential effect. Before the questions were developed, the writer did some analysis. This analysis was aimed to identify the student needs in developing the questions. In analysis procedure, there were four analyses conducted, they were document analysis, students’ need analysis, students’ reading level and readability level. The first analysis was document analysis, in document analysis the writer analyzed the questions related to Higher Order Thinking Skill (HOTS) available in the students’ textbook. From the textbook, the writer found that the number of HOTS questions available in the students’ textbook were still insufficient where as, Carver and Orth (2017) state that questioning is a ENGLISH REVIEW: Journal of English Education p-ISSN 2301-7554, e-ISSN 2541-3643 Volume 9, Issue 1, December 2020 https://journal.uniku.ac.id/index.php/ERJEE 155 powerful tool for helping students to better understand concepts while setting high expectations and promoting critical and creative thinking. It means that the number of the questions available in the textbook were not enough to improve students’ HOTS, beside analyzing HOTS questions in the textbook, the writer also analyzed several documents, the documents included are 2013 curriculum and syllabus. The developed questions were analyzed in terms of basic competence, learning indicators, and learning objectives which related to the developed questions by checking the syllabus of eighth grade students. Next, there were some aspects adapted from Tessmer (1993) which were taken into consideration to obtain information about students’ need. They were (1) Students opinion towards reading, (2) Students knowledge about Higher Order Thinking Skill, (3) Students opinion about the importance of Higher Order Thinking Skill, and (4) Students opinion about Higher Order Thinking Skill question. In obtaining the information, a questionnaire consisting of 14 questions was administered to 51 eighth grade students consisting of 6-7 representative students from each class. From the results of the needs analysis the developed Higher Order Thinking Questions was needed to be developed since the students were agree that solving Higher Order Thinking questions often can help them improve their Higher Order Thinking skill. As for the student reading level analysis, Jennings Informal Reading Assessment developed by Dr. Joyce, was given to the students which included reading texts at level 1, 2, 3, 4, and 5. From the distribution of the reading level on table 2. It can be concluded that students reading level was at level 2 since the highest percentage of the instructional stage was in level 2. Next, readability level. The adventure stories narrative text that used in developing the questions were calculated automatically by using online tool named Flesch Kincaid (i.e.: https://www.webfx.com/tools/read-able/) to determine the readability levels. There were five adventure story narrative text from different title that being used in developing Higher Order Thinking questions, since the reading level of the students was level 2. So, in developing higher order thinking skill questions the writer used the reading text for level 1,2,3,4, and 5. The next procedure was designing the developed product. At the question design stage, the writer designed the questions according to the Higher Order Thinking Skill (HOTS) indicator. The writer developed HOTS questions based on (HOTS) module guidelines of the Ministry of Education and Culture in 2017. In developing the question, the writer analyzed the basic competencies that HOTS questions can be developed. Then the writer chose an interesting stimulus, the writer used adventure story narrative text as an interesting stimulus. Then, the writer arranged the HOTS questions card based on HOTS question indicators level that correspond to C4 (analyzing), C5 (evaluating), C6 (creating). The last, the writer wrote the questions in accordance with HOTS question grids. The questions developed were 60 questions. Last stage, formative evaluation by Tessmer (1993) consisting of self-evaluation, one-to-one evaluation, small group evaluation, and field test was applied. After the writer has finished developing the questions, the next stage was self-evaluation, In self- evaluation, the writer evaluated the developed Higher Order Thinking Skill questions using adventure story narrative text. From this self- evaluation, some misspellings and ungrammatical sentences. When the writer finished evaluating the developed questions herself, the writer gave the questions developed to the expert to be evaluated its validity in terms of its instructional design, language and content. And the result of validity of the developed questions in experts review stage were very high practical. Besides, developed questions was also very high practical after being evaluated in one-to-one and small group evaluation. In addition, there were 10 questions and 11 questions were revised by the writer, there were still a few students who have difficulty understanding the purpose of the questions and what being asked was because they did not know the meaning of the word and they said that the words were new for them. After the writer revised the questions the writer continued to the next stage. The last evaluation was field test. A field test was conducted in order to find out whether or not the developed questions of HOTS questions had https://www.webfx.com/tools/read-able/ Octa Risfa Ria Ringkisa, Machdalena Vianty, & Bambang A. Loeneto Developing questions of high order thinking skill using adventure story narrative text for junior high school students in Palembang 156 the potential effect to the students. In the field test, the students were answering the questions with the length of time was one week and the students do it after the students finished their online class. The results showed that all of the students passed the minimum score which is 77 It means that based on the average scores, the developed HOTS questions by using adventure story narrative text was categorized as had high potential effect on the students. It was caused by the student passed the minimum mastery criterion (KKM) which is 75. CONCLUSIONS Results of study that have been discussed previously can be concluded as: the developed of higher order thinking skill questions was validated for its instructional design, language and content in which the validity for each was 4 and 4.7. The average score of validity was 4.3 implying the product was highly valid and the developed questions of higher order thinking skill questions was evaluated in one-to-one and small group evaluation for its practicality. In one-to-one evaluation, the practicality was 3.30 indicating very highly practical. The practicality in small group was 3,33 it also indicating very highly practical. The judgment of potential effect was seen from the mean score of the students in the field test. The resulting score of students were calculated to find out the percentage of the students who passed the score criteria which is 70. The results showed that the average score was 73. It means that the developed questions of higher order thinking skill questions as very had high potential effect to the students. REFERENCES Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., et al. (2001). A taxonomy for learning, teaching, and assissing: A revision of bloom's taxonomy of educational objectives. New York: Longman. Akker, J. (1999). Principle and methods of development research. In J Akker, R. Branch, K. Gustalfon, T. Nieven N, & Plomp (Eds), Design methodology and development research. (pp. 1-14). London: Kluwer. Bloom, B. (1956). Taxonomy of educational objectives: The classification of educational goals. New York, United States: Longman, Green and Co ltd. Butt, G. (2010). Making assessment matter. London: Continuum International Publishing Group. Carver, L., & Orth, J. (2017). Coaching: making a difference for k-12 students and teachers. London: The Rowman & Littlefield Publishing Group, Inc. Conklin, W. (2011). Higher-order thinking skills to develop 21st century learners. Huntingon Beach, USA : Shell International Publishing, Inc. Conklin, J. (2005, Spring). Book Reviews: A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. Educational Horizons. The Quest for Strengths: The Dawn of a Talent-Based Approach to K-12 Education, 83, 154-159. Ilma, F. (2018). An analysis of reading exercises in bright course book for the seventh grade of junior high school of 2013 curriculum, (Unpublished Degree, English Language Education Department, Walisongo State Islamic University, Semarang) Krathwohl, D. R. (2002). A Revision of Bloom’s Taxonomy: An Overview. Theory into Practice, 41(4), 212-218. Llewellyn, D. (2013). Teaching high school science through inquiry and argumentation. (2nded.). Thousand Oaks, CA: SAGE Publications. Ministry of National Education. (2013). Permendikbud No.65/2013 tentang standar proses pendidikan dasar dan menengah. Retrieved from https://www.slideshare.net/alvinnoor/permendikb ud-nomor-65-tahun-2013-tentang-standar- proses-23256577 OECD (2015). Programme for International Student Assessment (PISA). Diakses dari http://www.oecd.org/pisa/PISA-2015- Indonesia.pdf Partnership for 21st Century Learning (2015). P21 Framework Definition. [Online]. Retrieved from: https://www.imls.gov/assets/1/AssetManager/Bis hop%20Pre-Con%202.pdf Purwani, A., Rochsantiningsih., D. Kristina, D. (2017). A content analysis of bright 1: A 2013 curriculum-based textbook for VII grade students. Journal of Social Science, Educationa and Humanities, 158, 88-100 Retrieved from https://www.researchgate.net/publication/321841 238_A_Content_Analysis_of_Bright_1_A_2013 _CurriculumBased_Textbook_for_VII_Grade_St udents Tessmer, M. (1993). Planning and conducting formative evaluation. Philadelphia, PA: Kogan Page https://www.slideshare.net/alvinnoor/permendikbud-nomor-65-tahun-2013- https://www.slideshare.net/alvinnoor/permendikbud-nomor-65-tahun-2013- https://www.slideshare.net/alvinnoor/permendikbud-nomor-65-tahun-2013-tentang-standar-proses-23256577 https://www.slideshare.net/alvinnoor/permendikbud-nomor-65-tahun-2013-tentang-standar-proses-23256577 https://www.imls.gov/assets/1/AssetManager/Bishop%20Pre-Con%202.pdf https://www.imls.gov/assets/1/AssetManager/Bishop%20Pre-Con%202.pdf https://www.researchgate.net/publication/321841238_A_Content_Analysis_of_Bri https://www.researchgate.net/publication/321841238_A_Content_Analysis_of_Bri https://www.researchgate.net/publication/321841238_A_Content_Analysis_of_Bright_1_A_2013_Curriculum-Based_Textbook_for_VII_Grade_Students https://www.researchgate.net/publication/321841238_A_Content_Analysis_of_Bright_1_A_2013_Curriculum-Based_Textbook_for_VII_Grade_Students https://www.researchgate.net/publication/321841238_A_Content_Analysis_of_Bright_1_A_2013_Curriculum-Based_Textbook_for_VII_Grade_Students