ENGLISH REVIEW: Journal of English Education p-ISSN 2301-7554, e-ISSN 2541-3643 Volume 10, Issue 2, June 2022 https://journal.uniku.ac.id/index.php/ERJEE 645 DEVELOPING HOTS-BASED DESCRIPTIVE READING MATERIALS IN INDONESIAN CONTEXT FOR TENTH GRADERS Irnansyah Universitas Sriwijaya, Indonesia Email: irnansyah250@gmail.com Soni Mirizon (Corresponding author) Universitas Sriwijaya, Indonesia Email: smirizon@unsri.ac.id Ismail Petrus Universitas Sriwijaya, Indonesia Email: ismailpetrus@yahoo.com APA Citation: Irnansyah., Mirizon, S., & Petrus, I. (2022). Developing HOTS-based descriptive reading materials in Indonesian context for tenth graders. English Review: Journal of English Education, 10(2), 645-654. https://doi.org/10.25134/erjee.v10i2.6301. Received: 02-02-2022 Accepted: 28-04-2022 Published: 30-06-2022 INTRODUCTION The rapid growth of science and technology and the accessibility of the global market in the 21st century require people to master some specific skills, such as critical thinking, to communicate and compete in international competition. Duncan (2009) believes the skills that progressively request creativity, persistence, and problem-solving combined with performance as a part of a team are called 21st- century skills. This is in line with the Indonesian Curriculum 2013, which requires students to be active, innovative, creative, and effective in learning. Besides, National Education Association (2012) states that in addition to strong content mastery, every student must own 21st century skill such as critical thinking. Critical thinking skill means the skill that requires one’s ability to judge or respond to a matter. Rivas, Saiz, and Ossa (2022) believe that critical thinking is effective to solve a problem or reach goal. Wahyuni, Miarsyah, & Adisyahputra (2018) in their study revealed that critical thinking skill gave positive correlation with senior high school students’ literacy skills. Critical thinking skill is strongly related to the three top levels of cognitive dimension of the revised Bloom's Taxonomy (Watson, 2019). The three top levels analyzing, evaluating and creating are commonly called Higher Order Thinking Skills (HOTS). Higher order thinking skill (HOTS) plays an important role in effectively applying, connecting, or manipulating prior knowledge. It is one of the important components for an individual to solve new problems in the 21st century (Brookhart, 2010). HOTS is also an instrument of learning assessment that help students to increase their higher thinking ability (Kusuma, Rosidin, Abdurrahman, & Suyatna, 2017). The application of HOTS can be handled through one of the language skills such as reading. Abstract: The objective of this study was to find out the validity, practicality, and potential effect of the developed HOTS-based descriptive texts in Indonesian context for senior high school students. In conducting this study Akker (1999) development concept consisting of analysis, design, evaluation, and revision was applied. The formative evaluation consisted of self-evaluation, expert review, one-to-one evaluation, small group evaluation, and field test adopted from Tessmer (1993) was used to see the validity, practicality, and the potential effect of the developed product. The result of the expert review showed that the average score of all aspects of the developed product was 3.71> 2.51, which means it was categorized as very high. Based on the practicality evaluation results in the one-to-one and small group evaluations, the product was highly practical since the average score in both phases was 2.9 and 3.07, respectively. The potential effect of the product was good enough (81%), although 4 out of 21 students in the field test could not reach the minimum mastery ≥70. Keywords: HOTS; material development; reading materials; descriptive text mailto:irnansyah250@gmail.com Irnansyah, Soni Mirizon, & Ismail Petrus Developing HOTS-based descriptive reading materials in Indonesian context for tenth graders 646 The students’ reading skill competence generally determines their reading literacy. In fact, the students’ reading literacy in Indonesia is still low. Based on the result of Programme for International Student Assessment (PISA) in 2018, the score of Indonesian students on reading literacy was 371, below the average 487 (Kemdikbud, 2019). To promote students’ reading literacy and improve their cognitive thinking skills, reading materials could be developed based on HOTS questions. Utari and Gustiningsih (2021) found in their study that HOTS questions could support students’ creative thinking. While Putra and Abdullah (2019) believe that the provision of HOTS-based items help student to have good thinking skills. Some studies discussed English textbooks for secondary schools concerning higher order thinking skills such as (Wu, 2018; San, 2019; Hambali, Mirizon, & Heryana, 2020; Febriyani, Yunita, & Damayanti, 2020). In general, their findings revealed that most of the English textbooks provided less HOTS-based questions. Some previous development studies such as (Margana & Widyantoro, 2017; Indriyana & Kuswandono, 2019; Mursyid & Kurniawati, 2019) also dealt with HOTS, but their reading text materials did not reflect Indonesian context. Two similar studies dealing with the developing descriptive reading materials were conducted by Azizah, Inderawati, and Vianty (2021); Razak, Hermandra, and Elmustian (2021). The first study selected local culture for the reading materials and the result revealed that the developed product was valid and practical and gave potential effect. Meanwhile, the second one developed descriptive reading texts based on constructivism approach. This later study concluded that the developed product gave good contribution to the development of education. The study present study concentrated on choice of reading materials about Indonesian descriptive texts accompanied with constructed–multiple- choice item questions based on HOTS. By having this kind of materials the students could motivate their reading interest and improve their critical thinking skill. Tomlinson (2012) defines reading materials as anything used in language learning, including textbooks, videos, graded readers, flashcards, games, websites, and mobile phone interactions. The development of reading materials was urgently required to fulfill the students’ needs in a given school. Therefore, the focus of this study was on the development of reading material particularly reading texts in descriptive genre. Checaria, Petrus and Inderawati (2021) state descriptive texts aim to give information by describing a particular person, place, thing or animal. In this kind of text, the students are required to list the characteristics of something and usually deal with the physical appearance of the described thing. In developing HOTS in a language classroom, teachers not only need to have subject matter of the knowledge but also to know what HOTS contains and how it can be included into the curriculum (Daud, 2017; Tyas, 2019; Hambali, et al., 2020). In the curriculum of 2013, Higher-Order Thinking Skills has become the demand to be developed or implemented in the teaching process. Since the implementation of HOTS in the teaching process can be done by teaching reading skills, the reading materials should be well designed to reach the demand of the 2013 curriculum. HOTS-based descriptive reading materials could be characterized by the level of low or academic word frequency in the text. Nation (2001) defines high-frequency words are words that cover a very large proportion of the running words in spoken and written texts and occur in all kinds of uses of the language and if the words occur very infrequently and cover only a small proportion of any text, they are called low-frequency words. Therefore, it is believed that the descriptive text in the Indonesian context containing HOTS also depends very much on word choice. Another characteristic of HOTS-based descriptive reading text can be determined by the availability of high dimensions of the thinking process in reading tasks. The reading task usually accompanies the reading text given to the students for reading comprehension. The development of descriptive text in the Indonesian context based on HOTS must include the task whose questions reflect the level of cognitive thinking. Ramadhianti (2017) in her study found that reading comprehension gave positive effect toward students’ cognitive competence. The HOTS questions are usually identified by the verb choice used as the stem in the question items. The above brief explanation discussed the importance of HOTS to be applied in descriptive ENGLISH REVIEW: Journal of English Education p-ISSN 2301-7554, e-ISSN 2541-3643 Volume 10, Issue 2, June 2022 https://journal.uniku.ac.id/index.php/ERJEE 647 reading texts. Therefore, this development study aimed to find whether the product was valid, practical, and effective which could be used for tenth graders to improve their reading skills with high cognitive thinking levels. METHOD This study employed development research. The development research label has been used to refer to various kinds of research approaches related to design and development work (Akker, 1999; Gall., Borg., & Gall, 2003. They assert that development research has two main objectives: (1) to develop a product and (2) to test the effectiveness of the product. There were 33 tenth-grade students of SMA Negeri 11 Palembang participating in this study. Ten reading texts in Indonesian context were designed to develop HOTS-Based questions of descriptive texts. The quality of the product was determined by three common criteria: validity, practicality, and effectivity (Akker, 1999). This study used formative evaluation as suggested by Tessmer (1993), which included three stages: one-to-one evaluation, small group evaluation, and field test. A preliminary phase was used to determine the place and subject of the study, examine some of the related literature, analyze students, explore reading texts, and analyze the curriculum. The next procedure was the prototyping stage which included self-evaluation, expert reviews, one-to-one, small group, and field tests, as shown in the following figure. Figure 1. Formative evaluation design In the self-evaluation stage, the reading materials used for reading assessment, whether there were errors or not were evaluated, such as in vocabulary, spelling, and grammar. These results were referred to as the first prototype to be validated by experts for analysis of content, construct, and language. The results of the expert review were used for product revisions. Simultaneously with the expert review, reading comprehension test consisting of 50 higher levels of cognitive items was conducted to test students of one-to-one group evaluation. The test included reading comprehension aspects, such as a main idea, specific information/detail, references, sequences, inference, cause-effect, and vocabulary. In this one-to-one evaluation stage, three students from three different reading levels (one from frustration, one from instructional, and one from independent) were asked to answer the test and complete the questionnaire about the product. The results from one-to-one were used to revise the product. After validation by an expert review and one-to-one, the first prototype was said to be valid if the score from the questionnaire is at least ≥ 2.51. From expert reviews and one-to-one results, the second prototype was produced. The small group stage was carried out after an expert review and one-to-one group evaluation with the results of the second prototype to determine the practicality of the prototype. There were 9 students with three different reading levels involved at this stage doing the same activity as the previous one of one-to-one group evaluation. The results of the small group trial were revised based on the results of the trials and students’ comments on the second prototype. The result was then called the third prototype. The field test stage was the third prototype field as the final stage in developing reading texts for HOTS-based questions. This stage involved as many as 21 students who aimed to see the potential effects of the developed reading texts by analyzing the answers to the questions presented at the end of the reading texts and from the contextual questions on the reading assessment materials. This development study involved four different phases: need analysis, design, evaluation, and revision. The need analysis was conducted to get some information needed to develop the product, and reading materials. The analysis covered instructional analysis; students’ need analysis, and students’ reading level analysis. The instructional analysis was based on the curriculum used in the school, especially English for the natural science study program, the syllabus, and the English coursebook used. Students’ need analysis was done to see what materials to be developed by getting information from the students and the English teachers through questionnaires and interviews. Irnansyah, Soni Mirizon, & Ismail Petrus Developing HOTS-based descriptive reading materials in Indonesian context for tenth graders 648 Meanwhile, students’ reading level analysis was measured to match the students’ functional reading level with the readability of the developed reading materials. This study used a standardized reading comprehension test of Jennings Informal Reading Assessment to measure the students’ reading levels (Jennings, Caldwell, & Lerner, 2006). Data collection of this study included documentation, interview, questionnaire, and test. Documentation data were analyzed descriptively to see whether the developed text was based on the 2013 Curriculum and HOTS or not. Data from interviews as the results of experts’ judgment and students’ perceptions were analyzed based on the 2013 Curriculum and HOTS. The interview was given to the students after they had answered the developed HOTS-Based descriptive reading materials. Five categories of the Likert scale in the questionnaire were scored from 5 for the most positive expression to 1 for the least positive one (Sugiyono, 2012). The data of the questionnaire were analyzed using descriptive statistical analysis (percentage to see the validity, practicality, and effectiveness of the product. The developed instructional reading materials' practicality was evaluated using a questionnaire about the students’ perspective toward the developed instructional reading materials in one-to- one and small group evaluations. The product was considered practical if the average score was ≥ 2.51 (Kubiszyn & Borich, 1993). The effectiveness of the potential effect of the developed instructional reading materials can be obtained from either item difficulty index or minimum mastery criterion (KKM) of 70. This study chose KKM to measure the potential effect of the study which includes the following category (SMA Negeri 11 Palembang). RESULTS AND DISCUSSION Analysis phase The need analysis was conducted in order to get some information needed to develop the product and reading materials. The analysis involved instructional analysis, students’ need analysis, and students’ reading level analysis. Instructional analysis In this phase, the reading texts to be developed were analyzed in terms of their readability level, standard competence, basic competence, learning indicators, and learning objectives, and the descriptive texts available in curriculum 2013. Students’ reading level analysis Levels 1, 2, 3, 4, 5, and 6 of reading texts from the Independent Reading Inventory assessment (Jennings, Caldwell, & Lerner, 2006) were given to the students for their reading levels. The results of their reading level were distributed in the following table. Table 1. Distribution of the students’ reading level Text Level Students’ Reading Level Frustational Instructional Independen t (CN=≤6) (CN=7-8) (CN=9-10) NS* % NS* % NS* % Level 1 1 3.03 8 24.24 24 72.73 Level 2 3 9.09 25 75.76 5 15.15 Level 3 2 6.06 12 36.36 19 57.57 Level 4 12 36.36 18 54.54 1 3.03 Level 5 25 75.76 8 24.24 Level 6 20 60.60 13 39.40 TOTAL 63 31.81 84 42.2 49 24.91 NS=Number of students CN= Correct answer From Table 1 above, it can be seen that the students' reading level was either 3 or 4 since the highest total number of students (84) could complete the items of all levels correctly until the instructional reading level whose average percentage was 42,2. No student with independent level was in levels 5 and 6, so there was no reason to judge the students’ reading level in 5 and 6. Students’ need analysis Fifty items of questionnaires were distributed to the students to get information about their needs related to reading skills. The fifty items included nine aspects such as (1) students’ reading competence, (2) students’ reading motivation and interest, (3) teaching and learning process, (4) students’ knowledge of Indonesian descriptive texts and HOTS, (5) students’ strategies in solving difficulties in reading English texts, (6) students’ opinion about difficulties of reading materials and textbooks from the government, (7) students’ opinion about Indonesian descriptive texts, (8) illustrations, images or pictures in English reading texts, and (9) students’ opinions about types of questions. Each item was completed with four choices of letters A, B, C, and D which represented the most positive perception to the less positive one. ENGLISH REVIEW: Journal of English Education p-ISSN 2301-7554, e-ISSN 2541-3643 Volume 10, Issue 2, June 2022 https://journal.uniku.ac.id/index.php/ERJEE 649 Based on the analysis of students’ needs, it was revealed that their English competence was relatively good enough although their language aspects such as grammar and vocabulary were still low (43% of the students felt their grammar mastery was good and 62.5 % of them felt vocabulary mastery enough). 78% of the students thought their ability to read English texts good. The students’ perception of the importance of reading was positive. 75% of the students thought that having the ability of reading English was very important and 25% of them said important. Regarding students’ reading motivation and interest, although the students were not frequent to read texts at home, almost all of them (93.8%) said that they were interested or very interested in reading English texts. Besides, they were also motivated when reading through printed media as 79.2% of them agreed or even strongly agreed to say that printed media were more interested than electronic ones. In relation to teaching and learning process, with frequent lecture method used by the teacher, the students relatively felt interested. In fact, 59.4 percent of the students said that the English teacher sometimes provided English texts related to Indonesian context. For the aspect 4 related to HOTS, almost 70% of the students were familiar with HOTS-based questions and even 18% were very familiar. Meanwhile 93.8% of them were also very familiar with descriptive texts. When we refer to descriptive texts in Indonesian context, 65.6 % of them agreed and 25% said strongly agree to be used improve their knowledge of reading skills by providing HOTS-based questions. The students’ perception of the government's English textbook was relatively fair. They thought that the index difficulty of the textbook was easy enough (53.1% of them said easy) while the percentage of those who said difficult was only 43.8. In addition, the textbook provided fair number of factual information related to descriptive texts (≤ 5) and the topics related to Indonesian context were also poor. 25% of the students said nothing contained such topics and even 40.6 % of them said nothing. Concerning aspect of “Students’ Strategies in Solving Difficulties in Reading English Texts”, the students had difficulty reading English texts. 59.4 percent referred to their teacher, and only 15.6 percent did not consult their teacher. The frequency of asking their friends was relatively low (37.5% often and 37.5 % sometimes). They also made their effort often (34.4%) or very often (53.1%) although they sometimes felt frustrated or did nothing (50%). The result of the questionnaire for aspect 8 revealed that the students liked if the reading texts were completed with illustrations, images, or pictures. All the six items for this aspect earned high percentage of positive perception (average >90%). By having these kinds of texts, they could be motivated to read to improve their understanding of English texts. Finally, the result of students’ opinions about types of questions showed that they seemed to be more preferable with multiple-choice items than other questions. True-False type question got the least preferable of all types of questions. Besides, they liked short answer-type questions better than Essay ones. The interview results also supported the results of the questionnaires above as in the following: (1) Since the students have been familiar with descriptive texts, they felt motivated to read the texts, especially if they were in Indonesian context. (2) Although they were familiar with HOTS-based questions, they felt they were not accustomed to doing HOTS-based questions. (3) The conclusion of HOTS-based questions in descriptive texts was very important to increase their high thinking skills. Design phase In the design phase, ten descriptive reading texts in Indonesian context were selected for developing reading texts completed with HOTS-based questions. The texts were first retyped and checked their readability through an online automatic readability checker. The readability of reading texts was measured by using Flesch Kincaid to see whether they were appropriate enough for the tenth- grade students of SMA Negeri 11 Palembang. The average readability score is 57.03 which is appropriate enough for grade 10. The developed product passed two major stages i.e., preliminary and prototyping. The preliminary stage was done by studying the subject matter, curriculum, literature review, place, etc. The prototyping stage, the formative evaluation model by Tessmer (1993), was used to improve the quality of intervention, which included validity, practicality, and potential effect in the evaluation Irnansyah, Soni Mirizon, & Ismail Petrus Developing HOTS-based descriptive reading materials in Indonesian context for tenth graders 650 phase. The formative evaluation included self- evaluation, expert reviews, one-to-one, small group, and field test. In the self-evaluation phase, the activities covered checking the readability of descriptive texts to be appropriate to the students reading level, language use, grammar, vocabulary choice, content, and any revision related to the texts and the question items. By doing these, it was supposed to have any information related to the product's difficulty to be developed for validity, practicality, and potential effect. Expert review The expert review evaluated the design product for content, instructional design, and language. With few revisions such as language use, vocabulary choice, and distractors of HOTS question items, the product was finally good enough to be given to the students. The highest score for each item weighed 5 and the lowest one weighed 1. The results of expert review are summarized in the following Table 2. Table 2. Expert review No Aspect Average Score Remarks 1 Language 3,53 Highly valid 2 Content 3.76 Very highly valid 3 Instructional design 4 Very highly valid Total average score 3.71 Very highly valid From Table 2 above, it is apparent that the average scores were 3.53 for language, 3.76 for content, and 4.00 for instructional design. The total average score of the three aspects from the expert was 3.71 which was higher than 2.51. The score 3.71 was categorized as very highly valid. One-to-one evaluation The descriptive reading texts developed with HOTS-based questions and validated in terms of content, instructional design, and language were given to the group of one-to-one evaluation consisting of three students with reading level categories of frustational, instructional, and independent. The average reading test score was 54.7 as shown in Table 3 below, which is still far from the criterion of mastery ≥70.0. The questionnaire consisting of 30 items with three different aspects (students’ needs, attractiveness and material display, and evaluation) given to the students was aimed to evaluate the practicality of the product to be developed. The questionnaire required the answer on a Likert scale from 1 to 4, which generally gave the same value from the most positive meaning to the less one. However, since each item has different statement to ask, the expression of the choices 1, 2, 3, and 4 may vary from one and another. For example, the Likert scale 1 in items 1, 2, or 3 may means very good, very interesting, or very agree. The average score of the questionnaire was calculated and then interpreted. The result of the questionnaire revealed that the average score was 2, 84, which was higher than 2,51 as shown in Table 3 below. Table 3. The practicality of product in one-to-one evaluation phase Aspect Average Score Remarks Relevance with the students’ needs 2,7 high practicality Attractiveness and material display 3,2 high practicality Evaluation 2,67 high practicality Total average score 2,84 high practicality Small-group evaluation The reading tests and questionnaire results were then used to revise the product-developed reading texts. The revised product was called prototype two which was then given to the students in a small-group evaluation group with nine participants from three different reading level categories. The results showed that the average reading score was 69.11, earning 14 points from the one-to-one evaluation group. The item difficulty (ID) of the revised items was considered good enough since the lowest one was 0.4 (two items). The average reading score was 69.11 rounded 70 which achieved the minimum requirement of mastery, so there was not much work for revision except checking the spelling errors, format, etc. The result of the questionnaire revealed that the average score was 3, 05, which was higher than 2, 51. The reading scores of the small-group evaluation group can be shown in table 4. ENGLISH REVIEW: Journal of English Education p-ISSN 2301-7554, e-ISSN 2541-3643 Volume 10, Issue 2, June 2022 https://journal.uniku.ac.id/index.php/ERJEE 651 Notice that the lowest score was 50.0, gained by two students, and the highest one was 84, earned by one student. Similar to one-to-one evaluation, small group evaluation also participated in completing the questionnaire and it can be shown in Table 4 below. Table 4. The practicality of product in small group evaluation phase Aspect Average Score Remarks Relevance with the Students’ Needs 2,8 high practicality Attractiveness and Material Display 3,3 high practicality Evaluation 3,11 high practicality Total Average Score 3,05 high practicality Field test evaluation Twenty-one students in the field test evaluation group took the descriptive reading test based on HOTS questions in Indonesian context. The results revealed that their scores were distributed from the lowest score, 58 earned by one student, to the highest, 86 obtained by two students. Compared with the average score of the small-group evaluation group, the average score of this group gained two points which its score was 73.04 higher than the minimum requirement of mastery. The following Table 5 contains the distribution of reading scores of the students of three evaluation groups. Table 5. Distribution of reading scores of one-to-one, small groups, and field test One-to-one Small Group Field Test Valid Score Frequency Score Frequency Score Frequency Percent 52.00 1 50.00 2 58.00 1 4.8 54.00 1 58.00 1 60.00 2 9.5 58.00 1 72.00 2 62.00 1 4.8 Total 3 76.00 1 70.00 6 28.6 Mean 54.7 80.00 2 72.00 1 4.8 84.00 1 74.00 2 9.5 Total 9 78.00 2 9.5 Mean 69.11 80.00 1 4.8 82.00 3 14.3 86.00 2 9.5 Total 21 100.0 Mean 73.04 From Table 5 above, it can be seen that a score of 70 was in the field test group the biggest frequency and the highest percentage (28.6). Score 70 is the mode which occurred most often. There were four students whose scores were below and 19 students whose scores were above or equal to 70. 86.1% of the students could achieve the minimum grade of mastery (KKM). The mean reading scores of the whole group are presented in the following Figure. Figure 2. Percentage average of reading score in evaluation stages Figure 2 above shows that students in both small group evaluation (if the score is rounded) and field test evaluation generally could reach the minimum criterion of mastery of 70. Discussion This study was designed to develop a product of HOTS-based descriptive reading texts in Indonesian context for the tenth-grade students of SMA Negeri 11 Palembang. The development study is generally aimed to overcome any complex problem concerning education activities, so the study here was conducted to help the students to improve their cognitive thinking skills through HOTS-based questions. The HOTS-based questions were made through descriptive reading texts in Indonesian Irnansyah, Soni Mirizon, & Ismail Petrus Developing HOTS-based descriptive reading materials in Indonesian context for tenth graders 652 context about things, people, or places in Indonesia. By adopting descriptive texts in the Indonesian context, the students could be motivated to read and get easy to understand the texts better because they might have had prior knowledge or special interest. Tomlinson (2012) argues that a teacher could consider adapting such teaching materials as modifying the materials in a foreign language such as English to local or native language Bahasa Indonesia (L2 to L1). This product adopted the procedure offered by Akker (1999), which included instructional analysis, students’ needs, and students’ reading levels. The instruction analysis revealed that the descriptive texts were also taught to the tenth-grade students. Besides, the study also searched whether the English textbook or English reading materials used by the school contained some descriptive reading texts and HOTS-based questions. The government has instructed to include high cognitive levels in the teaching and learning process to increase learners’ thinking skills. The students’ need analysis questionnaire was distributed to the students to see their reading motivation and knowledge about HOTS. The questionnaire result revealed that they were very familiar with descriptive texts and even learned them during their studies in class. They felt motivated by descriptive texts accompanied by pictures and described things about Indonesia. They were also familiar with HOTS-based questions, although rarely did they have them. The language aspects to be improved were grammar and vocabulary as these aspects seemed to be low. The developed product was valid since the average scores of 2.84 and 3.05 from both one-to- one and small-group evaluations for the validity were higher than 2.51, as suggested by Kubyszyn and Borich (2003). This means that the developed product covered language, content (in this case, HOTS), and instructional design related to the curriculum. Concerning its practicality, the results from both one-to-one and small group evaluation groups showed that the practicality of the developed product was high enough. The average results of the questionnaire in both groups, one-to-one, and a small group, were higher than 2.51 as the category of high practicality. Based on the previous results of validity and practicality from both the experts and the students, the developed product was categorized as valid although there was a slightly significant difference between the average score of the experts (3.71/very high) and that of the students (2,95) from both groups of the one-to-one group and small group. Meanwhile, the result of the evaluation given to the field test group provided potential effects of the developed product for the following reasons. (1) Although the students' average score of validity and practicality was lower than that of the experts 2.95 < 3.71, the developed product was still high. (2) The appropriateness of the product was good enough for the tenth-grade students as the readability of the reading texts had been calculated and suited based on the students' reading level. Therefore, the readability of reading texts could potentially affect the developed product. (3) 86.1% of the students achieved the minimum requirement of mastery. This means 13.9% of the students were still weak in completing HOTS-based questions. This could happen due to the fact that based on the result of the interview, few of them often got difficulty in answering HOTS-based questions during their learning English. (4) Formative Evaluation Design, which included revision, helped improve the validity and practicality of the developed product. This could be determined by the average score in the small group evaluation which increased from 2.84 in one-to-one evaluation to 3.04. The average score of the reading test in the field test evaluation also gained more than 3 points, from 69.11 in the small group evaluation to 73.04. (5) Before revision (prototype one/one-to-one evaluation), the average index difficulty (ID) of the question items was 0.54, but after the revision (in the field test evaluation) the average ID was 0.61. The developed product contained 28% easy questions, 64% medium questions, and 8% difficult ones. Again, the evaluation stage helped to improve the quality of the product. (6) The developed product used descriptive texts with multiple-choice items. This type of question positively affected the students as they like this type of question better than the other types. However, although this type of question is preferable, the test maker should consider the weakness of multiple-choice items because test-takers often guess without thinking when they are in trouble. The probability of correct guessing is always open. In short, the validity, practicality as well as the potential effect of this development study were supported by the previous similar studies (Azizah, ENGLISH REVIEW: Journal of English Education p-ISSN 2301-7554, e-ISSN 2541-3643 Volume 10, Issue 2, June 2022 https://journal.uniku.ac.id/index.php/ERJEE 653 Inderawati, & Vianty, 2021; Razak, Hermandra, & Elmustian, 2021). CONCLUSION Based on the above findings and discussion, it could be concluded that the developed product of HOTS- based questions of descriptive reading materials in Indonesian context was valid, practical, and effective to be used in learning English for the tenth-grade students. The reading materials were adapted ones but were made readable and appropriate for the tenth-grade level. Some revisions related to long sentences or complex phrases and low frequency or difficult words in the reading text materials were made simpler from the original ones. In addition to improving students’ thinking skills, the product provided descriptive reading materials with various subjects accompanied and designed with pictures, so the students enjoyed reading them and could comprehend the questions easily. Although the questions were made for high cognitive levels, most of the students could achieve the target mastery of learning outcomes. Finally, it is suggested that the teacher should use HOTS-based learning as often as possible as demanded by the 2013 curriculum. Besides, they actively introduce HOTS to students, which can be done through descriptive reading materials as the instructional material. REFERENCES Akker, J. D. (1999). Design approaches and tools in education and training: Principles and methods of development research. Kluwer. Azizah, N., Inderawati, R., & Vianty, M. (2021). Developing descriptive reading materials in EFL classes by utilizing the local culture. Studies in English Language and Education, 8(2), 596-621. Brookhart, S. M. (2010). How to assess higher-order thinking skills in your classroom. ASCD. Checaria, K.A., Petrus, I., & Inderawati, R. (2021). Descriptive reading materials for tenth graders: Palembang tourist destinations. English Review: Journal of English Education, 10(1), pp. 255- 264. https://doi.org/10.25134/erjee.v10i1.5386 Daud, N. M. (2017). Integrating HOTS into language classes in the 21 st. the first education and language. International Conference Proceedings Center for International Language Development of Unissula (pp. 29–36). Duncan, A. (2009). Statement from US secretary of education Arne Duncan on results of NAEP Arts 2008 assessment. Retrieved from www.ed.gov/news/pressreleases/2009/06/061520 09.html Febriyani, R. A., Yunita, W., & Damayanti, I. (2020). An analysis on higher-order thinking skill (HOTS) in compulsory English textbook for the twelfth grade of Indonesian senior high schools. Journal of English Education and Teaching (JEET), 4(2), 170-183. https://doi.org/10.33369/jeet.4.2.170- 183 Gall, M. D., Borg, W. R., & Gall, J. P. (2003). Educational research: An introduction (7th ed). Pearson. Hambali, M., Mirizon, S., & Heryana, N. (2021). Difficulty index and cognitive skills of English textbook for senior high school. Indonesian Journal of EFL and Linguistics. 6(1). doi: http://dx.doi.org/10.21462/ijefl.v6i1.331 Indriyana, B. S., & Kuswandono, P. (2019). Developing students’ higher order thinking skills (HOTS) in reading: English teachers’ strategies in selected junior high schools. Journal of English Teaching, 5(3), 2622-4224. http://dx.doi.org/10.33541/jet.v5i3.1313 Jennings, J.H., Caldwell, J., & Lerner, J. (2006). Jennings informal reading assessment. reading problems: Assessment and teaching strategies. Pearson. Kemdikbud. (2019). Indonesia PISA results 2018. Retrieved from https://www.kemdikbud.go.id/main/blog/2019/12 /hasil-pisa-indonesia-2018-akses-makin-meluas- saatnya-tingkatkan- kualitas#:~:text=Hasil%20studi%20PISA%2020 18%20yang,rata%20skor%20OECD%20yakni% 20487. Kurbiszyn, T., & Borich, G. (1993). Educational testing and measurement classroom application and practice (4th ed.). Harper Collins College Publisher. Kusuma, M. D., Rosidin, U., Abdurrahman., & Suyatna, A. (2017). The development of higher order thinking skill (HOTS) instrument assessment in physics study. Journal of Research & Method in Education, 7(1) pp. 26-32. Retrieved from www.iosrjournals.org June 29, 2022 Margana & Widyantoro, A. (2017). Developing English textbooks oriented to higher order thinking skills for students of vocational high schools in Yogyakarta. Journal of Language Teaching and Research, 8(1), pp. 26-38. http://dx.doi.org/10.17507/jltr.0801.04 Mursyid, M., & Kurniawati, N. (2019). Higher order thinking skills among English teachers across generation in EFL classroom. Journal of English Education, 7(2). http://www.ed.gov/news/pressreleases/2009/06/06152009.html http://www.ed.gov/news/pressreleases/2009/06/06152009.html https://doi.org/10.33369/jeet.4.2.170-183 https://doi.org/10.33369/jeet.4.2.170-183 http://dx.doi.org/10.33541/jet.v5i3.1313 https://www.kemdikbud.go.id/main/blog/2019/12/hasil-pisa-indonesia-2018-akses-makin-meluas-saatnya-tingkatkan-kualitas#:~:text=Hasil%20studi%20PISA%202018%20yang,rata%20skor%20OECD%20yakni%20487 https://www.kemdikbud.go.id/main/blog/2019/12/hasil-pisa-indonesia-2018-akses-makin-meluas-saatnya-tingkatkan-kualitas#:~:text=Hasil%20studi%20PISA%202018%20yang,rata%20skor%20OECD%20yakni%20487 https://www.kemdikbud.go.id/main/blog/2019/12/hasil-pisa-indonesia-2018-akses-makin-meluas-saatnya-tingkatkan-kualitas#:~:text=Hasil%20studi%20PISA%202018%20yang,rata%20skor%20OECD%20yakni%20487 https://www.kemdikbud.go.id/main/blog/2019/12/hasil-pisa-indonesia-2018-akses-makin-meluas-saatnya-tingkatkan-kualitas#:~:text=Hasil%20studi%20PISA%202018%20yang,rata%20skor%20OECD%20yakni%20487 https://www.kemdikbud.go.id/main/blog/2019/12/hasil-pisa-indonesia-2018-akses-makin-meluas-saatnya-tingkatkan-kualitas#:~:text=Hasil%20studi%20PISA%202018%20yang,rata%20skor%20OECD%20yakni%20487 https://www.kemdikbud.go.id/main/blog/2019/12/hasil-pisa-indonesia-2018-akses-makin-meluas-saatnya-tingkatkan-kualitas#:~:text=Hasil%20studi%20PISA%202018%20yang,rata%20skor%20OECD%20yakni%20487 http://www.iosrjournals.org/ http://dx.doi.org/10.17507/jltr.0801.04 Irnansyah, Soni Mirizon, & Ismail Petrus Developing HOTS-based descriptive reading materials in Indonesian context for tenth graders 654 Nation, I. S. P. (2012). Learning vocabulary in another language. Cambridge University Press. https://doi.org/10.1017/CBO9781139524759 National Education Association. (2012). Preparing 21st century students for a global society: An educator’s guide to the “Four Cs.” http://www.nea.org/assets/docs/A-Guide-to- Four-Cs.pdf. Putra, T. K., & Abdullah, D. F. (2019). Higher-order thinking skills (HOTS) questions in English national examination in Indonesia. Journal Bahasa Lingua Scientia, 11(1), 145-160. https://doi.org/10.21274/ls.2019.11.1.145-160 Ramadhianti, A. (2017). The effects of cognitive competence and self-esteem towards students’ reading comprehenshion. Journal of English Language Teaching, 1(2). p-ISSN: 2541-0326. Razak, A., Hermandra., & Elmustian. (2021). Developing descriptive text reading materials based on constructivism approach. Jurnal PAJAR (Pendidikan Pengajaran), 5(3), 585-598. http://dx.doi.org/1033578/pjr.v5i3.8272 Rivas, S. F., Saiz, C., & Ossa, C. (2022). Metacognitive strategies and development of critical thinking in higher education. Front. Physchol.https://doi.org/10.3389/fpsyg.2022.913 219 San, K. M. (2019). The cognitive thinking levels demanded in the tasks in the coursebook global level 1 (A2). International Journal of Innovative Science and Research Technology, 4(2), 513- 519. Sugiyono. (2012). Metode penelitian pendidikan: pendekatan kuantitatif, kualitatif, dan R & D. Alfabeta. Tessmer, M. (1993). Planning and conducting formative evaluation. Philadehelphia, PA: Kogan Page. Tomlinson, B. (2012). Materials development for language learning and teaching. Language Teaching, 42(2), 143-179. Cambridge, UK: Cambridge University Press. Tyas, M. A., Nurkamto, J., Marmanto, S., & Laksani, H. (2019). Developing higher order thinking skills (HOTS) – based questions: Indonesian EFL teachers’ challenges, 2(1), 52-63. doi: 10.17501/26307413.2019.2106 Utari, R. S., & Gustiningsih, T. (2021). Developing of higher order thinking skills in relation and function to support student’s creative thinking. Mathematics Education Journal, 15(1). Wahyuni, S., Miarsyah, M., & Adisyahputra. (2018). Achievement motivation, critical thinking skills, and reading comprehension correlation with scientific literacy among senior high school students. Jurnal Pendidikan Sains, 6(2), 30-37. http://journal.um.ac.id/index.php/jps/ ISSN: 2338-9117/EISSN: 2442-3904. Watson, S. (2019). What Is the HOTS concept in american education reform? Thought co. Retrieved from https://www.thoughtco.com/higher-order- thinkingskills-hots-education-3111297 Wu, Y., & Pei, Z. (2018). An investigation of critical thinking manifested in the questions of EFL textbooks for tertiary-level English majors of China. American Journal of Education and Learning, 3(2), 72-84 file:///C:/Users/3rd%20Semester/Thesis%20proposal/Comment%20from%20pak%20Ismail%20Petrus/.%20https:/doi.org/10.1017/CBO9781139524759 file:///C:/Users/3rd%20Semester/Thesis%20proposal/Comment%20from%20pak%20Ismail%20Petrus/.%20https:/doi.org/10.1017/CBO9781139524759 http://www.nea.org/assets/docs/A-Guide-to-Four-Cs.pdf http://www.nea.org/assets/docs/A-Guide-to-Four-Cs.pdf https://doi.org/10.21274/ls.2019.11.1.145-160 http://dx.doi.org/1033578/pjr.v5i3.8272 https://www.thoughtco.com/higher-order-thinkingskills-hots-education-3111297 https://www.thoughtco.com/higher-order-thinkingskills-hots-education-3111297