Universitas Muhammadiyah Malang, East Java, Indonesia JPBI (Jurnal Pendidikan Biologi Indonesia) p-ISSN 2442-3750, e-ISSN 2537-6204 // Vol. 5 No. 3 November 2019, pp. 521-528 10.22219/jpbi.v5i3.10171 http://ejournal.umm.ac.id/index.php/jpbi jpbi@umm.ac.id 521 Research Article Higher-order thinking skills based on Marzano taxonomy in basic biology I course Metri Dian Insani a,1,*, Novida Pratiwi b,2, M. Muhardjito c,3 a Department of Biology, KU Leuven, Celestijnenlaan 200c-box 2406 3001, Leuven, Belgium (Ph.D Student) b Science Education Study Program, Faculty of Mathematics and Natural Science, Universitas Negeri Malang, Jl. Semarang 5 Malang, East Java 65145, Indonesia c Department of Biology Education, Faculty of Teacher Training and Education, Universitas Negeri Malang, Jl. Semarang 5 Malang, East Java 65145, Indonesia 1 metri.dian.fmipa@um.ac.id *; 2 novida.pratiwi.fmipa@um.ac.id; 3 muhardjito.fmipa@um.ac.id * Corresponding author INTRODUCTION Learning activities in the Science Education Study Program of Universitas Negeri Malang (UM) requires its students to do cooperative discussion, laboratory experiment, case study and project exploration to enact learning outcome in holistic and integrative ways. As the institution which plays its important function in producing teacher candidate students, Science Education Study Program-UM prepares the students to have competencies in planning, actuating, and evaluating natural science learning through scientific approach by utilizing media, A R T I C L E I N F O A B S T R A C T Article history Received October 16, 2019 Revised November 02, 2019 Accepted November 20, 2019 Published November 30, 2019 This research was aimed at profiling student’s thinking skills in dealing with Higher-Order Thinking Skills (HOTS) questions based on Marzano taxonomy by referring to 13 indicators. This pre-experimental research employed pretest-posttest design. The indicators included were comparison, classification, deductive reasoning, inductive reasoning, error analysis, construction, analysis perspective, abstraction, decision making, investigation, problem solving, inquiry experiment, and innovation finding. The instrument used was a 13-essay questions (r-Pearson= 0.79 and Cronbach alpha = 0.68). The research results conducted to 98 students of Natural Science Education which was analyzed using paired t-test showed the significant different between pre-test and posttest (sig. <0.01). As many as four HOTS indicators (i.e. deductive reasoning, error analysis, construction, and abstraction) were categorized as low level. Meanwhile, the eighth HOTS indicators were categorized as moderate level, namely: comparison, inductive reasoning, analysis perspective, decision making, investigation, solving problem, inquiry experiment, and innovative finding. In addition, the classification indicator was considered to fulfill the high level in the end of the course. Higher order thinking skills profile is crucial to be observed, along with students’ cognitive development, from the beginning of course, which then can be used as a basic information in designing learning approach which is HOTS-oriented for the next course. Copyright © 2019, Insani et al This is an open access article under the CC–BY-SA license Keywords Basic biology Higher-order thinking skills Marzano taxonomy How to cite: Insani, M. D., Pratiwi, N., & Muhardjito, M. (2019). Higher-order thinking skills based on Marzano taxonomy in basic biology I course. JPBI (Jurnal Pendidikan Biologi Indonesia), 5(3), 521-528. doi: https://doi.org/10.22219/jpbi.v5i3.10171 http://ejournal.umm.ac.id/ http://u.lipi.go.id/1422867894 http://u.lipi.go.id/1460300524 https://doi.org/10.22219/jpbi.v5i3.10171 http://ejournal.umm.ac.id/index.php/jpbi mailto:jpbi@umm.ac.id http://creativecommons.org/licenses/by-sa/4.0/ https://doi.org/10.22219/jpbi.v5i3.10171 http://creativecommons.org/licenses/by-sa/4.0/ https://crossmark.crossref.org/dialog/?doi=10.22219/jpbi.v5i3.10171&domai JPBI (Jurnal Pendidikan Biologi Indonesia) Vol. 5, No. 3, November 2019, pp. 521-528 522 Insani et al (Higher-order thinking skills …) science-and-technology based learning resources and environmental potentials so that they possess knowledge, scientific processing skill, scientific behavior as well as high level thinking, in term of critical-creative thinking. A good teaching-learning process enables students in mastering knowledge and intrinsic motivation in academic field. Besides that, it can also create a student-centered learning atmosphere which, in turn, affects to the external motivation as the consequences of cooperative project activities. These activities enhance students' enthusiasm to cooperate, discuss, as well as experiences sharing to complete their task together. Therefore, students' creativity and independent learning are also improved (Fox, 2013). During the teaching-learning processes, one of teachers' role is a facilitator in terms of giving supports in material resources, motivation, and guiding students to gain learning experiences through challenging project (Arends, 2012). Thus, students are habituated in facing proper cognitive conflicts to bring up ideas based on previous obtained experiences. However, based on the previous study, scientific thinking ability of Science Education Study Program students of UM needs to be improved. Munzil, Mustikasari, and Insani (2016) reported that as high as 63.67% student was categorized concrete, while 21.27% was low formal, 13.67% was upper formal, and 0.39% was post formal. This describes that the students' thinking skill needs to be raised; moreover, students' higher level thinking mapping was not conducted yet. Therefore, an effort to encourage students' thinking skill, especially for the category which suits the Piaget's cognitive development, in formal or post formal operational level. Higher level thinking skill is also known as Higher-Order Thinking Skills (HOTS). This is a holistic thinking performance ability to find and tackle a new challenge. It has been proved that by gaining these skills, students are aided to achieve the learning outcomes determined (Husamah, Fatmawati, & Setyawan, 2018; Madhuri, Kantamreddi, Prakash Goteti, & Goteti, 2012). This skills demand students to implement new information and knowledge they possessed to be manipulated as the possible solution for various issues faced (Heong et al., 2012). Moreover, Brookhart (2010) stated that HOTS is someone's ability in associating and implementing his/her knowledge to construct "thinking framework" in the context which never been faced before. In adult level, high thinking ability should more dominate compared to low thinking ability. Furthermore, it is a thinking competency required in the 21st-century. ATC21S (Assessment & Teaching of 21st-Century Skills) has classified the 21st- century competencies in four categories, in which one of them is thinking way. This showed that mastering HOTS is a crucial provision for social life (Griffin & Care, 2015) as well as one of intellectual barometer for a nation. However, HOTS was proven rarely implemented in small learning communities (Fischer, Bol, & Pribesh, 2011). There are several HOTS versions known among education practitioners. One of them is Bloom's Taxonomy which focuses on cognitive domain product. This taxonomy then was revised by involving thinking skills in terms of analyzing (C4), evaluating (C5), and creating (C6) (Anderson & Krathwohl, 2001). Anderson and Krathwohl have conducted revision by changing learning outcomes. This is crucial to be done as the original Bloom's Taxonomy focus is the description of thinking processes. Moreover, taxonomy sequence shifting which describes thinking processes from the lower level (low order thinking) to the higher level (high order thinking) was also done. Having considerations about Bloom’s Taxonomy limitations, Marzano and Kendall (2008) developed new taxonomy which comprised of 13 high order thinking indicators. In this taxonomy, various factors which influenced the way students gain their knowledge in every stage were also integrated so that they can hone their thinking competencies. Not only deals with cognitive domain, but Marzano's taxonomy does also cover three systems (i.e. self-system, metacognitive, and cognitive), as well as adds and emphasizes self-system and metacognition (Irvine, 2017). Generally, there were several previous researches measured HOTS based on Bloom's Taxonomy (Ghani, Ibrahim, Yahaya, & Surif, 2017; Narayanan & Adithan, 2015; Roets & Maritz, 2017). However, there were only few studies done based on Mazarno's. Yet Mazarno's-based HOTS can measure thinking skill in more complex and multitasking. Hence, study of high thinking skills based on Mazarno's taxonomy in Basic Biology I course is necessary to conduct which is expected to motivate students internal and externally as their thinking skills are sharpened from the beginning of semester. This will be high motivation to stimulate the creativity of students' thinking ability as their important competency. To go further, it also as the lecturers’ responsibility of Natural Science Education Study Program in generating their future graduates to be teacher candidates who are mastering science and possessing higher order thinking skills in accordance with science and technology advancement in the 21st-century (Insani, 2016). Based on the mentioned conditions, this research aimed at profiling student’s thinking skills in dealing with Higher-Order Thinking Skills (HOTS) questions based on Marzano taxonomy by referring to 13 indicators. This research will give considerable contribution to the future education scientific field, especially in Indonesia which has special characteristics in its learning culture. This will open a more focus discourse among researchers and do further research to pursue the better quality of education. Moreover, this can be a good reference for the JPBI (Jurnal Pendidikan Biologi Indonesia) Vol. 5, No. 3, November 2019, pp. 521-528 523 Insani et al (Higher-order thinking skills …) Preliminary study Research subjects determination Instrument construction Validity and reliability test Data collection Data analysis Conclusion educators to develop their learning in term of make sure their objective learning in more detail by referring Marzano’s taxonomy. METHOD This quasi experimental research employed pretest-posttest design. As many as 98 students of Natural Science Study Program, Faculty of Mathematics and Natural Science, Universitas Negeri Malang were involved as research subjects. This research was carried out in Basic Biology I Course for one semester which was comprised of 14 times class sessions and twice sessions for pretest-posttest (two credits, 100 minutes per week). The data collection for pretest was conducted in the beginning of semester while posttest was done in the end of the semester. There were 12 topics delivered to the students. The course activities were varied i.e. lecturing, discussion, pairing group work, case study, project, and laboratory experiment. The teaching plan has been approved by the lecturer and head of department. The research procedure is served in Figure 1. Figure 1. Research procedure. The initial stage in this research was a preliminary study to find the urgency of the problem under study. This stage was done by conducting a reflection and analyzing student learning outcomes of Basic Biology Course achieved in the previous academic year. The preparation of the instrument used was carried out by involving three experts (i.e. Basic Biology I lecturers). The instrument compiled consisted of 13 essay items which was then analyzed to determine the validity and reliability using Pearson product moment correlation and Cronbach alpha value. In addition, the data gained was analyzed using paired samples t-test to emphasize the difference between pre-test and posttest results. Furthermore, the students’ HOTS profile was assessed based on Marzano’s taxonomy which comprised of 13 indicators, namely, comparative ability, classification, deductive reasoning, inductive reasoning, error analysis, construction, perspective analysis, abstraction, decision making, investigation, problem solving, inquiry experimentation, and discovery/innovation (Marzano & Kendall, 2008). The level achievement determination of each indicator was conducted based on Heong et al. (2011) as served in Table 1. Table 1. Marzano HOTS level category Mean score HOTS level 0,00 – 1,00 Low 1,01 – 2,00 Moderate 2,01 – 3,00 High The Marzano's HOTS indicators were in line with HOTS indicators of revised Bloom's taxonomy i.e. C4 (analysis), C5 (evaluation), and C6 (creating). The analysis competency of Bloom's taxonomy was in line with: 1) comparison (distinguishing the relevance of elements/parties within materials; 2) classification (organizing and finding the coherence of elements one another in a unity of structure; 3) Analysis perspective (correlating and reassembly elements in a new situation); 4) Deductive reasoning (causality correlation based on general fact to the specific one; 5) inductive reasoning, generalization by inferring conclusion based on specific facts. The evaluation competence was in accordance with: 1) error analysis (the ability in recognizing the improperness or inconsistency an element in a structure; 2) Decision making (the ability in determining the most proper procedure based on determined criteria or standard; 3) Problem solving. The creating ability was suitable with: 1) JPBI (Jurnal Pendidikan Biologi Indonesia) Vol. 5, No. 3, November 2019, pp. 521-528 524 Insani et al (Higher-order thinking skills …) Constructing (the ability in constructing elements in a coherent structure); 2) abstraction (the ability in organizing elements from its abstract form to the concrete one; 3) investigation; 4) problem solving; 5) inquiry experiment; 6) Innovative discovery (the ability in designing a solutive and creative procedure or product. RESULTS AND DISCUSSION This students’ HOTS profiling research was done by employing a valid and reliable instrument. This has been proven as the test results gained. The validity test results showed that the Pearson product moment correlation value was 0.79; meanwhile, the Cronbach alpha value was 0.68 which represented the reliability of instrument. By considering these results, the instrument was valid and reliable to use. Hence, the data collection was done. The pre-test results obtained from the students involved were served in Table 2. The table provides an overview of the initial students’ HOTS based on 13 indicators. It is clearly shown that among the 13 HOTS indicators measured, there only two indicators categorized as moderate level, while the 11 remain indicators were classified in low level. To be more detail, the highest HOTS mean score was shown in the classification ability (1.51) with moderate level. Contrarily, the two lowest student abilities were construction and abstraction in which the mean values of each was 0.05. This evidence illustrates that students' HOTS need to be improved. This information is a valuable asset for lecturers to evaluate their management in facilitating learning process starting from the clarity of the objectives determined (Anderson & Krathwohl, 2001). Martin (1989) stated that the cognitive education as a new emphasize and a systematic exchange element of teacher candidate education is crucial from the viewpoint of its potential to affect adults as teachers, as considerable as on the thinking outcomes of students whom they will teach throughout their carrier. Students need to master complex judgemental skills to achieve higher-order thinking skills (Martin, 1989). Brookhart (2010) stated that HOTS cover logic and reasoning skills, analysis, evaluation, creation, problem solving, and decision making. Notwithstanding that neither are easy these skills to learn nor to teach, but they are valuable as are more likely to be usable in novel situations in terms of the situations other than those in which the skills were learnt Tonissen, Lee, Woods, and Osborne (2014) sated that higher level learning is not merely transferring knowledge, but there must be a development and reproduction of new knowledge. These skills can be achieved by determining proper learning objectives which cover all competencies needed by students (Marzano & Kendall, 2008). Table 2. HOTS score and pretest level No Indicator Mean SD HOTS level 1 Comparison 1.34 0.51 Moderate 2 Classification 1.51 0.82 Moderate 3 Deductive reasoning 0.26 0.04 Low 4 Inductive reasoning 0.60 0.05 Low 5 Error analysis 0.26 0.04 Low 6 Construction 0.05 0.01 Low 7 Analysis perspective 0.34 0.03 Low 8 Abstraction 0.05 0.01 Low 9 Decision making 0.97 0.56 Low 10 Investigation 0.80 0.29 Low 11 Problem solving 0.31 0.03 Low 12 Inquiry experiment 0.46 0.04 Low 13 Innovative discovery 0.14 0.06 Low After getting the pre-test data, to support further analysis, another data collection was held for students’ posttest in the end of the semester. The students’ HOTS posttest results are served in Table 3. Table 3 shows that, generally, the students HOTS profile were improved. Based on the table, it can be seen that more than 50% indicators have been classified as moderate category. The best students’ HOTS ability classification with the mean value was 2.3, while the lowest one was error analysis with the mean value was 0.53. In detail, as many as seven indicators (inductive reasoning, analysis perspective, decision making, investigation, problem solving, inquiry experiment, and innovative discovery) were improved from low to moderate level; meanwhile, one indicator was improved from moderate to high level (classification). Notwithstanding that there were five indicators remained in the same position as the previous levels (comparison in moderate level; deductive JPBI (Jurnal Pendidikan Biologi Indonesia) Vol. 5, No. 3, November 2019, pp. 521-528 525 Insani et al (Higher-order thinking skills …) reasoning, error analysis, construction, and abstraction in low level), but the five indicators’ mean values were higher compare to the pre-test results. Table 3. HOTS score and posttest level No Indicator Mean SD HOTS Level 1 Comparison 1.39 0.62 Moderate 2 Classification 2.37 0.90 High 3 Deductive reasoning 0.74 0.29 Low 4 Inductive reasoning 1.45 0.32 Moderate 5 Error analysis 0.53 0.20 Low 6 Construction 0.73 0.18 Low 7 Analysis perspective 1.55 0.90 Moderate 8 Abstraction 0.82 0.09 Low 9 Decision making 1.66 0.83 Moderate 10 Investigation 1.57 0.50 Moderate 11 Problem solving 1.32 0.26 Moderate 12 Inquiry experiment 1.77 0.53 Moderate 13 Innovative discovery 1.61 0.46 Moderate The students’ HOTS indicators which classified into low level means that the students were still lacking in the ability to analyze in the form of deductive reasoning. Likewise with the ability to evaluate, the students found it difficult to sort out relevant and irrelevant facts, consequently they cannot detect any information mismatch in well manner. This phenomenon is in line with the previous findings which reported that the obstacles in generating conclusion, as the key factor in affecting the achievements of the students’ assignments (Heong et al., 2012), by utilizing and connecting information was witnessed by postgraduate students (Narayanan & Adithan, 2015; Roets & Maritz, 2017).This because HOTS is characterized as a complex, non-algorithmic, self-regulative, meaningful, multiple criteria and uncertainties, as well as effortful and providing multiple solutions. Thus, HOTS must be improved by involving many subjects and teachers (Abosalem, 2015) while empowering students’ metacognitive as it creates self-regulated learner as well as improves learning outcome (Bahri & Corebima, 2015). The most common obstacles encountered by students were in the category of abstraction and construction, these two categories require them to be able to plan, design, and apply new ideas related to problems encountered in daily activities. Additionally, the results of the posttest revealed that the students' knowledge was still at a concrete and theoretical level, so that the processing of information that was not clearly illustrated was difficult to understand and could lead to misconceptions. The HOTS indicators with moderate level imply that students have been able to compare objects one another quite well based on certain criteria. They can also link and connect several different concepts simply. Some of them were also able to link cause and effect of problems precisely accompanied by a quite logic reasons. The ability to evaluate, in term of determination and evaluation the solutions was good, as well as the ability to create such as conducting investigations and experiments. The imagination of students was still not fully honed, the habit of giving answers such as in textbooks or lecture notes was often the first choice rather than trying to develop new ideas that are less familiar and risky. The tendency to be afraid of being wrong and not yet daring to make decisions in group or classical discussion activities can give an idea of how HOTS is closely related to student learning habits and culture. HOTS cannot be obtained instantly; in other words, this skills must be stimulated and trained constantly (Brown, Lawless, & Boyer, 2013; Saido, Siraj, Nordin, & Al_Amedy, 2013). Therefore, a systematic, constructive, and continuous learning process obviously gives a positive impact on the success of students’ mental processes. Meanwhile, the high level HOTS means that the students have a good classification skill. They have been able to find similarities and differences of elements, determine the coherence of how these elements relate to each other in a larger structure. Figure 2 shows the comparison of pre-test and posttest results of students’ HOTS profile. It can be obviously seen that the increase of students’ HOTS occurred from pre-test to posttest. Of 11 HOTS indicators classified in low level in pretest, there only four remained in low level as the posttest results obtained, while the other seven indicators were classified into the better one (moderate). Similarly, of two indicators categorized in moderate level, there only one which was in the same level, while the other one was better in term of improved to be classified into high level. This improvement is strengthened by the results of t-test analysis (served in Table 4). Table 4 provides the t-test results of students’ HOTS profile based on Marzano’s taxonomy. The results implied that there was significant difference between pretest and posttest (p < 0.05). In the end of learning processes, students were able to construct better theoretical and applied understanding through a series of JPBI (Jurnal Pendidikan Biologi Indonesia) Vol. 5, No. 3, November 2019, pp. 521-528 526 Insani et al (Higher-order thinking skills …) creative and innovative learning activities. The undergirding reason of significant improvement occurred was the detail indicators and various activities designed by lecturer to measure the students’ achievement determined based on Marzano’s taxonomy. This is one of Marzano’s taxonomy’s advantages compare to the other taxonomies exist (Irvine, 2017). Figure 2. Comparison between pretest and posttest scores Table 4. The t-test analysis results of students HOTS profile Paired Differences t df Sig. (2- tailed) Mean Std. deviation Std. error mean 95% Confidence interval of the difference Lower Upper Pair 1 pretest - posttest -.80154 .39805 .11040 -1.04208 -.56100 -7.260 12 .000 Higher-order thinking skills have an important role in determining students’ success in processing information. A good learning is that which is able to cover a series of meaningful activities so that they can be understood, mastered, and applied well (Tiantong & Siksen, 2013). Students should be accustomed to deal with difficult situations in term of complex cognitive conflicts; thus, they can solve problems (Freahat & Smadi, 2014) based on experiences they have had through project activities or case studies provided and laboratory activities practiced (Ganapathy, Singh, Kaur, & Kit, 2017). In addition, good learning also accommodates students cooperatively and collaboratively (İlhan, 2014). As the positive consequences, various ideas that were successfully put forward by students in the discussion will be able to motivate the other students in a teamwork. This, in turn, will elevates students achievement (Tanujaya, Mumu, & Margono, 2017). They are also able to compare ideas with regards to identify similarities and differences of the phenomena discussed together, the ability to compare a problem with other supporting information which, in turn, leads to categorical classification skill (Heong et al., 2011). This is the undergirding reason supports the classification indicator placed the highest position of the 13 HOTS indicators of the students’ posttest results. The results of the final reflection of lectures showed that various learning activities were not optimal with the regards to the characteristics of constructivist approaches that lead to meaningful learning. Students should be more focused and more motivated in conducting investigations that have never been done before so they are able to record a lot of hands-on activities done. Lecturers, somehow, still need to provide guidance during discussions, as well as direct the students during case studies/projects. Waugh (2012) stated that educators should encourage students' enthusiasm in planning, academic achievement in both independently and socially. Moreover, internal and external motivation for students can trigger great curiosity and high interest in the effort to find knowledge so that they can achieve learning goals in better way (Levin-Goldberg, 2009). Additionally, the learning atmosphere during lectures needed to be designed as good as possible so that it is more conducive for students to do their learning activities. Learning innovations are also required to be developed, not only the updated-technological facilities and media as learning resources, but also the appropriate learning assessment (Hopson, Simms, & Knezek, 2001; Saul & Wuttke, 2011), learning models (Chinedu, Kamin, & Olabiyi, 2015; Husamah et al., 2018) in constructing knowledge, attitudes, and higher-order thinking skills as a unity (Arends, 2012). Learning is not merely to pursue cognitive goals, but also must be able to accommodate the students’ metacognitive intelligence to contribute to the long life-learning (Marzano & Kendall, 2008). 1.34 1.51 0.26 0.6 0.26 0.05 0.34 0.05 0.97 0.8 0.31 0.46 0.14 1.39 2.37 0.74 1.45 0.53 0.73 1.55 0.82 1.66 1.57 1.32 1.77 1.61 0 0.5 1 1.5 2 2.5 1 2 3 4 5 6 7 8 9 10 11 12 13 M e a n S co re HOTS Indicator Pre-test Posttest JPBI (Jurnal Pendidikan Biologi Indonesia) Vol. 5, No. 3, November 2019, pp. 521-528 527 Insani et al (Higher-order thinking skills …) CONCLUSION Based on the results of the analysis and discussion related to higher order thinking skills (HOTS), there was a significant improvement in students' HOTS from their pretest to their posttest assessed based on 13 HOTS Marzano indicators. Deductive reasoning, error analysis, construction, and abstraction indicators were still at low level, thus, they need to be improved. Meanwhile, the comparison, inductive reasoning, perspective analysis, decision making, investigation, problem solving, inquiry, and innovative findings were at a moderate level. Hence, they still need to be improved as well. The classification indicator was at high level which proved that students have very good ability in this category. This results were not met the expectation yet optimally. Thus, there must be more efforts to design the more effective and meaningful learning to stimulate the students’ ability to achieve high level of HOTS. Moreover, the interaction between students and lecturers should be conducted in more effective way. The carrying capacity of the environment, facilities and learning must be optimally contributed to support the learning activities. The mapping of students’ HOTS based on Marzano taxonomy can be utilized as the basic information for further to optimize the efforts. REFERENCES Abosalem, Y. (2015). Assessment techniques and students’ higher-order thinking skills. ICSIT 2018 - 9th International Conference on Society and Information Technologies, Proceedings, 4(1), 61–66. doi: https:// doi.org/10.11648/j.ijsedu.20160401.11 Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing (A. E. Burvikovs & A. Martinez-Ramos, Eds.). Retrieved from https://www.uky.edu/~rsand1/china2018/texts/Anderson- Krathwohl - A taxonomy for learning teaching and assessing.pdf Arends, R. I. (2012). Learning to Teach (Ninth). Retrieved from https://joitonk.gq/file-ready/learning-to-teach- 9th-edition Bahri, A., & Corebima, A. D. (2015). The contribution of learning motivation and metacognitive skill on cognitive cearning outcome of students within different learning strategies. Journal of Baltic Science Education, 14(4), 487–500. Retrieved from http://www.scientiasocialis.lt/jbse/?q=node/448. Brookhart, S. M. (2010). How to assess higher-order thinking skills in your classroom. ASCD. Retrieved from https://eric.ed.gov/?id=ED512044 Brown, S. W., Lawless, K. A., & Boyer, M. A. (2013). Promoting positive academic dispositions using a web- based PBL environment: The GlobalEd 2 Project. Interdisciplinary Journal of Problem-Based Learning, 7(1), 3–15. doi: https://doi.org/10.7771/1541-5015.1389 Chinedu, C. C., Kamin, Y., & Olabiyi, O. S. (2015). Strategies for improving higher order thinking skills in teaching and learning of design and technology education. Journal of Technical Education and Training (JTET), 7(2), 35–43. Retrieved from https://pdfs.semanticscholar.org/8570/aec24c06b8b387880f3c0aa eb23260577e23.pdf Fischer, C., Bol, L., & Pribesh, S. (2011). An Investigation of Higher-Order Thinking Skills in Smaller Learning Community Social Studies Classrooms. American Secondary Education, 39(2), 5–26. Retrieved from https://www.jstor.org/stable/i23100396 Fox, T. G. (2013). Project based learning in primary grades (Northern Michigan University). Retrieved from https://www.nmu.edu/education/sites/DrupalEducation/files/UserFiles/Fox_Tim_MP.pdf Freahat, N. M., & Smadi, O. M. (2014). Lower-order and higher-order reading questions in secondary and university level EFL Textbooks in Jordan. Theory and Practice in Language Studies, 4(9), 1804–1813. doi: https://doi.org/10.4304/tpls.4.9.1804-1813 Ganapathy, M., Singh, M. K. M., Kaur, S., & Kit, L. W. (2017). Promoting higher order thinking skills via teaching practices. The Southeast Asian Journal of English Language Studies, 23(1), 75–85. doi: https://doi.org/10. 17576/3L-2017-2301-06 Ghani, I. B. A., Ibrahim, N. H., Yahaya, N. A., & Surif, J. (2017). Enhancing students’ HOTS in laboratory educational activity by using concept map as an alternative assessment tool. Chemistry Education Research and Practice, 18(4), 849–874. doi: https://doi.org/10.1039/c7rp00120g Griffin, P., & Care, E. (2015). Assessment and teaching of 21st century skills. doi: https://doi.org/10.1007/978- 94-017-9395-7 Heong, Y. M., Othman, W. B., Yunos, J. B. M., Kiong, T. T., Hassan, R. Bin, & Mohamad, M. M. B. (2011). The level of Marzano higher order thinking skills among technical education students. International Journal of https://doi.org/10.11648/j.ijsedu.20160401.11 https://doi.org/10.11648/j.ijsedu.20160401.11 https://www.uky.edu/~rsand1/china2018/texts/Anderson-Krathwohl%20-%20A%20taxonomy%20for%20learning%20teaching%20and%20assessing.pdf https://www.uky.edu/~rsand1/china2018/texts/Anderson-Krathwohl%20-%20A%20taxonomy%20for%20learning%20teaching%20and%20assessing.pdf https://joitonk.gq/file-ready/learning-to-teach-9th-edition https://joitonk.gq/file-ready/learning-to-teach-9th-edition http://www.scientiasocialis.lt/jbse/?q=node/448 https://eric.ed.gov/?id=ED512044 https://doi.org/10.7771/1541-5015.1389 https://pdfs.semanticscholar.org/8570/aec24c06b8b387880f3c0aaeb23260577e23.pdf https://pdfs.semanticscholar.org/8570/aec24c06b8b387880f3c0aaeb23260577e23.pdf https://www.jstor.org/stable/i23100396 https://www.nmu.edu/education/sites/DrupalEducation/files/UserFiles/Fox_Tim_MP.pdf https://doi.org/10.4304/tpls.4.9.1804-1813 https://doi.org/http:/doi.org/10.17576/3L-2017-2301-06 https://doi.org/http:/doi.org/10.17576/3L-2017-2301-06 https://doi.org/10.1039/c7rp00120g https://doi.org/10.1007/978-94-017-9395-7 https://doi.org/10.1007/978-94-017-9395-7 JPBI (Jurnal Pendidikan Biologi Indonesia) Vol. 5, No. 3, November 2019, pp. 521-528 528 Insani et al (Higher-order thinking skills …) Social Science and Humanity, 1(2), 121–125. doi: https://doi.org/10.7763/ijssh.2011.v1.20 Heong, Y. M., Yunos, J. M., Othman, W., Hassan, R., Tee, T. K., & Mohamad, M. M. (2012). The needs analysis of learning higher order thinking skills for generating ideas. Procedia - Social and Behavioral Sciences, (59), 197–203. doi: https://doi.org/10.1016/j.sbspro.2012.09.265 Hopson, M. H., Simms, R. L., & Knezek, G. A. (2001). Using a technology-enriched environment to improve higher-order thinking skills. Journal of Research on Technology in Education. doi: https://doi.org/10.108 0/15391523.2001.10782338 Husamah, H., Fatmawati, D., & Setyawan, D. (2018). OIDDE learning model: Improving higher order thinking skills of biology teacher candidates. International Journal of Instruction, 11(2), 249–264. doi: https:// doi.org/10.12973/iji.2018.11217a İlhan, I. (2014). A study on the efficacy of project-based learning approach on Social Studies Education: Conceptual achievement and academic motivation. Educational Research and Reviews, 9(15), 487–497. doi: https://doi.org/10.5897/err2014.1777 Insani, M. (2016). Studi pendahuluan identifikasi kesulitan dalam pembelajaran pada guru IPA SMP Se-Kota Malang. Jurnal Pendidikan Biologi, 7(2), 81–93. doi: https://doi.org/10.17977/jpb.v7i2.723 Irvine, J. (2017). A comparison of revised Bloom and Marzano’ s new taxonomy of learning. Research in Higher Education Journal, 33(08), 16. Retrieved from http://www.aabri.com/copyright.html Levin-Goldberg, J. (2009). Five ways to increase civic engagement. Social Studies and The Young Learner, 22(1), 15–18. Retrieved from https://www.socialstudies.org/system/files/publications/articles/yl_220115 .pdf Madhuri, G. V., Kantamreddi, V. S. S. . S. N., Prakash Goteti, L. N. S., & Goteti, L. N. S. P. (2012). Promoting higher order thinking skills using inquiry-based learning. European Journal of Engineering Education, 37(2), 117–123. doi: https://doi.org/10.1080/03043797.2012.661701 Martin, D. S. (1989). Restructuring teacher education programs for higher-order thinking skills. Journal of Teacher Education, (May-June), 1–8. doi: https://doi.org/10.1177/002248718904000301 Marzano, R. J., & Kendall, J. S. (2008). Designing & assessing educational objectives: Applying the new taxonomy. In D. Alpert (Ed.), Designing and assessing educational objectives. Retrieved from http://mirlyn.lib.umich.edu/Record/005808499 Munzil, M., Mustikasari, V. R., & Insani, M. D. (2016). Analisis kemampuan berfikir sains mahasiswa IPA ditinjau dari nilai UN IPA SMA. Unpublished report. Malang. Narayanan, S., & Adithan, M. (2015). Analysis of question papers in Engineering Courses with respect to hots (higher order thinking skills). American Journal of Engineering Education (AJEE), 6(1), 1–10. doi: https://d oi.org/10.19030/ajee.v6i1.9247 Roets, L., & Maritz, J. (2017). Facilitating the development of higher-order thinking skills (HOTS) of novice nursing postgraduates in Africa. Nurse Education Today, 49, 51–56. doi: https://doi.org/10.1016/j.nedt.20 16.11.005 Saido, G. M., Siraj, S., Nordin, A. B. Bin, & Al_Amedy, O. S. (2013). Higher order thinking skills among secondary school students in science learning. The Malaysian Online Journal of Educational Science, 3(3), 13–20. Retrieved from https://www.researchgate.net/publication/321423293%0AHigher Saul, C., & Wuttke, H.-D. (2011). Personalized assessment of higher-order thinking skills. CSEDU 2011 - Proceedings of the 3rd International Conference on Computer Supported Education. doi: https://doi.org/ 10.5220/0003480204250430 Tanujaya, B., Mumu, J., & Margono, G. (2017). The relationship between higher order thinking skills and academic performance of student in Mathematics instruction. International Education Studies, 10(11), 78- 85. doi: https://doi.org/10.5539/ies.v10n11p78 Tiantong, M., & Siksen, S. (2013). The online Project-based Learning model based on student’s multiple intelligence. Clinical Insights: Stereotactic Body Radiation Therapy: Lung Cancer, 3(7), 53–72. https://doi. org/10.2217/EBO.13.11 Tonissen, K. F., Lee, S. E., Woods, K. J., & Osborne, S. A. (2014). Development of scientific writing skills through activities embedded into biochemistry and molecular biology laboratory courses. International Journal of Innovation in Science and Mathematics Education, 22(4), 1–14. Retrieved from https://open journals.library.sydney.edu.au/index.php/CAL/article/view/7564/8365 Waugh, R. (2010). Applications of rasch measurement in education. In Applications of Rasch Measurement in Education. Retrieved from http://www.novapublishers.org/catalog/product_info.php?products_id=25176 https://doi.org/10.7763/ijssh.2011.v1.20 https://doi.org/10.1016/j.sbspro.2012.09.265 https://doi.org/10.1080/15391523.2001.10782338 https://doi.org/10.1080/15391523.2001.10782338 https://doi.org/10.12973/iji.2018.11217a https://doi.org/10.12973/iji.2018.11217a https://doi.org/10.5897/err2014.1777 https://doi.org/10.17977/jpb.v7i2.723 http://www.aabri.com/copyright.html https://www.socialstudies.org/system/files/publications/articles/yl_220115.pdf https://www.socialstudies.org/system/files/publications/articles/yl_220115.pdf https://doi.org/10.1080/03043797.2012.661701 https://doi.org/10.1177/002248718904000301 http://mirlyn.lib.umich.edu/Record/005808499 https://doi.org/10.19030/ajee.v6i1.9247 https://doi.org/10.19030/ajee.v6i1.9247 https://doi.org/10.1016/j.nedt.2016.11.005 https://doi.org/10.1016/j.nedt.2016.11.005 https://www.researchgate.net/publication/321423293%0AHigher https://doi.org/10.5220/0003480204250430 https://doi.org/10.5220/0003480204250430 https://doi.org/10.5539/ies.v10n11p78 https://doi.org/10.2217/EBO.13.11 https://doi.org/10.2217/EBO.13.11 https://openjournals.library.sydney.edu.au/index.php/CAL/article/view/7564/8365 https://openjournals.library.sydney.edu.au/index.php/CAL/article/view/7564/8365 http://www.novapublishers.org/catalog/product_info.php?products_id=25176