Indonesian Journal of Learning and Instruction Volume 2, Issue 1, April 2019 p-ISSN 2614-8250, e-ISSN 2614-5677 https://journal.uniku.ac.id/index.php/IJLI 65 THE ANALYSIS OF ITEM PROBLEMS IN HIGH SCHOOL MATHEMATICS TEXTBOOK IN INDONESIA (2016 REVISION EDITION) REVIEWED FROM THE COGNITIVE ASPECT OF TIMSS Siti Khodaria Department of Math Education, Universitas Swadaya Gunung Djati, Cirebon, Indonesia E-mail: sitihakhoridah@gmail.com Anggita Maharani Department of Math Education, Universitas Swadaya Gunung Djati, Cirebon, Indonesia E-mail: anggi3007@yahoo.co.id H. Sulaiman Department of Math Education, Universitas Swadaya Gunung Djati, Cirebon, Indonesia APA Citation: Khodaria, S., Maharani, A., & Sulaiman, H. (2019). The analysis of item problems in high school Mathematics textbook in Indonesia (2016 revision edition) reviewed from the cognitive aspect of TIMSS. Indonesian Journal of Learning and Instruction, 2(1), 65-70. doi: 10.25134/ijli.v2i01.1685. Received: 23-12-2018 Accepted: 20-02-2019 Published: 01-04-2019 Abstract: This study aims to analyze the question items on two supplementary Indonesian mathematics textbooks (revised edition 2016) of 2013 curriculum for XI graders and compare the analysis using 2015 TIMSS assessment framework. This study apply qualitative study with naturalistic approach. The analysis results consist of 104 question items and 85 items of practice test. On a book published by PT. SEWU Bandung shows 3.5% of knowing cognitive domain, 4.71% applying, and 91.76% reasoning. The analysis on 19 items of practice test on a book is published by PT. Bumi Aksara Jakarta. It shows 10.53% applying, 89.47% reasoning, but not showing a percentage on knowing cognitive domain. Results of the question items analysis on knowing and applying cognitive domain for both the books are less percentage than reasoning cognitive domain. Thus, the results of analysis of two textbooks are not yet in accordance with TIMSS. Nevertheless, the comparison of the analysis on knowing cognitive domain in a book published by PT. SEWU Bandung is close to what have expected by TIMSS. Meanwhile, on applying and reasoning cognitive domain the book published by PT. Bumi Aksara Jakarta is close to what have expected by TIMSS. Keywords: cognitive aspects of TIMSS; dimensions; domains; items; textbooks. INTRODUCTION Mathematics has a vital role in the world of education because all branches of science use it both in the scale of theory and in the implementation of daily life. In recent years, students' cognitive abilities have become a measuring tool to determine the level of educational progress of countries. As a result, the role of mathematics becomes essential. Therefore, every country needs to evaluate its national education system to find out how far the success level that has been carried out. To evaluate the quality of education, since 1999 Indonesia has participated in international research is organized by the Trends International Mathematics and Science Study (TIMSS). TIMSS is an international study of mathematics and science which researches every countries member (Wardhani, 2014). The research is every four years for fourth and eighth-grade mailto:anggi3007@yahoo.co.id mailto:anggi3007@yahoo.co.id https://doi.org/10.25134/ijli.v2i01.1685 Siti Khodaria, Anggita Maharani, & H. Sulaiman The analysis of item problems in high school Mathematics textbook in Indonesia (2016 revision edition) reviewed from the cognitive aspect of TIMSS 66 students. There are two dimension characteristics of TIMSS, that is, content and cognitive. Content is an assessment of the feasibility of the material presented both in textbooks and in the learning process. The cognitive is an assessment of students' thinking abilities which include three domains; knowing, applying, and reasoning. Based on Indonesia's participation in 2015's TIMSS international study, the quality of education in Indonesia is considered to be not optimal. Indonesia student’s competence is still considered as weak (Priyani & Ekawati, 2018). Of the 49 countries, Indonesia was ranked 44th in mathematics achievement with an average score of 397 points from the international average of 500 points. Indonesia's average score lags far behind neighbour countries, such as Singapore, Malaysia and Thailand, which each score 611, 440 and 427 points. This result is not much different from Indonesia's participation in previous years that presented in the following tables. Table 1. Indonesian participation results in TIMSS Tahun Indonesia’s rank Countries Member Average Score Indonesia International Difference 1999 34 38 403 point 487 point 84 point 2003 35 46 411 point 467 point 56 point 2007 36 49 397 point 500 point 103 point 2011 38 42 386 point 500 point 114 point 2015 44 49 397 point 500 point 103 point Based on the table above, the mathematics achievement of Indonesian students is left behind and underdeveloped. The Indonesian government's efforts in addressing this by improving the quality of education through the curriculum in the national education system which merged in the form of learning tools. Textbooks as one element contain questions that important as a measuring tool to train students' cognitive level abilities. However, in the implementation of the 2013 curriculum revised edition of 2016, required textbooks for learning resources in schools both the teacher and students handle, in fact, until now it has not been distributed evenly by the government. So that the teacher in the learning process at school only uses commercial books from various kinds of publishers available in bookstores, while students use the Student Worksheet book and only rely on the explanations from the teacher in the learning process. Thus, the mathematics textbooks used in schools have not been as appropriate. According to Pepin in Cahyono and Adilah (2016), most teachers rely on the textbooks in the learning process, decide what to teach, how to teach, and only practice questions for students based on the textbooks they choose. So, the only source for learning is textbooks from the teacher even though there are many other sources. A textbook is essential and strategic to improve the quality of education concerning students' mathematical achievements at school. It is necessary to conduct a study of analysis or study of mathematics textbooks. The study not only focuses on the dimensions of content but also focused on the quality of the items that are following the cognitive dimensions of the TIMSS assessment framework. METHOD The research method used is the naturalistic method involving two reference books or two research data sources. the achievement of the cognitive aspects of TIMSS was measured based on the items contained in the final evaluation form in the form of multiple choice questions in the conical section chapter covering circles, satellite dishes, ellipses, and hyperboles. Data collection is carried out by observation with natural conditions derived from primary data. Spradley’s stages were carried out in this study, namely the stages of description, Indonesian Journal of Learning and Instruction Volume 2, Issue 1, April 2019 p-ISSN 2614-8250, e-ISSN 2614-5677 https://journal.uniku.ac.id/index.php/IJLI 67 reduction, and selection. Data analysis is carried out according to spradley’s stages namely domain, taxonomy, and compound analysis. Objectivity of the findings is then analyzed through credibility tests (internal validity), transferability (external validity), dependability (reliability) and confirmability (objectivity). RESULTS AND DISCUSSION Mathematics textbook published by PT. SEWU Bandung has some questions on the conic section which is found in the final evaluation of the chapter/competency test in the form of objective (multiple choice) as many as 87 items. Whereas in the mathematics textbook published by PT. Bumi Aksara Jakarta, the conic section questions are 25 items. As much as 104 of 112 total numbers of items have investigated. Eight other items are not to analyse, because the questions had no answers, so they did not contain the cognitive aspects of TIMSS. The results of domain analysis are in the following table. Tabel 2. Recapitulation of items based on TIMSS cognitive domains Publisher Assessment Framework TIMSS 2015 Total Items Knowing Applying Reasoning ∑ Item % ∑ Item % ∑ Item % ∑ Item % PT. SEWU, Bandung 3 3,53% 4 4,71% 78 91,76% 85 100% PT. Bumi Aksara, Jakarta 0 0% 2 10,53% 17 89,47% 19 100% Based on the table above, the book published by PT. SEWU Bandung, out of 85 items analyzed, resulting in three items or 3.53% for the knowing cognitive domain, four items or 4.71% for applying, and 78 items or 91.76% for reasoning. Whereas in the book published by PT. Bumi Aksara Jakarta, of the 19 items analyzed there was no knowing cognitive domain, but produced two items or 10.53% for applying, and 17 items or 89.47% for reasoning. The results of the above domain analysis are presented more fully on the results of the taxonomic and compound analysis presented below. Table 3. Recapitulation of question items based on cognitive aspects of TIMSS Cognitive Domain Cognitive Aspects Publisher PT. SEWU, Bandung PT. Bumi Aksara, Jakarta ∑ Item % ∑ Item % K n o w in g Recall 0 - 0 - Recognize 0 - 0 - Compute 1 1,18% 0 - Retrieve 0 - 0 - Classify/order 2 2,35% 0 - Measure 0 - 0 - A p p l y -i n g Determine 0 - 0 - Represent/model 1 1,18% 0 - Implementation 3 3,53% 2 10,53% R e a so n in g Analyze 0 - 0 - Integrated/synthesize 55 64,70% 13 68,42% Evaluate 0 - 0 - Draw conclusions 12 14,12% 1 5,26% Generalize 0 - 0 - Justify 11 12,94% 3 15,79% Total 85 100% 19 100% Based on the table above, the results of the analysis of the "knowing" domain in the book published by PT. SEWU Bandung contains cognitive aspects of "compute" one Siti Khodaria, Anggita Maharani, & H. Sulaiman The analysis of item problems in high school Mathematics textbook in Indonesia (2016 revision edition) reviewed from the cognitive aspect of TIMSS 68 item or 1.18%, and "classify/order" two items or 2.35%. For items in the "applying" domain, the cognitive aspects "represent/model" one item or 1.18%, and the "implementation" is three items or 3.53%. The items in the "reasoning" domain contain cognitive aspects of "integrated/synthesise" is 55 items or 64.70%, "draw conclusions" is 12 items or 14.12%, and "justify" is 11 items or 12.94%. Furthermore, for the results of the analysis on "knowing" domain does not contain any cognitive aspects in the book published by PT. Bumi Aksara Jakarta. For items in the "applying" domain, only the aspects of cognitive "implementation" as much as two items or 10.53%. The items in the "reasoning" domain contain cognitive aspects "integrated/synthesise" is13 items or 68.42%, one item of "draw conclusions" or 5.26% and "justify" is three items or 15.79%. The comparison results of the analysis of the items in the two mathematics textbooks with specialisation in high school class XI 2013 curriculum (revised edition 2016) are in the following table. Table 4. Comparative results from analysis of question items Cognitive Domain The expectation of TIMSS 2015 Analysis Result Difference Close to TIMSS A B A B Knowing 35% 3,53% 0% −31,47% −35% A Applying 40% 4,71% 10,53% −35,29% −29,47% B Reasoning 25% 91,76% 89,47% 66,76% 64,47% B Informati on: A= PT. SEWU, Bandung; B= PT. Bumi Aksara, Jakarta. The difference between the results of the analysis of textbooks and TIMSS, which obtained a lower percentage of the expectations of TIMSS. The difference in the results of the analysis of textbooks and TIMSS, which obtained a percentage higher than the expectations of TIMSS. Based on the table above, there is no single cognitive domain in the book, which corresponds to the TIMSS achievement target. The results of the analysis of the two mathematics textbooks obtained scores of far percentages as expected by TIMSS, so the mathematics curriculum in Indonesia does not refer to TIMSS. In "knowing" domain, the results of the analysis of books published by PT. SEWU Bandung reached 3.53% and 0% for the results of the analysis of books published by PT. Bumi Aksara Jakarta, while expected TIMSS 35%, so that the difference of -31.47% for books published by PT. SEWU Bandung and -35% for books published by PT. Bumi Aksara Jakarta, where -35% <-31.47% <35%. In the "applying" domain, the results of the analysis of the book published by PT. SEWU Bandung reached 4.71% and the book published by PT. Bumi Aksara Jakarta is 10.53%, while TIMSS is expected to be 40% so that the difference of -35.29% for books published by PT. SEWU Bandung and - 29.47% for books published by PT. Bumi Aksara Jakarta, where -35.29% <-29.47% <40%. In reasoning domain results from the analysis of books published by PT. SEWU Bandung reached 91.76% and the book published by PT. Bumi Aksara Jakarta reaches 89.47%, while TIMSS expected 25%, so the difference of 66.76% for books published by PT. SEWU Bandung and 64.47% for books published by PT. Bumi Aksara Jakarta, where 25% <64.47% <66.76%. Based on the difference in the analysis of the two books, the "knowing" domain of the book published by PT. SEWU Bandung is closer to what TIMSS expected. Whereas in the "applying" and "reasoning" domains, the results of the analysis from the book published by PT. Bumi Aksara Jakarta is close to what TIMSS expected. Basic Competence (BC) on cone slices material written in the syllabus at the level of SMA/MA class XI 2013 curriculum (revised Indonesian Journal of Learning and Instruction Volume 2, Issue 1, April 2019 p-ISSN 2614-8250, e-ISSN 2614-5677 https://journal.uniku.ac.id/index.php/IJLI 69 edition 2016). BC 3.3 analyses cone slices (circles, ellipses, parabola, and hyperbole), and BC 4.3 complete problems related to cone slices. In the two BC, each contains indicators "analyzing" and "solving problems", which are in the cognitive dimension TIMSS, that is, the "reasoning" domain. Following the results of the analysis in this study, the percentage of cognitive "reasoning" domains was 91.76% for books published by PT. SEWU Bandung and 89.47% for books published by PT. Bumi Aksara Jakarta. The results of the study are dominated by the percentage of cognitive "reasoning" domains so that they are not as expected by TIMSS. Several other studies at the junior high school level also produced the same conclusions, although the Ministry of Education and Culture published the books analysed in the study. The results of the research from Padmawati on class IX mathematics textbooks semester I and II and research from in the first semester VIII mathematics textbooks obtaining the "knowing" cognitive domain higher than expected TIMSS meanwhile, "applying" and "reasoning" are lower than TIMSS expected (Padmawati, Murtiyasa, & Kom, 2017; Cahyono & Adilah, 2016). It is inversely proportional to the results of Rohmah (Rohmah & Murtiyasa, 2017), which conducted research on junior high school mathematics national exam items in 2015/2016 and obtained "knowing" lower than expected TIMSS, with higher "applying" and "reasoning" domains than expected TIMSS. Ideally, the result of cognitive aspects research of the items in the junior high school mathematics textbook published by the Ministry of Education and Culture in the year of 2015 is following the results research of the 2015/2016 junior high school national exam items. The results indicate that the mathematics curriculum in Indonesia has not referred to TIMSS, so it is necessary to make adjustments. It is opposite the Murtiyasa’s opinion that Indonesia had renewed mathematics lessons in the 2013 curriculum by referring to TIMSS(Murtiyasa, 2015). According to the study(Tatsuoka, Corter, & Tatsuoka, 2004), Indonesian students ranked 18th out of 20 TIMSS sample countries in setting the average standard for composite achievement variables. They obtained low scores on skills process, spatial, and reading. So the TIMSS scores was ambiguous for them(Meisenberg & Woodley, 2013). It indicates that Indonesian students are not accustomed to completing items with the characteristics of TIMSS. According to Yeom (Mailizar, Alafaleq, & Fan, 2014), since the 1970s the policy of education has been changing in the context of the expansion of human resources for it national development purposes. In 2015, Indonesia collaborated with the Asian Economic Community (AEC) with countries in Southeast Asia. Hutabarat say that AEC is a form of economic integration of the Association of Southeast Asian Nations (ASEAN), whose goal is to produce a production-based market by 2020 with the free movement of services, goods, capital, investment and skilled labour in the Southeast Asia region(Ariyanti, 2016). One of the most significant challenges related to the implementation of AEC is human resource readiness(Suratman, Trisnawati, & Wulandari, 2016). The AEC opens both opportunities and challenges for educated Indonesian workers to have work in ASEAN countries. Regarding work preparation, the teacher took the initiative to choose the subject matter in school based on students' needs in facing AEC challenges (Ariyanti, 2016). On this basis, it does not rule out the possibility that the curriculum in Indonesia is designed to prepare Indonesian workers in the face of AEC. So it is appropriate if the implementation of the curriculum in Indonesia does not pay attention and does not adjust as expected by TIMSS international level research studies. Although Indonesia has participated in the TIMSS international study from 1999 to the present and obtained the value of the conclusion that Indonesian Siti Khodaria, Anggita Maharani, & H. Sulaiman The analysis of item problems in high school Mathematics textbook in Indonesia (2016 revision edition) reviewed from the cognitive aspect of TIMSS 70 students' mathematics achievements are lagging behind and underdeveloped. CONCLUSION The percentage of achievement of the items problem in the two high school (SMA/MA) mathematics textbooks in class XI 2013 curriculum (revised edition 2016) is not following the expected TIMSS 2015 assessment framework. From the results of the analysis of the items in the two textbooks, there is no cognitive domain obtained in the book, which is following the target percentage of TIMSS. The results of the analysis are far-reaching percentage values as expected by TIMSS. Consequently, the curriculum applied in Indonesia does not refer to TIMSS international studies but is designed to prepare Indonesian workers in facing AEC. The results of the comparison of the analysis of the items in the two mathematics textbooks show that the "knowing" domain in the book published by PT. SEWU Bandung is closer to what TIMSS expected. Whereas in the "applying" and "reasoning" domains, the results of the analysis from the book published by PT. Bumi Aksara Jakarta is close to what TIMSS expected. So in this case, the book published by PT. Bumi Aksara Jakarta contains closer quality items like those expected by TIMSS international studies. Badan Standar Nasional Pendidikan (BSNP) as the government representative in assessing the feasibility of a book should consider the quality of the items problem in the mathematics textbooks. The items are assessed based on the cognitive aspects of the international research. The BSNP needs to improve the curriculum by collaborating with international research institutions (related to the implementation of mathematics learning). They involvement are to increase the mathematics ratings of Indonesian students in the TIMSS research study. REFERENCES Ariyanti, R. (2016). Exploring opportunities and challenges in incorporating sociocultural issues into ELT in a Vocational High School in Salatiga (Unpublished thesis). Universitas Kristen Satya Wacana, Salatiga. Cahyono, B., & Adilah, N. (2016). Analisis soal dalam buku siswa matematika kurikulum 2013 kelas VIII semester I berdasarkan dimensi kognitif dari TIMSS. JRPM (Jurnal Review Pembelajaran Matematika), 1(1), 86–98. Mailizar, M., Alafaleq, M., & Fan, L. (2014). A historical overview of mathematics curriculum reform and development in modern Indonesia. Inovacije U Nastavi-Časopis Za Savremenu Nastavu, 27(3), 58–68. Meisenberg, G., & Woodley, M. A. (2013). Are cognitive differences between countries diminishing? Evidence from TIMSS and PISA. Intelligence, 41(6), 808–816. Murtiyasa, B. (2015). Tantangan pembelajaran matematika era global. Prosiding Seminar Nasional Matematika dan Pendidikan Matematika Universitas Muhammadiyah Surakarta. Padmawati, A., A., Murtiyasa, B., & Kom, M. (2017). Analisis deskriptif butir soal pada buku ajar matematika kelas IX kurikulum 2013 ditinjau dari aspek kognitif TIMSS (Unpublished paper). Universitas Muhammadiyah Surakarta, Surakarta. Priyani, H. A., & Ekawati, R. (2018). Error analysis of mathematical problems on TIMSS: A case of Indonesian secondary students. IOP Conference Series: Materials Science and Engineering, 296, 12-10. Rohmah, I. F. Z., & Murtiyasa, B. (2017). Pemetaan aspek kognitif trends Internasional mathematics and science study pada ujian nasional matematika SMP 2015/2016. Universitas Muhammadiyah Surakarta, Surakarta. Suratman, B., Trisnawati, N., & Wulandari, S. S. (2016). Revolusi mental dan kurikulum pendidikan ekonomi dalam menghadapi MEA. Prosiding Seminar Nasional Himpunan Sarjana Ilmu-ilmu Sosial, 1, 124–131. Tatsuoka, K. K., Corter, J. E., & Tatsuoka, C. (2004). Patterns of diagnosed mathematical content and process skills in TIMSS-R across a sample of 20 countries. American Educational Research Journal, 41(4), 901–926. Wardhani, S. (2014). Rumiati 2011 instrumen penilaian hasil belajar matematika SMP: Belajar dari PISA dan TIMSS. Kementerian Pendidikan Nasional.