http://pmat.ustjogja.ac.id/jurnal/index.php/indomath Vol 4, No. 2, August 2021, pp. 81-94 Copyright © Authors. This is an open access article distributed under the Attribution-NonCommercial- ShareAlike 4.0 International (CC BY-NC-SA 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Mathematical Refractive Thinking Ability in Multivariable Calculus: Instrument Design and Quality of Use Isra Nurmai Yenti Pendidikan Matematika, Institut Agama Islam Negeri Batusangkar, isranurmaiyenti@iainbatusangkar.ac.id Yaya S. Kusumah Pendidikan Matematika, Universitas Pendidikan Indonesia, yayaskusumah@yahoo.com Jarnawi Afgani Dahlan Pendidikan Matematika, Universitas Pendidikan Indonesia, jarnawi@upi.edu ABSTRACT Mathematical refractive thinking ability must be given in every class because it is needed in decision-making. This study aims to develop a mathematical refractive thinking ability instrument for multivariable calculus material. This study is a research and development (R & D) with three research steps: instrument design, expert instrument validity testing, and pilot testing. This study involved 3 material experts and 32 mathematics education students using the purposive sampling technique. Analysis of research data using the Q-Cochran test, Product Moment correlation, and Cronbach's alpha. The results of the design and development of the mathematical refractive thinking ability instrument are in the form of four essay tests. After the trial was conducted, this study produced three test items for mathematical refractive thinking ability that was valid and reliable. Keywords: Mathematical Refractive Thinking Ability, Multivariable Calculus, Validity, Reliability ABSTRAK Kemampuan berpikir refraktif matematis harus diberikan pada setiap kelas perkuliahan karena sangat dibutuhkan dalam pengambilan keputusan. Penelitian ini bertujuan untuk mengembangkan instrumen kemampuan berpikir refraktif matematis untuk materi kalkulus multivariabel. Penelitian ini merupakan penelitian dan pengembangan (R & D) dengan tiga langkah penelitian, yaitu: merancang instrumen, melakukan validitas instrumen oleh pakar, dan melakukan ujicoba. Penelitian ini melibatkan 3 orang ahli materi dan 32 orang mahasiswa pendidikan matematika dengan teknik purposive sampling. Analisis data penelitian menggunakan uji Q-Cochran, korelasi Product Moment, dan Cronbach's alpha. Hasil perancangan dan pengembangan instrumen kemampuan berpikir refraktif matematis berupa empat butir tes uraian. Setelah dilakukan ujicoba, penelitian ini menghasilkan tiga butir tes kemampuan berpikir refraktif matematis yang valid dan reliabel. Kata Kunci: Kemampuan Berpikir Refraktif Matematis, Kalkulus Multivariabel, Validitas, Reliabilitas INTRODUCTION The Indonesian National Qualification Framework establishes the description of qualifications for level 6, one of which is being able to make right decisions based on analysis of information and data, and being able to provide guidance in choosing alternative solutions independently and in groups (Presidential Regulation of the Republic of Indonesia Number 8). Lecturers must train students to make appropriate and professional decisions based on the results of the analysis of information and data in all mathematics courses. Determining right decision in solving problems of learning mathematics, requires a variety of considerations and settlement options. This is the main goal of refraction, because the main purpose 82 Isra Nurma Yenti, Yaya S. Kusumah, and Jarnawi Afgani Dahlan Mathematical Refractive Thinking Ability in Calculus: Instrument Design and Quality of Use of refraction is to offer various solutions, considerations and / or observations of the problems encountered (Pagano & Roselle, 2009). Refraction is a mental activity experienced by someone to make decisions through reflective thinking and critical thinking (Maslukha et al., 2018). As a mental activity, refraction can be categorized as a type of thinking. Thinking that is characterized by reflective thinking followed by critical thinking and producing decisions is called refractive thinking (Prayitno, 2015). Thus, refractive thinking is critical thinking about things that are obtained from reflective thinking to produce decisions. According to Prayitno (2015), the main components of refractive thinking are reflective thinking and critical thinking. The results of the refractive thinking construction (Prayitno et al., 2014) are divided into three categories, as shown in Table 1. The problem identification component combines description and problem definition from reflective thinking with information exploration from critical thinking. Recording the information given in the problem, organizing the information, visualizing the information in mathematical symbols, and recognizing some concepts or principles inherent in the problem are part of the problem identification component (Prayitno, et al., 2014). Components compile the strategy obtained from the results of information gathering and linking some relevant information. Formulating a strategy can be done by identifying the relationship between information, several statements or concepts, and elements that are considered important in solving problems, even proposing some possible solutions to problems. The component of evaluating is the combination of the component believing the conclusion in reflective thinking with the component evaluating and clarifying critical thinking. Evaluating means evaluating the information generated based on relevant information to make a decision. Table 1. Refractive Thinking Construction Components of Reflective Thinking Components of Critical Thinking Components of Refractive Thinking Description of problem Define the problem Exploration of information Identification of problem Collection Relevance of information Strategy Conclusion belief Evaluation Clarification Evaluation Refractive thinking will arise when someone is given an open-ended problem (Prayitno, 2015). The open-ended problem has several alternatives for the initial and / or final situation and the solution method (Laine et al., 2018). When students are asked to solve open-ended problems, they may find several ways to solve the problem, then they analyze several ways to solve the problem. These solutions can be mutually supportive and complementary to one another or focused on choosing one method until a result (decision) is found. Refractive thinking in solving mathematical problems or mathematical refractive thinking can be measured through the ability of mathematical refractive thinking. The ability of mathematical refractive thinking is the ability of a person to solve complex mathematical problems through the analysis of critical thinking of some alternative solutions obtained from reflective thinking. For example, students are asked to solve the problem: “A sphere with a center point (2,4,6) is in the first Indomath: Indonesian Mathematics Education – Volume 4 | Issue 02 | 2021 83 octane. The sphere touches one of the coordinate planes. Move the sphere so that its center is at the origin and then draw the sphere that has been shifted." This problem is complex because it contains several concepts, namely sphere, Cartesian coordinates in three-space, translation, and drawing graphs in three-space. The problem is also open-ended because a sphere can touch the xy- plane, or yz-plane, or xz-plane. To solve the problem, students can draw a sphere and perform translations. In addition, students can also use the equation of a sphere and draw a sphere graph by referring to the equation of a sphere. Because there are multiple possible solutions, students must think critically when selecting the tangent planes and settlement strategy to reach the correct result. Thus, the above problems can demonstrate students' mathematical refractive thinking processes by addressing mathematical refractive thinking ability. Mathematical refractive thinking ability is measured through 8 indicators, namely: 1) identifying mathematical problems, 2) interpreting mathematical situations rationally, 3) representing ideas in the form of mathematical symbols, figures or tables, 4) identifying the relationship between statements, questions and mathematical concepts, 5) proposes several possible alternative solutions in solving mathematical problems, 6) chooses alternative solutions that are most relevant and is used to solve mathematical problems, 7) evaluates alternative solutions and answers produced by considering relevant information, and 8) explain again the information generated. Indicators (1), (2) and (3) to measure components identify problems, indicators (4), (5) and (6) to measure components compile strategies, while indicators (7) and (8) to measure components conduct evaluations . Several studies have examined the refractive thinking in solving mathematical problems (Maslukha et al., 2018; Prayitno, 2015; Prayitno et al., 2014). The process of refractive thinking has been investigated on data problems (Prayitno, 2015) and geometry (Maslukha et al., 2018; Prihati & Wijayanti, 2017). The research above resulted in the process of refractive thinking with one strategy, two strategies, and many strategies (Prayitno, 2015; Prayitno et al., 2014) as well as the occurrence of differences in refractive thinking based on mathematical ability (Maslukha et al., 2018) and type Keirsey's personality (Prihati & Wijayanti, 2017). No research has been found that addresses mathematical refractive thinking ability in calculus concepts. Calculus courses are a prerequisite for several advanced mathematics courses (Kashefi, et al., 2012). If students' mathematical refractive thinking abilities are still weak in calculus material, it is suspected that they also experience problems with these abilities in advanced mathematics. Mathematical refractive thinking ability was evaluated on multivariable calculus material in this study because the learning outcomes of multivariable calculus included critical thinking, analytical thinking, and creative thinking in problem-solving, as well as confidence in decision making. As a result, a thorough examination of students' mathematical refractive thinking ability in calculus content, particularly multivariable calculus, is required. To obtain correct and unbiased data on mathematical refractive thinking ability, researchers need to develop an instrument for mathematical refractive thinking ability that meets quality standards. The quality of measurement instruments is determined by validity and reliability (Colton & Covert, 2007; Karanicolas et al., 2009; Scholtes et al., 2011; Kimberlin & Winterstein, 2008) and responsiveness (Scholtes et al., 2011; Kimberlin & Winterstein, 2008). However, in this study, 84 Isra Nurma Yenti, Yaya S. Kusumah, and Jarnawi Afgani Dahlan Mathematical Refractive Thinking Ability in Calculus: Instrument Design and Quality of Use researchers only measured the quality of instruments validly and reliably, because valid and reliable instruments could provide accurate information about the weaknesses and shortcomings of educational programs (Setiawan & Mardapi, 2019). The validity and reliability of the instruments are two things that are important in improving the effectiveness of the data collection process (Sugiyono, 2015; Burton & Mazerolle, 2011; Pardimin et al, 2017; Purwaningsih et al, 2018). Validity requires a reliable instrument, but an instrument can be reliable even though it is not valid (Kimberlin & Winterstein, 2008). Validity is related to the ability of instruments to measure what should be measured while reliability is related to the consistency of measurements (Colton & Covert, 2007). In this study, the validity used is content validity, construct validity, face validity, and empirical validity. Content validity is the degree to which an instrument is representative of the topic and process being investigated (Colton & Covert, 2007). The validity of the contents for the instrument in the form of a test can be done by comparing the contents of the instrument with the material taught (Sugiyono, 2015; Lestari & Yudhanegara, 2015). Construct validity is the validity associated with the ability of a measuring instrument to measure the understanding of a concept that is measured (Colton & Covert, 2007; Sugiyono, 2015). Face validity is the accuracy of the arrangement of sentences or words used on a question item or statement in an instrument (Lestari & Yudhanegara, 2015). Empirical validity is validity obtained through observation or observation that is empirical and reviewed based on certain criteria (Lestari & Yudhanegara, 2015). Empirical validity in this study aims to see the use of instruments in small groups in the field. The reliability used in this study is internal reliability (internal consistency) which describes consistency between instrument items (Gall & Borg, 2003). Internal consistency allows researchers to compare results between items from one instrument and only one instrument trial (Colton & Covert, 2007). Thus, the purpose of this research is to develop a valid and reliable instrument of mathematical refractive thinking ability for multivariable calculus. METHOD This research belongs to the research and development (R & D) group. R & D aims to produce new technologies, services, products, or systems (Mumu & Tanujaya, 2019). The focus of this paper is to design a test instrument for mathematical refractive thinking abilities and scoring criteria, so that the result may be used as a guide for the development of similar instruments. The researchers in this study consisted of 3 stages of research: designing instruments, conducting instrument validity by experts, and pilot testing. Table 2 shows the activities involved in designing instruments, validating instruments, and pilot testing instruments. Testing the content validity, construct validity and face validity can be done using the opinions of experts (Sugiyono, 2015; Lestari & Yudhanegara, 2015; Azwar, 2016; Hoogland et al., 2016; Wichit et al., 2018). The instrument was designed for validity by three multivariable calculus experts (Hamdi et al., 2018). Indomath: Indonesian Mathematics Education – Volume 4 | Issue 02 | 2021 85 Table 2. Description of Activities Carried Out on the Design and Validation of the Instrument Research Prosedure Description of Research Activities Types of Instrument Quality Design 1. Selecting multivariable calculus material that can be combined into aproblem of mathematical refractive thinking ability 2. Designing open-ended problem according to indicators of mathematical refractive thinking ability Content validity Contruct validity Validity by experts 1. Checking the correctness of the material tested on the instrument 2. Adjusting questions and statements on the instrument with indicators of mathematical refractive thinking ability 3. Checking the accuracy of language use Content validity Contruct validity Face validity Pilot testing instruments 1. Conducting instrument pilot testing on 32 students 2. Calculating the Pearson Product Moment correlation coefficient 3. Calculating Cronbach's Alpha values Empirical validity and internal consistency The process of designing an instrument The process of designing the instrument begins with selecting multivariable calculus material. Researchers choose geometry material in space and derivatives for the function of two or more variables. These two materials are the starting material for multivariable calculus. Each indication of learning achievement is calculated by combining many indicators whose material is connected, rather than by a single problem of mathematics refractive thinking ability. Furthermore, a four-question problem was designed. These questions indicate eight indicators of mathematical refractive thinking ability. The validity of the instrument by experts Material experts need to provide an assessment of the content validity, construct validity and face validity of the mathematical refractive thinking ability instrument that has been designed. This aims to get the conformity of the instrument with the material provided, the objectives to be achieved, indicators of ability to be measured, the accuracy of the arrangement of sentences or words listed in the problem (language clarity / editorial) and clarity of the image (representation) so it is not confusing and has a double meaning. The material expert filled out the validation sheet on a two-value scale: 1 = valid instrument and 0 = invalid instrument. Expert validation data were processed using SPSS software Q-Cochran test. Instrument pilot testing The pilot testing of the instrument was limited. The recommended sample size for instrument trials is at least 10 respondents per item (Burton & Mazerolle, 2011). In this research, pilot testing participants numbered 32 people from 3 universities in West Sumatra, namely 12 students of Padang State Islamic University, 10 students of Padang State University and 10 students of Batusangkar State Islamic Institute. This selection is based on accreditation and students' willingness to become respondents. All participants are students who have taken multivariable calculus courses. 86 Isra Nurma Yenti, Yaya S. Kusumah, and Jarnawi Afgani Dahlan Mathematical Refractive Thinking Ability in Calculus: Instrument Design and Quality of Use Determination of the validity of each test item mathematical refractive thinking ability is done by calculating the correlation between the scores of each test item with its total score. Calculation of this correlation can be done using Product Moment Correlation from Pearson. Pearson's correlation coefficient is very suitable to be used for determining the validity of items on the measurement of student achievement (Muntholib et al., 2018). Researchers used the help of IBM SPSS 20 Software to calculate the Pearson’s correlation coefficient. The interpretation of the correlation coefficient follows Table 3 (Schober, Boer, & Schwarte, 2018). Table 3. Correlation Coefficient Criteria for Instrument Validity Correlation Coefficient Criteria 0,90 ≤ 𝑟𝑥𝑦 ≤ 1,00 Very strong 0,70 ≤ 𝑟𝑥𝑦 ≤ 0,89 Strong 0,40 ≤ 𝑟𝑥𝑦 ≤ 0,69 Moderate 0,10 ≤ 𝑟𝑥𝑦 ≤ 0,39 Weak 𝑟𝑥𝑦 < 0,1 Negligible Internal consistency tests of mathematical refractive thinking ability were calculated using Cronbach's alpha. Cronbach's alpha is a reliability test that requires only a single test administration to provide a unique estimate of the reliability for a given test (Gliem & Gliem, 2018). In addition, Cronbach's alpha is best used for instruments with a single construction scale and is less informative when reported for instruments that measure several constructs at once (Adams & Wieman, 2011; Taber, 2018). Cronbach's alpha calculation using the help of IBM SPSS 20. The category of reliability values obtained are: if less than 0.5 then the reliability is low, if more than or equal to 0.5 but less than or equal to 0.8 then the reliability is moderate, and if it is greater than 0.8 then reliability is high (Adams & Wieman, 2011; Salvucci et al., 1997). RESULT AND DISCUSSION The process of designing an instrument Before researchers design instruments of mathematical refractive thinking ability, researchers need to prepare test specifications in the form of determining the test objectives, compiling the grid and selecting the form of the test (Harjo et al., 2019). The test specification serves as a practical guide for the test compiler in planning the content of the subject being tested, aspects of the behavior being measured, the form of the test, and the length of the test (Mardapi, 2008). In addition, there are several steps that need to be considered to get a good test, namely: 1) identification of objectives and scope of measurement, 2) mapping of basic competencies and indicators, 3) determining competency limits to be measured, 4) determining test design based on measurements which will be used, 5) making the mathematics test instrument grids, 6) making the mathematical essay according to the plan (Hamdi et al., 2018). In this study, researchers designed the mathematical refractive thinking ability instrument for geometrical material in space and its derivatives for the function of two or more variables. Learning outcomes indicators for geometry in space are able students: 1) Explain Cartesian coordinates in Indomath: Indonesian Mathematics Education – Volume 4 | Issue 02 | 2021 87 three-space; 2) Determine the distance of two points in three-space; 3) Explain about and their equations; 4) Draw surfaces in three-space; 5) Explain the cylindrical and spherical coordinate system; 6) Change the coordinates of a point from the Cartesian coordinate system into a cylindrical or spherical coordinate system and vice versa; and 7) Change an equation of the Cartesian coordinate system into a cylindrical or spherical coordinate system and vice versa. While learning outcomes indicators for differential for functions of two or more variables are able students: 1) Explain the form of functions of two or more variables; 2) Determine the domain of the functions of two or more variables; 3) Describe a graph of the functions of two variables using the level curve (contour map); 4) Determine the partial derivatives of the functions of two or more variables; 5) Determine the limits and continuity of the functions of two variables; 6) Determine the derivatives of a function of two or more variables with chain rules; 7) Determine the derivatives of implicit functions; 8) Determine the extreme value of a function of two or more variables using the second partial derivative test; and 9) Determine the extreme value of a function of two or more variables using the Lagrange multiplier method. This indicator is compiled based on the learning outcomes that have been set in the curriculum. Based on the indicators of learning achievement above, the researcher made four items of mathematical refractive thinking ability questions, that each question has four questions which are presented in Table 4. Question a is to measure indicators of mathematical refractive thinking numbers (1), (2) and (3). Question b to measure indicators of mathematical refractive thinking ability number (4). Question c to measure indicators of mathematical refractive thinking ability numbers (5) and (6). Question d to measure indicators of mathematical refractive thinking ability numbers (7) and (8). Scoring student answers refers to the guidelines in Table 5. Table 4. Grid Tests for Mathematical Refractive Thinking Ability Question number Problems Subject matter Indicator of subject matter 1 Seorang arsitek akan menggambar sketsa sebuah kubah mesjid di koordinat Cartesius dengan bidang-xy sebagai alasnya. Ia hanya diberitahu oleh pengurus mesjid bahwa kubah mesjid tersebut memiliki diameter alas 6 m dan tinggi kubah 4 m. Berikut ini diberikan beberapa persamaan sebagai alternatif yang dapat membantu arsitek untuk membuat gambar kubah mesjid. 𝑧 = √16 − 16 9 𝑥2 − 16 9 𝑦2 𝜌 = 3 16𝑟 2 + 9𝑧2 = 144 a. Apakah situasi matematis dari menggambar sketsa sebuah kubah mesjid sudah memadai? Jika sudah, tulislah semua informasi yang diberikan dalam bahasa matematika. b. Jelaskan konsep-konsep yang dapat digunakan untuk menyelesaikan masalah menggambar sketsa sebuah kubah mesjid tersebut! c. Tuliskan beberapa alternatif cara yang dapat dilakukan untuk menyelesaikan masalah menggambar sketsa sebuah kubah mesjid di atas! Berikan alasan terhadap Geometry in space 1, 2, 3, 4, 5, 6 and 7 88 Isra Nurma Yenti, Yaya S. Kusumah, and Jarnawi Afgani Dahlan Mathematical Refractive Thinking Ability in Calculus: Instrument Design and Quality of Use alternatif mana yang kamu pilih dan selesaikan permasalahannya berdasarkan cara tersebut. d. Periksa kembali hasil kerjamu dengan menggunakan informasi yang diberikan soal. Berikan penjelasan tentang langkah-langkah yang dilakukan. 2 Ada beberapa permukaan seperti bola, paraboloida eliptik dan konik eliptik yang memiliki peta kontur seperti berikut. a. Tafsirkan situasi matematis dari permukaan yang memiliki peta kontur di atas! Nyatakan dalam bahasa matematika! b. Jelaskan konsep-konsep yang dapat digunakan untuk menyelesaikan masalah permukaan yang memiliki peta kontur di atas! c. Tentukan 𝑓(𝑥, 𝑦) yang memiliki grafik seperti peta kontur di atas apabila daerah asal dari 𝑓(𝑥, 𝑦) adalah {(𝑥, 𝑦): 𝑥2 + 𝑦2 ≤ 9}. d. Periksalah bahwa 𝑓(𝑥, 𝑦) kontinu di titik asal. Differential for functions of two or more variables 1, 2, 3, 4 and 5 3 Secara umum bagian pohon yang digergaji menjadi kayu adalah bagian batangnya, yakni bentuk benda yang hampir menyerupai tabung lingkaran tegak. Jari-jari batang suatu pohon tertentu tumbuh 0,5 cm per tahun dan tinggi bertambah 8 cm per tahun. Berapa cepat volume bertambah ketika jari-jari batang 20 cm dan tinggi batang 400 cm? a. Apakah situasi matematis dari masalah penambahan volume batang kayu sudah memadai? Jika sudah, tulislah semua informasi yang diberikan dalam bahasa matematika. b. Jelaskan konsep-konsep yang dapat digunakan untuk menyelesaikan masalah penambahan volume batang kayu tersebut! c. Tuliskan beberapa alternatif cara yang dapat digunakan untuk menyelesaikan masalah penambahan volume batang kayu di atas! Berikan alasan terhadap alternatif mana yang kamu pilih dan selesaikan permasalahannya berdasarkan cara tersebut. d. Periksa kembali hasil kerjamu dengan menggunakan informasi yang diberikan soal. Berikan penjelasan tentang langkah-langkah yang dilakukan. Differential for functions of two or more variables 6 and 7 4 Anton ingin membuat kotak infak suatu mesjid. Biaya bahan untuk alas kotak tiga kali lebih mahal daripada biaya bahan untuk sisi dan atas kotak. Tentukan volume terbesar yang dimiliki kotak tersebut jika banyak uang yang tersedia Differential for functions of two or 8 and 9 Indomath: Indonesian Mathematics Education – Volume 4 | Issue 02 | 2021 89 adalah 1,2 juta rupiah dan biaya bahan untuk alas adalah Rp. 60.000,- per dm2. a. Tafsirkan situasi matematis dari masalah kotak infak suatu mesjid di atas! Nyatakan dalam bahasa matematika! b. Tulislah konsep-konsep yang dapat digunakan untuk menyelesaikan masalah kotak infak suatu mesjid tersebut! c. Tuliskan beberapa alternatif cara yang dapat dilakukan untuk menyelesaikan kotak infak suatu mesjid di atas! Berikan alasan terhadap alternatif mana yang kamu pilih dan selesaikan permasalahannya berdasarkan cara tersebut. d. Periksa kembali hasil kerjamu dengan menggunakan informasi yang diberikan soal. Berikan penjelasan tentang langkah-langkah yang dilakukan. more variables Table 5. Guidelines for Scoring Mathematical Refractive Thinking Ability Tests Indicators of Mathematical Refractive Thinking Ability Description of student answers Score No answer 0 Identifying mathematical problems Identify data that is known and asked for a problem 0-2 Interpreting mathematical situations rationally Interpret mathematical situations rationally 0-2 Representing ideas in the form of mathematical symbols, figures or tables Representing ideas in the form of mathematical symbols, figures or tables 0-2 Identifying the relationship between statements, questions and mathematical concepts Identify the relationship between statements, questions and mathematical concepts / principles contained in the problem 0-2 Proposes several possible alternative solutions in solving mathematical problems Identify several strategies that can be used to solve problems 0-2 Chooses alternative solutions that are most relevant and is used to solve mathematical problems Choose alternative solutions that are most relevant and used to solve problems 0-3 Evaluates alternative solutions and answers produced by considering relevant information Check the answers obtained by using known data 0-1 Explain again the information generated Make conclusions on the results of a given problem 0-1 One point score on mathematical refractive thinking ability 0-15 The validity of the instrument by experts Content validity, construct validity, and face validity is all evaluated by experts' judgment. The Q-Cochran test results using SPSS software can be seen in Table 6. 90 Isra Nurma Yenti, Yaya S. Kusumah, and Jarnawi Afgani Dahlan Mathematical Refractive Thinking Ability in Calculus: Instrument Design and Quality of Use Table 6. Q-Cochran Test Results for Content Validity, Construct Validity and Face Validity of the Mathematical Refractive Thinking Ability Test Statistics Content Validity Construct Validity Face Validity N 16 16 16 Cochran's Q 2.000a 2.000a 5.200a Df 2 2 2 Asymp. Sig. 0.368 0.368 0.074 a. 1 is treated as a success. From the test results in Table 6, it can be seen that the Asymp value. Sig. respectively 0.368, 0.368 and 0.074; these three values are greater than α = 0.05. This means that the experts agree on the content validity, construct validity, and face validity of the mathematical refractive thinking ability test. Table 7 shows some expert suggestions as well as the results of their revisions. Table 7. Suggestions for Improvement of the Mathematical Refractive Thinking Ability Test from Experts and the Revised Results Expert Suggestion Revision Learning achievement indicators about determining the distance of two points in three- space as well as determining the limit and continuous function of two variables are discarded because the two indicators are not measured in the problem Problem 1a “Is the mathematical situation of drawing a sketch of a mosque dome adequate? If so, write down all the information provided in mathematical language. "Is the mathematical situation of drawing a sketch of a mosque dome adequate? If so, identify the situation or given mathematical equation!" Problem 2a "Interpret mathematical situations from surfaces that have contour maps above! Write down in mathematical language!" "Identification of situations or mathematical equations from surfaces that have contour maps above!" Problem 2d "Check that 𝑓(𝑥, 𝑦) is continuous at the origin." "Check your answer by drawing a contour map for each alternative answer." Problem 3a "Is the mathematical situation of the problem of increasing the volume of trunk sufficient? If so, write down all the information provided in mathematical language." "Is the mathematical situation of the problem of increasing the volume of trunk sufficient? If so, identify the situation or given mathematical equation!" Problem 3d "Double check your work using the information provided about the problem. Provide an explanation of the steps taken." "Check your work by writing a conclusion about your answer." Problem 4a "Interpret the mathematical situation of the mosque infaq problem problem above! Express in mathematical language!" "Identify the situation or mathematical equation of the infaq box problem of a mosque above!" In general, many suggestions for improvement are given on face validity, that is, the sentence editor in question a. According to the expert, the sentence "Express in mathematical language" is less familiar to students, so students will have difficulty in answering these questions. While the sentence editor for the problem has been stated properly and correctly. An instrument is said to have good face validity if the arrangement of sentences or words in questions or statements is clear, can be understood and does not cause other interpretations (Lestari & Yudhanegara, 2015). In addition, face validity is related to the ability to read (Burton & Mazerolle, 2011; Drost, 2011). Another suggestion from experts is on content validity and construct validity. According to the expert, there are two indicators of learning outcomes that need to be discarded because they were not tested in the questions. The items in the test must cover the entire content domain to be Indomath: Indonesian Mathematics Education – Volume 4 | Issue 02 | 2021 91 measured (Azwar, 2016). In addition, there is a change of questions because it is not in accordance with indicators of mathematical refractive thinking ability. The content of the test items must match the contents of the items designed to be measured (Gall & Borg, 2003). Next, researchers improved the mathematical refractive thinking ability instrument in the editorial section of the question, grid and key answers to the mathematical refractive thinking ability test. The revision of this instrument will be tested in the field. Instrument pilot testing The revised instrument of validity by experts was tested on 32 students. The results of calculations using Pearson Product Moment correlation can be seen in Table 8. Table 8. Pearson Product Moment Correlations Score Item Number 1 Score Item Number 2 Score Item Number 3 Score Item Number 4 Total score Score Item Number 1 Pearson Correlation 1 0.743** 0.380* -0.070 0.845** Sig. (2-tailed) 0.000 0.032 0.702 0.000 N 32 32 32 32 32 Score Item Number 2 Pearson Correlation 0.743** 1 0.328 -0.192 0.767** Sig. (2-tailed) 0.000 0.067 0.293 0.000 N 32 32 32 32 32 Score Item Number 3 Pearson Correlation 0.380* 0.328 1 0.061 0.760** Sig. (2-tailed) 0.032 0.067 0.741 0.000 N 32 32 32 32 32 Score Item Number 4 Pearson Correlation -0.070 -0.192 0.061 1 0.119 Sig. (2-tailed) 0.702 0.293 0.741 0.516 N 32 32 32 32 32 Total score Pearson Correlation 0.845** 0.767** 0.760** 0.119 1 Sig. (2-tailed) 0.000 0.000 0.000 0.516 N 32 32 32 32 32 **. Correlation is significant at the 0.01 level (2-tailed). *. Correlation is significant at the 0.05 level (2-tailed). The Pearson Product Moment correlation coefficient for each successive mathematical thinking ability test item is 0.845, 0.767, 0.760 and 0.119. Based on Table 3, if the values above are interpreted according to the correlation coefficient criteria, then questions number 1, 2 and 3 have a correlation value in the high category, while question number 4 has a correlation in the low category. In other words, after being tested in the field, the mathematical refractive thinking ability test for questions number 1, 2 and 3 already have good empirical validity. Meanwhile, question number 4 still has very poor empirical validity. If the validity coefficient is less than 0.30, it is usually considered as unsatisfactory (Azwar, 2016). Test questions that have low correlation coefficients are revised and used to collect data (Muntholib et al., 2018), so problem number 4 must be corrected and retried to find out its empirical validity. The results of the calculation of the internal consistency tests of mathematical refractive thinking abilities using Cronbach's Alpha can be seen in Table 9. Based on the criteria of Gliem & 92 Isra Nurma Yenti, Yaya S. Kusumah, and Jarnawi Afgani Dahlan Mathematical Refractive Thinking Ability in Calculus: Instrument Design and Quality of Use Gliem (2003) and Salvucci et al. (1997), the alpha value is in the medium category, meaning that the reliability of the mathematical refractive thinking ability test is moderate. In other words, the Cronbach’s Alpha correlation coefficient values above are not sufficient for a reliability. The minimum reliability coefficient is 0.70 to indicate the correct reliability (Scholtes et al., 2011; Setiawan & Mardapi, 2019). Table 9. Reliability Statistics Cronbach's Alpha N of Items 0.593 4 Another opinion states that the results of the calculation of high Cronbach's Alpha coefficients do not guarantee the reliability of an instrument (Adams & Wieman, 2011). In addition, reliability is relative, so the interpretation of the reliability coefficient is relative, meaning the test user can determine whether the reliability coefficient of the score obtained is satisfactory for his needs or not (Azwar, 2016). Table 10. Item-Total Statistics Scale Mean if Item Deleted Scale Variance if Item Deleted Corrected Item-Total Correlation Cronbach's Alpha if Item Deleted Score Item Number 1 12.66 34.814 0.604 0.293 Score Item Number 2 15.19 47.190 0.572 0.388 Score Item Number 3 13.81 38.738 0.399 0.522 Score Item Number 4 17.50 76.323 -0.063 0.708 However, if question number 4 is discarded, the reliability of the mathematical refractive thinking ability test will increase to 0.708 (see Table 10). That is, the reliability coefficient is satisfactory because it is above a predetermined limit (see opinion Scholtes et al., 2011; Setiawan & Mardapi, 2019). One effort to improve test reliability scores is to write tests clearly and make test instructions easy to understand (Drost, 2011). Thus, the mathematical refractive thinking ability test instrument has 3 valid test questions, 1 invalid question and medium reliability. Although the internal consistency of the test instrument is not high, researchers cannot discard question number 4 because otherwise there are unmeasured learning outcomes. So, researchers continues to use this instrument by replacing question number 4 following the suggestion (Drost, 2011). Researchers find previous studies that still use instruments with reliability not high, such as research Muntholib et al., (2018) using instruments with a reliability coefficient of 0.470 (low). CONCLUSION The instrument designed to measure the mathematical refractive thinking ability is a test consisting of 4 essay questions and scoring guidelines. Both instruments have content validity, construct validity, and face validity based on expert judgment. Empirical validity resulted in three valid test items and one test item that had to be improved. Calculations with Cronbach's Alpha gave the Indomath: Indonesian Mathematics Education – Volume 4 | Issue 02 | 2021 93 results of the reliability of the test instrument in the medium category. Thus, this research has developed an instrument to measure the mathematical refractive thinking ability in multivariable calculus material. The next researcher can try out the mathematical refractive thinking ability instrument in a larger sample to obtain higher validity and reliability values. In addition, the development of mathematical refractive thinking ability instruments for multivariable calculus lectures should be a reference for the development of instruments in other subjects. ACKNOWLEDGEMENT The authors would like to acknowledge the financial support from Institut Agama Islam Negeri Batusangkar. The authors would also like to thank the multivariable calculus material experts who have provided many suggestions for the design of this research instrument. REFERENCES Adams, W. K., & Wieman, C. E. (2011). Development and Validation of Instruments to Measure Learning of Expert‐Like Thinking. International journal of science education, 33(9), 1289-1312. https://doi.org/10.1080/09500693.2010.512369. Azwar, S. (2016). Reliabilitas dan Validitas, Edisi Keempat. Yogyakarta: Pustaka Pelajar. Burton, L. J., & Mazerolle, S. M. (2011). Survey Instrument Validity Part I: Principles of Survey Instrument Development and Validation in Athletic Training Education Research. Athletic Training Education Journal, 6(1), 27-35. https://doi.org/10.4085/1947-380X-6.1.27. Colton, D., & Covert, R. W. (2007). Designing and Constructing Instruments for Social Research and Evaluation. John Wiley & Sons. Drost, E. A. (2011). Validity and Reliability in Social Science Research. Education Research and Perspectives, 38(1), 105-123. Gall, M. D. & Borg, W. R. (2003). Educational Research: An Introduction. Longman Publishing. Gliem, J. A., & Gliem, R. R. (2003). Calculating, Interpreting, and Reporting Cronbach’s Alpha Reliability Coefficient for Likert-Type Scales. Midwest Research-to-Practice Conference in Adult, Continuing, and Community Education. Hamdi, S., Kartowagiran, B. & Haryanto (2018). Developing a Testlet Model for Mathematics at Elementary Level. International Journal of Instruction, 11(3), 375-390. -390. https://doi.org/10.12973/iji. 2018.11326a. Harjo, B., Kartowagiran, B., & Mahmudi, A. (2019). Development of Critical Thinking Skill Instruments on Mathematical Learning High School. International Journal of Instruction, 12(4), 149-166. https://doi.org/10.29333/iji.2019.12410a. Hoogland, K., Pepin, B., Bakker, A., de Koning, J., & Gravemeijer, K. (2016). Representing Contextual Mathematical Problems in Descriptive or Depictive Form: Design of an Instrument and Validation of its Uses. Studies in Educational Evaluation, 50, 22-32. https://doi.org/ 10.1016/j.stueduc. 2016.06.005. Laine, A., Näveri, L., Pehkonen, E., Ahtee, M., & Hannula, M. S. (2018). Connections of Primary Teachers’ Actions and Pupils’ Solutions to an Open Problem. International Journal of Science and Mathematics Education, 16(5), 967-983. https://doi.org /10.1007/s10763-017-9809-3. Lestari, K.E. & Yudhanegara, M. R. (2015). Penelitian Pendidikan Matematika: Panduan Praktis Menyusun Skripsi, Tesis dan Karya Ilmiah dengan Pendekatan Kuantitatif, Kualitatif, dan Kombinasi Disertai dengan Model Pembelajaran dan Kemampuan Matematika. Bandung: Refika Aditama. Karanicolas, P. J., Bhandari, M., Kreder, H., Moroni, A., Richardson, M., Walter, S. D., ... & Collaboration for Outcome Assessment in Surgical Trials (COAST) Musculoskeletal Group. (2009). Evaluating Agreement: Conducting a Reliability Study. JBJS, 91(Supplement_3), 99- 106. doi: 10.2106/JBJS.H.01624. Kashefi, H., Ismail, Z., Yusof, Y. M., & Rahman, R. A. (2012). Fostering Mathematical Thinking in the Learning of Multivariable Calculus through Computer-Based Tools. Procedia-Social and Behavioral Sciences, 46, 5534-5540. https:// doi.org/10.1016/j.sbspro.2012.06.47. https://doi.org/10.1080/09500693.2010.512369 https://doi.org/10.4085/1947-380X-6.1.27 https://doi.org/10.12973/iji.%202018.11326a https://doi.org/10.29333/iji.2019.12410a https://doi.org/%2010.1016/j.stueduc.%202016.06.005 https://doi.org/%2010.1016/j.stueduc.%202016.06.005 94 Isra Nurma Yenti, Yaya S. Kusumah, and Jarnawi Afgani Dahlan Mathematical Refractive Thinking Ability in Calculus: Instrument Design and Quality of Use Kimberlin, C. L., & Winterstein, A. G. (2008). Validity and Reliability of Measurement Instruments Used in Research. American Journal of Health-System Pharmacy, 65(23), 2276-2284. https://doi.org/10.2146/ajhp070364. Mardapi, D. (2008). Teknik Penyusunan Instrumen Tes dan Non Tes. Yogyakarta: Mitra Cendekia Press. Maslukha, M., Lukito, A., & Ekawati, R. (2018). Refractive Thinking Profile in Solving Mathematical Problem Reviewed from Students Math Capability. Journal of Physics: Conference Series, 947(1), hlm. 012022. IOP Publishing. doi: 10.1088/ 1742-6596/947/1/012022. Mumu, J., & Tanujaya, B. (2019). Measure Reasoning Skill of Mathematics Students. International Journal of Higher Education, 8(6), 85-91. https://doi.org/10.5430/ijhe.v8n6p85 Muntholib, J. M., Pratiwi, Y. N., Muchson, R. J., & Yahmin, S. R. (2017). Development of Simple Multiple-Choice Diagnostic Test of Acid-Base Concepts to Identify Students’ Alternative Conceptions. In 1st Annual International Conference on Mathematics, Science, and Education (ICoMSE 2017). Atlantis Press (pp. 251-268). Pagano, M. & Roselle, L. (2009). Beyond Reflection: Refraction and International Experiential Learning. Frontiers: The Interdisciplinary Journal of Study Abroad. 18, 217-229. Pardimin, P., Widodo, S. A., & Purwaningsih, I. E. (2017). Analisis butir soal tes pemecahan masalah matematika. Wacana Akademika: Majalah Ilmiah Kependidikan, 1(1). Peraturan Presiden Republik Indonesia Nomor 8 Tahun 2012 tentang Kerangka Kualifikasi Nasional Indonesia. Prayitno, A. (2015). Proses Berpikir Refraktif Mahasiswa dalam Menyelesaikan Masalah Matematika. (Disertasi). Pascasarjana Universitas Negeri Malang. Prayitno, A., Sutawidjaja, A., Subanji & Muksar, M. (2014a). Konstruksi Teoritik tentang Berpikir Refraksi dalam Matematika. Dalam Widodo, dkk (Eds.), Prosiding Seminar Nasional Pendidikan Matematika Ke-2 (hlm. 58-68). Yogyakarta: PPPPTK Matematika. Prayitno, A., Sutawidjaja, A., Subanji & Muksar, M. (2014b). Proses Berpikir Refraksi Siswa Menyelesaikan Masalah Data “Membuat Keputusan”. Dalam Prosiding Seminar Nasional TEQIP (Teachers Quality Improvement Program) dengan tema “Membangun Karakter Bangsa melalui Pembelajaran Bermakna TEQIP (hlm. 154-162). Malang: Universitas Negeri Malang. Prihati, C. N., & Wijayanti, P. (2017). Profil Berpikir Refraktif Siswa SMP dalam Memecahkan Masalah Geometri Ditinjau dari Tipe Kepribadian Keirsey. Mathedunesa: Jurnal Ilmiah Pendidikan Matematika, 1(6), 48-57. Purwaningsih, I., Widodo, S., Harini, E., Kusumaningrum, B., Putrianti, G., & Muanifah, M. (2018). Adaptation Measuring Instrument Keyrsey Temperament Sorter. In Proceedings of the 1st International Conference on Science and Technology for an Internet of Things. European Alliance for Innovation (EAI). Salvucci, S., Walter, E., Conley, V., Fink, S., & Saba, M. (1997). Measurement Error Studies at the National Center for Education Statistics. Washington: National Center for Education Statistics. Schober, P., Boer, C., & Schwarte, L. A. (2018). Correlation coefficients: appropriate use and interpretation. Anesthesia & Analgesia, 126(5), 1763-1768. https://doi.org/10.1213/ANE. 0000000000002864. Scholtes, V. A., Terwee, C. B., & Poolman, R. W. (2011). What Makes a Measurement Instrument Valid and Reliable?. Injury, 42(3), 236-240. https://doi.org/10.1016/j.injury.2010.11.042. Setiawan, A., & Mardapi, D. (2019). The Development of Instrument for Assessing Students' Affective Domain Using Self-and Peer-Assessment Models. International Journal of Instruction, 12(3), 425-438. https://doi.org/10.29333/iji.2019.12326a. Sugiyono. (2015). Penelitian dan Pengembangan. Bandung: Alfabeta. Taber, K. S. (2018). The Use of Cronbach’s Alpha When Developing and Reporting Research Instruments in Science Education. Research in Science Education, 48(6), 1273-1296. https://doi.org/10.1007/s11165-016-9602-2. Wichit, N., Mnatzaganian, G., Courtney, M., Schulz, P., & Johnson, M. (2018). Psychometric Testing of the Family‐Carer Diabetes Management Self‐Efficacy Scale. Health & Social Care in the Community, 26(2), 214-223. https://doi.org/10.1111/hsc.12511. https://doi.org/10.2146/ajhp070364 https://doi.org/10.5430/ijhe.v8n6p85 https://doi.org/10.1213/ANE.%200000000000002864 https://doi.org/10.1213/ANE.%200000000000002864 https://doi.org/10.1016/j.injury.2010.11.042 https://doi.org/10.29333/iji.2019.12326a https://doi.org/10.1007/s11165-016-9602-2 https://doi.org/10.1111/hsc.12511