How to cite item: Faralina, A., Kadri, A., & Yap, A. (2016). Validity and reliability of pre-class reading tasks for waves and optics (PRT- WO). Research and Evaluation in Education, 2(1), 42-52. doi: http://dx.doi.org/10.21831/reid.v2i1.8466 Research and Evaluation in Education e-ISSN: 2460-6995 Research and Evaluation in Education Volume 2, Number 1, June 2016 (pages 42-52) Available online at: http://journal.uny.ac.id/index.php/reid VALIDITY AND RELIABILITY OF PRE-CLASS READING TASKS FOR WAVES AND OPTICS (PRT-WO) 1 Asrab Ali Nur Faralina; 2Ayop Shahrul Kadri; 3Abdullah Nurul Syafiqah Yap 1,2,3 Universiti Pendidikan Sultan Idris 1 aralina91@gmail.com; 2 shahrul.kadri@upsi.fsmt.edu.my; 3 syafiqah@upsi.fsmt.edu.my Abstract This study aims to produce a valid and reliable instrument in preparing students prior to class at university level of introductory Waves and Optics course. The instrument so called Pre-class Reading Task for Waves and Optics (PRT-WO) was used to probe students’ know-ledge acquired through targeted reading activities. In practice, PRT-WO was given in a series before actual face-to-face class as a reading assignment. PRT-WO was content validated through expert review which was analyzed using the inter-rater reliability Cohen’s kappa. An item analysis was done to identify inappropriate items further evaluated using the reliability test of Kuder- Richardson 20 (KR20). The finding reveals that the value of kappa is 0.66 and the value of KR20 is 0.68, indicating that the developed instrument is valid and reliable. Keywords: validity, reliability, pre-reading mailto:1aralina91@gmail.com mailto:2shahrul.kadri@upsi.fsmt.edu.my Research and Evaluation in Education Validity and Reliability of PRT-WO - 43 Asrab Ali Nur Faralina, Ayop Shahrul Kadri, Abdullah Nurul Syafiqah Yap Introduction Students come to class with little or no prior knowledge on the subject. Most stu- dents do not read before entering the class even though they are aware of the importance of reading. According to Sikorski et al (2002, pp.312-320), majority of students do not use their textbooks frequently. It is expected that reading before entering the class may help the students to understand the topics better. Podolefsky and Finkelstein (2006, p.341) find that students who read before entering the class perform better in the course. However, with huge number of assignments and activ- ities in university, students might not read before the class. Stelzer, Gladding, Mestre, and Brookes (2009, pp.184-190) agree that it is difficult to make the undergraduate stu- dents read their textbooks. As reported by Hoeft (2012, p.12) in her research, more than 50% of students in first year undergraduate class did not read assigned materials before class. This indicates that if students hardly read the assigned materials, volunteer reading is much more difficult to achieve. Therefore, it is crucial to find a solu- tion to the engagement of students with reading before class. According to the studies conducted by Ryan (2006, pp.135-141) and Philips (1995, pp.484-489), some instructors leave with dilemma in finding the effective methods to motivate students. The best way is by encouraging a strong reading habit in and off campus for reading index. In addition, most importantly, their status as a wise person will not be disputed by the community after graduation. Various methods were proposed to prepare students before class, such as Just in Time Teaching or known as JiTT, quizzes, emails and presentations. These strategies may engage students to read before entering the class. Heiner, Banet, and Wieman (2014, p.989) implemented JiTT which consisted of regularly reading and online quiz parts for students to complete prior to class. Moravec, Williams, Aquilar-Roca and O’Dowd (2010, pp.473-481) employed learning before lecture (LBL) assignment as an instrument to engage with and to promote reading to the students. They report a significant increase in students’ learning gain and level of satisfaction en- couraging them to continue this method. Ryan (2006, p.136) used three different strategies in motivating students to read their textbooks and be prepared before entering the class. The strategies were as follows: (1) The use of general global assignments, (2) the use of focused explicit homework assignment with less comment from instructor, and (3) the use of focused, explicit assignment with the ample instructor comments. They find that those strategies are the most effective strategy. There also other researchers who employed technologies to encourage students to read their textbooks. Henderson and Rosenthal (2006, p.46) used email in order to submit a reading question based on the assigned reading before enter class. As a re- sult, this method increases student’s reading and has significantly higher outcomes com- pared to other universities on the assessment tool. Special technology developed Web was used by Howard (2004, pp.385-390) in his study. It was similar to the Blackboard and was used at least two hours before the class begin. Students need to answer two quizzes question for each section and the responses are graded with some points used in class discussion. The percentage of students who read textbook before class increases to 98% by using this technique. In this research, pre-class reading task is developed to engage students in reading the assigned material before coming to the face- to-face class. In the next study, the effect of pre-class reading on students’ achievement will be determined. Prior to that, a valid and reliable instrument that can measure the pre- reading effectiveness is required. For that pur- pose, an instrument so called the pre-class reading task for waves and optics (PRT-WO) was developed. The instrument was specific- ally designed for the course of Waves and Optics which is using university physics textbook (Young & Freedman, 2013). Pre- reading effectiveness is defined as the score obtained by students in answering questions regarding the given targeted reading. This instrument is useful for institutions offering a similar course and which use the same refer- Research and Evaluation in Education 44 − Volume 2, Number 1, June 2016 ence book. This article will describe the process of developing the valid and reliable PRT-WO instrument. Method This study was conducted at the Department of Physics, Faculty of Science and Mathematics, Universiti Pendidikan Sultan Idris (UPSI), Malaysia. The PRT-WO was designed to be short with a clear link to the material to be covered in immediately up- coming classes of SFT3023 Vibration, Waves and Optics course. The course is compulsory for students who take physics or science as their major or minor in their undergraduate programs. PRT-WO was prepared in English since the medium of instruction for the course is English even though the mother tongue and national language of the students is Bahasa Malaysia. Therefore, the language was not the barrier for the students to acquire knowledge from the main textbook written in English. There were eight chapters in the PRT- WO, consisting of: (1) Periodic Motion, (2) Mechanical Waves, (3) Sound and Hearing, (4) Electromagnetic Waves, (5) The Nature and Propagation of Light, (6) Geometric Optic, (7) Interference, and (8) Diffraction. These chapters correspond to the eight chap- ters in the main reference book (Young & Freedman, 2013). Each chapter in PRT-WO was split into two parts, except for Chapter 4. Therefore, a total of 15 parts of the instrument were prepared. These 15 parts were released to students one by one along the whole semester of 14 weeks of instruction prior to the corres- ponding chapter being discussed in the face- to-face class. Each release must be submitted at least a night before the class. This will give an ample time to the instructor to analyze students’ answers and discuss them in the class on the next day. Each release of PRT-WO parts was divided into three sections as follows: Section 1 is Targeted Reading, Section 2 is Quiz, and Section 3 is Feedback Review. Section 1 contained instruction of important concept to be focused during reading. The concept will be asked in Section 2. Section 2 contained items regarding a specific chapter. These items required students to open the main text- book. In this way, students will need to read with guidance from Section 2. After Section 2 is done, Section 3 will give feedback on how to get the answer to Section 2. During the actual instruction, PRT-WO was distributed using Google Form. Figure 1 shows snap- shots of Google Form for PRT-WO Part 1 of Chapter 1 (see the ‘Supplementary material’ at references for full PRT-WO material). Google Form was used to distribute PRT-WO due to its advantages such as simple management and unlimited free form cre- ation. Google Form includes low cost (free) and support for multiple formats questions (Kim, 2011). Google Docs spreadsheets and forms provide an easy and flexible way (Bonham, 2011, pp.22-23). Even though there were other more suitable simple tools, most of them were limited in their function. Some instructors thought that Google Form is only for survey research purposes, but in fact, it can also be used as a learning tool. Besides, all the answers and responses are automatic- ally collected in an Excel spreadsheet and it makes it easier for the instructor to analyze large sets of data using charts and other com- plex spreadsheet functions. Instructors can upload a wide range of questions because forms in Google Docs support it, including scale and grid that are generally not available in other web polling services (at least the free ones) such as Survey Monkey and Poll Every- where. As the study involved specific students only, Google Form is very suitable because it can automatically record the email addresses of students who fill out the form and limit the target to specific email domain only. Google Form employs cloud com- puting which required the Internet access, and with cloud computing, there is no physical location of the documents shared. Moreover, many institutions may prefer the idea of data stored on site where they are in control of it, rather than at a remote location where some- one else is hosting it. Although there were some disadvantages, the advantages surpassed those. Google Forms suits the simplicity of delivering PRT-WO to students. Research and Evaluation in Education Validity and Reliability of PRT-WO - 45 Asrab Ali Nur Faralina, Ayop Shahrul Kadri, Abdullah Nurul Syafiqah Yap There are 113 items crafted which covered all the eight chapters. The crafted items were given to the class instructor and underwent a content review which had to be parallel and in accordance with the syllabus covered for the course. Some modifications were done after getting feedback from the class instructor. The items were crafted to fit the first two levels of Bloom’s taxonomy: Remembering (level 1) and understanding (level 2) (Krathwohl, 2002 p.214&218). Since PRT-WO was aimed to measure reading effectiveness and also prepare students before class by targeted reading, the items must not be very difficult especially in level 3 of the taxonomy. This was not to penalize students, but to provide prior knowlegde to them. Examples of the developed items are shown in Examples 1 and 2. Example 1 was taken from Chapter 2 Part 2. Once students receive an instruction to complete their read- ing assignment, they will open the given Google Form link. The first section will guide them what they have to read in order to answer Section 2. This item was rated as level 2 of the taxonomy since the answer required students to formulate the concept of super- position in mathematical form. After the answer is submitted, feedback of Section 3 will appear to give hint where the correct answers can be found for students to revise. Example 1 Section 1: Targeted reading You must understand that wave can interfere on each other resulted in a new wave which can be understood from the principle of superposition Section 2: Chapter 2 Question 14 Which of the following describes the principle of superposition of two waves, 𝑦1(x,t) and 𝑦2(x,t) to produce the resulted wave of y(x,t)? A. y (x,t) = 𝑦1(x,t) = 𝑦2(x,t) B. y (x,t) = 𝑦2(x,t) - 𝑦1(x,t) C. y (x,t) = 𝑦1(x,t) - 𝑦2(x,t) D. y (x,t) = 𝑦1(x,t) + 𝑦2(x,t) Figure 1. Example of PRT-WO Part 1 of Chapter 1. (Left: The targeted reading section; Middle: Quiz section; Right: Feedback review section) Research and Evaluation in Education 46 − Volume 2, Number 1, June 2016 Section 3: Feedback review Pages 490-491. Principle of superposition in mechanical wave means that the amplitude of the resultant wave pattern created by the interference of travelling waves opens in a new window is the sum of the amplitudes of the travelling waves. Example 2 was taken from Chapter 1 Part 2. It required students to turn to a spe- cific page containing the figure in question. Therefore, students could not answer this question without referring to the main text- book. This item was rated as level 2 of the taxonomy. Example 2 Section 1: Targeted reading You must understand that in real oscillating system, friction does exist and it causes the oscillation to be damped. You must be able to recognize the behaviours of the oscillation at various degrees of damping. Section 2: Chapter 1 Question 9 Why will the heavy swinging bell (Figure 14.25) eventually stop oscillating? A. Due to the damping forces B. Due to the mass of the bell C. Someone stops the bell Section 3: Feedback review Page: 457. A swinging bell left to itself will eventually stop oscillating due to damping forces (air resistance and friction at the point of suspension), therefore, the energy is dissi- pated and the amplitude decreases and thus eventually stops. Each instrument needed to undergo validity and reliability testing. Before the PRT- WO was given to students, content validation on PRT-WO was carried out. The purpose of validation is to see how far the relationship between variables can be ensured. According to Merriam (2001), validity of equivalence re- lates the findings and reality. There are several methods that can be used to determine the validity, such as triangulation, expert examin- ation, expert review, statement of experience, and bias or hope. Content validity by the ex- pert review was employed in this study. The PRT-WO was content validated by the two subject-matter experts from UPSI. The valid- ation process required experts to evaluate whether each item is conceptually correct and appropriate to measure reading effectiveness. Meanwhile, each expert will rate either they agree or not on the items in PRT-WO. The agreement data received were used to calculate the Cohen’s kappa. Cohen’s kappa (Cohen, 1960, pp.37-46) was introduced as a measure of agreement which avoids the problems described above by adjusting the observed proportional agreement to take account of the amount of agreement which would be expected by chance. After undergoing the internal content validity and Cohen’s kappa test, the PRT-WO went through the modification process again by inspecting items one by one. In this process, error is minimized on each item, for example grammatical correction, content revision, and language appropriateness. A pilot test was administered to deter- mine the reliability value of KR20. It involved a group of 90 undergraduate students who registered Vibration, Waves, and Optics (SFT3023) course in Semester 2 batch 2014/ 2015. The group consisted of two subgroups: 46 students in Group A and 44 students in Group B. The PRT-WO was tested on Group A only. However, only 24 students from Group A answered all items in the PRT-WO. The students need to log in the email pro- vided by UPSI (siswa-mail) every time before answering the PRT-WO. The Google Form was only permitted for the person who logs in via siswa-mail. The study was carried out in a semester of 14 weeks of instruction. Table 1. List of assessment components for Group A Coursework Percentage PRT-WO Assignment 3 Lab Report and Other Assignments 12 Mid-Term Test 15 Mini Project 15 Group Presentation 15 Final Examination 40 Total 100 Research and Evaluation in Education Validity and Reliability of PRT-WO - 47 Asrab Ali Nur Faralina, Ayop Shahrul Kadri, Abdullah Nurul Syafiqah Yap The 3% allocation mark was given as incentive to the students. The percentage is appropriately chosen to be neutral in course marks so that it does not seem to force students or either can be ignored by the stu- dents. This is important to make sure that students give their best contribution to the pilot test. Table 2. Distribution and division of PRT-WO Chapter Sub-chapter Release Week-Expired Week 1 Part 1 1.1 Describing Oscillation 1.2 Simple Harmonic Motion 1.3 Energy in Simple Harmonic Motion W1-W2 Part 2 1.5 The Simple Pendulum 1.7 Damped Oscillations 1.8 Forced Oscillations and Resonance W1-W2 2 Part 1 2.1 Types of Mechanical Waves 2.2 Periodic Wave 2.3 Mathematical Description of A Wave 2.4 Speed of a Transverse Wave W2-W3 Part 2 2.5 Energy in s Motion 2.6 Wave Interference, Boundary Condition and Superposition 2.7 Standing Waves on a String 2.8 Normal Modes of a String W2-W3 3 Part 1 3.1 Sound Waves 3.2 Speed of Sound Waves 3.3 Sound Intensity 3.4 Standing Sound Waves and Normal Modes W4-W5 Part 2 3.5 Resonance of Sound 3.6 Interference of Waves 3.7 Beats 3.8 Doppler Effect W5-W5 4 Part1 4.1 Maxwell’s Equation and Electromagnetic Waves 4.2 Plane Electromagnetic Waves and the Speed of Light 4.3 Sinusoidal Electromagnetic Waves W6-W6 5 Part 1 5.1 The Nature of Light 5.2 Reflection and Refraction 5.3 Total Internal Reflection W7-W8 Part 2 5.4 Dispersion 5.5 Polarisation 5.6 Scattering of Light 5.7 Huygens’s Principle W7-W8 6 Part 1 6.1 Reflection and Refraction at a Plane Surface 6.2 Reflection at a Spherical Surface 6.3 Refraction at a Spherical Surface 6.4 Thin Lenses W8-W9 Part 2 6.5 Cameras 6.6 The Eye 6.7 The Magnifier 6.8 Microscopes and Telescopes W9-W9 7 Part 1 7.1 Interference and Coherent Sources 7.2 Two-Source Interference Of Light 7.3 Intensity in Interference Patterns W9-W10 Part 2 7.4 Interference in Thin Films 7.5 The Michelson Interferometer W10-W11 8 Part 1 8.1 Fresnel and Fraunhofer Diffraction 8.2 Diffraction from a Single Slit 8.3 Intensity in the Single-Slit Pattern W12-W12 Part 2 8.4 Multiple Slits 8.5 The Diffraction Grating 8.6 X-Ray Diffraction W12-W13 Research and Evaluation in Education 48 − Volume 2, Number 1, June 2016 At the beginning of the semester, week 1, all the students were given a brief regarding the methods and steps that they need to know. Every week, they were informed that PRT-WO was available via siswa-mail and social media tool (such as Whatsapp) as a reminder. This type of communication tech- niques aligns with the 21st century learning and literacy skills. If they face some technical problems, they can directly contact the PRT- WO administrator through Whatsapp Group. Table 2 shows the chapter and sub- chapter, number of question for each part, release and due date of the PRT-WO. There were 15 parts of PRT-WO which were distri- buted almost periodically throughout the semester. As soon as the students answered all the PRT-WO, the reliability test was ad- ministered. Before that, the item analysis was done to eliminate inappropriate items through the determination of difficulty index of each item. Only 24 undergraduate students answer- ed all items in the PRT-WO pilot test. Re- sponses from these groups of students were analyzed for KR20. Reliability refers to a mea- sure of consistency and stability of study or test as the measuring instrument. It aims to determine whether these measures give the same answer as when it is used to measure the same concept to population or respondent which are alike. There are several types of reliability, including internal consistency, test- retest, equivalent and stability, equivalent consistency and scorer. Internal consistency was chosen to be used for this instrument reliability test. It measured the extent to which the items in the test were consistent with each other and overall. Internal consistency re- liability consists of three types; Split Half reliability, Kuder Richardson reliability, and also Cronbach's Alpha reliability. Kuder- Richardson 20 (KR20) reliability was used for PRT-WO since the items were given in two options: Correct or wrong. Besides, according to Sabri (2013, pp.1-14) Kuder-Richardson 20 is a formula which is based on item difficulty where it is used to analyze the internal con- sistency of section A in the string instrument comprehensive test. The summarized process to produce PRT-WO is illustrated in Figure 2. Figure 2. Methodology framework Research and Evaluation in Education Validity and Reliability of PRT-WO - 49 Asrab Ali Nur Faralina, Ayop Shahrul Kadri, Abdullah Nurul Syafiqah Yap Findings and Discussion Instrument validity and reliability are important aspects in a social science study, especially to produce a correct instrument. Validity or legitimacy is a concept that refers to the extent to which instruments measure what you want to measure or study the extent to which it meets the job’s purpose (Anastasi & Urbina, 1997). A pilot study is considered as a small study carried out with the aim of improving and increasing the validity and reliability of the instrument (Fraenkel, Wallen & Hyun, 1993). Cohen's kappa index analysis was also used to determine the degree of agreement between evaluators (experts). Steven (1958, pp.177-196) states that the agreement between evaluators is important to determine the value of high reliability for every unit used to describe a theme. To deter- mine the level of agreement kappa, the value recommended by Landis and Koch (1977, pp.159-174) was used. The Cohen’s kappa value for PRT-WO is 0.66 and this value was the value of good reliability. Cohen’s kappa index analysis was carried out to determine the degree of agreement between two raters on the item in the PRT-WO. This supported the reliability of the PRT-WO. The obtained Cohen’s kappa for PRT-WO is rated as good level of instrument reliability (Landis & Koch, 1977, pp.159-174). The item analysis for each question in PRT-WO was carried out by determining the difficulty level (p) as suggested by Macintosh and Morrison (1969). The item analysis was done before running the KR20 to eliminate too difficult items. The difficulty level is defined as follows: p = N1/N where p is the difficulty level, N1 is the number of correct responses, N is the total number of students taking the test. The result is tabulated in Table 3. The items with p<0.39 were rejected. As many as 13 out of 113 items as highlighted in the table were removed. These items were considered as very difficult. Low p items indicated either the question itself is difficult to be understood in the aspect of language or the answer was difficult to be found in the main text. As the purpose of the instrument is to access stu- dents’ acquired knowledge through reading, very difficult items were removed so that students were not demotivated to continue their reading assignment. The following Example 3 (C3Q4) was among the rejected items taken from Chapter 3 Part 1. This item was rated as level 2 of the taxonomy. The difficulty level of the item was p=0.00, where nobody got it correct. Even though the answer was implicitly written in the reference, students might be confused with other options. Example 3 Section 1: Targeted reading Sound travels at different speed in gas, fluid and solid and also in different temperatures. Therefore, you have to understand how the sound travels in each of the medium. Section 2: Chapter 3 Question 4 The restoring term in the expression (page 514) for the wave speed of the sound waves A. exhibits the difficulties in the fluid compression. B. exhibits the massiveness of a bulk fluid. C. has actually no relation to the sound wave. D. follows Newton second law. Section 3: Feedback review Page: 514. To check you can derive the speed of sound waves in a fluid in a pipe. For your information, human speech works on the same principle. Another example of the rejected item is shown in Example 4 (C1Q?) taken from Chapter 1 Part 1. This item was initially rated as level 1 of the taxonomy with p=0.29. After revision, the item should be rated as level 3 which required students to interpret and analyse the equation of energy for simple harmonic motion. Research and Evaluation in Education 50 − Volume 2, Number 1, June 2016 Table 3. Item analysis of PRT-WO with difficulty level (p) Items Difficulty level Items Difficulty level C1Q1 0.92 C4Q6 0.67 C1Q2 0.88 C4Q7 1.00 C1Q3 0.96 C4Q8 1.00 C1Q4 0.46 C5Q1 0.08 C1Q5 0.63 C5Q2 0.92 C1Q6 0.29 C5Q3 0.96 C1Q7 0.96 C5Q4 0.96 C1Q8 1.00 C5Q5 1.00 C1Q9 1.00 C5Q6 0.88 C1Q10 0.67 C5Q7 0.96 C1Q11 0.46 C5Q8 0.96 C1Q12 0.46 C5Q9 0.96 C2Q1 0.92 C5Q10 0.96 C2Q2 0.42 C5Q11 0.29 C2Q3 1.00 C5Q12 0.92 C2Q4 0.71 C5Q13 0.29 C2Q5 1.00 C5Q14 0.25 C2Q6 0.79 C5Q15 0.42 C2Q7 0.71 C6Q1 1.00 C2Q8 0.96 C6Q2 0.92 C2Q9 1.00 C6Q3 1.00 C2Q10 0.92 C6Q4 0.83 C2Q11 0.96 C6Q5 0.92 C2Q12 0.67 C6Q6 0.92 C2Q13 0.54 C6Q7 0.96 C2Q14 0.96 C6Q8 1.00 C2Q15 0.67 C6Q9 1.00 C2Q16 0.79 C6Q10 0.96 C2Q17 1.00 C6Q11 0.92 C2Q18 0.96 C6Q12 0.96 C3Q1 0.96 C6Q13 1.00 C3Q2 0.96 C6Q14 0.96 C3Q3 0.71 C6Q15 1.00 C3Q4 0.00 C6Q16 1.00 C3Q5 0.79 C7Q1 0.96 C3Q6 0.88 C7Q2 1.00 C3Q7 1.00 C7Q3 0.96 C3Q8 0.96 C7Q4 0.83 C3Q9 - C7Q5 0.88 C3Q10 0.75 C7Q6 0.58 C3Q11 0.96 C7Q7 0.08 C3Q12 1.00 C7Q8 0.75 C3Q13 0.88 C7Q9 0.96 C3Q14 0.75 C7Q10 0.42 C3Q15 0.92 C8Q1 0.75 C3Q16 1.00 C8Q2 0.58 C3Q17 0.96 C8Q3 0.96 C3Q18 0.96 C8Q4 0.75 C3Q19 0.38 C8Q5 0.75 C3Q20 0.50 C8Q6 0.46 C3Q21 0.46 C8Q7 0.58 C3Q22 0.04 C8Q8 0.79 C4Q1 1.00 C8Q9 0.63 C4Q2 0.50 C8Q10 0.96 C4Q3 0.33 C8Q11 0.21 C4Q4 1.00 C8Q12 0.33 C4Q5 1.00 Research and Evaluation in Education Validity and Reliability of PRT-WO - 51 Asrab Ali Nur Faralina, Ayop Shahrul Kadri, Abdullah Nurul Syafiqah Yap Example 4 Section 1: Targeted reading You must understand the type of energy involved in SHM and how it changes at different locations along the motion, yet the total energy is kept constant. Section 2: Chapter 1 Question 6 The total mechanical energy E of SHM is also directly related to the ________ of oscillation. I. amplitude II. angular frequency III. velocity A. I only B. I and II C. II and III D. I, II, and III Section 3: Feedback review Page: 446. Refer to Equation 14.20 and (14.9) which relates the total mechanical energy to other physical quantities of oscillation. After the process of elimination in the item analysis, the remaining 100 items were analyzed for KR20. It was found that the value of the KR20 was rtest = 0.68. A value close to 0.80 is common to test the partici- pant’s heterogeneous classrooms, whereas common values as low as 0.50 to test for a homogeneous group of participants. A test of reliability higher than 0.7 is considered to be reliable for group of measurement and it is considered as a widely accepted criterion (Ding & Beichner, 2009, pp.020103-2). The result was approximately aligned with the results of the studies which are conducted by Ding and Beichner (2009, pp.020103-2). Conclusion and Recommendation Conclusion The process of validity and reliability of PRT-WO was discussed. The PRT-WO was crafted and given to the class instructor for the first step of content validation. The instru- ment was revised according to instructor’s comment. Then, PRT-WO was content valid- ated by two experts in the field. At the same time, the experts rated their agreement on each item. These agreement data were used to obtain kappa’s value of 0.66. PRT-WO was again revised according to experts’ review. The revised PRT-WO was then distributed to the students in series along the semester for a pilot test. At the end of the semester, PRT- WO responses were analyzed using the item analysis and KR20. From 113 items, PRT- WO reduced to 100 items due to unsuitable difficulty index. Finally, KR20 was done on the remaining items resulted in rtest = 0.68. PRT-WO was examined under thorough pro- cess to measure the preclass reading effective- ness in term of score and available for futher related study in the teaching and learning waves and optics. The validated and reliable PRT-WO is available via provided link in the reference. Recommendation Any institution that uses the same sylla- bus and reference book may use PRT-TWO to access student pre-class reading effective- ness before entering class. The students are expected to be prepared and actively engaged during the class discussion. At the time this paper was written, the 14th edition of University Physics book was already publish- ed. Since this study used the 13th edition, slight modification such as figure and page numbering can be done to adapt to this new edition. Acknowledgements The researchers would like to acknow- ledge the Ministry of Higher Education for providing MyBrain15 scholarship to support the first author in this study. References Anastasi, A. & Urbina, S. (1997). Psychology testing (7 th ed.). New Jersey, NJ: Prentice Hall. Bonham, S. (2011). Whole class laboratories with Google Docs. The Physics Teacher, 49(1), 22-23. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychosocial Measurement, 20, 37-46. Research and Evaluation in Education 52 − Volume 2, Number 1, June 2016 Heiner, C.E., Banet, A.I., & Wieman, C. (2014). Preparing students for class: How to get 80% of students reading the textbook before class. American Journal of Physics, 82(10), 989-996. Henderson, C. & Rosenthal, A. (2006). Read- ing questions. Journal of College Science Teaching, 35(7), 46. Hoeft, M.E. (2012). Why university students don't read: What professors can do to increase compliance. International Journal for the Scholarship of Teaching and Learning, 6(2), 12. Howard, J.R. (2004). Just-in-time teaching in sociology or how I convinced my stu- dents to actually read the assignment. Teaching Sociology, 32(4), 385-390. Ding, L., & Beichner, R. (2009). Approaches to data analysis of multiple-choice ques- tions. Physical Review Special Topics-Physics Education Research, 5(2), 020103. Fraenkel, J.R., Wallen, N.E., & Hyun, H.H. (1993). How to design and evaluate research in education (Vol. 7). New York, NY: McGraw-Hill. Kim, D. (2011). Using Google Forms for student engagement and learning. EDUCAUSE Quarterly, 34(1). Krathwohl, D.R. (2002). A revision of Bloom's taxonomy: An overview. Theory into Practice, 41(4), 212-218. Landis, J.R. & Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics, 159-174. Macintosh, H.G. & Morrison, R.B. (1969). Objective testing. London: University of London Press. Merriam, S. (2001). Qualitative research and case studies in education: Revised and expanded from case study research in education. San Francisco, CA: Jossey Bass. Moravec, M., Williams, A., Aguilar-Roca, N., & O'Dowd, D.K. (2010). Learn before lecture: A strategy that improves learn- ing outcomes in a large introductory biology class. CBE-Life Sciences Edu- cation, 9(4), 473-481. Philips, G. (1995). Using open book tests to encourage textbook reading in col- lege. Journal of Reading, 38(6), 484. Podolefsky, N. & Finkelstein, N. (2006). The perceived value of college physics text- books: Students and instructors may not see eye to eye. The Physics Teacher, 44(6), 338-342. Ryan, T.E. (2006). Motivating novice students to read their textbooks. Journal of In- structional psychology, 33(2), 135-141. Sabri, S. (2013). Item analysis of student com- prehensive test for research in teaching beginner string ensemble using model based teaching among music students in public Universities. International Journal of Education and Research, 1(12), 1-14. Sikorski, J., Rich, K., Saville, B., Buskist, W., Drogan, O., Davis, S.F., ... & Geller, E.S. (2002). Faculty forum. Teaching of Psychology, 29(4), 312-320. Stelzer, T., Gladding, G., Mestre, J.P., & Brookes, D.T. (2009). Comparing the efficacy of multimedia modules with traditional textbooks for learning intro- ductory physics content. American Journal of Physics, 77(2), 184-190. Steven, S.S. (1958). Problems and method of psychophysics. Psychological Bulletin. LV, 177-196. Young, H.D. & Freedman, R.A. (2013). University physics with modern physics (13 th ed.). San Francisco, CA: Pearson Higher Ed. Supplementary: PRT-WO. Retrieved from https://goo.gl/2mQftJ. https://goo.gl/2mQftJ