188 | JISAE. Volume 6 Number 2 September 2020. https://doi.org/10.21009/JISAE JISAE (Journal of Indonesian Student Assessment and Evaluation) ISSN : P-ISSN: 2442-4919│E-ISSN: 2597-8934 Vol 6 No 2 (2020) Website : http://journal.unj.ac.id/unj/index.php/jisae THE VALIDITY OF INSTRUMENTS AS A MEASURING INSTRUMENT FOR THE EVALUATION ON STUDENTS JOB TRAINING PROGRAM AT SMK NEGERI 1 CARIU BOGOR REGENCY Sefrianto 1 Aip Badrujaman 2 Komarudin 3 123Universitas Negeri Jakarta ABSTRACT This research is part of the preparation before evaluating On the Job Training program. The goal is to validate the instrument before used as a raw instrument to obtain field data. This research is carried out with quantitative approach. The data is collected by using questionnaire instruments to evaluate the impact of evaluation on the job training program at the behavior, outcome levels, and results. The instrument is tested on 30 respondents, namely Honda Putra Merdeka Cibubur, Honda Lestari Motor Cariu and Honda Citra Indah Jonggol. The test validity of the instrument using the product moment correlation formula. The results of the study show not all tested instruments are valid. Out of 35 instruments tested, 4 items of instruments are invalid. The instrument is in the dimensions of precision and work accuracy, increase in company revenue, and the level of curiosity at work. The result of the instrument validity concludes that the instrument that can be used to evaluate the on the job training program is as many as 31 items with the reliability value that can be received in a very good category of 0.928. Keywords: Evaluation Of On The Job Training Program, Impact Evaluation, Instrument Validation Address for Correspondence: sefrianto_pep17s2@mahasiswa.unj.ac.id1, Aip.bj@unj.ac.id2 , komarudin@unj.ac.id3 INTRODUCTION Vocational high School or SMK is part of the national education system which plays an important role in preparing and developing human resources. UUSPN No. 20 year 2003 Article 15, said, SMK aims to prepare students mainly to work with expertise in a particular field. One of the Government's efforts in preparing and developing qualified Vocational high School students and ready to be a workforce that is ready to jump into the workplace or the world of industry is by doing Cooperation between the school and the business world and the Industrial World in the form of implementing learning programs. The learning Program is in the form of industrial visits, field work practices, internships or On the Job Training. The goal is to provide work experience and to develop its competence directly. Thus, with expertise, experience and competence, it is expected that Vocational high School graduates will have good self quality. Based on the results of the initial observation conducted by researchers at SMK Negeri 1 Cariu, the learning based On the Job Training Department of Light Vehicle Engineering is basically well-maintained. Although there are some records that should be of concern for organizers or implementing programs. This is in line with the research analysis results of several other schools that have carried out the same program, it shows the success of the program viewed from the aspect of process, preparation and results. One of them is in SMK N 3 Buduran Sidoarjo (Febrikha: 2015) http://journal.unj.ac.id/unj/index.php/jisae mailto:sefrianto_pep17s2@mahasiswa.unj.ac.id mailto:Aip.bj@unj.ac.id mailto:komarudin@unj.ac.id 189 | JISAE. Volume 6 Number 2 September 2020. On the Job Training Program in other schools have been conducted evaluation by the previous research. The evaluation research generally uses CIPP (Context, Input, Process, Product) models. The results of some studies concluded that almost 90% of the On the Job Training Program was carried out well (Irfan, 2017; Febrikha, 2015; Emila, 2016; Senab, 2014; Mulyadi, 2018; Dadan, 2016; Alipuor, 2009). In addition, the program is implemented as a procedure and the results are able to prepare students to enter the workforce in accordance with the required competencies. As the follow- up of evaluation result, On the Job Training program can be continued even with some notes that must be considered to improve the implementation of the program in the future. The results of the study generally show a level of success in the context, input, process and product dimensions. The context dimension shows approximately 80% of its success rate (Irfan, 2017; Febrikha, 2015; Emila 2016; Senab, 2014; Mulyadi, 2018; Darmawan, 2016). Some of the problems that occur in this dimension are; student selection tools On the Job Training has not been effective and the journals used by participants On the Job Training are not yet relevant. However, this can be solved well by the program organizer. The level of success on input dimensions is approximately 80% (Irfan, 2017; Febrikha, 2015; Emila, 2016; Senab, 2014; Mulyadi, 2018; Darmawan, 2016). The problems that occur in this dimension include; the execution of the curriculum that needs to be updated and mental of students who are still lacking in facing the implementation of the program. The organizer can overcome the problem with more well preparations in the learning and supply process for prospective participants On the Job Training before the program is implemented. In the dimension of the process, the success rate is approximately 75% (Irfan, 2017; Febrikha, 2015; Emila, 2016; Senab, 2014; Mulyadi, 2018; Darmawan, 2016). As a record in the On the Job Training process, process dimension among others; the needs of students’ skills and discipline improvement, the time allocated for the supply is not appropriate, and the lack of optimal supervision from the school as program organizer. On the product dimensions, it is concluded that the success rate reaches more than 90% (Dimas 2018; Irfan, 2017; Febrikha, 2015; Emila, 2016; Senab, 2014; Mulyadi 2018; Darmawan, 2016; Septian, 2018). The indication is the achievement of the program's main objectives On the Job Training, which prepares students to become graduates with skills, knowledge competencies and ready to enter the workforce. The achievement is seen from the results of the final assessment or test of the students after implementing the program. Results of the research interview with the head of Light Vehicle Engineering Department in SMKN 1 Cariu and the vice principal of industry relations, supported by documentation On the Job Training activities show the program was well-prepared, organized and well-planned. Results of the research interview with the head of Light Vehicle Engineering Department in SMKN 1 Cariu and vice principal of industry relations and supported by documentation On the Job Training activities show the program was well-prepared, organized and well-planned. Ranging from socialization to students and parents, submission of proposals to the relevant industries, formation of organizers, schedules of activities, the creation of the teacher monitoring schedule, the system and selection process of participants On the Job Training. In addition, the process dimension illustrates a positive success rate. Students carried out activities actively, with discipline and responsibility so as not to interfere with the work atmosphere that is usually set by the industry even very helpful to the industry. Although some obstacles are still remain, among other there are some students who have not received basic knowledge while learning at school, so it needs to give a reunderstanding of basic knowledge by an instructor or industry supervisor. The value earned by the student when carrying out the exam at the end of the program belongs to the category above the graduation criteria, there are even some students who get excellent grades. Other obstacles were the lack 190 | JISAE. Volume 6 Number 2 September 2020. of time allocation of techers in providing guidance to students in the process of making final reports. In addition, the lack of time allocation of activities On the Job Training Program is proclaimed by the school. Since the industry often asks to give an additional dispensation of time for students who have completed the activities On the Job Training. One of the things that must be done in order to improve the quality of learning is to conduct an evaluation. Because evaluation can assess whether a program is good and useful, and so that after the conclusion an evaluation can be followed up on an education program that has been implemented. This is in line with the evaluation theory, according to Gall and Borg (2007: 559) "Educational evaluation is the process of making judgments about the merit, value, or worth of educational programs". Stufflebeam and Shinkfield (2007: 9) also quoted the Joint Committee on Standards for Educational Evaluation (JCSEE, 1994) defined evaluation as the "systematic investigation of the worth or merit of an object". Based on the statements, it can be concluded that an educational program will be qualified with the implementation of evaluation. If a program is not evaluated, it will not be known how and how well the program has been carried out and whether it has an impact on the people involved in the program. For this reason, the Job Training Program at SMK N 1 Cariu needs to be evaluated so that improvements can be made to implement it in a sustainable manner. Several evaluations about the Job Training have been conducted in several SMKs in Indonesia. In general, research on evaluating the Job Training uses the approach of context, input, process and product methods. There is still very little number of further evaluation of the outcome or impact of the program. Based on initial observation and analysis of previous research, researchers feel the need to do further research on On the Job Training Program. Mainly researchers focus on the main On the Job Training at SMK Negeri 1 Cariu, Department of Light Vehicle Engineering, Bogor Regency. The goal is to see the impact On the Job Training program on the work behavior of employee and the impact of the result for the company where employees work. Before conducting the research, of course researchers should prepare well. This form of preparation is to prepare a research instrument as a tool to collect information or field data needed. The preparation of instruments is conducted with systematic processes and procedures. Until the instrument is finally carried out test validation as a form of confirmation whether the instrument items can be used as a raw instrument. The raw instrument means it can be used to measure dimensions or indicators. Instrument validity test is one of the processes done in the effort to prepare the test instrument. A validity test is a test used to indicate the extent to which the measuring instrument is used in measuring of what is to be measured (Ghozali, 2009). In addition, validity is a measure that indicates that the variables measured are actually variables that are to be examined by researchers (Cooper and Schindler, in Zulganef, 2006). Ghozali (2009) also stated that the validity test is used to measure the valid absence of a questionnaire. Validity explains how well the data collected includes the actual area of the investigation (Ghauri and Gronhaug, 2005). Validity essentially means "Measuring what is meant to be measured" (Field, 2005). A questionnaire is said to be valid if a question or statement on the questionnaire is capable of revealing something to be measured by the questionnaire. Based on the expert explanation, It is concluded that instrument validity is important to be conducted before used to collect research data. The goal is to produce a valid instrument and can measure the variables measured. METHOD This research is carried out with quantitative approach. The data collection method used in this study is questionnaire instrument. While data analysis is using the correlation formula of product moment. The analysis is conducted to determine the validity of each item on the 191 | JISAE. Volume 6 Number 2 September 2020. tested instrument. The instrument item is declared valid or invalid when it has a value of r count >r table at a significant level of 0.05 with N = 30 (Sugiyono, 2013). The samples used in the study were 30 employees (technicians). The instrument is tested on 30 respondents namely Honda Putra Merdeka Cibubur, Honda Lestari Motor Cariu and Honda Citra Indah Jonggol. The samples were taken in a random sampling. It is recognized by the consideration of the character's condemnation. The validity test with a correlation formula of the product moment from Kurl Pearson, as follows Description: R XY : The correlation of the Tangkar moment (Product Moment) N : Subject Count ∑ X : Total X (items score) ∑ X ² : Sigma X (squared) ∑ Y : Number Y (Factor score) ∑ Y ² : Sigma Y (squared) ∑ XY : Sigma Tangkar (multiplication) X with Y The result of analysis of the correlation product moment goal is to lay the value of r count >r table of each item tested. Before conducting the instrument validity test, a few steps must be placed in the research process. The procedures or instruments for the creation of the following studies: Picture 1 Research instrument Creation plot Analysis of the evaluation criteria is conducted in order to determine the dimensions, indicators and the grids of the instrument. The evaluation criteria of this study are determined based on the objectives of the On the Job Training Program. This criterion is derived from several theories about the On the Job Training Program and is seen its relevance to the program to be researched. In addition, the criteria are taken from previous researches that is relevance to the research to be implemented. After obtaining the dimension and indicators, the researchers then made the grids of instrument. This evaluation aims to see the impact of the On the Job Training Program on the behavior of employees who previously participated in the program and the impact for the company itself. Thus, the standard evaluation criteria is focused on the level of behavior and result. The standard criteria, dimensions, indicators and grid of the instrument can be seen in the following table: 192 | JISAE. Volume 6 Number 2 September 2020. Table 1 Dimensions, Indicators and Grids of Insrument Items The next step is to carry out expert testing. Expert judgment is conducted after the instrument is arranged based on the dimensions and indicators to be measured. The statement items that have been compiled are then consulted with lecturers or experts (Expert Judgment). The aim is to test the appropriateness of instruments that are made to be used in later stages. According to Djaali and Pudji (2008) construct validity is a validity that concerns the extent to which test items are able to measure what is really being measured according to a specific concept or conceptual definition that has been set. Besides that, Groth-Marnat (2010) also suggested that a test has construct validity if it shows the relationship between test scores and predictions of theoretical properties. Based on the opinions of several experts, the construct validation process of an instrument must be done through expert review or justification or through the assessment of a group of panels consisting of people who master the substance or content of the variable to be measured. This eligibility is based on construct criteria and language criteria. The construct criteria are aimed at assessing whether the statement items are in accordance with the dimensions measured, the limits of the expected answers are clear, and the suitability of the contents of the statement with the measurement objectives. Whereas, the purpose of language criteria are to assess whether statements use sentences, technical terms, and unfamiliar words that are easy to understand. Besides, it is to assess whether the words used are not confusing/ambiguous and the choice of answers in accordance with the statements submitted. Researchers tested the instruments by the help of three experts who were considered competent in the language and constructs. After the instrument items are collected based on the results of expert/expert tests, the next step is to test the validity and reliability. Validity test uses the calculation of the product moment correlation formula. The aim is to determine the feasibility of each item by looking at a comparative analysis of the values of r arithmetic and r tables. Elimination technique used with the provisions of the instrument is said to be valid if r ≥ r table, with the significant No r count value r table value Valid/Invali d No r count value r table value Valid/Inva lid 1 2,861 0,374 valid 19 5,871 0,374 valid 2 4,977 0,374 valid 20 2,714 0,374 valid 3 6,177 0,374 valid 21 4,238 0,374 valid 4 0,106 0,374 Invalid 22 0,012 0,374 Invalid 5 4,789 0,374 valid 23 3,576 0,374 valid 6 4,977 0,374 valid 24 2,576 0,374 valid 7 0,881 0,374 Invalid 25 5,871 0,374 valid 8 0,075 0,374 Invalid 26 2,656 0,374 valid 9 5,977 0,374 valid 27 4,288 0,374 valid 10 5,871 0,374 valid 28 3,789 0,374 valid 11 5,871 0,374 valid 29 3,802 0,374 valid 12 2,875 0,374 valid 30 3,802 0,374 valid 13 4,288 0,374 valid 31 2,875 0,374 valid 14 4,977 0,374 valid 32 6,177 0,374 valid 15 4,238 0,374 valid 33 2,834 0,374 valid 16 4,067 0,374 valid 34 2,415 0,374 valid 17 3,802 0,374 valid 35 4,045 0,374 valid 18 2,576 0,374 valid 193 | JISAE. Volume 6 Number 2 September 2020. level of 0.05. If the correlation value is below the r table, it can be concluded that the instrument is invalid, so it must be eliminated. The final step is grouping the instruments. The aim is to select and determine instruments that are declared valid and invalid. The instruments that are valid will be used in collecting field data. While the instruments that are invalid will be discarded. RESULTS AND DISCUSSION In this study the questionnaire was tested to 30 respondents. A total of 35 instruments have been tested. Before testing the instrument, the respondent was previously tested by an expert. Based on the expert test results, out of 40 items that are consulted with, 5 items of the instrument have been eliminated. The 5 instruments are divided into several dimensions, including; the accuracy of work completion consists of 2 items, namely item number 7 and 12, the dimensions of sincerity in working consist of 1 item, namely item number 20, and the dimension of great curiosity in working consist of 2, namely items numbers 33 and 36. The instrument is declared invalid because it does not meet the construction and language criteria. Therefore, the instrument used for the validaty test is 35 items. The results of instrument testing on 30 respondents, there were several instruments that were declared invalid. Some of these instruments should be eliminated because the results of the validity analysis are not eligible for the instrument. The following table shows the results of validity test analysis of each the items tested. Table 2 The result of r count value for each instrument item No r count value r table value Valid/Invalid No r count value r table value Valid/Invalid 1 2,861 0,374 valid 19 5,871 0,374 valid 2 4,977 0,374 valid 20 2,714 0,374 valid 3 6,177 0,374 valid 21 4,238 0,374 valid 4 0,106 0,374 Invalid 22 0,012 0,374 Invalid 5 4,789 0,374 valid 23 3,576 0,374 valid 6 4,977 0,374 valid 24 2,576 0,374 valid 7 0,881 0,374 Invalid 25 5,871 0,374 valid 8 0,075 0,374 Invalid 26 2,656 0,374 valid 9 5,977 0,374 valid 27 4,288 0,374 valid 10 5,871 0,374 valid 28 3,789 0,374 valid 11 5,871 0,374 valid 29 3,802 0,374 valid 12 2,875 0,374 valid 30 3,802 0,374 valid 13 4,288 0,374 valid 31 2,875 0,374 valid 14 4,977 0,374 valid 32 6,177 0,374 valid 15 4,238 0,374 valid 33 2,834 0,374 valid 16 4,067 0,374 valid 34 2,415 0,374 valid 17 3,802 0,374 valid 35 4,045 0,374 valid 18 2,576 0,374 valid Based on the results of data analysis, it can be concluded that from 35 instruments tested there are 31 items that are declared valid. An invalid intrument is because of it does not meet the valid criteria, r count > r table. The provisions of the r table value 30 respondents 30 are 0.374. The instrument which is declared as invalid is 4 items, namely item number 4.7, 8, 194 | JISAE. Volume 6 Number 2 September 2020. and 22. Instruments are located on the dimensions of accuracy of work completion, the level of curiosity in working and responsibility in working. In detail, the stated value of the instrument is calculated as follows: Item 4 = 0.106, item 7 = 0.881, item 8 = 0.075, and item 22 = 0.012. The instrument validity test is analyzed using the Microsoft Excel computer program. CONCLUSION This research is a type of quantitative study. The research results are limited to the validity stage of the instrument. This research aims to test the validity of instruments used in evaluating On the Job Training program. The results of the study show that not all instruments tested are valid. Out of 35 instruments tested, there were 4 items eclared invalid. The four instruments are in the dimensions of accuracy of work completion, the level of curiosity in working, and responsibility in working. The results of the test and the validity of the instrument concluded that the instrument that can be used to evaluate the On the Job Training program is 31 instruments. REFERENCES Alipour, M. (2009). A Study of On the Training Effectiveness: Empirical Evidence of Iran. Internasional Journal of Bussines Management, (4) 2. Diperoleh 12 Maret 2019, dari http://ccsenet.org.journal.html/ Ammitzbøll J, Skovgaard AM, Holstein BE, Andersen A, Kreiner S, Nielsen T (2019) Construct validity of a service-setting based measure to identify mental health problems in infancy. PLoS ONE 14(3): e0214112. https://doi.org/10.1371/journal.pone.021411 Cooper, Donald R., dan Pamela S. Schindler. (2006). Metode Riset Bisnis, (2) 9. Penerbit PT Media Global Edukasi. Jakarta Djaali & Pudji Muljono. (2008). Pengukuran Dalam Bidang Pendidikan. Jakarta: PT. Grasindo. Dimas Anugrah Ardian. (2018). Pengaruh On The Job Training Dan Off The Job Training Terhadap Kemampuan Kerja Dan Kinerja. Jurnal Administrasi Bisnis (JAB), 62 (1). Emilia. (2016). Investigating the extent to which on the job training influences job satisfaction and examining other contributory Factors in the Irish retail sector. Research Declaration Form for Submission of Dissertation National College of Ireland. Diperoleh 10 Maret 2019, dari https://core.ac.uk/reader/80766822 FIELD, A. P. 2005. Discovering Statistics Using SPSS, Sage Publications Inc. Ghauri, P. & Gronhaug, K. 2005. Research Methods in Business Studies, Harlow, FT/Prentice Hall. Gall, Meredith D., Gall, Joyce dan Borg, Walter R. 2007. Educational Research: An Introduction. New York: Pearson Education. Ghozali, Imam. (2012), Aplikasi Analisis Multivariate dengan Program IBM SPSS 20”. Semarang : UNDIP. Growth – Marnat, G. (2010). Handbook of Psychological Assessment. Terj. Soetjipto, H.P & Soetjipto, S.M. Yogyakarta : Pustaka Pelajar Irfan (2012). Implementasi Program On The Job Training (OJT) Dalam Mempersiapkan Siswa SMK Memasuki Dunia Kerja. Skripsi, Fakultas Keguruan dan Ilmu Pendidikan (FKIP), Universitas Sebelas Maret Surakarta. Mulyadi (2018). Evaluasi Program Magang Pada Penyelenggaraan Pendidikan Smk Dengan Model (3+1) Program Keahlian Kehutanan Di Smk Negeri 1 Pagelaran – Cianjur (Implementasi Model CIPPO). Jurnal evaluasi Pendidikan.(9)1. Permendistik. 2017. Undang-undang Ri No. 3 Tahun 2017. Tentang Pedoman dan Pengembangan Sekolah Menengah Kejuruan Berbasis Kompetensi. http://ccsenet.org.journal.html/ https://doi.org/10.1371/journal.pone.021411 https://core.ac.uk/reader/80766822 195 | JISAE. Volume 6 Number 2 September 2020. Stufflebeam, Daniel L. dan Shinkfield, Anthony J. 2007. Evaluation Theory, Models, and Application. San Francisco: Jossey-Bass. Sugiyono. 2010. Metode Penelitian Pendidikan Pendekatan Kuantitati, Kualitatif R & D. Bandung: Alfabeta.