Research and Evaluation in Education e-ISSN: 2460-6995 † Deceased 15 May 2015 Research and Evaluation in Education Volume 1, Number 2, December 2015 (pages 129-145) Available online at: http://journal.uny.ac.id/index.php/reid DEVELOPING A MODEL OF COMPETENCY CERTIFICATION TEST FOR VOCATIONAL HIGH SCHOOL STUDENTS 1) Pardjono; 2) Sugiyono; 3) †Aris Budiyono 1)2) Yogyakarta State University, Indonesia; 3) Semarang State University, Indonesia 1) pardjono@uny.ac.id; 2) sugiyono_1953@yahoo.com; 3) aries_budiy@yahoo.com Abstract This study aims to develop, produce, and investigate the appropriateness of model of competency and expertise certification tests for vocational high school (VHS) students of the Mechanical Engineering expertise competency. To attain the objectives, the researchers conducted a research and development study consisting of 10 steps. The research product was validated by experts, VHS teachers, and lecturers at Mechanical Engineering Education through Focus Group Discussion (FGD), and the field tryout conducted at Warga Surakarta VHS and Bhineka Karya Simo VHS, Boyolali, Central Java. The results of the study are: (1) The study produces a model of Competency and Expertise Certification Tests based on the School Production Unit (CECT_SPU) for VHS students of the Mechanical Engineering expertise competency; (2) The CECT_SPU model satisfies the criteria for a good model by a mean score of 3.557; (3) The mean score of the model implementation in the tryouts are 3.670 in the individual tryout and 3.730 in the small-group tryout; (4) The CECT_SPU model satisfies the criteria for an effective model by a mean score of 3.730; (5) The CECT_SPU model satisfies the criteria for an efficient model by a mean score of 3.780; (6) The CECT_SPU model satisfies the criteria for a practical model by a mean score of 3.700. Keywords: CECT model, VHS students, mechanical engineering, SPU mailto:1)aries_budiy@yahoo.com mailto:1)aries_budiy@yahoo.com mailto:sugiyono_1953@yahoo.com Research and Evaluation in Education Developing a model of competency certification test... - 130 Pardjono, Sugiyono, & †Aris Budiyono Introduction The implementation of vocational education, including Sekolah Menengah Kejuruan (SMK) -- Vocational High School (VHS) is now entering a crucial phase in which the graduates of vocational education are at stake in their readiness in the scene of the world's labour force in the regional and global levels, both within the context of China-Asean Free Trade Agreement (C-AFTA) and the Asean Free Labour Agreement (AFLA). In addition, they also have to face the demands of the use of new-findings-based technology for the efficient use of production, which requires the availability of renewable competencies in accordance with the demands of the 21 st century workforce competencies. Coombs in Gunawan (2006, p. 4) explains that a good quality of vocational education is when the students who have undergone such education process can be accepted in the world of work fitted with the field of their expertise. Based on the afore- mentioned statement, it can be put forward that the VHS as a producer of graduates should be able to make every individual student has the ability, skill, and expertise that are relevant to the demands and needs of the workforce. Thus, vocational education cannot be removed from the existing workforce development. The development of a market- able workforce should be made by vocational education based on the needs of the market (demand driven) through an increase in competence of graduates. Statistical data of February 2011 from Badan Pusat Statistik (BPS) -- the Bureau of Statistics Center (Bureau of Statistics Center, 2011, p. 39) shows that the formation of labour industrial sector has reached 13.71 million (12.32%) of the entire work figures of 111.28 million people. It hints that the needs of the job market in the industrial sector are still quite large. This condition provides an opportunity for VHS, especially those who are in the expertise areas of Technology and Engineering competence for Mechanical Engineering, to take a role in the fulfillment of labor in Indonesia. In 2008, the number of public and private VHS graduates in Central Java are between 95% and 100%, which is, passing the range of being absorbed into employment that matches their program that is 30% up to 50%, and the waiting period to get the first job is on average of 1-6 months; the rest of them continued into college, as well as some unknown activities. VHS graduates from Technology and Engineering in Mechanical Engineering study who are required by the industry are operators of machine tools manuals, operators of Comput- er Numerically Controlled (CNC) machines, electric welding, argon welding, metal casting, and, in addition, the required soft skills of perseverance, commitment, discipline, and the ability to work together (team work) (Central Java Bureau of Regional Research and Development, 2008, p. 21). A breakthrough has been made in order to prepare Indonesian workforce to enter the scene of the world's workforce at local, national, and regional, as well as global level. It is a formulation of policy for competency- based Human Resource (HR) development (Department of National Education, 2004b, p.1). The HR is embodied in the national framework through the establishment of the Institute of Profession Certification-National Board of Professions Certification (IPC- NBPC) -- Lembaga Sertifikasi Profesi-Badan National Sertifikasi Profesi (LSP-BNSP). NBPC is a non-structural independent institute and responsible directly to the President which was established based on the Government Regulation Number 23 Year 2004, State Gazette of the Republic of Indonesia Number 78 Year 2004. The HR development policy substance within the framework of the second standardization system is the agreement of NBPC national competence which will be- come reference for training institutions and agencies as well as other agencies testing which are tied to the development of HR. Law Number 20 Year 2003 on National Education System which states that the particular purpose of Vocational High Schools is preparing students to enter the workforce. As for the work acceptance, a person should be competent, and legally formal possession was proven by, for instance, a certificate of competency as ‘skill Research and Evaluation in Education 131 - Volume 1, Number 2, December 2015 passport’ which contains the competency- based skills that are owned by the holder through competency test and certification. A certificate of competency is issued in re- cognition of someone’s competent to per- form particular work through a process of competence and certification carried out by the accredited educational or certification agencies, which are clearly expressed in article 61, paragraph 3 of Law Number 20 Year 2003 which states that: ‘certificate of competence is given by organizers of education and training institutions to learners and citizens in re- cognition of competence to perform specific jobs after graduation from competency tests conducted by an accredited educational or unit certification agencies’. A study conducted by Samsudi, et al. (2009, pp. 41-42) indicates that the model/ approach to competency test shows total of 74.28% respondents affirm that they still use project work approach with internal and external verification; but at the same time, 36.52% of respondents also provide an alternative to use the approaches/models applied by the IPC-NBPC and in general, the cost of implementation of competence by organizing school collaborate with Business and Industry (B&I) is cheaper, while other models cost is more expensive. Furthermore, there is a difference beside the characteristics of the three models of CECT. There are also the advantages and disadvantages of each model which are reviewed from the completion of CECT model in VHS, assess- ors of the test, and infrastructure in the Competency Assessment Center (CAC). Test- ing material and scoring system are also used so that the existing CECT model needs to be developed. Development is needed for perfecting one CECT model that is currently underway, and can be implemented by all the VHS, that is, Final Task Project (FTP) model through the improvements from the disadvantages. Therefore, the developed CECT of VHS students is expected to be able to (1) measure the competence of students, according to the standards established by the workforce to avoid mismatch and under qualified graduates, (2) contain skill needed in the future (future skill), (3) be implemented by VHS in competence of Mechanical Engineer- ing in general, and (4) be accessed by all VHS students. Based on the previous description, the research problems are outlined as follows: (1) how to develop model of competence and certification for VHS students on Mechanical Engineering expertise competence?; (2) what is the effective, efficient, and practical model of competence and certification for VHS students on Mechanical Engineering expertise competence?; (3) what is the appropriateness of the developed model of competence and certification for VHS students on Mechanical Engineering competence?. In addition, the objectives of this study are: (1) developing a model of competence and certification for VHS students on Mechanical Engineering expertise competence based on the existing models; (2) producing an effective, efficient, and practical model of competence and certification for VHS students on Mechanical Engineering expertise competence; and (3) finding out the feasibility of the developed model of competence and certification for VHS students on Mechanical Engineering competence. Research Method Type of Research The target of this study was discovering an effective, efficient, and practical model of competence and certification for VHS students on Mechanical Engineering expertise competence. Therefore, the research method which was used in this study was Research and Develop-ment method (Borg and Gall, 1989, p.782). Subject of Research The subjects of this research were the students of Mechanical Engineering program of Vocational High School (VHS) – Sekolah Menengah Kejuruan (SMK) in Central Java, Indonesia, namely: SMKN 2 Wonogiri, SMKN 2 Purwokerto, SMKN 1 Adiwerna Tegal, SMKN 1 Semarang, SMKN 2 Surakarta, SMK St. Mikael Surakarta, SMK Research and Evaluation in Education Developing a model of competency certification test... - 132 Pardjono, Sugiyono, & †Aris Budiyono Warga Surakarta, SMK Ganeshatama Boyolali, and SMK BK2 Simo Boyolali. Procedure of Development The development procedures which were conducted in this study included: (1) Preliminary studies and collection of inform- ation (research and information collection); (2) planning; (3) initial product development; (4) initial field trials (preliminary field testing); (5) main product revision; (6) main product field trials (main field testing); (7) operational product revision; (8) operational product field trials (operational field testing); (9) final product revision; (10) dissemination and implementation. Technique of Data Analysis The techniques of data collection which were used were observation, interview, and documentation. The instrument which was used to perform the initial study was the guideline of the interview, while questions were employed for the instruments used in the validation of the design model, initial product trials, trials of the main product field test products and operations. Quantitative data and qualitative descriptive analysis were employed in analyzing the data. At the preliminary study stage, descript- ive introduction was analyzed interactively (interactive models of analysis) referred to five components: reduction of data analysis; triangulation; serving data; verification; and withdrawal of the conclusions, that carried out simultaneously and mutually interacting starts from the process of data collection. The collected data then was analyzed descriptive- qualitatively. The analysis was performed toward the research instruments, models’ validity, guides, modules, completion, effect- ivity, efficiency, and practicality. The analysis of assessment instrument of the model (validity and reliability research instrument) was performed by experts. The validity limit value is 0.8 (Guilford, 1936, p.279) and the limit value of the coefficient of reliability is 0.7 (Nunnaly, 1981, p.245), then the instruments which were used were valid and reliable. Effectivity, efficiency, and practicality of the model at the expanded time trials was determined through criteria which referred to score classification into level of evaluative meanings according to Azwar (2003, p.157). The score classification refers to (M + 1, 5s) < X; (M + 0, 5s) ≤ X ≤ (M-0, 5s); (M + 0, 5s) ≤ X ≤ (M + 1, 5s); (M- 1, 5s) ≤ X ≤ (M-0, 5s); ≤ (M + 1, 5s), with M = average rating, and s = standard deviation (Azwar, 2003, p.163). Findings and Discussion Preliminary Development Result The Urgency and Characteristics of CECT CECT model was developed based on the data which were found in the field through the preliminary study. A preliminary study of the activities was needed to formulate a model of CECT which already exists. Before examining the existing models, survey was held to identify the urgency of the carried out CECT for VHS, especially in Mechanical Engineering expertise competence and characteristics of existing CECT expected by VHS. Nine VHSs or 100% opined that the urgency of holding CECT was as a learning evaluation of a productive program, as a requirement of graduation, and as a measure- ment of competence achieved by students. Meanwhile, CECT for VHS students on Mechanical Engineering expertise competence were implemented to meet the requirements of working in the world of Business and Industry (B&I), five VHSs (55.56) agree, two VHSs (22.22%) strongly agree, and two VHSs (22.22%) disagree. The characteristics of CECT for VHS students on Mechanical Engineering expertise competence are described as follows: (1) it is conducted by the B&I, the Professions Association or the Institute of Profession Certification of Metal Machinery Indonesia (IPC-MMI) involving the disputing parties of VHS; (2) the involvement of B&I in the VHS is required in study plan, mainly determines the standard of competency and curriculum learning programs, productive implement- ation, and CECT for VHS students; (3) the graduation standards, testing material, and Research and Evaluation in Education 133 - Volume 1, Number 2, December 2015 assessment criteria of CECT by school, National Education Standards Board, and graduates user (B&I) are needed; (4) a place for competency test or Competency Assess- ment Center (CAC) takes place at school or in B&I; (5) the assessor of Competency and Expertise Certification Tests (CECT) is a prolific teacher who is certified as assessor, assessors from the B&I; (6) the requirements of students who can follow a CECT are not determined by the B&I; (7) a CECT is said to be successful if the students pass and get a certificate; (8) a certificate of competency plays not only as a condition for graduation but also as a work acceptance in B&I. Existing Model Description In relation to the components of the management of CECT which include plan- ning, organizing, implementing, evaluating and reporting, there are three models of CECT implementation found in the nine VHSs, especially in the expertise competence of Mechanical Engineering. The first model (01) is a model of CECT which follows models released by the Ministry of Education and Culture. The regulation used is the Standard Operational Procedure for National Examination (SOP_NE) published by the National Education Standardization Board, and the Technical Directive of Expertise Competency Test (TD_ECT) published continuously by the Fostering Directorate of Vocational High Schools (FDVHS). This model is implemented by seven of nine VHSs (78%) surveyed which were State Vocational High School (SVHS) -- Sekolah Menengah Kejuruan Negeri (SMKN) 2 Wonogiri, SMKN 2 Purwokerto, SMKN 1 Adiwerna Tegal, SMKN 1 Semarang, SMKN 2 Surakarta, SMK Ganeshatama Boyolali, and SMK BK2 Simo Boyolali. The second model (02) is a model of CECT that follows the models created by the Ministry of Education and Culture, coupled with the implementation of School Exam (test of competence at the level of school). This model uses the standard VHS product in the school's production units (SPU) starting from the type of work, materials and quality workmanship. Exam school that uses the existing SPU product standards allows students to do various works according to the work made by SPU, while institutionally SPU replaces the role of B&I because SPU does manufacture product as done by B&I. VHS that implements this model is SMK Warga Surakarta. The third model (03) is the model of CECT that follows the model developed by the Institute of Profession Certification (IPC). The involvement of B&I in this case is the Production Unit (PU) of St. Mikael Academy of Industrial Mechanical Engineering, Surakarta, Central Java -- Akademi Teknik Mesin Industri St. Mikael Surakarta, starting from: (a) Material testing, assessment criteria, graduation standard as required by PU, (b) the material appropriate to PU requirements, (c) Assessors by PU trainer, and (d) results/work pieces used by PU. Production Units as a representation of the B&I plays a very important role. VHS that implements this model is SMK St. Mikael Surakarta. SMK St. Mikael is also appointed by the National Board for Professions Certification (NBPC) and they serve the implementation of CECT for other VHSs that require it. Result of the Development Design of SPU-Based CECT Model The design of CECT model developed was based on the existing model, conceptual model, and research framework. The three basis of the development can be outlined as follows. Conceptual Model. Work Based Learning (WBL) approach is a series of learning as a whole through competency-based education and training from: planning (curriculum) (Work Based Curriculum), also known as competency based curriculum; work-based learning (Competency Based Training); and evaluation of competence-based assessment using the work (Work Based Assessment). WBL is a lesson or a college/school in which the program works together with the organization creating new opportunities of learning and experience in the place of work (Boud, 2001, p.6). Research and Evaluation in Education Developing a model of competency certification test... - 134 Pardjono, Sugiyono, & †Aris Budiyono The engagement of B&I is required from the very start of planning step to the implementation of the competence test. Given that school is the supplier of the resources which are needed, B&I need to fulfill the necessary competencies. Thus, it is needed in order to avoid mismatch and under qualified graduates. Regarding the particular implementation of the existing CECT in VHS, CECT should be implemented properly so that the goal of CECT can be achieved. In achieving its goal, CECT models which are effective, efficient, and practical are required. The management of CECT includes planning, organizing, implementing, evaluating, and reporting, as described by some scholars such as George, Terry, and Luther Gulick (in Handoko, 1999, p. 9) about the management functions. The components of each stage of the management are as follows: (1) planning, which comprises: engagement party user, testing material, assessment criteria and graduation standards, infrastructure, and the requirements of participants (students) who follow CECT; (2) a cooperation mechanism at the organizing industry in Competency and Expertise Certification Tests (CECT); (3) execution, which is consisting of: a place for competence test, assessors, and the duration of CECT; (4) evaluation, which consists of: competency certificate and evaluation program of CECT; (5) reporting in a form of graduation assessment and reporting CECT program. As a concept, it can be summarized that the conduct of vocational education on the basis of WBL which is ranging from planning, learning, evaluation, to the test of competence always involves B&I. Planning study in VHS involves B&I sync through competence and curriculum. Next, the learning process involves B&I in the form of a field-work practice, industrial-work practices, Double System Education, internships, and more. At the end of educational process, competence test is carried out. The competence test involves B&I, Profession Association, and the Institute of Profession Certification (IPC). Finally, the management of CECT is teaming up involving B&I, the Profession Association, and Institute of Profession Certification (IPC) using the principles of management as explained before. Existing Model. Based on the research data with questionnaires as data-collecting instrument, nine VHSs in Central Java, especially in Mechanical Engineering expertise competence associating with the component management of CECT including planning, organizing, implementing, evaluating and reporting; it can be said that the CECT management can be implemented in three models. The existing models were: the first model (01), second model (02), and third model (03). The first model (01) is a model that is widely used, in which a whole series of competencies and certifications test manage- ment which are ranging from planning, organizing, implementing, reporting, and evaluation, follows the rules which are issued by the Ministry of Education and Culture. The regulation which is used is heading the National Examination (NE) which is published by the National Education Standardization Bureau and the Technical Directive of Expertise Competency Test which is published by the FDVHS. As for the vocational practice exam at school level, it employs the pattern of students working on a practical matter, such as practices on the final exam of the semester. The second model (02) is a model of a whole series of the competencies and certifications test management, which are ranging from planning, organizing, implementing, reporting and evaluation, which is organized by the Ministry of Education and Culture, in this case is NBPC/FDVHS who published SOP_NE and TD_ECT; but the schools exam is planned to also take advantage of the existing potential in a school production unit (SPU) that has existed in the VHS. SPU, which has run well replacing B&I in planning, especially for a competence test, is conducted by the schools outside the national exam. The Schools Exam applies that: (a) the material tests in accordance with the needs of the SPU, (b) the material which is used is appropriate with the Research and Evaluation in Education 135 - Volume 1, Number 2, December 2015 SPU requirement, (c) the assessors are from SPU trainer, and (d) the results/work piece is used by SPU. The specificity of this model is on the implementation of a vocational school practice exam, which is carried out through SPU empowerment that had existed at the school. A job that is in the PU plays as a material which is made for school exams to test the expertise so that the resulting product/goods at the time of the test is used as a product of the SPU. The third model (03) is the CECT model that follows the model developed by the Institute of Profession Certification (IPC) and CECT models for schools that have a teaching factory as the representation of B&I. The involvement of B&I in this case is the Production Unit (SPU) of St. Mikael Academy of Industrial Mechanical Engineering, Surakarta, which is ranging from the planning stage until the determining graduation phase (competent or incompetent). The features of this model are: (a) testing material, assessment criteria, graduation standard as required by PU, (b) the material used is appropriate as PU requirement, (c) the assessors are from PU instructors, and (d) the results/work pieces are used by PU. The PU here is a factory which is located in an institution (St. Michael Academy of Industrial Mechanical Engineer- ing, Surakarta, Central Java). Testing material, assessment criteria and graduation standards are issued by National Board of Professions Certification (NBPC). VHS that implements this CECT model uses industry/PU as CECT or uses the facilities and infrastructure of the respective VHS with standards which are set forth by NBPC. Based on the existing three models, it can be inferred that one of the weaknesses in the competence test is the involvement of B&I. The involvement of B&I in the competence and skill certification should be formulated by means as follows: (1) determination of the criteria and graduation standards, (2) the creation of questions/ material of the test, (3) verification of the tools and infrastructure (tools and machinery) in CAC, and (4) the assessors. Framework of CECT Model Development The difference of the three models is analyzed from: the system, implementation process, resulting competencies, participants who are included, financing, and the recognition of B&I. Meanwhile, the advantages and drawbacks of the model are analyzed from: the implementation of CECT model in VHS, assessors, CAC infra- structures, testing material, and scoring system. Next, to formulate the development model, there will be grouping of the existing model. The groups consist of the superior model and models that can be implemented by all schools (see Figure 1). The parameters of the model are two groups: (1) a superior model, i.e. the model with the following parameters: the test material, the basis of competence creation in B&I, standards/ criteria determined by the B&I or Profession Association/IPC, assessors from the B&I or the productive teacher assessors-certified. The resulted competence is in accordance with the standards/competencies defined by the B&I. The recognition is shown with the publication of a certificate of competency; and (2) models that can be implemented, namely the model with the following parameter: the use of facilities and infrastructure for the existing school or in cooperation with other schools, all students can join CECT, the process of implementation in accordance with the schedule set by the Ministry of Education and Culture or in accordance with the Technical Directive of Expertise Competency Test issued by the FDVHS, the affordable financing, and assessors who come from B&I and prolific teacher in the VHS. The CECT superior model is an effective model because the model can be used to measure the students' competence in accordance with the standard of B&I and the certificate of competence issued by B&I. In addition, the models that can be implemented efficiently and practically are the inaugural time, cost and ease of implementation of CECT. Research and Evaluation in Education Developing a model of competency certification test... - 136 Pardjono, Sugiyono, & †Aris Budiyono 1) Testing material is made based on B&I competences 2) Assessment standard/ criteria is determined by B&I or Profession Association 3) The assessors are from B&I or productive assessor- certified teachers 4) The resulted competence is in accordance with the standard competence set up by B&I 5) The recognition of B&I is shown by issuing the certificate of competence 1) Using the infrastructure CAC which is available in the school or in collaboration with other schools 2) All students can have CECT 3) The organizing process is in accordance with the schedule set by Ministry of Education and Culture or in accordance with Competence Test from FDVHS 4) Reasonable financing is achieved 5) Assessors are from the B&I and VHS productive teacher Production Unit (PU) as the representation of B&I because: (1) To cope the common difficulties of VHS to collaborate with B&I in the making of lesson plan, learning activities, and learning evaluation (competency test) is necessary (2) Commonly, the VHS has already had PU as instructed by FDVHS (3) The implementation of school competence test available in existing model 02 has already been implemented properly (4) The PU which develops into teaching factory in a school as indicated in the third model Superior model Analysis a. The vocational education principles will be effective if: (1) the environment in which students get the training is the representation of the real situation they will face in the future, (2) the training tasks are done using the same way, tools, and machine as what are implemented in the workplace, and (3) training someone the way of thinking and working as needed in the workplace is done (Proser, 1925). b. The learning assessment of VHS should use competence based method (competence-based assessment). The learning assessment of VHS is done through the appropriate Expertise Competence Assessment. Model that can be implemented CECT model of VHS students Mechanical Engineering expertise competency based on PU Superior model and model that can be implemented Figure 1. Finding Flow of CECT_SPU Model The Formulation of Model Development Draft The draft of CECT model formulation (design) to VHS students on Mechanical Engineering expertise competence based on SPU herein after referred to ‘CECT_SPU’ was designed based on a conceptual model, the existing model, and direction of development as contained in the framework of the development. The draft is further analyzed with the use of two principles, namely the principles of vocational education and competency-based assessment to find the advantages and disadvantages of the existing model. As a result, the superior and applicable model can be found. The draft of model formulation (design) is described as follows: (1) a vocational secondary school (VHS) collaborates with School Production Units (SPU) to implement SPU-based CECT with regard to the regulation of the Ministry of Research and Evaluation in Education 137 - Volume 1, Number 2, December 2015 Education and Culture, including: determin- ing SOP_NE and standards from NBPC and also FDVHS in the form of TD_ECT and determining the financing sources of CECT; (2) joint planning between VHS and SPU in composing CECT, (3) VHS in collaboration with SPU organizes CECT, (4) VHS in collaboration with SPU carries out CECT, (5) VHS evaluates CECT, and (6) VHS reports the implementation of CECT. In order that the implementation of CECT uses a School Production Unit (SPU), its graduates are recognized by the B&I, and the SPU must be verified by the B&I in terms of facilities and infrastructure (CAC) which are owned, including: primary equipment, supporting equipment and room/space, and the certified assessors of the association profession. A sequential process of CECT can be noted in the chart as presented in Figure 2. Figure 2. The CECT_SPU Model Draft Figure Initial Product Development To discuss the draft of CECT_SPU model collaboratively, there was Focus Group Discussion (FGD) with the VHS principals in Central Java through the Principal Work Discussion Forum of VHS comprising 26 people. The initial product design of ‘CECT_SPU’ was developed from each of its components based on the theory of input and suggestions during the FGD. The development of each component of CECT_SPU is explained as follows: (a) the SPU-based CECT model of VHS students of Mechanical Engineering expertise competence is able to: (1) measure the students’ competence based on the standard, (2) contain the skills needed, (3) be implemented in VHS of Mechanical Engineering expertise competence in general, and (4) be accessed by all VHS students. Ministry of Education and Culture - Through BSNP sets up SOP_NE and standard - FDVHS publishes TD_ECT - Giving financial subsidy Vocational High School (VHS) along with School Production Unit (SPU) PLANNING ORGANIZING IMPLEMENTATION STUDENTS Material and Training Enrichment E V A L U A T IO N R E P O R T IN G Evaluation on organizing CECT Report on organizing CECT Module Expertise Certificate Competence Assessment Research and Evaluation in Education Developing a model of competency certification test... - 138 Pardjono, Sugiyono, & †Aris Budiyono planning of SPU-based CECT on VHS students on Mechanical Engineering expertise competency covers several aspects, namely personal (man), financing (money), material, methods, market, equipment (machine), and reporting (time). Planning the CECT involves two agencies/institutes namely VHS and SPU considering input in the form of regulation of the Ministry of Education and Culture and in the form of SOP_NE issued by NBPC, financing, and TD_ECT issued by FDVHS; (b) organizing is working mechanism between the organizers of the CECT in VHS and SPU that concerns with the results of the planning and preparation implementation of the SPU- based CECT by involving a wide range of party as formulated in the committee. The components include: CECT Committee, the material/test, assessment criteria, graduation standards, standards of infrastructure (CAC), the requirements of participants, and CECT schedule. Coordination is also undertaken to prepare the site, materials, practices and equipment to be used; (c) implementation is a stage of the implementation of the entire plan that had been drawn up. The activities in this stage is the implementation of students' competence test with the availability of the test place, test questions, practice materials, tools (machine used), assessors, duration of the time provided, and the guidelines of the assessment. The process of implementation of the SPU-based CECT can be observed in the flowchart presented in Figure 3; (d) evaluation was undertaken with regard to two aspects, namely: evaluation of the attainment of competencies using a predetermined assessment standards, this evaluation is to determine whether the participants of the test are ‘competent’ or ‘incompetent’ by using the assessment standard and graduation guide- lines made by the assessors, and evaluation of holding of CECT ranging from evaluation on the planning, organizing, and implementation of the test, such as the adequacy of materials, time and the assessors . This evaluation was expected to be a feedback for the CECT implementation in the next year. The evaluation of the organized test was made by the committee organizers; and (e) reporting is the last activity of organizing of PU-based CECT for VHS students of Mechanical Engineering expertise competence. Reporting concerns two things, namely: reporting on achievement of the students’ competencies, and CECT in the form of a certificate in accordance with the results of the evaluation of skills competence based on the graduation standard which has been established and given to the test participants (students). Certificates are issued by SPU and are known to the head of the VHS. They also report the implementation of CECT which is a feedback to the CECT organizing at the stage of planning, implementation, and organizing. This report is given to the related parties (the school, PU and Education Service). Validation (assessment) of the early products (draft CECT_SPU model) is performed by the experts, whhich consist of expert on Education of Mechanical Engineer- ing, Mechanical Engineering Learning, Education Management, CECT implementer in school, as well as B&I. The aspects that are assessed by the validators are: (a) the basic model development; (b) School Production Unit-based (SPU-based) CECT models for VHS students of Mechanical Engineering expertise competence; (c) SPU-based CECT planning component; (d) SPU-based CECT planning procedure; (e) SPU-based CECT organizing component; (f) SPU-based CECT implementation components; (g) SPU-based CECT implementation procedure; (h) the evaluation component; and (i) the reporting component. Based on the assessment data of the model draft which are obtained from the standard deviation, the entire data is 0.414 and average data is 3.557 on the evaluation scale of 1 to 4. In order to determine the assessment model, the analysis is conducted by grouping the score into several criteria that the model is not good if M ≤ 2.936; less if 2.936 ˂ M ≤ 3.350; well if ˂ 3.350 M ≤ 3.764; and very good if 3.764 ˂ M ≤ 4.178. With reference to the classification criteria, then the draft model of CECT_SPU belongs to the category of ‘good’. Research and Evaluation in Education 139 - Volume 1, Number 2, December 2015 The Assessment of Model Guidance and the Strengthening Material and Training Modules of CECT_SPU The CECT_SPU assessment guidance model was made by the principal and productive teachers of VHS on Mechanical Engineering expertise competencies consist- ing of 15 people. From the assessment results, the standard deviation of the entire data was 0.480 and the average data was 3.370. Based on the set of criteria, CECT_SPU guidance model belongs to ‘good’ category. The CECT_SPU assessment guidance model was done by professors of Mechanical Engineering Education and productive VHS teachers on Mechanical Engineering expertise competence consisting of 25 persons. From the assessment results, the standard deviation of the entire data was 0.478 and average data was 3.710. To determine the assessment of material reinforcement module and exercises, analysis was conducted by grouping the score into several criteria. Based on the criteria, the module reinforcement material and exercises CECT_SPU belong to ‘good’ category. Testing Result of CECT_SPU Model In this study, there are five testing associated with formulating the CECT_SPU model. First, the results of the validity testing of the CECT_SPU related to the substances of the CECT management; second, the results of the validity testing CECT_SPU reviewed from the implementation of CECT_SPU; third, the results of the effectiveness testing of CECT_SPU; fourth, the results of the efficiency testing of CECT_SPU; fifth, the results of the practicality testing of CECT_SPU. The results of those five tests are presented in table 1. Table 1. Testing Results of CECT_SPU No. Data Analysis Result Limit Value Conclusion 1. Product validation (CECT_SPU) by experts M = 3.557 3.350 The model is good 2. Evaluation of organizing model M = 3.670 (P) M = 3.730 (K) 3.422 3.422 The model can be implemented 3. Evaluation of effectivity M = 3.730 3.506 The model is effective 4. Evaluation of efficiency M = 3.780 3.569 The model is efficient 5. Evaluation of practicality M = 3.700 3.468 The model is practical The test results indicate that according to experts, CECT_SPU model is good. Seen from the CECT_SPU components consisting of planning, organizing, implementing, evaluating and reporting, each component has correlations (reciprocal links) between one another, and the respective attributes present- ed in the components have a correlation between inside and outside indicators present- ed in the other components. The CECT of VHS students on Mechanical Engineering expertise competency based on SPU covers several aspects, including personnel, funding, material, method, market, machines, and time. In planning, CECT involves two agencies: vocational school and SPU with regards to input in the form of the Ministry of Education and Cultures’ regulations. In the implementation of field trials, CECT_SPU is declared as enforceable in VHS samples. The next test was carried out after the extended CECT_SPU model is declared as effective, efficient and practical model. Thus, CECT_SPU model is a good model, so that it can be implemented effectively, efficiently, and practically. Product Revisions Revision of CECT_SPU Model In producing CECT_SPU model, the final product has been passed revisions several times. Revisions were done four times in accordance with the stage of the method used to develop the model. The first product revision was revision of the initial products Research and Evaluation in Education Developing a model of competency certification test... - 140 Pardjono, Sugiyono, & †Aris Budiyono resulted from the activities of the FGD planning model and expert judge-ment in the internal test. The second product revision was the revision of the main products made after the initial field products test (individual test). The third product revision was the revision of the operational product revisions made after the main product field test (a small group test). The fourth product revision was the revision of the final products made after the operational field test (extended test). The Guidelines of Revision and CECT_SPU Module The revision or refinement of guidelines and CECT_SPU modules was made based on input from the evaluator's guide and trial or experiment. Based on the revisions that have been made, the final model was produced as shown in Figure 3. Figure 3. Final CECT_SPU Model SPU-based CECT model of VHS students of Mechanical Engineering expertise competence is able to: (1) measure the students’ competence according to the standard, (2) contain the skills needed, (3) be implemented in VHS of Mechanical Engineering expertise competence in general, and (4) be accessed by all VHS students. C O M P E T E N T IN C O M P E T E N T Vocational High School (VHS) along with School Production Unit (SPU) Ministry of Education and Culture - Through BSNP set up SOP_NE and standard - FDVHS publishes TD_ECT - Giving financial subsidy Implement the CECT based on: a. Testing material, equipment, material and time allotment have been set b. Set assessor c. Assessment guideline STUDENTS Material and Training Enrichment E V A LU A T IO N R E P O R T IN G Evaluation on organizing CECT Report on organizing CECT Module Expertise Certificate IM P LE M E N T A T IO N ORGANIZING P LA N N IN G Construct the committee (man) and CECT assessor Set the CECT method Set the graduate market Construct the material for CECT: Material/tests questions, assessment guideline, graduation criteria and testing material Set the CECT Equipments Finance planning (money) Plan the reporting system (minute) Coordinating the planning results: Assessor, CECT place, testing resouces, testing material and equipments; place; practice material; tools Competence Assessment Research and Evaluation in Education 141 - Volume 1, Number 2, December 2015 The Analysis of Final Product The Technical Specification, Characteristics, and the Advantages of the Product Product Specifications. The product’s name is model of competence test and certification for VHS students on Mechanical Engineering expertise competency. The Competency and Expertise Certification Tests based on the School Production Unit (CECT_SPU) content model consists of CECT manage- ment components of VHS students, namely: planning, organizing, implementing, evaluat- ing, and reporting. The usage of the model is managing VHS students’ competence and certification test, especially in their Mechanical Engineering competence. The model tools include (1) the guidelines of the implementation of CECT_SPU and (2) the module for enrichment material and training. Product characteristics. CECT_SPU model was developed based on the concept of Work Based Learning (WBL) on vocational edu- cation using vocational education principles (Proser, 1925) and principles of Competency Based Assessment so that each component developed in CECT_SPU always involved B&I in this SPU. CECT_SPU model is a model working with SPU which is the representation of B&I so that it can cope with the difficulties of VHS in collaboration with B&I, especially in the implementation of CECT. It increases the relevance of graduates using the graduation standards, material tests, and assessments criteria done by the VHS with the SPU so the students are tested based on the competencies required by the B&I at the moment. CAC may use the existing infrastructures in VHS which are concerned or collaborated with other similar VHSs. The assessors who examine the students come from the industry as well as a prolific teacher who has been certified as an assessor obtained from industry through Professional Associations. All students can follow CECT as national exam requirements. A certificate of competence is given after the students are declared as competent. Those who were incompetent are given the opportunity to repeat or take a remedy. Product Advantages. CECT Model which involves B&I in this case is the SPU ranged from planning, organizing, implementing, evaluating, and reporting in the form of a certificate of competence. The CECT_SPU model is able to describe the VHS students’ skill competencies that are relevant to standard competencies needed by business/ industry (B&I). The device’s inputs that are required in the implementation of CECT, such as: material/test questions, assessment criteria, graduation standards, and infra- structure standard required by B&I become the reason of the choice, so that the aspects are tested to meet the requirements of B&I. The model can be implemented by all VHS which have had SPU at school or collaborated with SPU from another school. The CECT_SPU model which has assessors is recognized by B&I. It can be followed by all students who are eligible to take the exam nationwide. CECT_SPU model is a form of CECT model which provides an opportunity for the participants who have not passed the test to follow remedial tests and later, they are provided the module for enrichment training and materials. CECT_SPU Model The skill passport contains the competency-based skills owned by the holder. It is obtained through competence tests and certification. A certificate of competency is issued in recognition of someone who is competent to perform a particular work through a process of competence test and certification carried out by the accredited educational or certification agencies. The regulation is clearly explained in article 61, paragraph 3 of Law Number 20 Year 2003 which explains: ‘the certificate of competence is given by organizers of education and training institutions to learners and citizens in recognition of competence to perform a specific work after they were passed the competency tests conducted by an accredited educational or unit certification agencies’. Research and Evaluation in Education Developing a model of competency certification test... - 142 Pardjono, Sugiyono, & †Aris Budiyono The graduation standards imposed on CECT by the school are created by NBPC (NBPC under the Ministry of National Education), whereas the world of business/ industry (B&I) has a National Working Competency Standard of Indonesia developed by the Department of Labor. Therefore, both of the standards need to be set up to avoid ‘mismatch’ between the expertise competenci- es yielded by the education world and skills competencies required by the workforce. The distinction of the standards is also related to (a) the types of jobs that are realistic (real world), while the school provides only simulation practice, (b) the quality of the work results in the industry which is measured by stating accepted or rejected, while in the school, the quality is stated by scores (0-100), and (c) the risk of financial failure in the real industrial world, while in school, there is still a lot of tolerance to redo the work (Sidi, 2000, p.3). The students are declared that they are passed or competent in the competence and skill certification if they meet the criterion of graduation which has been set. The graduation criterion is the minimum require- ment for being passed (Department of Education and Culture, 2014). The graduation criterion is set in the score of 7.0. Some CECT models have been set and developed until now, including (Mone, 2004c, p.5): (a) the implementation of certification is carried out by the school along with B&I which becomes its partner institution; (b) the implementation of competency certification is organized by a particular industry that has the nationwide recognition, for example, in the field of machinery done by St. Mikael Academy of Industrial Mechanical Engineer- ing, Surakarta; (c) the implementation of certification is carried out by IPC. Because the model developed is focus- ed on the management, based on the validation of the CECT indicators and components when the survey is conducted, the main components of CECT are planning, organizing, implementing, evaluating, and reporting. This statement is in accordance with the statement of Luther Gulick (in Handoko, 1999, p.9) who defines manage- ment as one of science branches because management is regarded as a field of knowledge which systematically understands why and how people work collaboratively to achieve goals and make the system work better and benefit humans. The CECT management which is proposed is the process of planning, organizing, conducting, and controlling the CECT including all its aspects so that the goals of CECT competency test in which to perform and certify students could be achieved effectively and efficiently. The development concept of the CECT model was based on three things. They are the conceptual model, the existing model, and the development direction. Conceptual Model The conceptual model as the reference is constructed from the concept of work- based learning, also known as WBL. WBL is a series of a whole learning through the competency-based education and training including: (1) planning (curriculum) or (Work Based Curriculum), known as Competency- Based Curriculum, (2) Competency Based Training, and (3) the competency evaluation using Work Based Assessment. The implementation of WBL in VHS cannot be separated from the role of B&I as explained by Proser (1925) that (1) vocational education will be efficient if the environment where the students are trained is a replica of an environment where later they will work, (2) an effective vocational education can only be given when tasks are carried out in a manner, device, and machine that are the same as those set out at work and (3) vocational education will be effective if it coaches someone in the habit of thinking and working as required in the work. Existing Model There are three existing models which are found in the implementation of CECT for VHS students on Mechanical Engineering expertise competence. The models are then named as the first model (01), the second model (02), and the third model (03). After examining the advantages and disadvantages of each model, it can be deduced that the Research and Evaluation in Education 143 - Volume 1, Number 2, December 2015 models which are effective and efficient can be implemented or practiced. By using the analysis based on the principles of vocational education (Proser, 1925) and competency- based assessment (Directorate of VHS, 2013), it can be concluded that there are two patterns of CECT model: efficient and effective model, and model that can be implemented. The Development Direction of CECT The model development has been directed to produce CECT model which is effective, efficient, and practical. It is compatible with the purpose implemented by CECT which is strengthened by the conceptual model of work-based learning and the existing models that are later analyzed based on the principles of vocational and work-based assessment. The development was focused on components and indicators of the management for VHS students of mechanical engineering expertise competence including planning, organizing, implementing, evaluating and reporting. Conclusions and Suggestions Conclusions Based on the developmental studies using the R&D approach and the study of the products as stated before, it can be concluded as follows: (1) the development of a model of Competence and Expertise Certification Test (CECT) for students of mechanical engineer- ing expertise competence is based on three things: (a) the concept of Work Based Learn- ing (WBL) on vocational education principles (Proser, 1925) and principles of competency- based assessment. Each component which was developed in the model always involves B&I which in this case is the school production units (SPU); (b) the existing CECT models which have been found is CECT model that is implemented based on the SOP_NE, published by the National Standardization Education, and Technical Guidelines Agency for ECT, published by FDVHS, the CECT model-based SOP_NE, and technical guidelines for ECT and also test-based SPU in every school, and CECT model developed by the Institute of Profession Certification (IPC); (c) the development direction of the CECT model is excelled (effective and efficient) and can be implemented (practical) by VHS in particular mechanical engineering competency; (2) Produced models of Competence Test and Expertise Certification for VHS students of mechanical engineering competency based on the School Production Unit (CECT_SPU) consists of management components, which include: planning, organizing, implementing, evaluating, and reporting. It is used as a model to manage the testing and certification expertise competency for VHS students, especially mechanical engineering expertise competence. The models are equipped with a manual on the CECT_SPU implementation and the modules for enriching materials and training; (3) The CECT_SPU Model meets the eligibility requirements to be implemented based on the assessments as follows: (a) the validation results indicate that CECT_SPU model is a good model. It is in accordance with the results of the expert judgment 3.557. By using the criteria of a good model if 3.350 < M ≤ 3.764, the assessment results have an M above 3.350. Therefore, the CECT_SPU model belongs to ‘good’ category. If it is seen from the CECT_SPU components which consist of planning, organizing, implementing, evaluating and reporting; then each component has correlations (reciprocal links) between one another. The management system of CECT_SPU includes several important aspects, namely personnel or man, funding or money, material, methods, market, machine and time; (b) the assessment result indicates that CECT model is a model that can be implemented. This result is in accordance with the result of the implement- ation model assessment in the trial stage. The average value is 3.670 in individual trial and 3.730 in small group trials, by using the criteria that the model can be implemented if 3.422 ˂ M ≤ 3.907 for individual trial and 3.954 < M ≤ 3.499 to test small groups, then the CECT_SPU model can be implemented; (c) the assessment results indicate that the CECT_SPU model is a model which is effective. It is based on the results of the Research and Evaluation in Education Developing a model of competency certification test... - 144 Pardjono, Sugiyono, & †Aris Budiyono assessment model of M 3.730 which is above 3.506 making the model operational field test on CECT_SPU (an expanded test) belongs to ‘effective’ category. The indicators of effectiveness of the model is a cooperation between VHS and business/industry world (B&I) which in this case is the SPU as the representation of B&I, the required competencies at SPU as needed in B&I, VHS which has an SPU can apply this CECT_SPU model, the test materials depends on works provided in the SPU, the CECT_SPU can measure VHS student competencies, and there is the materials and training enrichment and remedial tests for the students who are incompetent or failed; (d) the assessment results indicate that the CECT_SPU model is a model which is efficient. It is in accordance with the results of the assessment model of M above 3.569 in this case is 3.780, so the CECT_SPU Model on the operational field test (test the extended) belongs to ‘efficient’ category. The CECT-SPU Model is efficient because there is time savings in the management of CECT because of the SPU is located in the school environment. It can be implemented in accordance with the agenda of the school. The costs are cheaper because the School Production Unit (SPU) was already in the respective schools or students can carry out their own SPU in their school because the SPU is in their school environment and does not require a great effort to establish a partnership with B&I because the B&I has already been represented by SPU. The assess- ment results show that the CECT_SPU model is practical. It is in accordance with the results of the assessment model of the Mpr over 3.468 that is 3.700, so CECT_SPU models in the operational field test (the extended test) belong to the category of ‘practical’. The indicator of practicality of CECT_SPU model is that it can be implemented by VHS, in the very own school, does not require a complicated bureaucracy because SPU is owned by the school, is easy to communicate to B&I which is represented by SPU, and the assessors who can be derived from SPU. Suggestions The utilization of research results and products are intended to the following: (1) CECT_SPU is expected to be utilized by the VHS on Mechanical Engineering expertise competency particularly as alternative models for overcoming adversity VHS in collaborat- ing with B&I which happened so far. It is very important because in fact, vocational education must continue to collaborate with B&I so that the resulting graduates do not mismatch and under qualified according to the B&I requirements; (2) CECT_SPU is expected to be utilized by the Directorate of Construction of the VHS as a reference in order to continue to develop appropriate models of CECT assembled on the field and the dynamics of the industry. This model is a model of early CECT development, according to the policy development of the Directorate Construction of VHS i.e. developing School Production Units into School Teaching Factory. References Azwar, S. (2003). Tes prestasi: Fungsi dan pengembangan pengukuran prestasi belajar [Performance test: Function and development of learning performance measurement]. Yogyakarta: Pustaka Belajar. Bureau of Statistics Center. (2011). Data strategis BPS [Strategic data of the Bureau of Statistics Center]. Jakarta. Bureau of Regional Research and Develop- ment. (2008). Laporan penelitian tentang keterkaitan pendidikan dan penyedia lapangan kerja di Jawa Tengah [Research report on the interrelatedness of education and workforce provider in Central Java]. Retrieved from http://gerbangtani.com/litbang/hasil_ penelitian/2-pendidikandanlapkerja.pdf. Borg, W.R. & Gall, M.D. (1989). Educational research: An introduction (4 th ed.). New York, NY: Longman. Department of Education and Culture. (1997). Ketrampilan menjelang 2020 untuk http://gerbangtani.com/litbang/hasil_penelitian/2-pendidikandanlapkerja.pdf http://gerbangtani.com/litbang/hasil_penelitian/2-pendidikandanlapkerja.pdf Research and Evaluation in Education 145 - Volume 1, Number 2, December 2015 era global [Approaching 2020 skill for global era]. Jakarta. Department of National Education. (2004b). Direktori lembaga sertifikasi profesi dan tempat uji kompetensi [Directory of institute of profession certification and competency test center]. Jakarta: Ditektorat Jendral Pendidikan Dasar dan Menengah. Guilford, J.P. (1950). Fundamental statistics in psychology and education (2 nd ed.). New York: McGraw-Hill. Gunawan, R. (2006). Relevansi kompetensi lulusan SMK dengan tuntutan dunia kerja [The relevance of VHS graduates with workforce demand]. Retrieved from http://file.upi.edu/Direktori/FPTK/J UR._PEND._TEKNIK_MESIN/1951 05011980021-RICKY_GUNAWAN/ MAKALAH_SEMNAS_PTK_2006.pd f. Handoko, T.H. (1999). Manajemen [Management] (2 nd ed.). Yogyakarta: BPFE. Nunnally, J.C. (1981). Psychometric theory. New Delhi: Tata McGraw-Hill. Samsudi, et al. (2009). Uji kompetensi siswa SMK dalam rangka Ujian Nasional [VHS students competency test in relation to National Examination]. Kajian SMK, Direktorat Pembinaan SMK Depdiknas. http://file.upi.edu/%20Direktori/FPTK/JUR._PEND._%20TEKNIK_MESIN/195105011980021-RICKY_GUNAWAN/%20MAKALAH_SEMNAS_PTK_%202006.pdf http://file.upi.edu/%20Direktori/FPTK/JUR._PEND._%20TEKNIK_MESIN/195105011980021-RICKY_GUNAWAN/%20MAKALAH_SEMNAS_PTK_%202006.pdf http://file.upi.edu/%20Direktori/FPTK/JUR._PEND._%20TEKNIK_MESIN/195105011980021-RICKY_GUNAWAN/%20MAKALAH_SEMNAS_PTK_%202006.pdf http://file.upi.edu/%20Direktori/FPTK/JUR._PEND._%20TEKNIK_MESIN/195105011980021-RICKY_GUNAWAN/%20MAKALAH_SEMNAS_PTK_%202006.pdf http://file.upi.edu/%20Direktori/FPTK/JUR._PEND._%20TEKNIK_MESIN/195105011980021-RICKY_GUNAWAN/%20MAKALAH_SEMNAS_PTK_%202006.pdf