126 | pISSN: 1858-3598  eISSN: 2502-5791 Jurnal Ners Vol. 14, No. 3, Special Issue 2019 http://dx.doi.org/10.20473/jn.v14i3(si).17046 This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License Original Research Implementation of the CIPP evaluation model in Indonesian nursing schools Yakobus Siswadi, Grace Solely Houghty and Triulan Agustina Universitas Pelita Harapan, Jakarta, Indonesia ABSTRACT Introduction: The implementation of the Indonesian National Nursing Competency Test (INNCT) has faced several challenges, especially related to the low pass rate. The pass rate has decreased over time, but the number of examinees has increased. The aim of this study was to evaluate the nursing schools’ performance in INNCT using the Context Input Process Product (CIPP) evaluation model. Methods: A quantitative description was used in this study. The Performance Evaluation of Nursing Program Questionnaire based on CIPP was developed and used to collect the data. The participants in this study were faculty members and alumni from the nursing schools that were a member of AINEC within region V. The variables of this study were the nursing school pass rate and the school’s performance. Results: 320 participants were involved in this study. The passing rate of the nursing school in INNCT was in the moderate category. The CIPP evaluation shows that out of the four aspects, there were two (2) aspects, namely “context and input” that were in the category of having met the requirements and two (2) aspects, namely “process and product”, were in the category exceeding the requirements. Conclusion: Nursing schools have fulfilled the government regulation based on the CIPP evaluation and the nursing schools need to make a strategic plan to improve their performance and to increase their pass rate in INNCT. ARTICLE HISTORY Received: Dec 26, 2019 Accepted: Dec 31, 2019 KEYWORDS context input process product; model; implementation; model CONTACT Yakobus Siswadi  yakobus.siswadi@uph.edu  Universitas Pelita Harapan, Jakarta, Indonesia Cite this as: Siswadi, Y., Houghty, G. S., & Agustina, T. (2019). Implementation of the CIPP evaluation model in Indonesian nursing schools.Jurnal Ners, 14(3si), 126-131. doi:http://dx.doi.org/10.20473/jn.v14i3(si).17046 INTRODUCTION The Indonesian National Nursing Competency Test (INNCT) has been implemented since 2014. The INNCT is part of a standardization of registration and practices for nurses in Indonesia. The INNCT has been developed as a computer-based test. All Indonesian nursing students must take this test at the end of their program to get their nursing license. The Indonesian government expecting that by implementing INNCT, this will increase the quality (Ristekdikti, 2016). The implementation of the NNCT has been beset with several challenges, especially the declining passing rate. The passing rate in the period IV/2015 was 53.61% or 6,222 out of 10,571 examinees and in the period XI/2018, it was 41.41%, where 5,957 had passed out of the 14,383 examinees (Ristekdikti, 2018). Low exam pass rates in the licensure examination have the potential for serious negative adverse effects on the school’s reputation, student body, accreditation and the continuity of a school’s program (Brown-O’Hara, 2013; Grant, 2015; Wade, 2011). The quality of a nursing education program is evidenced by the performance of its graduates on the licensure examination (Bahari, 2015). From the Indonesian perspective, a low passing rate indicates that there are gap and disparities in the performance of nursing education. On other hand, this will impact on the availability of nurses to fulfill the needs of the country. A low passing rate will impact on the availability of nurses to fulfill the needs of the country. Indonesia has a lack of nurses; there is gap between the target and reality in term of nurses and the population ratio. The Ministry of Health stated that the nurse : population ratio was 158/100.000 in 2014 but the reality was that it was 87.65/100.000 in 2015. This means that there was a lack of 70.35 nurses. In 2019, the target is 180/100.00 (Ministry of Health of the Republic of Indonesia, https://creativecommons.org/licenses/by/4.0/ https://creativecommons.org/licenses/by/4.0/ mailto:jalina@ppukm.ukm.edu.my http://dx.doi.org/10.20473/jn.v14i3(si). JURNAL NERS http://e-journal.unair.ac.id/JNERS | 127 2016). Failures in the INNCT potentially add to the number of unemployed and this could be a burden for both their family and country. Based on this background, there is a pressing need to evaluate the Indonesian Nursing School Performance in the INNCT. The purpose of this study was to provide an insight into the Indonesian nursing school performance focused on the Indonesian National Nursing Competency Test (INNCT) using the CIPP evaluation model. MATERIALS AND METHODS A descriptive quantitative design was used in this study. The population of this study consisted of the faculty staff and alumni from the nursing schools who were a member of AINEC within region V. The convenience sample included all faculty staff and alumni who took part in the INNCT in the period 2017 through to 2018 and who either failed or passed. Variables of the study were the nursing school passing rate and school performance in INNCT. The Performance Evaluation of Nursing Program Questionnaire (PENPQ) based on the CIPP evaluation model was developed and used as an instrument in the data collection. The results are categorized based on mean score: exceeded requirement = 3.26-4.00; met requirement = 2.51 – 3.25; partially met requirement = 1.76 – 2.50; and did not meet requirement = 1.00 – 1.75. The validity test of the instrument showed that the Cronbach’s Alpha value was .977. The data collection was done by SurveyMonkey. The link to the instrument was sent to the faculty and alumni coordinator or person in charge as assigned by the nursing school administrator through SMS or Whatsapp. The informed consent form was done electronically. The participants were required to fill out the informed consent form by clicking on the “AGREE” button on the screen after reading the research information and before being given full access to the instrument. The data was analyzed using the frequency, percentage and mean. The study was approved by the institutional review board of the Mochtar Riady Institute of Nanotechnology (MRIN) protocol number 04.1807188. Table 1. Frequency and Percentage Distribution of the Socio-Demographics of the Participants according to Faculty (n=56) Characteristics n % Sex: Male Female 7 49 12.50 87.50 Age: 25 year and below 26 – 35 years old 36 – 45 years old 46 – 55 years old 56 – 65 years old 66 years and above 1 29 14 7 3 2 1.78 51.78 8.13 12.50 5.34 3.57 Education Background: Bachelor (S1) Master (S2) Doctor (S3) 10 42 4 17.85 75.00 7.15 Teaching Experience: 5 years and below 6 – 10 years 11 – 19 years 20 years and above 22 18 8 8 39.28 32.14 14.29 14.29 Table 2. Frequency and Percentage Distribution of the Socio-Demographic Qualities of the Participants who were Alumni (n=264) Characteristics n % Sex: Male Female 51 213 19.31 80.69 Age: 25 year and below 26 – 35 years old 36 – 45 years old 46 – 55 years old 209 41 12 2 79.17 15.53 4.54 .76 Alumni type: Regular Non-regular 226 38 85.60 14.40 INNCT Status: Pass Fail 201 63 76.13 23.87 Y. SISWADI, ET AL. 128 | pISSN: 1858-3598  eISSN: 2502-5791 RESULTS A total of 320 participants from six different nursing schools were involved in this study. The participants were both faculty staff (17.50%) and alumni (82.50%). Most of the faculty staff were female (49 or 87.50%). The age of the majority of the participants ranged from 26 – 35 years old 29 (51.78%. Only 2 or 3.57% were 66 years old and above. In terms of educational background, most of the participants had finished a Master’s degree or S2 with 42 (75%); 4 or 7.15% hadf inished a doctorate, or S3. With regard to teaching experience, 22 or 39.28% had experience of 5 years and below, followed by 18 participants or 32.14% who had 6 – 10 years. See on Table 1. The table shows that that the majority of the participants were 213 (80.69%). The majority of the alumni were 25 years old and below (209 or 79.17%). The alumni-participants were mostly of the regular type (226 or 85.60%) and the non-regular type consisted of 38 or 14.40%. The majority of them had passed the NNCT (201 or 76.13%), while 63 or 23.87 failed (Table 2). Table 3 shows there were two (2) nursing schools in the high category for their passing rate and four (4) nursing schools who were in the moderate category. The highest average passing rate was 91.07% and the lowest was 37.80%. Table 4 shows that the overall mean of the aspects “Context” and “Input” were in the met requirement category, while the aspects of “Process” and “Product” were in the exceeded requirement category. The sub Aspect of “Input”, such as leadership, student, faculty staff and the facility, were in the met requirement category. DISCUSSION The results of the study show that the majority (four; 66.66%) of the nursing schools were at a moderate level in terms of the average passing rate of INNCT. The highest passing rate was 91.07% and the lowest was 37.80%. These results show that there are disparities in terms of the passing rate and the gap was 56.5%. These disparities corroborate that there is still variety in the process of learning and in the education standard for every nursing program. The government, professional organization and nursing school association have released guidance or regulations such as the national standard of education and the blue print of INNCT as the basis for running a nursing program. The results also show that there were many retakes because of failure in the INNCT. This means that there were those who cannot work as professional nurses who could be potentially be an unemployed person. A strategy plan could be performed to support and help those who fail the NNCT and the nursing school should have the responsibility to help them. The preview studies confirm that many contributing factors are involved in the licensure examination. These factors can be categories sorted into academic and non-academic. The academic factors include cumulative GPA (CGPA). CGPA significantly contributes to the success of the licensure examination (Amankwaa, Agyemang- Dankwah, & Boateng, 2015; Foley, 2016; Grant, 2015; Penprase, Meghan Harris, & Qu, 2013; Ristekdikti, 2018; Siswadi Y, 2018). Moreover, the nursing subject grade was a significant predictor for licensure examination (Breckenridge, Wolf, & Roszkowski, 2012; McGahee, Gramling, & Reid, 2010; Schooley & Dixon Kuhn, 2013; Simon, McGinniss, & Krauss, Table 3. Percentage Distribution of the Nursing School Grouped According to Passing Rate School Passing Rate Average Category 2017-1 2017-2 2018-1 A 94.03 81.58 93.65 91.07 High B 59.09 28.57 45.83 45.63 Moderate C 50.38 62.26 53.76 Moderate D 30.77 32.73 43.37 37.80 Moderate E 93.18 81.07 75.00 82.17 High F 63.16 53.13 37.14 54.29 Moderate Table 4. Mean Distribution of the Assessment of the Participants with Respect to School Performance based on CIPP Evaluation Aspect Mean Category Alumni Faculty Overall Context 3.17 3.17 3.17 Met Requirement Input: 3.25 2.23 3.24 Met Requirement Curriculum 3.30 3.35 3.32 Exceeded Requirement Leadership 3.26 3.22 3.24 Met Requirement Student 3.25 3.23 3.24 Met Requirement Faculty 3.23 3.19 3.21 Met Requirement Facility 3.25 3.18 3.22 Met Requirement Process 3.29 3.24 3.27 Exceeded Requirement Product 3.23 3.38 3.30 Exceeded Requirement JURNAL NERS http://e-journal.unair.ac.id/JNERS | 129 2013). Other studies found differently, in that there were no significant correlations between CGPA and academic achievement (Siswadi Y, Sommers L C, 2017; Ukpabi, 2008). Non-academic factors that contribute to the licensure examination include socio- demography, which includes gender, age and the educational background of the parent (Amankwaa et al., 2015; Breckenridge et al., 2012). School accreditation level also contributed to the performance of the licensure examination (Dator, 2016; Gutierrez N P, 2016). The results of the CIPP evaluation showed that two aspects (50%) such as “Context” and “Input” were in the met requirement category. The aspect “context” was the focus on the vision, mission and objective of the study program. Clarity and socialization of the vision, mission and objectives to all civitas academica is needed to ensure the school’s achievements. Vision helped the administrator and their team to become inspired and committed concerning a shared goal. The vision was a strong driving force for ongoing and systematic practice development and thus it established a culture that favored quality and safety improvement in patient care (Martin, Mccormack, Fitzsimons, & Spirig, 2014). The finding implies that the nursing school has not yet reached the highest standard or exceeded requirements. The aspect “input” includes the curriculum, administrator, students, faculty and facilities. Four (4) sub aspects such as the administrator, students, faculty and facilities were rated as having met the requirements, and only curriculum exceeded requirements with a mean score 3.32. The category mean of the students and faculty was closed (3.25:3.23). The majority, or 4 out of 5, category means of the alumni were higher compared to the category mean of faculty. The overall category mean of the aspect “input” was 3.24 or met requirement. These findings corroborate that nursing schools have a problem in the majority sub aspect of “input”. The previous studies utilized the aspect of “input” and the relevant evaluation data for several proposes, such as to make decisions regarding subsequent program implementation regarding End-of-Life education program (Lippe M, 2017), for structuring decisions (Patil Y, 2015), and to help prescribe a project to address the identified needs (“Using the Context, Input, Process, and Product Evaluation Model (CIPP) as a Comprehensive Framework to Guide the Planning, Implementation, and Assessment of Service-learning Programs,” 2011). Moreover, this evaluation was used as the bases for the recommendation of revising the course's curriculum and for increasing the facilities that included a library (Mohebbi, Akhlaghi, Yarmohammadian, & Khoshgam, 2011). The aspect “process” covers the implementation of the program either in the classroom activities, in the class laboratory and or in clinical practice; it also covers the monitoring, documentation and evaluation of the implementation of the program. There were 19 items used to evaluate the implementation of the process. The category mean was 3.27, or exceeded requirement. This finding implies that the nursing schools have exceeded the requirements of the national standard in terms of the aspect of “process”. The program evaluation monitors the project implementation process to help the staff carry out activities and for users to be able to judge the program’s performance (Stufflebeam, 2003). The “process” evaluation is important in order to provide feedback to allow the program to be implemented correctly, to improve the program and to verity accountability in the work plan (Chinta, Kebritchi, & Ellias, 2016; Kahn et al., 2014; O’Sullivan, 2013; Pfitzinger M, 2016). In addition, another researcher explained that the “process” evaluation concerns the link between theory and practice and the implementation of the curriculum (Stavropoulou & Stroubouki, 2014). The aspect “product” is focused on the graduation rate, CGPA, passing rate and employment rate. There were 6 items reated to the ‘product’ aspect where the overall category mean was 3.30, or exceeded requirement. This implies that the aspect “product” was evaluated as having exceeded requirements but there was still problems related to the passing rate on the INNCT. The majority of the passing rates of INNCT were moderate. The passing rate is an indicator of the school’s quality. The quality of a nursing education program is evidenced by the performance of its graduates on the licensure examination (Bahari, 2015). A nursing school’s reputation and standing with the national board can be at risk; students have a lot of money invested in their education and they may have to wait to gain employment (Wade, 2011). The CIPP Evaluation Model has been accepted worldwide and it is used in several settings. A clear picture and detailed explanation of the application of the CIPP Evaluation model in the nursing education setting was presented (Singh, 2004). A limitation is that since the data collection used an online survey, it was a challenge to encourage the participants to participate in this study. Not all alumni were a part of the WhatsApp group or other social media in related groups. CONCLUSION The nursing schools have been fulfilling the national standard of education based on the CIPP evaluation model. There are areas of CIPP that need to be improved in order to provide a higher level of standard, especially for the aspects of “context” and “input”. Nursing schools should make a strategic plan to increase the pass rate of INNCT. REFERENCES Amankwaa, I., Agyemang-Dankwah, A., & Boateng, D. (2015). Previous Education, Sociodemographic Characteristics, and Nursing Cumulative Grade Point Average as Predictors of Success in Nursing Y. SISWADI, ET AL. 130 | pISSN: 1858-3598  eISSN: 2502-5791 Licensure Examinations. Nursing Research and Practice. https://doi.org/10.1155/2015/682479 Bahari, G. M. (2015). A Plan to Promote NCLEX-RN Success of Saudi Arabian Nurse Graduates. Austin Jpurnal of Nursing & Health Care, 2(1), 1–8. Breckenridge, D. M., Wolf, Z. R., & Roszkowski, M. J. (2012). Risk assessment profile and strategies for success instrument: Determining prelicensure nursing students’ risk for academic success. Journal of Nursing Education. https://doi.org/10.3928/01484834-20120113- 03 Brown-O’Hara, P. (2013). The influence of academic coaching on: Baccalaureate nursing students’ academic success, perceptions of the academic coaching relationship, perceived NCLEX-RN exam readiness and success on the NCLEX-RN exam. Chinta, R., Kebritchi, M., & Ellias, J. (2016). A conceptual framework for evaluating higher education institutions. International Journal of Educational Management. https://doi.org/10.1108/IJEM-09-2015-0120 Dator, W. (2016). Accreditation and its Relationship to the Performance in the Nurse Licensure Examination of Higher Education Institutions in the Philippines. International Journal of Evidence- Based Healthcare. https://doi.org/10.1097/01.xeb.0000511624.93 121.49 Foley, D. M. (2016). Predicting student success: Factors influencing NCLEX-RN rates in an urban university’s pre-licensure programs. Grant, A. R. (2015). NCLEX-RN predictor test scores and NCLEX-RN success for first attempt test takers. Available from ProQuest Dissertations & Theses Global. Gutierrez N P. (2016). Level of accreditation and board performance on the colleges of nursing in the national capital region. International Education & Research Journal, 2(5), 21–29. Kahn, K. L., Mendel, P., Weinberg, D. A., Leuschner, K. J., Gall, E. M., & Siegel, S. (2014). Approach for conducting the longitudinal program evaluation of the US department of health and human services national action plan to prevent healthcare- associated infections: Roadmap to elimination. Medical Care. https://doi.org/10.1097/MLR.00000000000000 30 Lippe M, C. P. (2017). Using the CIPP Model to Assess Nursing Education Program Quality and Merit. Teaching and Learning in Nursing, 1–5. Martin, J., Mccormack, B., Fitzsimons, D., & Spirig, R. (2014). The importance of inspiring a shared vision. International Practice Development Journal. McGahee, T., Gramling, L., & Reid, T. (2010). NCLEX- RN success: Are there predictors. Southern Online Journal of Nursing Research. Ministry of Health of the Republic of Indonesia. (2016). Indonesian Health Profile 2015. Mohebbi, N., Akhlaghi, F., Yarmohammadian, M. H., & Khoshgam, M. (2011). Application of CIPP model for evaluating the medical records education course at master of science level at Iranian medical sciences universities. Procedia - Social and Behavioral Sciences. https://doi.org/10.1016/j.sbspro.2011.04.287 O’Sullivan, C. A. M. (2013). Evaluation of a successful high risk nursing student assistance program: One ADN program’s journey. Patil Y, K. S. (2015). CIPP MODEL FOR SCHOOL EVALUATION. Scholarly Research Journal for Humanity Science & English Language, 2(2), 2615– 2619. Penprase, B. B., Meghan Harris, M. H., & Qu, X. (2013). Academic success: Which factors contribute signify- cantly to NCLEX-RN success for ASDN students? Journal of Nursing Education and Practice. https://doi.org/10.5430/jnep.v3n7p1 Pfitzinger M. (2016). Evaluating Baccalaureate Curriculum End-of-Life Care Content based upon End of Life Nursing Education Consortium (ELNEC) Guidelines. The University of Texas at Austin. Retrieved from https://repositories.lib.utexas.edu/bitstream/ha ndle/2152/39604/PFITZINGER-DISSERTATION- 2016.pdf?sequence=1 Ristekdikti. (2016). Implementasi Uji Kompetensi Nasional bidang Kesehatan sebagai Langkah Konkrit Penjaminan Mutu Pendidikan Tinggi Kesehatan. Ristekdikti. (2018). No Title. Retrieved from http://ukners.dikti.go.id/pages/statistik_lulus# Schooley, A., & Dixon Kuhn, J. R. (2013). Early indicators of NCLEX-RN performance. Journal of Nursing Education. https://doi.org/10.3928/01484834-20130819- 08 Simon, E. B., McGinniss, S. P., & Krauss, B. J. (2013). Predictor variables for NCLEX-RN readiness exam performance. Nursing Education Perspectives. https://doi.org/10.5480/1536-5026-34.1.18 Singh, M. D. (2004). Evaluation framework for nursing education programs: Application of the CIPP model. International Journal of Nursing Education Scholarship. https://doi.org/10.2202/1548-923X.1023 Siswadi Y, Sommers L C, H. G. S. (2017). Faktor-Faktor Prediksi Keberhasilan Uji Kompetensi Nasional. In Penguatan sistem uji kompetensi dalam meningkatkan kualitas profesi tenaga kesehatan untuk memperkuat daya saing bangsa di era global. Bandung: LPUK-NAKES and UNPAD. Siswadi Y, T. A. (2018). Determinants of nursing licensure examination performance: A literature review. In H. Malimi, K. Abdullah, J. McFarlane, J. Evans, & Y. Sari (Eds.), Strengthening research capacity and disseminating new findings in nursing and public health. London: Group, Taylor & Francis. Stavropoulou, A., & Stroubouki, T. (2014). Evaluation of educational programmes - The contribution of history to modern evaluation thinking. Health JURNAL NERS http://e-journal.unair.ac.id/JNERS | 131 Science Journal, Vol. 8, pp. 193–204. Technological Educational Institute of Athens. Stufflebeam, D. L. (2003). The CIPP Model for Evaluation. In International Handbook of Educational Evaluation. https://doi.org/10.1007/978-94-010-0309-4_4 Ukpabi, C. V. (2008). Predicotrs of successful nursing education outcomes: A study of the North Carolina Central Universit’s Nursing Program. Educational Research Quarterly. Using the Context, Input, Process, and Product Evaluation Model (CIPP) as a Comprehensive Framework to Guide the Planning, Implementation, and Assessment of Service- learning Programs. (2011). Journal of Higher Education Outreach and Engagement, 15(4), 57– 84. Wade, R. J. (2011). Predicting NCLEX-RNreg pass rates: A regional study in the United States.