Paper—Mobile Phone-Based CIPP Evaluation Model in Evaluating the Use of Blended Learning at… Mobile Phone-Based CIPP Evaluation Model in Evaluating the Use of Blended Learning at School in Bali https://doi.org/10.3991/ijim.v11i4.6796 Dewa Gede Hendra Divayana, Dewa Bagus Sanjaya Universitas Pendidikan Ganesha, Singaraja, Indonesia hendra.divayana@undiksha.ac.id Abstract—Through the use of information technology, the students no longer find it difficult to look for sources/learning materials, interact and dis- cuss with each other any time and any where they are. One form of the roles of information technology which can facilitate students in individual learning, group learning and learning together with the teacher in the classroom and out- side of the classroom is blended learning. The fact in schools shows that not all of the learning process can be optimized by using blended learning. Therefore, there is a need to evaluate comprehensively any time and any where they are (evaluators). For this purpose, the appropriate evaluation model to be used is mobile phone-based CIPP. By using this model, then the evaluator can study evaluating the use of blended learning overall through of the context, input, process and product that will be able to provide appropriate recommendations for the improvement of the learning process using blended learning in school. Keywords—Blended Learning, Evaluation, CIPP. 1 Introduction The presence of information technology at school broadens the insight of the school into the importance of information technology in the technology development era today. The development of information technology today help very much the learning process at school, both in the classroom between the teacher and the stu- dents and outside of the classroom between one student with another in a group/ community. Through the use of information technology, the students no longer find it difficult to find learning materials which are very easy to obtain through an in- ternet access. In addition, they can interact and discuss with each other whenever and where ever they are through social media facilities. With the ease of information access through internet, then it cannot be denied that may facilities are available to be accessed which give learning services both online and offline. One of the forms of learning services that can facilitate the students in learning independently, in groups, and learning together with the teacher in the classroom and outside the classroom both fully online and offline is blended learning. Through blended learning the teacher and the students can use information tech- nology in the learning process online and offline. In the use of blended learning, iJIM ‒ Vol. 11, No. 4, 2017 149 Paper—Mobile Phone-Based CIPP Evaluation Model in Evaluating the Use of Blended Learning at… learning quality obtained at school, that is, through direct interaction between the teacher and the students is always maintained, but the use of information technolo- gy used to facilitate the students in seeking/accessing learning resources and makes it easy for the students to seek/access learning resources and to interact/discuss with the their groups/learning communities at school and outside. By using blended learn- ing at school, the students will have increasingly more knowledge and insights so that they do not only focus on the information/ knowledge given by the teacher in the classroom, but can obtain comprehensive knowledge from various reliable sources , through social media, internet and from a community. There are some choices of platforms that can be used for blended learning in the learning process at school, which among others are Edmodo, quipper school, Moodle, Kelase, etc [1]. These platforms, obviously have their strengths and weaknesses. Gen- erally, the platforms are able to offer and provide facilities needed for learning pro- cess at school. However, the fact at school shows that not all learning processes that use blended learning can run optimally. Therefore, there is a need to do a comprehensive evalua- tion of the use of blended learning at school. The evaluation process that needs to be done by educational evaluators is an evaluation that is capable of seeking infor- mation comprehensively and accurately on the extent of effectiveness of the use of blended learning at school viewed from the components of context, input, process and output. There are some aspects evaluated concerning the use of blended learn- ing at school viewed from the component of context, which cover: vision, missions, goal, support from the school community and legal foundation for the implemen- tation of blended learning. The aspects evaluated in the component of input cover: the structure of management, readiness in terms of infrastructure and facilities, teacher’s ability, students’ readiness and management’s ability. The aspects evalu- ated in the component of process cover: the process of introducing blended learn- ing to the users, the process of teaching the users to operate blended learning, and the process of financial management in blended learning implementation. The aspects evaluated in the component of product covers blended learning service quality from the points of view of tangibles, reliability, responsiveness, assurance, and empathy. The evaluation process of the use of blended learning at school is expected to be done anytime and anywhere the evaluators are by interested parties. To overcome the problem, then the evaluation model which is appropriate to use is mobile-phone-based CIPP. By using the model, the educational evaluators can evalu- ate the use of blended learning comprehensively in terms of the components of context, input, process and product, that is done anytime and anywhere they are without any pressure from the interested parties so that they can give an appropriate recommendation to improve the process of learning by using blended learning at school. 150 http://www.i-jim.org Paper—Mobile Phone-Based CIPP Evaluation Model in Evaluating the Use of Blended Learning at… 2 Literature Review 2.1 Blended Learning Definitions of blended learning range from the very broad where practically any learning experience that integrates some use of ICTs qualifies, to others that focus on specific percentages of online curriculum and face to face instruction. Most people agree that blended learning combines teaching and learning methods from both face to face, mobile and online learning and that it includes elements of both synchronous and asynchronous online learning options [2]. Blended learning is also defined as learning that can conventionally be defined in the classroom combined with online learning both independently and in a collaboration by using information and com- munication technology facilities [3]. Blended Learning is a student-centered learning using a systematic approach that combines face-to-face learning and online learning aided by ICTs [1]. Blended learning is a student-centered approach to creating a learning experience whereby the learner interacts with other students, with the in- structor, and with content through thoughtful integration of online and face-to-face environments [4]. Blended learning is a strategic and systematic approach to combin- ing times and modes of learning, integrating the best aspects of face-to-face and online interactions for each discipline, using appropriate ICTs [5]. The core of blended learning theory is the integration of multiple teaching means, teaching media, and teaching environments [6]. Blended learning has many ad- vantages over E-learning; the most important one is that blended learning participants being able to socialize face-to-face interaction in order to motivate the less independ- ent student [7]. Based on the opinions above, then blended learning is a learning approach which combines direct face to face learning in the classroom and online learning through information technology facility and learning interaction keeps on being developed well both in the classroom and outside of the classroom. 2.2 Evaluation Evaluation is an activity for collecting, understanding and reporting the result of analysis of a particular program/object in such a way that the result can be used for consideration in making a decision whether the program will be continued or stopped [8]. Evaluation is an activity for collecting, analyzing and presenting infor- mation about a particular object under an investigation to be used as a consideration in making a decision [9]. The evaluation is an activity for data collecting, data analyzing and data presenting into information about a particular object under study so that the results can be used to take a decision [10]. Evaluation is an activity for collecting, analyzing, and explaining comprehensively information about a par- ticular object/program/policy being studied and the result can be used for a consid- eration in making a decision to continue or stop the object/program/ policy[1]. iJIM ‒ Vol. 11, No. 4, 2017 151 Paper—Mobile Phone-Based CIPP Evaluation Model in Evaluating the Use of Blended Learning at… From the various opinions above it can be concluded in general that evaluation is an activity for collecting, analyzing, and presenting, information about a particular object to be used for a consideration in making an appropriate and accurate decision. 2.3 CIPP (Context-Input-Process-Product) Evaluation Model Basically, the CIPP evaluation model requires that a series of questions be asked about the four different elements of the model on context, input, process, and product [11]. When compared with professional standards for project evaluation, and after being rated by their utility, feasibility, propriety, and accuracy, the best approach that has surfaced is the Context, Input, Process, and Product evaluation model. The CIPP evaluation model belongs in the improvement/accountability category, and is one of the most widely applied evaluation models [12]. The CIPP model is the most widely known and applied model that is used by evaluators. This model was developed by Stufflebeam. CIPP is an acronym using an initial letter of each of four words, that is, Context, Input, Process and Product [13]. One of the strengths of CIPP model is, especially, that it is a useful and simple tool for helping evaluators produce ques- tions of vital importance to be asked in an evaluation process [14]. Based on the various opinions above, CIPP evaluation model is a model that es- sentially has four stages of evaluation are: evaluation of the component context, com- ponent input, component process and component product. 3 Research Methodology 3.1 Object of Study The object of this study was blended learning at school. 3.2 Subject of Study The subjects of this study were teachers, students, and blended learning manage- ment. The subjects were determined by Purposive Sampling. 3.3 Location of Study The locations of this study were vocational high schools in Singaraja, Bali. 3.4 Methods of Data Collection The methods of data collection used were questionnaire distribution, observation, and documentation. 152 http://www.i-jim.org Paper—Mobile Phone-Based CIPP Evaluation Model in Evaluating the Use of Blended Learning at… 3.5 Design of the Study The study used CIPP model design which can be shown as follows. Fig. 1. A Design Model of Mobile Phone-Based CIPP Evaluation Model Context • Vision • Mission • Purpose • Support from the school community • Legal Foundation Main & additional instruments Degree of Effectiveness Description of the constraints Recomendations Mobile Phone Aspects being evaluated: Input E va lu at io n C om po ne nt s • Structure of management • Readiness in terms of infrastructure and facilities • Readiness in terms of teacher’s ability • Readiness in terms of student’s ability • Readiness in terms of management’s ability Aspects being evaluated: Process • The process of introducing blended learning • The process of teaching how to operate blended learning • The process of financial management Aspects being evaluated: Product • Quality of blended learning service in terms of: - Tangibles - Reliability - Responsiveness - Assurance - Empathy Aspects being evaluated: iJIM ‒ Vol. 11, No. 4, 2017 153 Paper—Mobile Phone-Based CIPP Evaluation Model in Evaluating the Use of Blended Learning at… 3.6 Technique of Data Analysis Data analysis about the implementation of an evaluation of the use of blended learning at school by using quantitative descriptive technique. 4 Results and Discussion 4.1 Results 1. The standard of the effectiveness of the use of blended learning at school The Standard effectiveness of the use of blended learning at school can be shown in Table 1 as follows. Table 1. The Standard of Effectiveness of the Use of Blended Learning at School by Using CIIP Model No Evaluation Components Evaluation Aspects Effectiveness Standard (%) 1. Context Vision 85-100 Missions 85-100 Goal 85-100 Support from the school community 85-100 Legal Foundation 88-100 2. Input Structure of management 85-100 Readiness in terms of infrastructure and facilities 85-100 Readiness in terms of teacher’s ability 86-100 Readiness in terms of Student’s ability 86-100 Readiness in terms of management’s ability 88-100 3. Process The process of introducing blended learning 85-100 The process of teaching how to operate blended learning 85-100 The process of financial management 90-100 4. Product Tangibles 88-100 Reliability 88-100 Responsiveness 88-100 Assurance 88-100 Empathy 88-100 Note: Categories of Effectiveness Standard Scales Very High 90%-100% High 80%-89% Satisfactory 70%-79% Low 60%-69% Very Low 0%-59% 154 http://www.i-jim.org Paper—Mobile Phone-Based CIPP Evaluation Model in Evaluating the Use of Blended Learning at… 2. Results of the Evaluation of the Use of Blended Learning at School The results of the evaluation of the use of blended learning at school by using Mobile Phone- based CIPP Model can be seen in Table 2 as follows: Table 2. Results of the Evaluation of the Use of Blended Learning at School by Using Mobile Phone -Based CIPP Model N o Evaluation Components Evaluation Aspects Result of Evaluation (%) B1 B2 B3 B4 1. Context Vision 86.4 85.2 85.6 85.4 Missions 85.2 85.4 85.8 85.6 Goal 85.8 85.6 85.2 85.2 Support from the school commu- nity 85.6 85.4 85.2 85.4 Legal Foundation 90.2 89.4 88.6 88.6 Average of the Context Component 86.6 86.2 86.1 86.0 2. Input Structure of management 85.6 85.2 85.4 85.2 Readiness in terms of infrastruc- ture and facilities 85.4 85.2 84.6 85.2 Readiness in terms of teacher’s readiness 87.4 86.6 85.4 86.2 Readiness in terms of student’s ability 88.2 86.8 85.6 87.8 Readiness in terms of manage- ment’s ability 89.6 88.8 87.8 88.2 Average of the Input Component 87.2 86.5 85.8 86.5 3. Process The process of introducing blend- ed learning 87.2 86.4 86.2 87.4 The process of teaching how to operate blended learning 86.4 87.2 85.8 86.8 The process of funds management 92.8 93.2 92.4 91.6 Average of the Process Component 88.8 88.9 88.1 88.6 4. Product Tangibles 89.2 88.4 87.6 88.6 Reliability 88.8 89.2 88.6 88.2 Responsiveness 89.6 88.4 89.8 88.6 Assurance 89.2 88.6 88.8 88.4 Empathy 89.8 88.8 88.4 88.2 Average of the Product Component 89.3 88.7 88.6 88.4 Notes : Types of Blended Learning Platforms : B1 : Edmodo B2 : Quipper School B3 : Moodle B4 : Kelase iJIM ‒ Vol. 11, No. 4, 2017 155 Paper—Mobile Phone-Based CIPP Evaluation Model in Evaluating the Use of Blended Learning at… 3. The Simulation of Mobile Phone Application in Evaluating the Use of Blended Learning at School by Using CIPP Model The description about the implementation of mobile phone application in evaluat- ing the use of blended learning at school by using CIPP model can be visualized in Figure 2 as follows: Fig. 2. A Simulation of Mobile Phone Application-Based CIPP Evaluation Model 156 http://www.i-jim.org Paper—Mobile Phone-Based CIPP Evaluation Model in Evaluating the Use of Blended Learning at… 4.2 Discussion Based on the results that have been described above there are some points that have to be discussed, which are among others are: a. The Structure of Mobile Phone - Based CIPP Model Application This mobile phone CIPP model application was made by using Android appli- cation with SQLite database. b. The highest result of Effectiveness Computation. The highest result of effectiveness in the use of blended learning at school by using Android application-based CIPP model can be explained as follows: 1) In the component of context the highest percentage of effectiveness ob- tained for the use of blended learning at school was 86.6% by using edmo- do platform, so that edmodo falls into the high category of blended learning platform. 2) In the component of input the highest percentage of effectiveness obtained for the use of blended learning at school was 87.2% by using edmodo plat- form, so that Edmodo falls into the high category of blended learning plat- form. 3) In the component of process, the highest percentage of effectiveness ob- tained was 88.9% for the use of blended learning at school by using quip- per school platform, so that quipper school falls into the high category of blended learning platform. 4) In the component of product, the highest percentage of effectiveness ob- tained was 89.3% for the use of blended learning at school by using edmo- do platform, so that edmodo platform falls into the high category of blend- ed learning platform. c. Constraints in each component being evaluated The constraints faced in each component being evaluated in the Android appli- cation-based CIPP evaluation can be explained as follows: 1) In the component of context there was no constraint found in the imple- mentation, since the score of effectiveness of the evaluation result exceeds the standard of effectiveness that has been specified. 2) In the component of input there were constraints in the implementation, i.e, : a) the aspect of readiness in terms of infrastructure and facilities to support moodle platform which is not yet optimal, b) in the aspect of readiness in terms of teacher’s readiness, the teachers were not yet compe- tent enough to use moodle platform. c) the aspect of readiness in terms of student’s ability, the students were not yet competent enough to use moodle platform, and d) in the aspect of readiness in terms of the devel- oper’s ability in developing moodle platform used for blended learning was not yet optimal. The evidence for the presence of the constraints is shown by the score of effectiveness in the results of evaluation which was below the effectiveness standard that has been specified. iJIM ‒ Vol. 11, No. 4, 2017 157 Paper—Mobile Phone-Based CIPP Evaluation Model in Evaluating the Use of Blended Learning at… 3) In the component of process there was no constraint found in the imple- mentation, since the score of effectiveness of the evaluation result exceeds the effectiveness standard that has been specified. 4) In the component of product there was a constraint faced in the implemen- tation, i.e., the aspect of quality in blended learning service in terms of tan- gibles by using moodle platform which was not yet optimal. This is shown by the score of effectiveness of the evaluation result which was below the effectiveness standard that has been specified. 5 Conclusions Based on the results of the study and the discussion the following points can be concluded: a. The percentage of effectiveness of the use of blended learning at school viewed from the components of context, input, process, and product can be counted validly and accurately, whenever and wherever the evaluators do the evaluation by using mobile phone-based CIPP model. b. From some platforms that can be used for blended learning at school, it turns out that the most appropriate platform used at school is edmodo due to the ease in using it, while the most difficult to use is moodle. 6 Acknowledgment I would like to extend thanks to all academic staffs at Universitas Pendidikan Ganesha who have given me material and morale supports in the course of the im- plementation of this study. I would dedicate this research for my son “Dewa Gede Agung Suharta Divayana”. 7 References [1] Ardana, I.M., Ariawan, I.P.W., and Divayana, D.G.H. (2016). “Development of Decision Support System to Selection of the Blended Learning Platforms for Mathematics and ICT Learning at SMK TI Udayana,” International Journal of Advanced Research in Artificial Intelligence, Vol. 5, No. 12, pp. 15-18. [2] Department of Education and Early Childhood Development. (2012). Blended Learning: A Synthesis of Research Findings in Victorian Education 2006-2011. Melbourne: NEALS. [3] Prayitno, W. 2015. Penerapan Blended Learning Dalam Pengembangan Pendidikan dan Pelatihan (Diklat) Bagi Pendidik dan Tenaga Kependidikan (PTK). Yogyakarta: LPMP D.I.Yogyakarta. [4] Garrison, R., and Kanuka, H. (2004). “Blended Learning: Uncovering its Transformative Potential in Higher Education,” Internet and Higher Education 7, pp. 95–105. https://doi.org/10.1016/j.iheduc.2004.02.001 [5] Saliba, G., Rankine, L., and Cortez, H. (2013). Fundamentals of Blended Learning. Syd- ney: University of Western Sydney. 158 http://www.i-jim.org Paper—Mobile Phone-Based CIPP Evaluation Model in Evaluating the Use of Blended Learning at… [6] Liu, H. (2016). “An Analysis on Blended Learning Pattern Based on Blackboard Network Platform,” International Journal of Emerging Technologies in Learning (iJET), Vol. 11, No. 9, pp. 4-8. https://doi.org/10.3991/ijet.v11i09.6124 [7] Obiedat, R., Nasir Eddeen, L., Harfoushi, O., Koury, A., AL-Hamarsheh, M., and AlAs- safSaliba, N. (2014). “Effect of Blended-Learning on Academic Achievement of Students in the University of Jordan,” International Journal of Emerging Technologies in Learning (iJET), Vol. 9, No. 2, pp. 37-44. https://doi.org/10.3991/ijet.v9i2.3220 [8] Divayana, D.G.H., and Sugiharni, G.A.D. (2016). “Evaluasi Program Sertifikasi Komputer Pada Universitas Teknologi Indonesia Menggunakan Model CSE-UCLA,” Jurnal Pendidi- kan Indonesia, Vol. 5, No. 2, pp. 865-872. https://doi.org/10.23887/jpi-undiksha.v5i2.8586 [9] Divayana, D.G.H. (2015). “Penggunaan Model CSE-UCLA Dalam Mengevaluasi Kualitas Program Aplikasi Sistem Pakar,” SNATIA 2015, pp.165-168. [10] Sanjaya, D.B., and Divayana, D.G.H. (2015). “An Expert System-Based Evaluation of Civics Education as a Means of Character Education Based on Local Culture in the Uni- versities in Buleleng,” International Journal of Advanced Research in Artificial Intelli- gence, Vol. 4, No. 12, pp. 17-21. [11] Tiantong, M., and Tongchin, P. (2013). “A Multiple Intelligences Supported Web-based Collaborative Learning Model Using Stufflebeam’s CIPP Evaluation Model,” International Journal of Humanities and Social Science, Vol. 3, No.7, pp. 157-165. [12] Zhang, G., et al. (2011). “Using the Context, Input, Process, and Product Evaluation Model (CIPP) as a Comprehensive Framework to Guide the Planning, Implementation, and Assessment of Service-learning Programs,” Journal of Higher Education Outreach and Engagement, Vol. 15, No. 4, pp. 57-84. [13] Arikunto, S., and Jabar, C.S.A. (2009). Evaluasi Program Pendidikan Pedoman Teoritis Praktis Bagi Mahasiswa dan Praktisi Pendidikan Edisi Kedua. Jakarta: Bumi Aksara. [14] Hakan, K., and Seval, F. (2011). “CIPP Evaluation Model Scale: Development, Reliability and Validity,” Procedia Social and Behavioral Sciences, Vol.1 No.1, pp.1-8. https://doi.org/10.1016/j.sbspro.2011.03.146 8 Authors Dewa Gede Hendra Divayana and Dewa Bagus Sanjaya are with Universitas Pendidikan Ganesha, Singaraja, Indonesia (hendra.divayana@undiksha.ac.id). Article submitted 22 February 2017. Published as resubmitted by the authors 27 March 2017. iJIM ‒ Vol. 11, No. 4, 2017 159 iJIM – Vol. 11, No. 4, 2017 Mobile Phone-Based CIPP Evaluation Model in Evaluating the Use of Blended Learning at School in Bali