Paper—An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Tech… An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Technology-based CSE-UCLA Model https://doi.org/10.3991/ijim.v11i6.6697 Dewa Gede Hendra Divayana!!" Universitas Pendidikan Ganesha, Singaraja, Indonesia hendra.divayana@undiksha.ac.id Baso Intang Sappaile Universitas Negeri Makassar, Makassar, Indonesia I Gusti Ngurah Pujawan, I Ketut Dibia Universitas Pendidikan Ganesha, Singaraja, Indonesia Luh Artaningsih Universitas Panji Sakti Singaraja, Singaraja, Indonesia I Made Sundayana STIKES Buleleng, Singaraja, Indonesia Gusti Ayu Dessy Sugiharni STMIK STIKOM Bali, Denpasar, Indonesia Abstract—One of various types of application program which could facili- tate human activities is known as expert system application program. The con- cept of the expert system of the application program so important that it re- quires to be shared to other people by developing a course program called ex- pert system conducted in particular by an institution which specialized in the field of information technology. This course program of expert system which is offered to the students need to be designed in terms of good instructional pro- cess to improve its effectiveness. In fact, not all existing course programs con- ducted based on expert system could run more effectively than it was expected in order to achieve higher quality of learning outcomes. This could be explained as a numerous problems occurred during the process of implementation. In or- der to discover the extend of program effectiveness in the expert system of the course, a through program evaluation is required to be conducted. One model called CSE-UCLA will be recommended in making deeper instructional pro- gram evaluation in relation to expert system of course program. Through this model the instructional affectivity of application program of expert system could be measured by involving system assessment, program planning, program implementation, program improvement, and program certification. Keywords—Evaluation, Instruction Process, CSE-UCLA, Mobile Technology 18 http://www.i-jim.org Paper—An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Tech… 1 Introduction In the era of high speed development of technology a lot of application programs had been invented which could easily facilitate every human in his daily life either in the social activities, economic aspect, government activities, in the health sector, as well as education, and others [1]. One of the various types of the famous application program developed nowadays which could be able to support every single human activity is known as an expert system. The concept of this program is considerably important that a broad dissemination is required by developing and designing a course program called expert system, in particular around an institution where a field of information technology is greatly being concerned. A course of expert system pre- sented for the students’ needs to be designed into a more interesting instructional process that it could improve its’ effectiveness. An instructional process is called effective when the outcomes of the process itself bring an effect to the students in understanding the goal of instruction, the platforms of the instruction, the concept being presented, and they could manage the results, preparing all the related soft and hardware required in learning an expert system, they could also manage to set up either the software as well as hardware, they could be able to make rules based on the ability to manage the basis of knowledge, and finally they are able to present information as accurately as possible. However, in reality not all processes of expert system courses developed in this in- stitution could be able to run as effectively as expected. This indication is the results of the immersion of unexpected problems around the process activities. To find out the extend of how effective the process of instruction of an expert system course an evaluation is indeed required. By conducting an evaluation to a process of an instruc- tion to a related course the problems could be identified then solutions and recom- mendations could be provided in order to improve the running process of instruction as well as to develop further the well effective running program of instruction. One model proposed in this study in order to evaluate the process effectiveness of an expert system of course instruction is known as a mobile technology-based CSE- UCLA model. By using this model the extend of process effectiveness of the expert system course program could be assessed based on the five different dimensions, such as assessment system, program planning, program implementation, program im- provement and program certification. In support with a mobile technology, the eval- uation process by involving five dimensions of CSE-UCLA to the process of expert system course of instruction by evaluators could be managed quickly and easily, whenever and wherever possible. In term of system assessment, several aspects of expert system course of instruction to be evaluated include the students’ ability to understand: the goal of instruction, the platform of instruction, and the basic concept the knowledge. In term of program planning, several aspects of expert system course of instruction to be evaluated in- clude the students’ ability to prepare all the required soft-wares and hard-wares in learning an expert system subject matter. In terms of program implementation, several aspects of expert system course of instruction to be evaluated include the students’ ability to set up the required soft-wares and hard-wares in learning an expert system iJIM ‒ Vol. 11, No. 6, 2017 19 Paper—An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Tech… subject matter. In terms of program improvement, several aspects of expert system course of instruction to be evaluated include the students’ ability to manage the basic knowledge and rules in relation to the course being concerned. In terms of program certification, the aspects of expert system course of instruction to be evaluated include the students’ ability to present as accurately as possible the information based on the basic knowledge and rules being concerned. The problems of this research, among others: 1) How is the effectiveness of the standard implementation of instructional process of expert system course program by using CSE-UCLA model based on mobile technology?, 2) How is the result of Evaluation of Expert System Learning Process using Model CSE-UCLA with manual calculation and the calculation based on mobile technology?, 3) What are the constraints found in the implementation of evaluation? Some of the previous researches that underlies the implementation of this research include: research conducted in 2015 by Divayana [1] is about the use of CSE-UCLA model in evaluating the quality of expert system application program, where the rese- arch was conducted with the aim to know the good or bad quality of expert system application program. The result of this research is through the use of CSE-UCLA model in evaluating expert system application program, so it can be seen the percen- tage of program quality as a whole and also can be seen the percentage of program quality from five dimension that is: system assessment, program planning, program implementation, program improvement, and program certification. The constraints in the research is the absence of technology used to evaluate the quality of expert system application program is still in the form of application program, so it has not been able to conduct the evaluation process quickly and easily. Another research in 2016 which is also done by Divayana [2] about evaluation of computer certification program at Universitas Teknologi Indonesia using CSE-UCLA model, where that research is conducted with the aim to know effectiveness of com- puter certification program at Universitas Teknologi Indonesia and obtained evalua- tion result is the implementation of the computer certification program at the Univer- sity of Technology Indonesia as a whole at the level of effectiveness that is good categorized. The constraints found in the research that is not yet found a technology capable of performing evaluation quickly and easily using CSE-UCLA model without being limited by a certain time and place. From some constraints that are still found in previous research, then in this research is a breakthrough in the utilization of mobile technology to evaluate the learning process of expert system course, so that evaluation can be done easily and quickly, whenever and wherever. There are several studies related to this research, namely research conducted by Ariawan, Sanjaya, and Divayana in 2016 about an evaluation of the implementation of practice teaching program for prospective teachers at Ganesha University of Edu- cation Based on CIPP-Forward Chaining [3], where the study has similarities with this research in the object of research is related to the learning process. The difference lies in the utilization of the evaluation model used to perform the evaluation, where Ariawan, Sanjaya, and Divayana use CIPP evaluation model, while in this research use CSE-UCLA model. Research conducted by Divayana in 2017 about evaluation of blended learning implementation at SMK TI Udayana using CSE-UCLA model [4], 20 http://www.i-jim.org Paper—An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Tech… where the research have similarities with this research in terms of utilization of CSE- UCLA model to conduct an evaluation. The difference lies in the object under study, where the object of the study is blended learning, while the object of this study is the learning process of expert system course. Based on research objectives, research problems, previous researches base, and re- lated researches, the researchers are interested in conducting research about evalua- tion of instructional process of expert system course program by using mobile tech- nology based on CSE-UCLA model. 2 Literature Review 2.1 Evaluation An Evaluation is an activity to collect information whether something could man- age to work or not, and further the information will be used to decide an accurate alternative to make decision [5]. Evaluation is a mean for understanding how things going. Based on the evaluation goal, criteria should be identified before evaluation could be conducted [6]. Evaluation is a research in order to collect, analyse, and pre- sent worthwhile information in relation to an object of evaluation, judging by making comparison with the indicators of evaluation and the results would be utilized when a decision should be made related to the object of evaluation [7]. Evaluation is an activ- ity conducted by an evaluator in collecting, analyzing, and presenting information related to the program/object/policy that the results can be used to take a decision [8]. Evaluation is an activity conducted by the evaluator to collect, analyze, and present complete and accurate information about a particular object/program/ service/policy being studied, thus the results could be used as a recommendation in making a deci- sion [9]. Evaluation is an activity of data collection, data processing, data analysis, presentation of data into information that used as a recommendation in taking a right decision [10]. Evaluation is an activity to collect, analyze, and present information about an object to be evaluated, where the results of these evaluations are used for consideration in making a decision that is precise, accurate, and reliable [11]. Evaluation is an activity for collecting, analyzing, and explaining comprehensively information about a particular object/program/policy being studied and the results of an evaluation can be used for the consideration in making a decision to continue or to stop the object/program/policy [12]. Evaluation is an activity that collects, analyzes, and presents data into useful information in making decisions based on recommenda- tions obtained from these activities [13]. Evaluation is an activity for collecting, ana- lyzing, and presenting, information about a particular object to be used for a consider- ation in making an appropriate and accurate decision [14]. Evaluation is one of the measurement activities conducted through the process of data collection, data analy- sis, and interpret it into an information so that the results can be used as recommenda- tions for decision making [15]. Evaluation is an important activity undertaken in measuring the quality of learning through the process of data collection, data analysis, and presentation of information that the results can be used for consideration in taking iJIM ‒ Vol. 11, No. 6, 2017 21 Paper—An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Tech… a decision in the improvement of learning process towards a better quality [16]. Eval- uation is an activity to collect, understand, and report the results of an analysis about a particular program/object so that the results can be used for consideration in taking a decision whether the program is continued or stopped [2]. Evaluation is an activity that consists of the process of gathering, describing, and explaining various pieces of information about the effectiveness of something that can be used later as the con- sideration for making a decision and a recommendation [3]. Evaluation is an activity for data collecting, data analyzing and data presenting into information about a particular object under study so that the results can be used to take a decision [17]. Evaluation is a process of understanding, giving meaning, obtaining and com- municating an information for the purpose of taking a decision [18]. Evaluation is a process of judging something based on a predetermined criteria or purpose, and then followed by making decision to the object being evaluated [19]. We can define two key criteria to evaluate the quality of educational resources: veracity of learning activ- ities, and content of the educational resource [20]. Mardapi states that the evaluation can be interpreted as a determination of the conformity between the results achieved and goals to be achieved [21]. Based on the above definitions a conclusion could be drawn that in general an evaluation is an activity to collect, analyse, and present an information about a partic- ular object being studied and the result of an evaluation could be used as an aspect of consideration to make a decision. 2.2 CSE-UCLA (Center for the Study of Evaluation-University of California in Los Angeles) Evaluation Model Evaluation framework was developed by Alkin in (1969, 1991) at University of California-Los Angeles. This framework was called the UCLA Evaluation Model and was extensively used by the Center for the Study of Evaluation at UCLA. The Alkin framework has five types of evaluations within it (see Table 1) [22]. Table 1. Alkin’s UCLA Evaluation Framework Evaluation Type Description System Assessment Provide information about the state of a system Program Planning Assist in the selection of particular programs likely to be effective in meeting specific educational needs Program Implementation Provide information about whether a program was introduced to an appropri-ate group in the manner intended Program Improvement Provide information about how a program is functioning, whether interim objectives are being achieved, and whether unanticipated outcomes are ap- pearing Program Certification Provide information about the value of the program and its potential for use elsewhere An evaluation CSE-UCLA is a model developed by Alkin used to evaluate a pro- gram in five different stages of evaluation which include: system assessment, pro- 22 http://www.i-jim.org Paper—An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Tech… gram planning, program implementation, program improvement, and program certi- fication [23]. CSE-UCLA model evaluation was accomplished in several phases, namely: system assessment, program planning, program implementation, program improvement, and program certification [24]. Tayibnapis stated that the evaluation model of CSE-UCLA developed by Alkin has five evaluation stages: system assess- ment, program planning, program implementation, program improvement, and pro- gram certification [25]. System assessment is evaluation component that provides information about the state or position of the system. Program planning is evaluation component that helps the selection of certain programs that may successfully meet the needs of the program. Program implementation is evaluation component that prepares information whether the program has been introduced to the appropriate group as planned. Program improvement is evaluation component that provides information about how the program works, works or runs, whether to a certain achievement. A Program certification is evaluation component which provides information on the value or use of the program. A CSE-UCLA is a model of evaluation having five dif- ferent evaluations (system assessment, program planning, program implementation, program improvement, program certification) and is appropriately reliable to evalu- ate service program and instructional program [26]. CSE-UCLA model is evaluation model that has five dimension of evaluation, such as: 1) system assessment, which provides information about the state of the system, 2) program planning, which selec- tion of specific programs to meet program needs, 3) program implementation, which provides information to introduce programs, 4) improvement program, which pro- vides information about program functions/performance, 5) certification program, which provides information about benefits or uses of program [2]. Based on the above statements it could be concluded that in general an evaluation of CSE-UCLA a form of evaluation focusing on five aspects including,: providing information about the condition of the program being evaluated (system assessment), selection of effective programs relevant to meet the needs in planning the program (program planning), giving information/ introducing the program to the groups being appointed about the program implementation according to the re-determined planning (program implementation), providing information about the program performances (program improvement), and providing information about the results and significance of the program (program certification). 3 Research Methodology 3.1 Object of the Study The object of the study include expert system of course program. 3.2 Subject of the Study The subjects of the study include: teaching staff members and students. The selec- tion of the subjects was conducted based on a technique called Purposive Sampling, iJIM ‒ Vol. 11, No. 6, 2017 23 Paper—An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Tech… such as all the related parties with the instructional process of expert system of course program. 3.3 Location of the Study The study was conducted at the Department of Information Technology, Universi- tas Teknologi Indonesia, Denpasar, Bali. 3.4 Methods of Collecting Data The data of the study were collected by distributing questionnaires, making obser- vation, and documentation. 3.5 Design of the Study The study utilized a model designed called as CSE-UCLA which could be present- ed as follows: Fig. 1. A Design Model of Evaluation Program CSE-UCLA based on Mobile Technology Program Imple- mentation Program Planning Learning Process System Assessment Program Improvement Program Certification 1. Understood about purpose of learning 2. Understood about learning plat- form 3. Understood about knowledge base 1. Understood about hardware setup 2. Understood about software setup 1. Understood about hardware installation 2. Understood about software instal- lation 1. Understood about knowledge base management 2. Understood about rule manage- ment Got the information Main Instru- ments & Complementary Effectiveness Level Problems Encountered Recommendations Android Application 24 http://www.i-jim.org Paper—An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Tech… 3.6 Technique Data Analysis The data analysis of evaluation instructional process of expert system course pro- gram was conducted based on five different components, such as system assessment, program planning, program implementation, program improvement, and program certification by using descriptive quantitative technique. 4 Results and Discussion 4.1 Results Standard effectiveness of instructional process of expert system course pro- gram according the CSE-UCLA model on the basis of mobile technology. A pro- gram implementation is called successful when at least the pre-determine effective- ness standard was met. The criteria of effectiveness standard used in the process of expert system of instruction based on CSE-UCLA model based on mobile technology is presented in the following table. Table 2. Effectiveness Standard of The Implementation of Expert System Course of Instruc- tions by Using CSE-UCLA Model No Dimensions Evaluation Aspects Effectiveness Standard (%) 1. System Assesment Understood about purpose of learning >=85.00 Understood about learning platform >=85.00 Understood about knowledge base >=85.00 2. Program Planning Understood about hardware setup >=85.00 Understood about software setup >=85.00 3. Program Implementation Understood about hardware installation >=85.00 Understood about software installation >=85.00 4. Program Improvement Understood about knowledege base management >=88.00 Understood about rule management >=88.00 5. Program Certification Got the information >=90.00 The Category of Effectiveness Standard Scales: Very High : 90.00%-100.00% High : 80.00%-89.00% Sufficient : 70.00%-79.00% Low : 60.00%-69.00% Very Low : 0.00%-59.00% The Results of Evaluation of Process of Expert System Course Instruction Us- ing CSE-UCLA Model based on Mobile Technology. Based on the process of eval- uation of expert system course program by using CSE-UCLA model, the results based on manual calculation is presented in the following table 3 and the results based on the mobile technology supported by Android application program could be seen in the figure 2. iJIM ‒ Vol. 11, No. 6, 2017 25 Paper—An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Tech… Table 3. The Results of Evaluation of Instructional Process of Expert System Course Program by Using CSE-UCLA Model With Manual Calculation No Dimensions Evaluation Aspects Score of Respondents X % R1 R2 R3 R4 R5 1 System Asses- ment Understood about purpose of learning 4 4 5 4 4 4.2 84.00 Understood about learning platform 4 4 4 4 5 4.2 84.00 Understood about knowledge base 4 4 5 5 4 4.4 88.00 Effectiveness Percentage in System Assessment 85.33 2 Program Plan- ning Understood about hardware setup 4 4 4 5 4 4.2 84.00 Understood about software setup 5 5 4 4 4 4.4 88.00 Effectiveness Percentage in Program Planning 86.00 3 Program Im- plementation Understood about hardware installation 4 4 4 5 4 4.2 84.00 Understood about software installation 5 5 4 4 4 4.4 88.00 Effectiveness Percentage in Program Implementation 86.00 4 Program Im- provement Understood about knowledege base management 4 4 5 4 5 4.4 88.00 Understood about rule management 4 5 5 4 5 4.6 92.00 Effectiveness Percentage in Program Improvement 90.00 5 Program Certifi- cation Got the information 5 5 4 5 4 4.6 92.00 Effectiveness Percentage in Program Certification 92.00 Effectiveness Percentage in Learning Process 87.87 Calculation of System Assessment Calculation of Program Planning Calculation of Program Implementation Calculation of Program Improvement Calculation of Program Certification The Final Results of Evaluation Fig. 2. The Results of Evaluation of Instructional Process of Expert System Course Program With Automatic Calculation Supported by Android Application 26 http://www.i-jim.org Paper—An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Tech… 4.2 Discussion Based on the results of the study described above, some points are required to be discussed, like: 1. The results of calculation about the instructional process effectiveness of the expert system course program both manually as well as automatically using mobile tech- nology-based CSE-UCLA model indicated the same and accurate results such as: a) On the system assessment dimension, the extend of effectiveness was about 85.33%, this indicated a high category; b) On the program planning dimension, the extend of effectiveness was about 86.00%, this indicated a high category; c) On the program implementation aspect, the extend of effectiveness was about 86.00%, this indicated a high category; d) On the program improvement aspect, the extend of effectiveness was about 90.00%, this indicated a high category; e) On the pro- gram certification dimension, the extend of effectiveness was about 92.00%, this indicated a high category; f) The effectiveness of the instructional process of expert system course program as a whole totally about 87.87%, this indicated a high cate- gory. 2. Some problems encountered during the implementation of Evaluation In general the extend of effectiveness of instructional process of expert course pro- gram belongs to a high category, however, according to the results of analysis made based on effectiveness standard of expert system course program implementation, presented on the Table 2, compared with the results of evaluation of the program implementation, as presented on Table 3, the results of quantitative analysis indicated that a problem was still found, in particular on the system assessment dimension, on the aspect of understanding the goal of learning, since the effectiveness score was still 84.00 % lower than the pre-determined effectiveness standard which was about >=85.00%. It was also proved from the results of observation that there were found many students did not understand deeply about the goal of learning. In addition, on the aspect of “understanding the learning platform” a problem also found, since the effectiveness score was about 84.00 % lower than the level of the pre-determined effectiveness standard which was about >=85.00%. It was also proved from the re- sults of observation that some students still did not understand about the base of in- struction. On the dimension of program planning, a problem was also found in partic- ular on the aspect of “Understanding about hardware setup” since the score effective- ness was about 84.00 % lower than the pre-determined effectiveness standard which was about >=85.00%. It was also proved from the results of observation that some students still did not understand about how to prepare the hard-wares required to learn expert course. On the dimension of program implementation, a problem was also found in particular on the aspect of “Understanding about hardware set up” since the effectiveness score was about 84.00 % lower than the pre-determined effectiveness standard which was about >=85.00%. It was also proved from the results of observa- tion that some students still did not understand how to install the hard-wares required to learn expert course. Other problems encountered from the application of CSE- UCLA model based on mobile technology when using Android facility that the stu- iJIM ‒ Vol. 11, No. 6, 2017 27 Paper—An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Tech… dents were unable to indicate the information about the extend of coherent effective- ness scores from the lowest to the highest scores. 5 Conclusion and Recommendations 5.1 Conclusion Based on the results of the study as well as its previous discussion, some conclu- sion could be made such as: 1. By using mobile technology-based CSE-UCLA in evaluating the instructional pro- cess of expert system course program, the effectiveness percentage of the process implementation of expert system course could be seen as a whole as well as every single aspect from five different dimensions such as: system assessment, program planning, program implementation, program improvement and program certifica- tion. 2. Having identified the percentage score of every single aspect, improvement could be made on whatever aspect are still considered to have problems and others which were required to be resumed. When the low percentage were found improvement needed to be done, however, when the effectiveness scores were high development was needed to be conducted. 5.2 Recommendations Based on the problems found in the process of evaluation of expert system course program by using mobile technology-based CSE-UCLA model at the Department of Information Technology, Universitas Teknologi Indonesia, Denpasar, Bali, some recommendations were proposed as a follow up to improve the instructional process, such as: 1. The teaching staffs are required to explain clearly in detail the purposes of goal of instruction to the students. 2. The teaching staffs should explain clearly strong platforms about the significance and benefit the students obtained after the process of learning the expert system subject matter completed. 3. The teaching staffs have to provide greater opportunities and times for the students to practice installing the hard-wares needed for learning expert system course ma- terial. 4. Developing a model of evaluation using mobile technology-based CSE-UCLA is required in order to provide coherent information from the lowest effectiveness level until the highest, to facilitate in deciding the priority the process of improve- ment. 28 http://www.i-jim.org Paper—An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Tech… 6 Acknowledgment Thank you for the entire academicians at Universitas Pendidikan Ganesha, Univer- sitas Negeri Makassar, Universitas Panji Sakti Singaraja, STIKES Buleleng, and STMIK STIKOM Bali who has supported this research. 7 References [1] Divayana, D.G.H. (2015). “Penggunaan Model CSE-UCLA Dalam Mengevaluasi Kualitas Program Aplikasi Sistem Pakar,” SNATIA 2015, pp.165-168. [2] Divayana, D.G.H., and Sugiharni, G.A.D. (2016). “Evaluasi Program Sertifikasi Komputer Pada Universitas Teknologi Indonesia Menggunakan Model CSE-UCLA,” Jurnal Pen- didikan Indonesia, Vol. 5, No. 2, pp. 865–872. https://doi.org/10.23887/jpi-undiksha. v5i2.8586 [3] Ariawan, I.P.W, Sanjaya, D.B., and Divayana, D.G.H. (2016). “An Evaluation of the Im- plementation of Practice Teaching Program for Prospective Teachers at Ganesha Universi- ty of Education Based on CIPP-Forward Chaining,” International Journal of Advanced Research in Artificial Intelligence, Vol. 5, No. 2, pp. 1-5. [4] Divayana, D.G.H. (2017). “Evaluasi Pelaksanaan Blended Learning di SMK TI Udayana Menggunakan Model CSE-UCLA,” Jurnal Pendidikan Vokasi, Vol. 7, No. 1, pp. 64-77. https://doi.org/10.21831/jpv.v7i1.12687 [5] Arikunto, S., and Jabar, S.A. (2009). Evaluasi Program Pendidikan Pedoman Teoritis Praktis Bagi Mahasiswa dan Praktisi Pendidikan Edisi Kedua. Jakarta: Bumi Aksara. [6] Kuo, L.H., et.al. (2012). “An Evaluation Model of Integrating Emerging Technology into Formal Curriculum,” International Journal of Education and Information Technologies, Vol. 6, No. 3, pp. 250-259. [7] Wirawan. (2011). Evaluasi Teori, Model, Standar, Aplikasi, dan Profesi. Jakarta: Rajawali Pers. [8] Divayana, D.G.H., Adiarta, A., and Abadi, I.B.G.S. (2017). “Conceptual and Physical De- sign of Evaluation Program for Optimizing Digital Library Services at Computer College in Bali Based on CSE-UCLA Model Modification with Weighted Product,” Journal of Theoretical and Applied Information Technology, Vol. 95, No. 16, pp. 3767-3782. [9] Divayana, D.G.H., Marhaeni, A.A.I.N., Dantes, N., Arnyana, I.B.P., and W. Rahayu, (2017). “Evaluation of Blended Learning Process of Expert System Course Program by Using CSE-UCLA Model Based on Mobile Technology”, Journal of Theoretical and Ap- plied Information Technology, Vol. 95, No. 13, 2017, pp. 3075-3086. [10] Suandi, I.N., Putrayasa, I.B., and Divayana, D.G.H. (2017). “Compiling a Dictionary of Loan Words in Balinese: The Evaluation Result of Effectiveness Testing in The Field Aid- ed by Mobile Technology,” Journal of Theoretical and Applied Information Technology, Vol. 95, No. 14, pp. 3186-3195. [11] Divayana, D.G.H., Sanjaya, D.B., Marhaeni, A.A.I.N., and Sudirtha, I.G. (2017). “CIPP Evaluation Model Based on Mobile Phone in Evaluating The Use of Blended Learning Platforms at Vocational Schools in Bali,” Journal of Theoretical and Applied Information Technology, Vol. 95. No 9, pp. 1983-1995. [12] Divayana, D.G.H., Ardana, I.M., and Ariawan, I.P.W. (2017). “Measurement of Effective- ness of a Lecturer in Transferring Algebra Knowledge Through of Multimedia Facilities iJIM ‒ Vol. 11, No. 6, 2017 29 Paper—An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Tech… by Using Certainty Factor-Formative-Summative Model,” Journal of Theoretical and Ap- plied Information Technology, Vol. 95. No 9, pp. 1963-1973. [13] Divayana, D.G.H. (2017). “Utilization of CSE-UCLA Model in Evaluating of Digital Li- brary Program Based on Expert System at Universitas Teknologi Indonesia: A Model for Evaluating of Information Technology-Based Education Services,” Journal of Theoretical and Applied Information Technology, Vol. 95, No. 15, pp. 3585-3596. [14] Divayana, D.G.H., and Sanjaya, D.B. (2017). “Mobile Phone-Based CIPP Evaluation Model in Evaluating the Use of Blended Learning at School in Bali,” International Jour- nal of Interactive Mobile Technologies, Vol. 11, No. 4, pp.149-159. [15] Arnyana, I.B.P., Sadia, I.W., Suma, I.K., and Divayana, D.G.H. (2017). “Determination of Effectiveness of Evaluation Results on School Culture and Character of Junior High School Students Using Character Assessment Instruments With The Local Wisdom of Bali Based on Mobile Phone,” Journal of Theoretical and Applied Information Technology, Vol. 95, No. 20, pp. 5348-5359. [16] Divayana, D.G.H., Agung, A.A.G., Sappaile, B.I., Simatupang, W., Sastrawijaya, Y., Sun- dayana, I.M., and Sugiharni, G.A.D. (2017). “Utilization of Open Source Technology in Determining of Validity and Reliability of Evaluation Model Instruments Based on ANEKA Values in Order to Evaluate The Quality of Computer Learning,” Journal of The- oretical and Applied Information Technology, Vol. 95, No. 20, pp. 5517- 5534. [17] Sanjaya, D.B., and Divayana, D.G.H. (2015). “An Expert System-Based Evaluation of Civics Education as a Means of Character Education Based on Local Culture in the Uni- versities in Buleleng,” International Journal of Advanced Research in Artificial Intelli- gence, Vol. 4, No. 12, pp. 17-21. [18] Sukardi, H.M. (2008). Evaluasi Pendidikan Prinsip & Operasionalnya. Yogyakarta: Bumi Aksara. [19] Djaali and Mulyono, P. (2004). Pengukuran Dalam Bidang Pendidikan. Jakarta: Program Pascasarjana Universitas Negeri Jakarta. [20] Abderrahim, E.M., Mohamed E., and Azeddine N. (2013). “An Evaluation Model of Digi- tal Educational Resources,” International Journal of Emerging Technologies in Learning (iJET), Vol. 8, No. 2, pp. 29-35. [21] Mardapi, D. (2012). Pengukuran, Penilaian, dan Evaluasi Pendidikan, Yogyakarta: Nuha Medika. [22] Dean, G.S. (2003). Strategies for The Development of Integrated Career and Technical Education Program Evaluation Systems. Blacksburg, Virginia : Virginia Polytechnic Institute and State University. [23] Suryanto A., Gafur A., and Sudarsono F.X. (2013). “Model Evaluasi Program Tutorial Tatap Muka Universitas Terbuka,” Jurnal Penelitian dan Evaluasi Pendidikan, Vol. 17, No. 2, pp. 198-214. [24] Kurniawan, D. (2013). “Evaluation on Foreign Language Development Program,” Jurnal Evaluasi Pendidikan, Vol. 4, No. 1, pp. 1-11. https://doi.org/10.21009/JEP.041.01 [25] Tayibnapis, F.Y. (2000). Evaluasi Program, Jakarta: PT. Rineka Cipta. [26] Divayana, D.G.H. (2016). Evaluasi Program Perpustakaan Digital Berbasis Sistem Pakar Pada Universitas Teknologi Indonesia, Jakarta : Universitas Negeri Jakarta. 8 Authors Dewa Gede Hendra Divayana is with Universitas Pendidikan Ganesha, Singaraja, Indonesia (hendra.divayana@undiksha.ac.id). 30 http://www.i-jim.org Paper—An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Tech… Baso Intang Sappaile is with Universitas Negeri Makassar, Makassar, Indonesia. I Gusti Ngurah Pujawan is with Universitas Pendidikan Ganesha, Singaraja, In- donesia. I Ketut Dibia is with Universitas Pendidikan Ganesha, Singaraja, Indonesia. Luh Artaningsih is with Universitas Panji Sakti Singaraja, Singaraja, Indonesia. I Made Sundayana is with STIKES Buleleng, Singaraja, Indonesia. Gusti Ayu Dessy Sugiharni is with STMIK STIKOM Bali, Denpasar, Indonesia. Article submitted 24 January 2017. Published as resubmitted by the authors 30 March 2017. iJIM ‒ Vol. 11, No. 6, 2017 31 iJIM – Vol. 11, No. 6, 2017 An Evaluation of Instructional Process of Expert System Course Program by Using Mobile Technology-based CSE-UCLA Model