Bulletin of Social Informatics Theory and Application ISSN 2614-0047 Vol. 4, No. 2, September 2020, pp. 83-90 83 https:doi.org/10.31763/businta.v4i2.287 Indonesian online learning system evaluation framework based on UTAUT 2.0 Felix Andika Dwiyanto a,1,*, Hakkun Elmunsyah b,2, Yoto c,3 a Graduate School, Universitas Negeri Malang, Malang, Indonesia b Depatment of Electrical Engineering, Universitas Negeri Malang, Malang, Indonesia c Depatment of Mechanical Engineering, Universitas Negeri Malang, Malang, Indonesia 1 felix.adwiyanto@gmail.com*; 2 hakkun@um.ac.id; 3 yoto.ft@um.ac.id * corresponding author 1. Introduction Currently, the role of information and communication technology (ICT) cannot be separated from any activities and works. Especially in education, the use of ICT is to communicate, create, disseminate, store, and manage information [1]–[3]. Moreover, a learning process with face-to-face method has been long time combined with technology assistance. In its application this, method commonly known as online learning. This method is able to conduct a flexible learning process that can be done anywhere and anytime as long as user connected to the internet. Moreover, it can provide a sufficient learning quality in a wider scope[4], [5]. In Indonesia, one of ICT utilization in online learning is SPADA referred as Sistem Pembelajaran Daring Indonesia. It is a program from Indonesian Ministry of Research, Technology, and Higher Education to increase equitable access of quality learning in higher education. SPADA is developed to answer several education challenges [6], which are (a) limited capacity of university, (b) limited scope of university due to uneven distribution, (c) there are still many universities that do not have adequate and quality educational resources, (d) higher quality universities are still concentrated in Java, and (e) a lack of equal and quality higher education services. Therefore, the use of ICT is expected to be a solution with a concept of networked higher education and online learning. One of SPADA application is to provide and accommodate a teacher profession education program which known as PPG. This program is actualization from law about teacher profession which state that teacher should have academic and competency qualification, teaching certificate, physically and mentally healthy, also have an ability to realizing the goals of national education. This regulation become a basic Indonesian teacher reformation that teacher should prepare through PPG after completed a bachelor degree. In other words, this program will prepare a graduates in either education or non-education majors who have an interest to become a professional teacher. A R T I C L E I N F O A B S T R A C T Article history Received August 6, 2020 Revised August 20, 2020 Accepted September 2, 2020 This study aims to propose an evaluation of Indonesian online learning system which known as SPADA. This system accommodate an online learning for teacher profession education program which known as PPG. The system evaluation is using unified theory of acceptance and use of technology 2 (UTAUT2) model with a few adjustments. This study provide an information such as required variables to evaluate the system and research design on SPADA. The proposed evaluation model using seven main constructs of UTAUT 2.0 and its influence on behavioral intention and use behavior. The data obtained from several questionnaires related to the variable and analyzed with path analysis technique. Moreover, this proposed study is expected can be as reference to conduct a similar study related to the system evaluation. This is an open access article under the CC–BY-SA license. Keywords UTAUT2 Online Learning system http://creativecommons.org/licenses/by-sa/4.0/ http://creativecommons.org/licenses/by-sa/4.0/ 84 Bulletin of Social Informatics Theory and Application ISSN 2614-0047 Vol. 4, No. 2, September 2020, pp. 83-90 Dwiyanto et.al (Indonesian online learning system evaluation framework based on UTAUT 2.0) The implementation of PPG is consists of three main stages which are online learning, workshop, and teaching practice at school. Fig. 1 show the implementation flow of PPG. Based on the flow, it can be seen that the early 12 weeks of the program is done by completed the course of pedagogic competency through an online learning. The online platform used is Brightspace web-based learning management system. A several features in this platform are discussion forum and virtual classroom. The advantage of this platform is instructor can monitoring an improvement of learning participant such as user activity, visited content, learning achievement, final exam, and quizzes. Fig. 1. The implementation stages of PPG Based on the program phase, it can be seen that the most proportion required is the first phase which are online course. From that phase, an instructor will evaluate the participants’ learning achievement and filter participants who deserve and succeed to continue to the next phase. In this case, the use of web-based platform has an important role in a learning process. Therefore, it is required to evaluate the system quality, also the user intentions to use a system. An evaluation is used to examine the effectiveness of content and features that provided in the system. Also, it will improve the system quality based on users’ technology acceptance. One of evaluation model used to test the system is unified theory of acceptance and use of technology (UTAUT). This model has four main factors in acceptance and use of technology, which are (a) performance expectancy, (b) effort expectancy, (c) social influence, and (d) facilitating conditions [7]. That four factors will affecting technology behavioral intention and use behavior. Nowadays, the use of UTAUT has been improved become UTAUT2. This model extends the previous model of UTAUT to study acceptance and use of technology [8]. Previous studies shows that UTAUT describes an acceptance and use of technology in the educational fields [9]–[14]. In this case, an evaluation of the SPADA system needs to be done because the learning implementation of PPG students are new and different. So, acceptance, use and adaptation will be different for each user. This evaluation will be useful for explaining the differences in use and acceptance so that it will produce a feedback that can be used as a reference for improving the system according to user needs. 2. Unified Theory of Acceptence and Use of Technology 2.0 (UTAUT2.0) Generally, UTAUT is a technology acceptance model. This model is a development and synthesis of several existing technology application models including theory of reasoned action (TRA), technology acceptance model (TAM), motivation model (MM), theory of planned behavior (TPB), combined TAM and TPB, model of PC utilization (MPTU), innovation diffusion theory (IDT) and social cognitive theory (SCT) [7], [15], [16]. Along with the development of information technology and communication, UTAUT also developed into new version known as UTAUT2. This developed model added three main factors in its implementation which are (a) hedonic motivation, (b) price value, and (c) habit [8]. It’s produced a variation of substantial improvement in behavioral intention and technology use. Fig. 2 shows the initial model of UTAUT2. Based on the initial model of UTAUT2, it consists of several constructs or variable. Those variables are independent, dependent, and moderating variable. Independent variables are seven main factors that affecting dependent variables (behavioral intention and use behavior). While, the three factors which are age, gender, and experience acts as moderating variable. ISSN 2614-0047 Bulletin of Social Informatics Theory and Application 85 Vol. 4, No. 2, September 2020, pp. 83-90 Dwiyanto et.al (Indonesian online learning system evaluation framework based on UTAUT 2.0) Fig. 2. Model of UTAUT2 The first variable is a performance expectancy (PE). This is a level of an individual's belief that using the system will help him get benefits at work [10], [17]. The higher level of trust will make him to use the technology longer. In its application, there are five constructs related to performance expectations. The five constructs are perceived usefulness, extrinsic motivation, job fit, relative advantage, and outcome expectations. The first construct, perceived usefulness, is related to the benefits when using a certain technology, which will facilitate his job. The second construct, extrinsic motivation is the perception of doing something to get results or rewards from the work such as wages, bonuses, and others [18]. Furthermore, job fit is the suitability of the system functions that will help someone's personal work. The fourth construct is the relative advantage associated with the extent to which the system development will be better than the previous one. The final construct is the outcome expectations associated with the expected results achieved by using a system. The second variable is effort expectancy (EE). This construct defined as the level of ease associated with using the system [9], [19]. In other words, it refers to the expectation of ease of doing work when using a particular system. This variable has three basic constructs, which are perceived ease of use, complexity, and ease of use. The perceived ease of use is related to the level of user confidence in the use of a system [20]. In other words, the use of a system can ease the work effort. The second construct is the complexity associated with the level of ease and use of a system. This is because the level of convenience varies between users. Each user will feel the difference in manual work with system assistance which impacts on their level of understanding of a system. The final construct is ease of use associated with innovation and development of a technology or system. This causes a different experience when a system is experiencing development or innovation. Furthermore, users will feel whether the developed system is better used or not. Social influence (SI) as a third variable is defined as how other people influence the use of a system [21]. This shows that the role of other people influences the user to use a system or not. This variable has three main constructs which are subjective norms, social factors, and views. Subjective norms mean the user's perception of someone who is important to him will think that the user must use a certain system. The second construct is social factors related to the social environment. This means that the social environment will influence someone to use a certain system. 86 Bulletin of Social Informatics Theory and Application ISSN 2614-0047 Vol. 4, No. 2, September 2020, pp. 83-90 Dwiyanto et.al (Indonesian online learning system evaluation framework based on UTAUT 2.0) The last construct is the view or impact of using the system on one's views. This is related to the use of a system that can increase the view of a person's social status. Next, the fourth variable is a facilitating condition (FC). It is defined as organizational and technical infrastructure will affecting and supporting a system [22]. This variable has three main construct [7]. The first construct is the perception of behavioral control that reflects internal and external perceptions including the condition of facilities, resources, and technological facilities. The second construct is a facilitating condition related to the environment that supports the use of a system both from the system itself and from others. The final construct is compatibility which is related to the consistency of innovation against the value, needs and user experience. In the fifth variable there is a hedonic motivation (HM). Hedonic motivation is a pleasure thing from using technology or systems. This will be an experience and an important factor for users in using this technology [23]. If the user experience is pleasant, then users will tend to use the technology. This will affect the acceptance level and use of a technology or system. The sixth variable is a price value (PV). From the consumer side, the price value is related to the costs required to use a system or technology [24]. This is related to the feasibility of a system for the costs incurred to access it. But, this does not apply if the system or technology used is free or open source. This factor also affects the user in the intention to use a system. In the seventh variable, there is a habit (H). This is related to the pattern of using the system in their daily life. In other words, this familiarity is a factor in using a system [25]. The more accustomed to users using a system, the impact on their daily work. Interest or behavioral intention (BI) as an eighth variable can be interpreted as a person's level of exploiting a system [7], [26]. The higher the interest and desire in using a system will have an impact on the intensity of its use in that system. This is the dependent variable which is influenced by the previous variables. And the last variable is a use behavior (UB). This is the way and intensity of a person using a system [10]. This is influenced by the factors or independent variables previously described. 3. Evaluation of Indonesian Online Learning System The implementation of ICT are related to the user acceptance and understanding of technology [27]. It becomes an important factors that affecting the success of technology implementation [27], [28]. The user acceptance is determined by two factors which are (a) user’s perception about technology that will assist their works, and (b) the level of difficulty in using technology. It means that the use of technology should be easy to operate. In a technology usability, there are five aspects to evaluate the system or technology. First aspect is a learnability related to the ease in learning a system. Second aspect is a rememberability related to the ease of remember to operate the system. Third aspect is an efficiency of use the system. Fourth aspect is a reability in use means that system should reduce the human error in their works. And the fifth aspect is a user satisfaction when using a system. As we mention before, this study proposes an evaluation for Indonesian online learning system (SPADA) for teacher profession education program (PPG). In its application, SPADA has a several features to support an online learning. These features such as learning content, discussions, course tools and so on. Fig. 3 show the preview of course overview in SPADA web page. Moreover, the next sub-section will discussed an evaluation design for SPADA. ISSN 2614-0047 Bulletin of Social Informatics Theory and Application 87 Vol. 4, No. 2, September 2020, pp. 83-90 Dwiyanto et.al (Indonesian online learning system evaluation framework based on UTAUT 2.0) Fig. 3. Preview of SPADA overview page 3.1. Evaluation Model This study uses a quantitative approach with the UTAUT2 model for evaluating the SPADA. In its implementation, there are several variables tested according to the UTAUT2 model. However, not all variables are tested because they will be adjusted to the needs and conditions of system use. Fig. 4 shows the model used in this study and adopted from the UTAUT2 model. Based on Fig. 4, there are several differences from the UTAUT2 model. The first difference is that from the independent variable there is no price value variable. This is because in using the SPADA users do not need to pay extra for accessing it. The use of this learning system is an integral part of the PPG program. Then the moderating variable such as age, gender, and experience also not measured because it does not directly affect the dependent variable. Therefore, further research can be done to discuss a moderating variables in UTAUT2. Fig. 4. Evaluation model adopted from UTAUT2 88 Bulletin of Social Informatics Theory and Application ISSN 2614-0047 Vol. 4, No. 2, September 2020, pp. 83-90 Dwiyanto et.al (Indonesian online learning system evaluation framework based on UTAUT 2.0) 3.2. Evaluation Design A quantitative approach is used to analyze data using numerical data and processed by statistical analysis methods. After the data is processed, the results will be described and conclusions based on the statistical analysis. The data collection method used in this study is to collect information from the sample using a questionnaire. Meanwhile, the data analysis used is path analysis [29], [30]. Path analysis is a form of multiple regression statistical analysis used to evaluate causal models by examining the relationship between the dependent variable and two or more independent variables [31]. Moreover, Fig. 5 illustrates the evaluation design in this study. Fig. 5. Evaluation Design The population is selected for participants who finished the online learning stage through the system to be tested. Then, a purposive sampling technique is used with the selection of a group of subjects based on certain considerations. The number of samples with an error rate of 5% by looking at the table for determining the number of samples. The next important thing is the evaluation instrument to measure and analyze a data. A questionnaire is used as an evaluation instrument that would be distributed to the research sample. The questionnaire contains questions arranged based on the adoption of the UTAUT2 model. The questions are structured by adjusting the variables used only in this study. Table 1 describes the indicators of the variables in this research model. Furthermore, the weighting uses a Likert scale in the questionnaire using five answer choices consisting of strongly disagree (1), disagree (2), neutral (3), agree (4) and strongly agree (5). Table.1 Indicator of each Variable Variable Code Indicator of the Variable Performance Expectancy PE1 Perceived usefulness PE2 Extrinsic motivation PE3 Job-fit PE4 Relative advantage Effort Expectancy EE1 Perceived ease of use EE2 Complexity EE3 Ease of use Social Influence SI1 Subjective Norm SI2 Social Factor SI3 Image Facilitating Conditions FC1 Perceived behavioral control FC2 Facilitating conditions FC3 Compatibility Hedonic Motivation HM1 Repurchase intentions HM2 Positive word-of-mouth communication HM3 Service quality Habit HT1 Fun HT2 Entertain HT3 Interest Behavioral Intention BI1 Quality BI2 Price BI3 Value Use Behavior UB1 Prior Use UB2 Addiction UB3 Behavior to be automatic ISSN 2614-0047 Bulletin of Social Informatics Theory and Application 89 Vol. 4, No. 2, September 2020, pp. 83-90 Dwiyanto et.al (Indonesian online learning system evaluation framework based on UTAUT 2.0) Based on Table 1, it can be seen that there are several indicators from each variable. These indicators are the basis for making the questionnaire questions to be tested. Each indicator can be broken down into several questions that refer to that topic. An example of the question such as “using the system in my job would enable me to accomplish tasks more quickly” for PE1; “it is easy for me to learn operating the system” for EE1; and “I use the system because of the coworkers who use the system” for SI1. The question is adjusted to the intent of each indicator. After collecting the data, in the last stage the data will be processed using statistical techniques. The analysis used is path analysis, wherein the results of the analysis will be described. Then, based on the results, a conclusion and suggestions can be used as a reference for system development. 4. Conclusion This paper proposed an evaluation model for Indonesian online learning system (SPADA). It is used for accommodate online learning of teacher profession education program (PPG). The evaluation is to discover user acceptance and understanding of technology. It becomes an important factors that affecting the success of technology implementation. The user acceptance is determined by two factors which are (a) user’s perception about technology that will assist their works, and (b) the level of difficulty in using technology. The evaluation is adopted from UTAUT2 as technology acceptance model to explain user intentions to use an information system and usage behavior. By evaluating the system it is expected the system can be improved better. Furthermore, it is necessary to implement this evaluation model to get the expected results. Also, this study also can be as reference to conduct similar study related to the use of information system. References [1] A. Bertram and B. Waldrip, “ICT for ICT’s sake: Secondary teachers’ views on technology as a tool for teaching and learning,” Aust. Educ. Comput., vol. 28, no. 1, 2013. [2] H. Elmunsyah, “A national education policy-based ICT model for Indonesian vocation1al high schools (VHS),” Glob. J. Eng. Educ., vol. 16, no. 3, pp. 136–140, 2014. [3] T. Teo, Technology Acceptance in Education. AW Rotterdam: Sense Publishers, 2011. [4] D. Xu and S. S. Jaggars, “The impact of online learning on students’ course outcomes: Evidence from a large community and technical college system,” Econ. Educ. Rev., vol. 37, pp. 46–57, 2013. [5] T. Nguyen, “The effectiveness of online learning: Beyond no significant difference and future horizons,” MERLOT J. Online Learn. Teach., vol. 11, no. 2, pp. 309–319, 2015. [6] M. I. Mustofa, M. Chodzirin, and L. Sayekti, “Formulasi Model Perkuliahan Daring Sebagai Upaya Menekan Disparitas Kualitas Perguruan Tinggi (Studi terhadap Website pditt.belajar.kemdikbud.go.id),” Walisongo J. Inf. Technol., vol. 1, no. 2, pp. 151–160, 2019. [7] V. Venkatesh, M. G. Morris, G. B. Davis, and F. D. Davis, “User Acceptance of Information Technology: Toward a Unified View,” MIS Q., vol. 27, no. 3, pp. 425–478, 2003. [8] V. Venkatesh, J. Y. L. Thong, and X. Xu, “Consumer Acceptance and Use of Information Technology: Extending the Unified Theoryof Acceptance and Use of Technology,” MIS Q., vol. 36, no. 1, pp. 157–178, 2012. [9] B. Bervell and I. N. Umar, “Validation of the UTAUT model: Re-considering non-linear relationships of Exogeneous variables in higher education technology acceptance research,” Eurasia J. Math. Sci. Technol. Educ., vol. 13, no. 10, pp. 6471–6490, 2017. [10] S. A. Salloum and K. Shaalan, “Factors affecting students’ acceptance of e-learning system in higher education using UTAUT and structural equation modeling approaches,” in International Conference on Advanced Intelligent Systems and Informatics, 2018, pp. 469–480. [11] R. Bendi and S. Andayani, “Penerapan Model UTAUT untuk memahami perilaku pengguna sistem informasi akademik,” J. HOAQ Teknol. Inf., vol. 2, no. 1, pp. 144–151, 2013. [12] N. Ain, K. Kaur, and M. Waheed, “The influence of learning value on learning management system use: An extension of UTAUT2,” Inf. Dev., vol. 32, no. 5, pp. 1306–1321, 2016. 90 Bulletin of Social Informatics Theory and Application ISSN 2614-0047 Vol. 4, No. 2, September 2020, pp. 83-90 Dwiyanto et.al (Indonesian online learning system evaluation framework based on UTAUT 2.0) [13] M. El-Masri and A. Tarhini, “Factors affecting the adoption of e-learning systems in Qatar and USA: Extending the Unified Theory of Acceptance and Use of Technology 2 (UTAUT2),” Educ. Technol. Res. Dev., vol. 65, no. 3, pp. 743–763, 2017. [14] S. Yang, “Understanding undergraduate students’ adoption of mobile learning model: A perspective of the extended UTAUT2,” J. Converg. Inf. Technol., vol. 8, no. 10, p. 969, 2013. [15] V. Venkatesh, J. Y. Thong, and X. Xu, “Unified theory of acceptance and use of technology: A synthesis and the road ahead,” J. Assoc. Inf. Syst., vol. 17, no. 5, pp. 328–376, 2016. [16] Y. K. Dwivedi, N. P. Rana, A. Jeyaraj, M. Clement, and M. D. Williams, “Re-examining the unified theory of acceptance and use of technology (UTAUT): Towards a revised theoretical model,” Inf. Syst. Front., vol. 21, no. 3, pp. 719–734, 2019. [17] O. Oechslein, M. Fleischmann, and T. Hess, “An application of UTAUT2 on social recommender systems: Incorporating social information for performance expectancy,” in 47th Hawaii international conference on system sciences, 2014, pp. 3297–3306. [18] V. Venkatesh, M. G. Morris, G. B. Davis, and F. D. Davis, “User Acceptance of Information Technology: Toward a Unified View,” J. Chem. Inf. Model., vol. 27, no. 3, pp. 425–478, 2013. [19] A. A. Alalwan, Y. K. Dwivedi, and N. P. Rana, “Factors influencing adoption of mobile banking by Jordanian bank customers: Extending UTAUT2 with trust,” Int. J. Inf. Manage., vol. 37, no. 3, pp. 99– 110, 2017. [20] A. M. Elkaseh, K. W. Wong, and C. C. Fung, “Perceived ease of use and perceived usefulness of social media for e-learning in Libyan higher education: A structural equation modeling analysis,” Int. J. Inf. Educ. Technol., vol. 6, no. 3, p. 192, 2016. [21] H. Khechine and M. Augier, “Adoption of a social learning platform in higher education: an extended UTAUT model implementation,” in Proceedings of the 52nd Hawaii International Conference on System Sciences, 2019. [22] M. Kuciapski, “Students acceptance of m-learning for higher education–UTAUT model validation,” in EuroSymposium on Systems Analysis and Design, 2016, pp. 155–166. [23] H. Khechine, B. Raymond, and M. Augier, “The adoption of a social learning system: Intrinsic value in the UTAUT model,” Br. J. Educ. Technol., 2020. [24] S. Hu, K. Laxman, and K. Lee, “Exploring factors affecting academics’ adoption of emerging mobile technologies-an extended UTAUT perspective,” Educ. Inf. Technol., pp. 1–21, 2020. [25] D. Chávez Herting, R. Cladellas Pros, and A. Castelló Tarrida, “Habit and social influence as determinants of PowerPoint use in higher education: A study from a technology acceptance approach,” Interact. Learn. Environ., pp. 1–17, 2020. [26] C. M. Chao, “Factors determining the behavioral intention to use mobile learning: An application and extension of the UTAUT model,” Front. Psychol., vol. 10, p. 1652, 2019. [27] R. Rauniar, G. Rawski, J. Yang, and B. Johnson, “Technology acceptance model (TAM) and social media usage: an empirical study on Facebook,” J. Enterp. Inf. Manag., vol. 27, no. 1, pp. 6–30, 2014. [28] I. I. Ushatikova, A. R. Rakhmanova, V. S. Kireev, A. O. Chernykh, and M. A. Ivanov, “Pedagogical bases of formation of key information technology competencies polytechnic institute graduates,” Int. J. Econ. Financ. Issues, vol. 6, no. 2, pp. 283–289, 2016. [29] T. Thomas, L. Singh, and K. Gaffar, “The utility of the UTAUT model in explaining mobile learning adoption in higher education in Guyana,” Int. J. Educ. Dev. using ICT, vol. 9, no. 3, 2013. [30] S. Saxena and M. Janssen, “Examining open government data (OGD) usage in India through UTAUT framework,” Foresight, vol. 19, no. 4, 2017. [31] J. W. Creswell, Research design: Qualitative, quantitative, and mixed methods approaches. Sage publication, 2013.