International Journal of Interactive Mobile Technologies (iJIM) – eISSN: 1865-7923 – Vol. 14, No. 20, 2020 Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in the Social Distancing https://doi.org/10.3991/ijim.v14i20.15787 Dewa Gede Hendra Divayana Universitas Pendidikan Ganesha, Singaraja, Indonesia hendra.divayana@undiksha.ac.id Abstract—The selection of a suitable platform to facilitate online learning in social distancing becomes a very important thing to maintain the quality and smoothness of the learning process was carried out at home. Some educational evaluation models were able to be used to provide recommendations in choosing an online learning platform that was suitable to use in social distancing, such as CIPP, Countenance, and CSE-UCLA. However, those models are only able to provide recommendations based on narrative evaluation components so raised different understandings and high subjectivity in its implementation in the field. Those models haven’t formulas that specifically and accurately provide quantitative results in determining the most priority platforms is used in online learning. Therefore, it is essential to know there is a new formula in the field of educational evaluation to determine the suitable platform for online learning that is done at home. One of the new formulas that can be used and contributed to solving problems in the field of educational evaluation is the DIVAYANA formula. This formula can show an accurate calculation mechanism in determining one of the platforms that the most priority from the various choices of online learning platforms. The purpose of this research was to show the mechanism for calculating the DIVAYANA formula to determine the priority platform suitable for online learning. This research used an evaluative approach that focused on the nominate stage in the DIVAYANA model because the DIVAYANA formula is located in that nominate stage. Eighty respondents were involved in the initial data collection by evaluating the platform selection criteria for online learning. The subjects who were involved in testing the effectiveness of the DIVAYANA formula were eight experts. Questionnaires were used as the initial data collection tools and testing tools for the effectiveness of the DIVAYANA formula. The method was used to analyze the data of effectiveness test results on the DIVAYANA formula was by comparing that test results with the effectiveness standard that refers to five scales. The results of the effectiveness test showed the percentage of effectiveness level was 89.79%. It means that the DIVAYANA formula is effective to use in determining priority platforms suitable for online learning at home. Keywords—DIVAYANA Formula, Platforms, Online Learning, Social Distancing 50 http://www.i-jim.org https://doi.org/10.3991/ijim.v14i20.15787%0d mailto:ssr@online-engineering.org Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... 1 Introduction The entry of the COVID-19 pandemic into Indonesia made worried all Indonesian society. This outbreak had an impact on various fields of life, such as health, trade, transportation, economics, and also in the education sector. All elements of society in Indonesia made various efforts to prevent this outbreak from spreading further. One of the prevention efforts carried out was to do social distancing. We can minimize the contact or close interaction between people who have contracted the virus with others who have not been infected through social distancing. Social distancing actions in Indonesia were carried out by reducing activities that caused crowds of people. One of the activities that often causes crowd or meeting of many people is the activity of the learning process at schools or campuses. Therefore, the Ministry of Education and Culture of the Republic of Indonesia had taken a policy to temporarily close the learning activities through face to face in schools or campuses as an effort to prevent the spread of the COVID-19 pandemic. However, the face-to-face learning process at schools or campuses was replaced via online learning that was carried out by each student and teacher/lecturer at home. Universitas Pendidikan Ganesha, as one of the state universities in the northern Bali region, also had responsive and had implemented online learning since the policies of the Ministry of Education and Culture of the Republic of Indonesia had rolled out. Various platforms had been used by the academic community of Universitas Pendidikan Ganesha as an effort to support online learning at home. Several familiar platforms that are often used by students and teachers/lecturers in schools/campuses, included: Schoology, Moodle, Quipper School, Edmodo, and Kelase [1-16]. In principle, all of those platforms can be used as supporting facilities for online learning, but it’s just not possible that all of those platforms can be used simultaneously in one learning process. Therefore, it is necessary to determine a platform that is suitable for use in a learning process in schools/campuses that adjust to the conditions and characteristics of students. Determination of one platform that is suitable and appropriate to be used in the learning process is not only seen from the number of people who know the existence of the platforms, but also from various other criteria that influence it. Several of the other criteria that also affect in the platform’s selection, included: speed of access, completeness of learning features available, ease of operation, maximum capacity of material content, the maximum number of users who can become members, ease of manipulating data, data security, and visual display of the platform. The recommendation results will not be optimal if there is no appropriate model used to evaluate several platforms, although the criteria that influence platform selection have been known. Therefore, it is necessary to evaluate using an appropriate evaluation model. Several evaluation models that are often used in evaluating online learning platforms included: CSE-UCLA [17], Countenance [18], and CIPP [19,20]. The problem arises when each of those evaluation models is unable to show an accurate calculation process mechanism in determining one of the priority platforms from several choices of online learning platforms. Therefore, we need a formula in the evaluation model that can be used to determine the priority platform that is suitable iJIM ‒ Vol. 14, No. 20, 2020 51 Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... for use in online learning during the social distancing period. One formula that is an innovation in the field of educational evaluation is the DIVAYANA formula. This formula can show the calculation process to determine a suitable platform to use in online learning during social distancing so that later the learning process can still be maintained and well implemented. Based on that innovation, it raises research questions and research objectives. The research question: how was the DIVAYANA formula calculation mechanism in determining the most suitable platform for online learning during the social distancing? The purpose of this research was to show the detailed calculation mechanism of the DIVAYANA formula, starting from the initial data to the final result that shows a priority platform that suitable for use in online learning. 2 Literature Review There are several studies from previous research that are used as the basis for presenting this research. The research was conducted by Basilaia and Kvavadze [21] in 2020 showed a transfer of conventional and traditional face-to-face learning processes carried out in the classroom to online learning with various platform choices. The difficulty experienced in implementing online learning is ensuring the learning process can operate honestly and quality. Basilaia and Kvavadze’s research can become the reference research that is related to choosing a suitable platform to use to make quality online learning and avoid cheating in the learning process. The research was conducted by Kalogiannakis and Papadakis [22] in 2019 showed the use of TAM (Technology Acceptance Model) in evaluating teacher skills in using ICT and measuring teacher attitudes/perceptions of the use of mobile devices in teaching. It was also obtained an overview of mobile devices that can be used to provide learning content inside or outside of a traditional classroom environment. The results of Kalogiannakis and Papadakis’ research become a trigger for researching the selection of online learning platforms that can be accessed and operated easily using a mobile device. Some of the online learning platforms can be accessed from a mobile device, such as: Moodle, Kelase, Schoology, Quipper School, and Edmodo. Research was conducted by Papadakis et al. [23] in 2018 showed that Moodle was not able to be used as an effective learning tool and was impressed only as a medium for electronic document storage because of its limitations in terms of reliability and usability. The research results by Papadakis et al. become a trigger to conducted research more deeply about Moodle as a platform that can be used to support the learning process during the social distancing. Research was conducted by Ferdianto and Dwiniasih [24] in 2019 showed the visualization of Schoology as a platform that can be used in online learning. Schoology consist of four features, such as: features of tests and quizzes, features of content delivery, features of video viewing/link sources, and features of discussion. Research was conducted by Gunawan et al. [25] in 2019 showed that the Moodle platform has features that can increase the creativity of its users in the online learning process. Research was conducted by Mulyono [26] in 2016 showed the use of the 52 http://www.i-jim.org Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... Quipper School platform which can be used to facilitate the online learning process both inside and outside the classroom. The research was conducted by Ekici [27] in 2017 showed the use of Edmodo as a platform that can be used to create online communities in the learning process. Research by Ferdianto and Dwiniasih [24]; Gunawan et al. [25]; Mulyono [26]; and Ekici [27]; strengthening the foundation for the author to conducted research more deeply on the selection of priority platforms used to support the learning process during social distancing. Besides some of the studies above, the limitations of several previous studies have also been the trigger for the presence of this research. Limitation of the research that was conducted by Dodun et al. in 2015 [28], showed there is no mathematical and specific formula used in the calculation process to determine Moodle as the most suitable platform for e-learning based online learning. Limitation of the research that was conducted by Ouadoud et al. in 2016 [29] showed there is not yet demonstrated the use of mathematical formulas in the calculation process to determine the best platform for using as a facility that supports online learning. Limitation of the research that was conducted in 2018 by Chivu et al. [30] showed there is not yet a detail description of the calculation process to determine which platform is the main priority as a facility to support the online learning process. Based on the limitations in the research of Dodun et al. [28]; Ouadoud et al. [29]; and Chivu et al. [30]; it appears that innovation in the form of the DIVAYANA formula has a contribution and is suitable for solving problems/limitations in those studies. That is because the DIVAYANA formula is presented to show a mathematical calculation process in determining the priority platform. 3 Research Methodology This research approach was evaluative. The evaluative stage in this research refers to the DIVAYANA model stages, which consist of the Description stage, Input stage, Verification stage, Action stage, Yack stage, Analysis stage, Nominate stage, and Actualization stage [31]. At the Description stage, activity was carried out to identify the causes of the emergence of needs and problems related to online learning platforms. At the Input stage, activities were carried out to input several alternative solutions for problems in determining the online learning platform. At the Verification stage, a minimum standard was determined to achieve success in obtaining a priority platform and matching the suitability of problem-solving alternatives with those minimum standards. At the Action stage, it was carried out field trial activities on several alternatives of problems solutions that had verified. At the Yack stage, focus group discussion activities were carried out between experts and evaluators to obtain the mutual agreement in giving weight to the minimum standard of evaluation success. Besides that, at this Yack stage, discussions were also held to obtain mutual agreement about qualitative data used to strengthen the quantitative data obtained in evaluating online learning platforms. At the Analysis stage, it was carried out analysis activities related to quantitative and qualitative data. Quantitative data in the form of the percentage level of effectiveness obtained from the Action stage. Qualitative data iJIM ‒ Vol. 14, No. 20, 2020 53 Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... in the form of arguments/opinions of experts and evaluators obtained from the Yack stage. At the Nominate stage, a calculation process was carried out using the DIVAYANA formula to determine the priority platform. At the Actualization stage, activities to implement the priority platforms that had been selected were carried out in the actual environment. Based on the questions and objectives of this research, the focus of this research was only on the Nominate stage contained in the DIVAYANA model. The Nominate stage was the focus of research because the DIVAYANA formula is located at this stage. The DIVAYANA formula consists of three equations, which are used as a basis in determining which platform is the best priority as an online learning facility. Equation (1) is used to find the weighted improvement scores, equation (2) is used to find normalization scores, and equation (3) is used to determine ranking scores. Those three equations can be shown as follows. (𝑊!"#$)% = &! ∑ &! " !#$ (1) Notes: �̅� = The average of weight given by the experts/evaluators through joint discussion 𝑊!"#$ = Scores of average weights improvement 𝐷( = ∏ &%! &'()*+,!" !#$ * (2) Where: å (𝑊!"#$)% = 1; and i=1,2,3,...,n. Notes: D = Vector-D 𝑥 = Assessment scores for each criterion m = Total of all experts 𝑅( = +% ∑ +% " %#$ (3) Notes: D = Vector-D R = Vector-R Initial data that was used to try to calculate the DIVAYANA formula was obtained from the results of an assessment involved 80 students who used online learning during social distancing. Those eighty respondents were the populations in this research. The populations of this research were all students of 2nd semester at the Informatics Education Department, Universitas Pendidikan Ganesha who had taken the Operating System courses at the 2019/2020 academic year. All students of 2nd semester were divided into 4 classes, included: class-A, class-B, class-C, and class-D with each class consisting of 20 students. The tools were used to collect initial data from 80 respondents were in the form of questionnaires that included nine items related to criteria for selecting platforms for online learning. Those nine questions can be seen in Appendix 6. The effectiveness test of the DIVAYANA formula was carried out by eight experts (four education 54 http://www.i-jim.org Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... experts and four informatics experts). The tools were used to conduct effectiveness tests were in the form of questionnaires, which included 12 questions. Those twelve questions can be seen in Appendix 12. The content validity of the questionnaires was tested using the Gregory formula. The Gregory formula can be seen in equation (4) [32]. The reliability of the questionnaire items was tested using the Cronbach Alpha formula. The Cronbach Alpha formula can be seen in equation (5) [33]. The interpretation of the results of the questionnaire content validity and reliability tests refers to the Guilford categorization [34]. The very high category is in the range of 0.800 < rxy ≤ 1.000; the high category in the range of 0.600 < rxy ≤ 0.800; the moderate category in the range of 0.400 < rxy ≤ 0.600; the low category in the range of 0.200 < rxy ≤ 0.400; and very low category in the range of 0.000 < rxy ≤ 0.200. 𝐶𝑜𝑛𝑡𝑒𝑛𝑡 𝑉𝑎𝑙𝑖𝑑𝑖𝑡𝑦 = + ,-.-/-+ (4) Notes: A = cell indicating disagreement between the two raters B and C = cells indicating the difference in views between the raters D = cell indicating valid agreement between the two raters 𝛼 = 0 012 ∗ 51 − ∑𝛔% - 𝛔.- 8 (5) Notes: a = The non-test instrument reliability coefficient n = Number of items s! " = The variant of the score of item-i s# " = The variant of the total score The content validity results of the questionnaires that were used for the initial data collection can be seen in Appendix 4, and its reliability can be seen in Appendix 5. The content validity results of the questionnaire that were used to test the effectiveness of the DIVAYANA formula can be seen in Appendix 10, and its reliability can be seen in Appendix 11. Analysis of the effectiveness of DIVAYANA formula test results was done by comparing the effectiveness of the test results with the effectiveness standard that refers to five scales. The effectiveness standard referring to the five scales can be seen in Table 1 [35-37]. Table 1. The formula effectiveness standard that refers to five scales Category Percentage Ineffective 0-54 Less effective 55-64 Sufficient 65-79 Effective 80-89 Very effective 90-100 iJIM ‒ Vol. 14, No. 20, 2020 55 Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... 4 Results and Discussion Data of weights that were given by eight experts and initial data obtained from the assessment of 80 respondents were needed to facilitate the process of calculating the DIVAYANA formula in determining the most appropriate platform to facilitate online learning for social distancing in Operation System learning at Informatics Education Department, Universitas Pendidikan Ganesha. Table 2 shows the data of weights from the experts, while Table 3 shows initial data from the assessment results by eighty respondents. Table 2. Data of weights from eight experts Names of Criteria C1 C2 C3 C4 C5 C6 C7 C8 C9 Weight from Experts Expert-1 4 4 5 4 4 5 5 4 4 Expert-2 5 5 4 5 4 5 4 4 4 Expert-3 4 5 5 5 5 4 4 5 5 Expert-4 5 4 5 5 5 5 4 5 4 Expert-5 5 5 4 5 4 4 4 5 5 Expert-6 4 4 5 5 5 4 5 4 5 Expert-7 5 5 5 5 4 5 4 5 4 Expert-8 4 5 5 4 4 4 4 5 5 Average 4.500 4.625 4.750 4.750 4.375 4.500 4.250 4.625 4.500 WYack 0.110 0.113 0.116 0.116 0.107 0.110 0.104 0.113 0.110 SWYack 1.000 Table 3. Initial data of assessment results from 80 respondents Platforms Quipper School Moodle Edmodo Schoology Kelase Platforms Selection Criteria C1 82.33 82.97 74.19 75.36 70.22 C2 82.33 82.97 74.19 75.36 70.22 C3 82.33 82.97 74.19 75.36 70.22 C4 82.33 82.97 74.19 75.36 70.22 C5 82.33 82.97 74.19 75.36 70.22 C6 82.33 82.97 74.19 75.36 70.22 C7 17.67 17.03 25.81 24.64 29.78 C8 82.33 82.97 74.19 75.36 70.22 C9 82.33 82.97 74.19 75.36 70.22 Notes: C1: Knowing the platform existence C2: Speed of platform access C3: Completeness of learning features available on the platform C4: Ease of platform operation C5: Maximum capacity of material content on the platform C6: The maximum number of users who can become members on the platform C7: Ease of data manipulation (input, edit, update, and delete) in platforms C8: Guarantee of data security in the platform C9: Display of platform visualization Vector-D can be calculated using the DIVAYANA formula, especially through equation (2), with data sourced from Table 2 and Table 3. The Vector-D calculation process entirely can be explained as follows. 56 http://www.i-jim.org Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... D1 = $82.330.110&$82.330.113&$82.330.116&$82.330.116&$82.330.107&$82.330.110&$17.670.104&$82.330.113&$82.330.110& 8 = 8.77 D2 = $82.970.110&$82.970.113&$82.970.116&$82.970.116&$82.970.107&$82.970.110&$17.030.104&$82.970.113&$82.970.110& 8 = 8.80 D3 = $74.190.110&$74.190.113&$74.190.116&$74.190.116&$74.190.107&$74.190.110&$25.810.104&$74.190.113&$74.190.110& 8 = 8.31 D4 = $75.360.110&$75.360.113&$75.360.116&$75.360.116&$75.360.107&$75.360.110&$24.640.104&$75.360.113&$75.360.110& 8 = 8.39 D5 = $70.220.110&$70.220.113&$70.220.116&$70.220.116&$70.220.107&$70.220.110&$29.780.104&$70.220.113&$70.220.110& 8 = 8.03 Based on the obtained Vector-D score, the Vector-R calculation process can be performed to determine to rank using the DIVAYANA formula, especially through equation (3). The Vector-R calculation process entirely can be explained as follows. 𝑅2 = +$ +$- +-- +/- +0- +1 𝑅2 = 5.77 5.77-5.58- 5.92 -5.9:-5.89 = 0.207 𝑅; = +- +$- +-- +/- +0- +1 𝑅; = 5.58 5.77-5.58- 5.92 -5.9:-5.89 = 0.208 𝑅9 = +/ +$- +-- +/- +0- +1 𝑅9 = 5.92 5.77-5.58- 5.92 -5.9:-5.89 = 0.196 𝑅< = +0 +$- +-- +/- +0- +1 𝑅< = 5.9: 5.77-5.58- 5.92 -5.9:-5.89 = 0.198 𝑅= = +1 +$- +-- +/- +0- +1 𝑅= = 5.89 5.77-5.58- 5.92 -5.9:-5.89 = 0.190 Based on the Vector-R score was obtained, then it can be seen the recapitulation ranking visualization of each platform that was used in online learning, especially in Operating System courses at the Informatics Education Department, Universitas Pendidikan Ganesha during the social distancing. The recapitulation ranking visualization of each platform can be seen in Figure 1. iJIM ‒ Vol. 14, No. 20, 2020 57 Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... Fig. 1. Ranking recapitulation of each online learning platform Based on Figure 1 above, it appears that the most suitable and priority platform to be used in online learning for Operating System courses at the Informatics Education Department, Universitas Pendidikan Ganesha, during the social distancing was Moodle. Generally, Moodle has advantages when compared to the other four learning platforms used in the learning process at the Informatics Education Department, Universitas Pendidikan Ganesha. That is because Moodle provides complete features to support the learning process. Besides, all components of Moodle can be easily configured and can be adjusted according to the needs of each institution that uses it. Those advantages of Moodle are reinforced by several research results conducted by Sari, Baedhowi, and Indrawati in 2017 [38]; Singh in 2016 [39]; Umek et al. in 2015 [40,41]; Goyal and Tambe in 2015 [42]; Gogan, Sirbu, and Draghici in 2015 [43]; Jebari, Boussedra, and Ettouhami [44]; Alghafis, Alrasheed, & Abdulghany [45]. The display of the Moodle platform that was used in the Operating System learning at the Informatics Education Department, Universitas Pendidikan Ganesha, can be seen in Figure 2. 0,207 0,208 0,196 0,198 0,190 0,000 0,050 0,100 0,150 0,200 0,250 0,300 0,350 0,400 0,450 0,500 Quipper School Moodle Edmodo Schoology Kelase Sc or es o f V ec to r- R Platforms 58 http://www.i-jim.org Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... Fig. 2. Moodle platform for online learning of operating system courses with the introduction language of Indonesian Figure 2 above shows the display of the Moodle platform that was used to facilitate the online learning process of Operating System courses at the Informatics Education Department, Universitas Pendidikan Ganesha. Online learning for Operating System courses was conducted in the 2nd semester of the 2019/2020 academic year using the language of instruction, namely Indonesian. All the features available on the Moodle platform was optimized its utilization, especially during the social distancing so that students were able to learn the Operating System material optimally. The features that were optimized, including: the features that provide learning resources in video format, digital modules, and features to facilitate discussion through forums. The effectiveness measurement of the DIVAYANA formula utilization to determine a suitable platform for online learning during social distancing was carried out by an effectiveness test that involved four informatics experts and four education experts. The effectiveness test results of the DIVAYANA formula completely can be seen in Table 4. iJIM ‒ Vol. 14, No. 20, 2020 59 Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... Table 4. Effectiveness test results on DIVAYANA formula in selection of suitable platforms for online learning in the social distancing Experts E1 E2 E3 E4 E5 E6 E7 E8 Items 1 5 4 4 5 4 5 4 5 2 5 4 4 4 4 5 5 4 3 4 4 4 5 4 4 5 5 4 4 5 4 5 4 5 5 4 5 5 4 5 4 4 5 5 4 6 5 5 5 4 5 4 5 5 7 5 4 5 5 4 5 5 5 8 5 5 5 4 4 4 5 4 9 5 5 4 5 5 4 4 5 10 5 4 5 4 4 4 5 4 11 4 5 4 4 5 4 5 4 12 4 4 5 4 4 4 4 5 ∑ 56 53 54 53 51 53 57 54 Effectiveness (%) 93.33 88.33 90.00 88.33 85.00 88.33 95.00 90.00 Average 89.79 Notes: E1: Education Expert-1 E2: Education Expert-2 E3: Education Expert-3 E4: Education Expert-4 E5: Informatics Expert-1 E6: Informatics Expert-2 E7: Informatics Expert-3 E8: Informatics Expert-4 The data of average weights improvement shown in Table 2 were obtained from the calculation of the DIVAYANA formula, especially in equation (1). The initial data shown in Table 3 were obtained from the results of 80 respondents’ assessment of the criteria to determine platforms suitable for online learning. The data shown in Table 4 shows the effectiveness level percentage of the DIVAYANA formula utilization to determine the most suitable platform for online learning was 89.79%. It is indicated that the DIVAYANA formula effective to be used to determine the suitable platform for online learning. Generally, the DIVAYANA formula is effectively used to determine the ranking or recommendation priority of the object being evaluated related to the field of informatics engineering education [31]. Therefore, it is true that the DIVAYANA formula is also effective for choosing the priority platform used in the online learning process during the social distancing period. That is because choosing a platform for online learning also an activity categorized in the scope of informatics engineering education. The results of this research can answer the limitations of Dodun et al. [28], Ouadoud et al. [29], and Chivu et al. [30] by showing the calculation process using the DIVAYANA formula in determining the priority platform that is most suitable as an online learning support facility. Besides the advantages shown from the results of this research, there are also limitations found in this research. The first limitation is 60 http://www.i-jim.org Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... the alternative platforms that are compared in this research still limited only on five platforms, whereas in fact, many other platforms also can be used as supporting facilities for online learning. The second limitation is a factor analysis has not been carried out to determine the latent factor on each platform. 5 Conclusion The form of contribution through this research results in the field of educational evaluation is the discovery of the DIVAYANA formula. Generally, this formula is used to determine the recommendation priority or ranking of the object being observed. Referring to the research question and the research results that had been carried out, it can be concluded that the DIVAYANA formula calculation mechanism had running well and produces appropriate decisions with the real conditions in the field. The decision made through the DIVAYANA formula calculation mechanism in this research is to obtain a priority platform that is suitable for online learning in the social distancing period. This is evidenced by the research results, which showed that calculation of the DIVAYANA formula was able to determine the Moodle become the priority platform suitable to use in the Operating System courses at Informatics Education Department, Universitas Pendidikan Ganesha. It was as a supporting facility of online learning in the social distancing. Future work that can be done to anticipate the limitation in this research is to conduct further research by adding other alternative platforms as a comparison of pre-existing platforms. Besides, in the future, factor analysis is needed to obtain latent factors. 6 Acknowledgment The author would like to thank profusely to all parties who provided support in completing this research. The author also expressed thanks to the Chairperson of the Institute for Research and Community Service, Universitas Pendidikan Ganesha that had given the research funding through Research Grant with contract number 760/UN48.16/LT/2020. 7 References [1] Essel, D.D., & Wilson, O.A. (2017). Factors Affecting University Students’ Use of Moodle: An Empirical Study Based on TAM. International Journal of Information and Communication Technology Education, 13(1), 14-26. https://doi.org/10.4018/IJICTE.2017 010102. [2] Sudarwati, N., & Rukminingsih. (2018). Evaluating E-Learning as a Learning Media: A Case of Entrepreneurship E-Learning using Schoology as Media. International Journal of Emerging Technologies in Learning, 13(9), 269-279. https://doi.org/10.3991/ijet.v13i09.77 83. iJIM ‒ Vol. 14, No. 20, 2020 61 https://doi.org/10.4018/IJICTE.2017010102 https://doi.org/10.4018/IJICTE.2017010102 https://doi.org/10.3991/ijet.v13i09.7783 https://doi.org/10.3991/ijet.v13i09.7783 Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... [3] Badia, A., Martín, D., & Gómez, M. (2019). Teachers’ Perceptions of the Use of Moodle Activities and Their Learning Impact in Secondary Education. Technology, Knowledge and Learning, 24(3), 483-499. https://doi.org/10.1007/s10758-018-9354-3. [4] Teo, T., Zhou, M., Fan, A.C.W., & Huang, F. (2019). Factors That Influence University Students’ Intention to Use Moodle: A Study in Macau. Educational Technology Research and Development, 67(3), 749-766. https://doi.org/10.1007/s11423-019-09650-x. [5] Karagiannis, I., & Satratzemi, M. (2018). An Adaptive Mechanism for Moodle Based on Automatic Detection of Learning Styles. Education and Information Technologies, 23(3), 1331-1357. https://doi.org/10.1007/s10639-017-9663-5. [6] Boloudakis, M., Symeon Retalis, S., & Psaromiligkos, Y. (2018). Training Νovice Teachers to Design Moodle-based Units of Learning using A CADMOS-Enabled Learning Design Sprint. British Journal of Educational Technology, 49(6), 1059-1076. https://doi. org/10.1111/bjet.12678. [7] Ma’azi, H., & Janfeshan, K. (2018). The Effect of Edmodo Social Learning Network on Iranian EFL Learners Writing Skill. Cogent Education, 5, 1-17. https://doi.org/10.1080/23 31186X.2018.1536312. [8] Sulisworo, D., Sulistyo, E.N., & Akhsan, R.N. (2017). The Motivation Impact of Open Educational Resources Utilization on Physics Learning Using Quipper School App. Turkish Online Journal of Distance Education, 18(4), 120-128. https://doi.org/10.17718/toj de.340399. [9] Alqahtani, A.S. (2019). The Use of Edmodo: Its Impact on Learning and Students’ Attitudes toward It. Journal of Information Technology Education: Research, 18, 319-330. https://doi.org/10.28945/4389. [10] Sugito, Susilowati, S.M.E., Hartono, & Supartono. (2017). The Learning Syntax through Edmodo in the Beginners Class. International Journal of Evaluation and Research in Education, 6(4), 299-305. https://doi.org/10.11591/ijere.v6i4.10773. [11] Ursavaş, Ö.F., & Reisoglu, I. (2017). The Effects of Cognitive Style on Edmodo Users’ Behaviour: A Structural Equation Modeling-Based Multi-Group Analysis. International Journal of Information and Learning Technology, 34(1), 31-50. https://doi.org/10.1108/IJI LT-06-2016-0019. [12] Trust, T. (2017). Motivation, Empowerment, and Innovation: Teachers’ Beliefs about How Participating in the Edmodo Math Subject Community Shapes Teaching and Learning. Journal of Research on Technology in Education, 49(1-2), 16-30. https://doi.org/10.1080/1 5391523.2017.1291317. [13] Yeou, M. (2016). An Investigation of Students’ Acceptance of Moodle in a Blended Learning Setting Using Technology Acceptance Model. Journal of Educational Technology Systems, 44(3), 300-318. https://doi.org/10.1177/0047239515618464. [14] López, G.A., Sáenz, J., Leonardo, A., & Gurtubay, I.G. (2016). Use of the “Moodle” Platform to Promote an Ongoing Learning When Lecturing General Physics in the Physics, Mathematics and Electronic Engineering Programmes at the University of the Basque Country UPV/EHU. Journal of Science Education and Technology, 25(4), 575-589. https://doi.org/10.1007/s10956-016-9614-8. [15] Buus, L. (2016). From Website to Moodle in a Blended Learning Context. International Journal of Web-Based Learning and Teaching Technologies, 11(1), 51-64. https://doi.org/ 10.4018/ijwltt.2016010104. [16] Horvat, A., Dobrota, M., Krsmanovic, M., & Cudanov, M. (2015). Student Perception of Moodle Learning Management System: A Satisfaction and Significance Analysis. Interactive Learning Environments, 23(4), 515-527. https://doi.org/10.1080/10494820.20 13.788033. [17] Hamid, A., Siregar, T.M., Purba, J., & Mukmin, B.A. (2019). Evaluation of Implementation of Blended Learning in Universitas Negeri Medan. Britain International of 62 http://www.i-jim.org https://doi.org/10.1007/s10758-018-9354-3 https://doi.org/10.1007/s11423-019-09650-x https://doi.org/10.1007/s10639-017-9663-5 https://doi.org/10.1111/bjet.12678 https://doi.org/10.1111/bjet.12678 https://doi.org/10.1080/2331186X.2018.1536312 https://doi.org/10.1080/2331186X.2018.1536312 https://doi.org/10.17718/tojde.340399 https://doi.org/10.17718/tojde.340399 https://doi.org/10.28945/4389 https://doi.org/10.11591/ijere.v6i4.10773 https://doi.org/10.1108/IJILT-06-2016-0019 https://doi.org/10.1108/IJILT-06-2016-0019 https://doi.org/10.1080/15391523.2017.1291317 https://doi.org/10.1080/15391523.2017.1291317 https://doi.org/10.1177/0047239515618464 https://doi.org/10.1007/s10956-016-9614-8 https://doi.org/10.4018/ijwltt.2016010104 https://doi.org/10.4018/ijwltt.2016010104 https://doi.org/10.1080/10494820.2013.788033 https://doi.org/10.1080/10494820.2013.788033 Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... Linguistics, Arts and Education (BIoLAE) Journal, 1(2), 224-231. https://doi.org/10.3325 8/biolae.v1i2.89. [18] Thanabalan, T.V., Siraj, S., & Alias, N. (2015). Evaluation of a Digital Story Pedagogical Module for the Indigenous Learners using the Stake Countenance Model. Procedia-Social and Behavioral Sciences, 176, 907-914. https://doi.org/10.1016/j.sbspro.2015.01.557. [19] Nkhosi, D.T. (2019). The Evaluation of a Blended Faculty Development Course using the CIPP Framework. International Journal of Education and Development using Information and Communication Technology, 15(1), 245-254. [20] Hartati, S.J., Sayidah, N., & Muhajir. (2018). The Use of CIPP Model for Evaluation of Computational Algorithm Learning Program. The 6th South East Asia Design Research International Conference (6th SEA-DRIC), IOP Conf. Series: Journal of Physics: Conf. Series, 1088, 1-6. https://doi.org/10.1088/1742-6596/1088/1/012081. [21] Basilaia, G., & Kvavadze, D. (2020). Transition to Online Education in Schools during a SARS-CoV-2 Coronavirus (COVID-19) Pandemic in Georgia. Pedagogical Research, 5(4), 1-9. https://doi.org/10.29333/pr/7937. [22] Kalogiannakis, M., & Papadakis, S. (2019). Evaluating Pre-service Kindergarten Teachers’ Intention to Adopt and Use Tablets into Teaching Practice for Natural Sciences. International Journal of Mobile Learning and Organisation, 13(1), 113-127. https://doi.org/ 10.1504/IJMLO.2019.096479. [23] Papadakis, S., Kalogiannakis, M., Sifaki, E., & Vidakis, N. (2018). Evaluating Moodle use via Smart Mobile Phones: A case study in a Greek University. EAI Endorsed Transactions on Creative Technologies, 18(16), 1-9. https://doi.org/10.4108/eai.10-4-2018.156382. [24] Ferdianto, F., & Dwiniasih. (2019). Learning Management System (LMS) Schoology: Why it’s Important and What It Looks Like. International Symposium on Sciences, Engineering, and Technology, IOP Conf. Series: Journal of Physics: Conf. Series, 1360, 1- 6. https://doi.org/10.1088/1742-6596/1360/1/012034. [25] Gunawan, G., Sahidu, H., Susilawati, S., Harjono, A., & Herayanti, L. (2019). Learning Management System with Moodle to Enhance Creativity of Candidate Physics Teacher. Mathematics, Informatics, Science, and Education International Conference (MISEIC) 2019, IOP Conf. Series: Journal of Physics: Conf. Series, 1417, 1-6. https://doi.org/10.10 88/1742-6596/1417/1/012078. [26] Mulyono, H. (2016). Using Quipper as an Online Platform for Teaching and Learning English as a Foreign Language. Teaching English with Technology, 16(1), 59-70. [27] Ekici, D.I. (2017). The Use of Edmodo in Creating an Online Learning Community of Practice for Learning to Teach Science. Malaysian Online Journal of Educational Sciences, 5(2), 91-106. [28] Dodun, O., Panaite, E., Seghedin, N., Nagîţ, G., Duşa, P., Neştian, G., & Slătineanu, L. (2015). Analysis of an E-Learning Platform Use by Means of the Axiomatic Design. 9th International Conference on Axiomatic Design (ICAD 2015), 244-249. https://doi.org/10. 1016/j.procir.2015.07.059. [29] Ouadoud, M., Chkouri, M.Y., Nejjari, A., & Kadiri, K.E.E. (2016). Studying and Comparing the Free E-learning Platforms. 4th IEEE International Colloquium on Information Science and Technology (CiSt), 581-586. https://doi.org/10.1109/CIST.2016. 7804953. [30] Chivu, R.G., Turlacu, L.M., Stoica, I., & Radu, A.V. (2018). Identifying the Effectiveness of E-learning Platforms among Students using Eye-Tracking Technology. 4th International Conference on Higher Education Advances (HEAd’18), 621-628. https://doi.org/10.4995/ head18.2018.8046. [31] Divayana, D.G.H. (2020). DIVAYANA Evaluation Model. Jakarta: Ministry of Law and Human Rights of the Republic of Indonesia. Copyright Number: 000197532. iJIM ‒ Vol. 14, No. 20, 2020 63 https://doi.org/10.33258/biolae.v1i2.89 https://doi.org/10.33258/biolae.v1i2.89 https://doi.org/10.1016/j.sbspro.2015.01.557 https://doi.org/10.1088/1742-6596/1088/1/012081 https://doi.org/10.29333/pr/7937 https://doi.org/10.1504/IJMLO.2019.096479 https://doi.org/10.1504/IJMLO.2019.096479 https://doi.org/10.4108/eai.10-4-2018.156382 https://doi.org/10.1088/1742-6596/1360/1/012034 https://doi.org/10.1088/1742-6596/1417/1/012078 https://doi.org/10.1088/1742-6596/1417/1/012078 https://doi.org/10.1016/j.procir.2015.07.059 https://doi.org/10.1016/j.procir.2015.07.059 https://doi.org/10.1109/CIST.2016.7804953 https://doi.org/10.1109/CIST.2016.7804953 https://doi.org/10.4995/head18.2018.8046 https://doi.org/10.4995/head18.2018.8046 Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... [32] Sugihartini, N., Sindu, G.P., Dewi, K.S., Zakariah, M., & Sudira, P. (2019). Improving Teaching Ability with Eight Teaching Skills. Advances in Social Science, Education and Humanities Research, 394, 306-310. https://doi.org/10.2991/assehr.k.200115.050. [33] Divayana, D.G.H., Adiarta, A., & Sudirtha, I.G. (2019). Instruments Development of Tri Kaya Parisudha-Based Countenance Model in Evaluating the Blended Learning. International Journal of Engineering Pedagogy, 9(5), 55-74. https://doi.org/10.3991/ijep.v9 i5.11055. [34] Fazlina, A. (2018). An Analysis of College Entrance Test. English Education Journal, 9(2), 192-215. [35] Fikri, H., Madona, A.S., & Morelent, Y. (2018). The Practicality and Effectiveness of Interactive Multimedia in Indonesian Language Learning at the 5th Grade of Elementary School. The Journal of Social Sciences Research, 2, 531-539. https://doi.org/10.32861/jssr. spi2.531.539. [36] Sugiharni, G.A.D. (2018). The Development of Interactive Instructional Media Oriented to Creative Problem-Solving Model on Function Graphic Subject. Journal of Educational Research and Evaluation, 2(4), 183-189. https://doi.org/10.23887/jere.v2i4.16694. [37] Mantasiah, R., Yusri, & Jufri. (2020). Semantic Feature Analysis Model: Linguistics Approach in Foreign Language Learning Material Development. International Journal of Instruction, 13(1), 185-196. https://doi.org/10.29333/iji.2020.13112a [38] Sari, A.P., Baedhowi, & Indrawati, C.D.S. (2017). The Use of Learning Media with Moodle Approach to Improve the Quality of Education: A Literature Study. Advances in Social Science, Education and Humanities Research (ASSEHR), 158, 54-59. https://doi. org/10.2991/ictte-17.2017.33. [39] Singh, E.G. (2016). Moodle as an E- Learning Approach for Training and Education. International Journal of Innovative Research in Computer and Communication Engineering, 4(10), 17163-17168. https://doi.org/10.15680/IJIRCCE.2016.0410010. [40] Umek, L., Keržič, D., Tomaževič, N., & Aristovnik, A. (2015). Moodle E-Learning System and Students’ Performance in Higher Education: the Case of Public Administration Programmes. International Conference e-Learning 2015, 97-104. https://doi.org/10.1504/i jil.2017.10002132 [41] Umek, L., Aristovnik, A., Tomaževič, N., & Keržič, D. (2015). Analysis of Selected Aspects of Students’ Performance and Satisfaction in a Moodle-Based E-Learning System Environment. Eurasia Journal of Mathematics, Science & Technology Education, 11(6), 1495-1505. https://doi.org/10.12973/eurasia.2015.1408a. [42] Goyal, E., & Tambe, S. (2015). Effectiveness of Moodle-Enabled Blended Learning in Private Indian Business School Teaching Niche Programs. The Online Journal of New Horizons in Education, 5(2), 14-22. [43] Gogan, M.L., Sirbu, R., & Draghici, A. (2015). Aspects Concerning the Use of the Moodle Platform-Case Study. Procedia Technology, 19, 1142-1148. https://doi.org/10.1016/j.pro tcy.2015.02.163. [44] Jebari, K., Boussedra, F., & Ettouhami, A. (2017). Teaching ‘Information Systems Management’ with Moodle. International Journal of Emerging Technologies in Learning, 12(4), 4-16. https://doi.org/10.3991/ijet.v12i04.6183. [45] Alghafis, A., Alrasheed, A., & Abdulghany, A. (2020). A Study on the Usability of Moodle and Blackboard – Saudi Students Perspectives. International Journal of Interactive Mobile Technologies, 14(10), 159-165. https://doi.org/10.3991/ijim.v14i10.14381 64 http://www.i-jim.org https://doi.org/10.2991/assehr.k.200115.050 https://doi.org/10.3991/ijep.v9i5.11055 https://doi.org/10.3991/ijep.v9i5.11055 https://doi.org/10.32861/jssr.spi2.531.539 https://doi.org/10.32861/jssr.spi2.531.539 https://doi.org/10.23887/jere.v2i4.16694 https://doi.org/10.29333/iji.2020.13112a https://doi.org/10.2991/ictte-17.2017.33 https://doi.org/10.2991/ictte-17.2017.33 https://doi.org/10.15680/IJIRCCE.2016.0410010 https://doi.org/10.1504/ijil.2017.10002132 https://doi.org/10.1504/ijil.2017.10002132 https://doi.org/10.12973/eurasia.2015.1408a https://doi.org/10.1016/j.protcy.2015.02.163 https://doi.org/10.1016/j.protcy.2015.02.163 https://doi.org/10.3991/ijet.v12i04.6183 https://doi.org/10.3991/ijim.v14i10.14381 Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... 8 Author Dewa Gede Hendra Divayana is an Associate Professor in the field of Informatics Education at Department of Informatics Education, Faculty of Technical and Vocational, Universitas Pendidikan Ganesha. He obtained his Doctoral Degree in Measurement and Evaluation in Education from Universitas Negeri Jakarta, at Jakarta in 2016. His research interest areas are Artificial Intelligence in Education, Evaluation in Education, and Informatics in Education. (email: hendra.divayana@undiksha.ac.id) Article submitted 2020-05-27. Resubmitted 2020-09-13. Final acceptance 2020-09-14. Final version published as submitted by the authors. iJIM ‒ Vol. 14, No. 20, 2020 65 mailto:hendra.divayana@undiksha.ac.id Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... 9 Appendix 1 9.1 Questions list of the questionnaires used for initial data collection (questions have not been validated) 1. Do you know about platforms for the following online learning? • Quipper School • Moodle • Edmodo • Schoology • Kelase 2. How is the access speed of the following platforms? • Quipper School • Moodle • Edmodo • Schoology • Kelase 3. How complete are the learning features available on the following platforms? • Quipper School • Moodle • Edmodo • Schoology • Kelase 4. Are the following platforms easy to operate? • Quipper School • Moodle • Edmodo • Schoology • Kelase 5. How is the maximum capacity of material content storage on the following platforms? • Quipper School • Moodle • Edmodo • Schoology • Kelase 6. Are the following platforms able to facilitate maximally the number of users who are members? 66 http://www.i-jim.org Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... • Quipper School • Moodle • Edmodo • Schoology • Kelase 7. Are input, edit, and delete data be done easily in the following platforms? • Quipper School • Moodle • Edmodo • Schoology • Kelase 8. How is the guarantee of data security in the following platforms? • Quipper School • Moodle • Edmodo • Schoology • Kelase 9. How is the visual display of the following platforms? • Quipper School • Moodle • Edmodo • Schoology • Kelase 10. How are the prices of the following platforms? • Quipper School • Moodle • Edmodo • Schoology • Kelase iJIM ‒ Vol. 14, No. 20, 2020 67 Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... 10 Appendix 2 Table 5. Results of content validation test by experts on the questionnaire questions used for initial data collection Items Rating Score from Experts Expert-1 Expert-2 Irrelevant Relevant Irrelevant Relevant 1 2 3 4 1 2 3 4 1 - - - Ö - - Ö - 2 - - Ö - - - - Ö 3 - - Ö - - - Ö - 4 - - - Ö - - Ö - 5 - - - Ö - - - Ö 6 - - Ö - - - - Ö 7 - - - Ö - - Ö - 8 - - Ö - - - - Ö 9 - - Ö - - - - Ö 10 Ö - - - - Ö - - 11 Appendix 3 Table 6. Compilation of content validation test results by the two experts to the questionnaire questions that were used for initial data collection Expert-1 Expert-2 Less Relevant (Score 1 - 2) Very Relevant (Score 3 - 4) Less Relevant (Score 1 - 2) Very Relevant (Score 3 - 4) 10 1,2,3,4,5,6,7,8,9 10 1,2,3,4,5,6,7,8,9 12 Appendix 4 Table 7. Cross tabulation of content validation test results by the two experts to the questionnaire questions that were used for initial data collection Expert -2 Less Relevant (Score 1 - 2) Very Relevant (Score 3 - 4) Expert -1 Less Relevant (Score 1 - 2) A 10 (1) B - (0) Very Relevant (Score 3 - 4) C - (0) D 1,2,3,4,5,6,7,8,9 (9) Based on the Gregory formula shown in equation (4), the cross-tabulation results can be calculated to determine the questionnaire content validity used for initial data collection. 68 http://www.i-jim.org Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... 𝐶𝑜𝑛𝑡𝑒𝑛𝑡 𝑉𝑎𝑙𝑖𝑑𝑖𝑡𝑦 = + ,-.-/-+ = : 2-8-8-: = : 28 = 0.900 The content validity value of 0.900 means that the questionnaire used for data collection was very valid. This is evidenced by Guilford’s categorization, where the category is very valid which in the range of 0.800 < rxy ≤ 1.000. 13 Appendix 5 Table 8. Reliability test results of the questionnaire items that was used to data collection Items σi2 I-1 0.682 I-2 1.691 I-3 2.296 I-4 2.452 I-5 2.086 I-6 1.994 I-7 2.037 I-8 1.486 I-9 1.023 I-10 0.484 ∑ σi2 16.230 Based on the calculation of the reliability test using Microsoft Excel, the score was obtained ∑𝜎( ; = 16.230; n = 80; ∑𝜎>; = 91.117; so that it obtains the calculation results of the reliability coefficient using the following Cronbach Alpha formula that reference to equation (5). 𝛼 = 0 012 ∗ 51 − ∑𝛔% - 𝛔.- 8 𝛼 = 58 5812 ∗ 51 − 2?.;98 :2.227 8 𝛼 = 58 7: ∗ 51 − 2?.;98 :2.227 8 𝛼 = 1.013 ∗ 0.822 𝛼 = 0.832 The reliability value of the questionnaire items is 0.832 means the questionnaires used for data collection were very reliable. This is evidenced by Guilford’s categorization, where the category is very reliable which in the range of 0.800 < rxy ≤ 1.000. iJIM ‒ Vol. 14, No. 20, 2020 69 Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... 14 Appendix 6 Based on the results of the content validity shown in Appendix 4, so there were nine questions retained, and one item was discarded. The item discarded was the item- 10. The nine items used can be shown as follows. Table 9. Final questions of the questionnaires that were used for initial data collection (questions had validated) Items Questions Rating Score 1 2 3 4 5 1 Do you know about platforms for the following online learning? Unknown Less Enough Know Very familiar Quipper School Moodle Edmodo Schoology Kelase 2 How is the access speed of the following platforms? Slow Less Enough Fast Very fast Quipper School Moodle Edmodo Schoology Kelase 3 How complete are the learning features available on the following platforms? Incomplete Less Enough Complete Very Complete Quipper School Moodle Edmodo Schoology Kelase 4 Are the following platforms easy to operate? Difficult Less Enough Easy Very Easy Quipper School Moodle Edmodo Schoology Kelase 5 How is the maximum capacity of material content storage on the following platforms? Not optimal Less Enough Optimal Very optimal Quipper School Moodle Edmodo Schoology Kelase 70 http://www.i-jim.org Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... Items Questions Rating Score 1 2 3 4 5 6 Are the following platforms able to facilitate maximally the number of users who are members? Not optimal Less Enough Optimal Very optimal Quipper School Moodle Edmodo Schoology Kelase 7 Are input, edit, and delete data be done easily in the following platforms? Difficult Less Enough Easy Very Easy Quipper School Moodle Edmodo Schoology Kelase 8 How is the guarantee of data security in the following platforms? Un-guaranteed Less Enough Guaranteed Very guaranteed Quipper School Moodle Edmodo Schoology Kelase 9 How is the visual display of the following platforms? Not Interesting Less Enough Interesting Very Interesting Quipper School Moodle Edmodo Schoology Kelase 15 Appendix 7 15.1 Questions list of the questionnaires used to effectiveness test the DIVAYANA formula (questions have not been validated) 1. Are the criteria used to obtain the initial data correctly? 2. Is the number of respondents involved in assessing the platform appropriate? 3. Are the initial data collected has been presented properly? 4. Is the weighted assessment for each criterion is following the agreement of the experts? 5. Is the number of experts involved in the weighting assessment appropriate? 6. Is the field of knowledge of the experts involved does not match the object under study? iJIM ‒ Vol. 14, No. 20, 2020 71 Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... 7. Is the weighting formula for determining Wyack correct? 8. Is the formula of Vector-D determination for normalization correct? 9. Is the formula of Vector-R determination for ranking correct? 10. Are the calculation results of each equation correct? 11. Is the platform chosen through the DIVAYANA formula calculation process following the existing reality empirically in the field? 12. Is the formula easy to understand? 13. Can the formula be combined with other formulas? 14. Is the formula easy to apply? 16 Appendix 8 Table 10. Results of content validation test by experts to the questionnaire questions that were used to effectiveness test of the DIVAYANA formula Items Rating Score from Experts Expert-1 Expert-2 Irrelevant Relevant Irrelevant Relevant 1 2 3 4 1 2 3 4 1 - - Ö - - - Ö - 2 - - - Ö - - Ö - 3 - - Ö - - - Ö - 4 - - - Ö - - Ö - 5 - - Ö - - - - Ö 6 - Ö - - Ö - - - 7 - - - Ö - - Ö - 8 - - Ö - - - - Ö 9 - - Ö - - - - Ö 10 - - - Ö - - Ö - 11 - - Ö - - - - Ö 12 - - - Ö - - Ö - 13 - Ö - - Ö - - - 14 - - - Ö - - - Ö 17 Appendix 9 Table 11. Compilation of content validation test results by two experts to the questionnaire questions that were used to effectiveness test of the DIVAYANA formula Expert-1 Expert -2 Less Relevant (Score 1 - 2) Very Relevant (Score 3 - 4) Less Relevant (Score 1 - 2) Very Relevant (Score 3 - 4) 6,13 1,2,3,4,5,7,8,9,10,11,12,14 6,13 1,2,3,4,5,7,8,9,10,11,12,14 72 http://www.i-jim.org Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... 18 Appendix 10 Table 12. Cross tabulation of content validation test results by two experts to the questionnaire questions that were used for the effectiveness test of the DIVAYANA formula Expert -2 Less Relevant (Score 1 - 2) Very Relevant (Score 3 - 4) Expert -1 Less Relevant (Score 1 - 2) A 6,13 (2) B - (0) Very Relevant (Score 3 - 4) C - (0) D 1,2,3,4,5,7,8,9,10,11,12,14 (12) Based on the Gregory formula shown in equation (4), the cross-tabulation results can be calculated to determine the questionnaire contents validity that was used to the effectiveness test of the DIVAYANA formula. 𝐶𝑜𝑛𝑡𝑒𝑛𝑡 𝑉𝑎𝑙𝑖𝑑𝑖𝑡𝑦 + ,-.-/-+ = 2; ;-8-8-2; = 2; 2< = 0.857 The content validity value of 0.857 means that the questionnaires used for data collection were very valid. This is evidenced by Guilford’s categorization, where the category is very valid which in the range of 0.800 < rxy ≤ 1.000. 19 Appendix 11 Table 13. Reliability test results of questionnaire items that were used to effectiveness test of the DIVAYANA formula Items σi2 I-1 3.125 I-2 1.071 I-3 0.125 I-4 0.500 I-5 3.071 I-6 0.554 I-7 1.643 I-8 1.071 I-9 0.268 I-10 0.554 I-11 1.643 I-12 1.696 iJIM ‒ Vol. 14, No. 20, 2020 73 Paper—Utilization of DIVAYANA Formula in Evaluating of Suitable Platforms for Online Learning in... Items σi2 I-13 0.214 I-14 0.411 ∑ σi2 15.946 Based on the calculation of the reliability test using Microsoft Excel, the score was obtained ∑𝜎( ; = 15.946; n = 8; 𝜎>; = 54.109; so that it obtains the calculation results of the reliability coefficient using the following Cronbach Alpha formula that reference to equation (5). 𝛼 = 0 012 ∗ 51 − ∑𝛔% - 𝛔.- 8 𝛼 = 5 512 ∗ 51 − 2=.: