Microsoft Word - 3NinaArticle_pp31_44_03212009 IJAHP ARTICLE: Begičević, Divjak & Hunjak: Comparing AHP and ANP in a Case Study of E-learning Implementation International Journal of the Vol. 1 Issue 1 2009 ISSN 1936-6744 Analytic Hierarchy Process 31 COMPARISON BETWEEN AHP AND ANP: CASE STUDY OF STRATEGIC PLANNING OF E-LEARNING IMPLEMENTATION Nina Begičević E-mail: nina.begicevic@foi.hr Blaženka Divjak E-mail: blazenka.divjak@foi.hr Tihomir Hunjak E-mail : tihomir.hunjak@foi.hr University of Zagreb Faculty of Organization and Informatics Pavlinska 2, 42 000 Varaždin ABSTRACT The specific objectives of the paper are: validation of the theoretical model for strategic planning of e-learning implementation by means of factor analysis, presentation of the structure of AHP and ANP models for decision making about e-learning implementation, comparison between developed AHP and ANP models, analysis of the results of group decision making supported by the software TeamEC2000 based on AHP model and the results of decision making supported by the software Super Decisions 1.6.0., which is based on an ANP model. The models can be applied at the course level, for the group of courses (department level), or at the study program (faculty level). At the same time, the proposed models are useful for structuring discussion on strategic decisions on e-learning implementation at the university level. Our aim is to analyze and identify advantages and disadvantages of using different models and tools in the process of decision making about e-learning at different organizational levels. Keywords: e-learning, AHP, ANP, factor analysis Rob Typewritten Text http://dx.doi.org/10.13033/ijahp.v1i1.5 IJAHP ARTICLE: Begičević, Divjak & Hunjak: Comparing AHP and ANP in a Case Study of E-learning Implementation International Journal of the Vol. 1 Issue 1 2009 ISSN 1936-6744 Analytic Hierarchy Process 32 1. Introduction E-learning is usually defined as a type of learning supported by information and communication technology (ICT) that improves quality of teaching and learning. Implementing e-learning contributes to the advancement of higher education (HE). An E- learning system is a powerful tool for achieving strategic objectives of the university (teaching, research and serving the society) and it contributes to progress on the institutional level as well as at the personal level for teaching staff and students (Divjak and Begicevic, 2006). It supports collecting, analyzing and applying information appropriately, and comprises different teaching methods, which include information management, creative thinking, critical thinking, problem solving and collaborative learning (Bates, 2005). Generally speaking, universities in Croatia and some other European universities are currently at the stage of strategic planning and deciding about the systematic implementation of e-learning in the existing academic activities. Strategic planning and decision making about e-learning implementation is one of the aims of the Tempus EQIBELT Project (EQIBELT, 2006), coordinated by the University of Zagreb, which provides a useful platform for our research. In our paper, we will present the possibility of using AHP and ANP models and statistical techniques in strategic planning and decision-making about e-learning. 2. Objectives and research methodology The overall objectives of the research are: • to provide a basis for decision making for members of the EQIBELT project team and university strategy teams in the process of creating e-learning vision and strategic documents • to develop the general model for decision making about e-learning implementation in HE based on theoretical findings and surveys results • to complete the factor analysis, validate the theoretical model and reduce a large number of variables to a smaller number of factors, i.e. design an improved theoretical model for modeling purposes • to develop AHP and ANP models for decision making about e-learning implementation in HE • to compare decision models for e-learning implementation in HE based on some other research methods or built-in questionnaires including experts from other countries The specific objectives of this paper are: • presentation and analysis of the results of questionnaire performed on expert group • presentation of the theoretical model for decision making about e-learning implementation in HE, by means of factor analysis • developed structure of AHP model for strategic planning of e-learning implementation on course and department level IJAHP ARTICLE: Begičević, Divjak & Hunjak: Comparing AHP and ANP in a Case Study of E-learning Implementation International Journal of the Vol. 1 Issue 1 2009 ISSN 1936-6744 Analytic Hierarchy Process 33 • presentation of the results of group decision making on e-learning implementation supported by the software TeamEC2000 • developed structure of ANP model for strategic planning of e-learning implementation on institutional level • presentation and analysis of the results of expert decision making on e-learning implementation supported by the software Super Decisions We have treated decision-making as consisting of four phases: (1) intelligence, (2) design, (3) choice, and (4) implementation. More details can be found in (Begicevic, Divjak, Hunjak, 2006). The alternatives in the decision making process on e-learning implementation is: • Face-to-face learning, • ICT supported face-to-face learning, • Blended learning, and • Fully online learning. In the statistical evaluation of the results we have used factor analysis to validate the theoretical model for decision making about e-learning implementation. We have connected the results of the survey using factor analysis and these results have served as input in the multicriteria decision model (AHP) that we have developed and described in (Divjak and Begicevic, 2006). In the decision making phase we have solved the problem of choosing the best option for e-learning implementation. This was done with the assistance of an AHP model developed and validated in the process of group decision-making supported by the software TeamEC2000. We have also developed an ANP model, which was used for defining the structure of the strategies for e-learning at the institutional level. The action plan and the monitoring system have followed the decision making phase. 3. Questionnaire description and response After we had developed the theoretical model for decision making about e-learning implementation, we created a questionnaire about the importance of the advantages and goals of e-learning implementation and about criteria and subcriteria essential for decision making about e-learning implementation. The alternatives were not included in the questionnaire, but an explanation of each criterion/subcriterion was attached to the questionnaire. We carried out the survey collecting a total of 90 questionnaires. The participants were vice-rectors, vice-deans, members of relevant university bodies, members of government bodies responsible for the implementation of e-learning methodology and technology, members of the EQIBELT project team and university strategy teams, university teachers and student representatives involved or interested in e-learning, etc. The criteria for the selection of experts were an expertise in e-learning, and familiarity with the HE environment. Hence, the survey represented a group of e-learning experts in Croatia. IJAHP ARTICLE: Begičević, Divjak & Hunjak: Comparing AHP and ANP in a Case Study of E-learning Implementation International Journal of the Vol. 1 Issue 1 2009 ISSN 1936-6744 Analytic Hierarchy Process 34 4. The results of the survey: ranking of criteria and subcriteria In this section, we present some of the results of the survey from 90 experts on e-learning in HE in Croatia. The complete results of the survey can be found in the paper “Development of AHP based model for decision making on e-learning implementation” (Begicevic, Divjak, Hunjak, 2006). In all questions, the discrete scale used for importance was from 1 to 5. All the proposed criteria were accepted as important, but four of them ranked above the average mark of four: Organizational readiness of environment, Development of human resources, Availability of human resources and Availability of basic ICT infrastructure. Legal and formal readiness of environment and Availability of specific ICT infrastructure ranked below average (Figure 1). This last ranking reflects the stage of development of e- learning in Croatia, which is generally below the EU level, and therefore the importance of legal framework and appropriate ICT infrastructure is not recognized. Details about ranking the proposed subcriteria are shown in Table 1. The importance of criteria 4,38 4,28 4,27 4,24 3,94 3,74 3,4 3,6 3,8 4 4,2 4,4 4,6 Criteria R at in gs Organizational readiness of environment Development of human resources Availability of human resources Availability of basic ICT infrastructure Legal and formal readiness of environment Availability of specific ICT infrastructure Figure 1 Results of the survey - importance of criteria. Table 1 Results of the survey - importance of subcriteria. ORGANIZATIONAL READINESS OF ENVIRONMENT Faculty strategy for development 4.54 Organizational readiness of universities/faculties for e- learning implementation 4.42 University framework for development 4.34 Financial readiness of universities/faculties for e-learning 4.21 IJAHP ARTICLE: Begičević, Divjak & Hunjak: Comparing AHP and ANP in a Case Study of E-learning Implementation International Journal of the Vol. 1 Issue 1 2009 ISSN 1936-6744 Analytic Hierarchy Process 35 implementation AVAILABILITY OF BASIC ICT INFRASTRUCTURE Network infrastructure 4.50 Teachers and students equipped with computers 4.43 Classrooms equipped for e-learning 4.17 Integral information system of universities/faculties 3.86 DEVELOPMENT OF HUMAN RESOURCES Continuous training of academic staff 4.63 Continuous training of support staff 4.17 Training of students for use of e-learning 4.04 LEGAL AND FORMAL READINESS OF ENVIRONMENT Evaluation and quality control at universities/faculties 4.20 System and criteria for academic staff promotion 4.04 Standardization of digital educational materials 4.03 Protecting intellectual property rights on state and academic level 3.49 AVAILABILITY OF HUMAN RESOURCES Specialized e-learning centers at universities 4.56 Availability of technical support staff for e-learning 4.36 Availability of support staff for graphic design. animation and video 4.09 Availability of support staff for methodology of e-learning 4.08 AVAILABILITY OF SPECIFIC ICT INFRASTRUCTURE Virtual learning environment 4.31 Managed learning environment 4.06 Library management system 3.97 Production of video and audio materials 3.61 Network videoconferencing system 3.60 Exam management system 3.57 Video and audio streaming 3.49 Systems for simulation and virtual environment 3.32 5. Results of the factor analysis Factor analysis is a statistical data reduction technique that can simultaneously manage over a hundred variables, compensate for random error and invalidity, and disentangle complex interrelationships into their major and distinct regularities (Rummel, 1967). It is used to explain variability among observed random variables in terms of fewer unobserved random variables called factors. We have used factor analysis to validate the theoretical model (Table 2), to reduce a large number of variables to a smaller number of factors for modelling purposes (AHP modelling), to specify the strength of the relationship between each factor and each variable and to determine which sets of items should be grouped together in the theoretical model. The complete results of the factor analysis are presented in the paper (Begicevic and Divjak, 2006). IJAHP ARTICLE: Begičević, Divjak & Hunjak: Comparing AHP and ANP in a Case Study of E-learning Implementation International Journal of the Vol. 1 Issue 1 2009 ISSN 1936-6744 Analytic Hierarchy Process 36 The extraction method, which was used in the factor analysis, was Principal Component Analysis (Brace et al., 2000) and the rotation method was the orthogonal Varimax rotation (Brace et al., 2000) with Kaiser normalization. The number of factors was specified, m = 5 (5 factors were recognized in the theoretical model). The factor analysis was performed with the support of the statistical program SPSS (Brace et al., 2000). We set the lower boundary for the projection of variable variance on the factor at 0.519 and noticed that 6 variables did not correlate above 0.519 with the principal components of the original correlation matrix and therefore we excluded them from the model. Moreover, 5 out of the above mentioned 6 variables related almost equally to two or three factors. Finally, the new theoretical model was reduced to 21 variables (Begicevic and Divjak, 2006). Experts did not agree upon the importance of Protecting intellectual property rights and Standardization of digital educational materials, and in our opinion, it shows that in general the present state of e-learning implementation in HE in Croatia is at a rather early stage. Furthermore, the variables Training of students for use of e-learning, Integral information system of universities/faculties, Virtual learning environment and Organizational readiness of universities/faculties for e-learning implementation were excluded because of redundancy with other variables in the theoretical model. The factor analysis results have also confirmed 5 factors of the theoretical model for decision making about e-learning implementation (Table 2). Not only does the factor analysis confirm the major findings of prior data acquisition and analysis, but it also refines and better restructures our first theoretical model. We assume that there are two reasons for correspondence between the two models: first, the fact that the qualitative analysis in the first part of research was thorough as a considerable sample of strategic documents on e-learning was used, and second, experts were involved in the survey. The latter were essential because of the highly specific area, which requires both familiarity with e-learning and expertise in the HE environment. Table 2 Results of factor analysis (rotated component matrix). F 1 F 2 F 3 F 4 F 5 F 1 - HUMAN RESOURCES Availability of support staff for methodology of e-learning .883 3.415E-02 5.202E-02 -1.120E-02 -4.832E-02 Availability of technical support staff for e-learning .835 6.881E-02 .119 2.543E-02 .103 Availability of support staff for graphic design. animation and video .761 .118 9.200E-02 .105 1.353E-02 Continuous training of support staff .709 .146 .164 .196 .106 Specialized e-learning centers .652 -1.242E-03 .176 .206 4.064E-02 Continuous training of academic staff .610 .175 .139 .238 .156 IJAHP ARTICLE: Begičević, Divjak & Hunjak: Comparing AHP and ANP in a Case Study of E-learning Implementation International Journal of the Vol. 1 Issue 1 2009 ISSN 1936-6744 Analytic Hierarchy Process 37 F 2 - SPECIFIC ICT INFRASTRUCTURE FOR E-LEARNING Video and audio streaming -.196 .840 9.800E-02 -1.927E-03 .108 Network videoconferencing system -5.610E-02 .806 .176 .204 .154 Systems for simulation and virtual environment .265 .784 -9.944E-02 9.253E-02 .153 Production of video and audio materials .214 .769 9.195E-02 -9.597E-03 -4.100E-02 Exam management system .160 .609 .254 .136 -.101 Library management system .242 .603 .179 9.750E-02 -.276 F 3 - BASIC ICT INFRASTRUCTURE FOR E-LEARNING Network infrastructure .163 .193 .778 .107 3.312E-02 Teachers and students equipped with computers .266 .105 .720 -6.693E-02 -1.287E-02 Classrooms equipped for e-learning -3.167E-02 .183 .625 2.887E-02 .564 Managed learning environment .268 .233 .528 .417 -.240 F 4 - STRATEGIC READINESS FOR E-LEARNING IMPLEMENTATION Faculty strategy for development .191 3.302E-02 5.800E-02 .792 .154 University framework for development 9.796E-02 .282 -.100 .662 -3.168E-02 Financial readiness of universities/faculties for e- learning implementation .194 -3.291E-02 .397 .558 7.218E-02 F 5 - LEGAL AND FORMAL READINESS FOR E-LEARNING IMPLEMENTATION System and criteria for academic staff promotion .123 -9.182E-02 -4.377E-03 2.484E-02 .807 Evaluation and quality control at universities/faculties .340 .251 6.778E-03 .289 .512 6. An AHP based model for decision making at the course level In the Choice phase, we developed an AHP based model for decision making on e- learning implementation based on the reduced and restructured theoretical model (21 variables). We have built the AHP model using the TeamEC2000 software (EC 2000) which is specially designed for making group decisions. We take into account that a group can generate a higher number of ideas and usually know more than an individual does. It is also important that a group is more likely to make the riskier decisions, since risk is shared among all group members. In our case of “Decision making on the most suitable option for implementing e- learning” for the Mathematics course on the Faculty of Organization and Informatics, we were using TeamEC2000 with wireless electronic keypads for 5 decision makers (participants) and top down structuring with numerical judgments mode. The model and the methodology can also be applied for a group of courses (department level). IJAHP ARTICLE: Begičević, Divjak & Hunjak: Comparing AHP and ANP in a Case Study of E-learning Implementation International Journal of the Vol. 1 Issue 1 2009 ISSN 1936-6744 Analytic Hierarchy Process 38 All participants in the group decision-making process have specific knowledge, which makes them competent to assess and judge the most suitable options for e-learning implementation in the course Mathematics on Faculty of Organization and Informatics. The participants were given detailed instructions on definitions of criteria and subcriteria and the tool that would be used, a week before the decision-making event, in order to familiarize themselves with the task. The competencies of the group members are as follows: One participant is an Associate Professor and main lecturer of Mathematics, she has a Ph.D. in Mathematics, and she is familiar with the strategic planning of e-learning at the Faculty and University level. The second participant is an assistant in Mathematics and has an MA in Mathematics. Two other participants have MAs in Information Science and they are also PhD students. One of them is an assistant at Informatics and one is an administrator of the Learning Management System (LMS) at the Faculty of Organization and Informatics. The fifth participant is a student at Faculty and a student tutor for Mathematics. During their studies and training, they were included several times in lectures where e-learning was used to support traditional classroom teaching. Four of them are involved in creating courses that integrate e–learning and traditional classroom teaching. All participants are working on e-learning projects. Three out of five are experts in programming and have experience in developing the necessary infrastructure to implement e-learning courses. All participants are authors or co-authors of several scientific and professional papers in the area of e-learning. These experts form a heterogeneous group of decision makers. The group possesses the knowledge and responsibility to initiate and implement decisions about the most suitable option for e-learning implementation at the course level. The results of the group decision making incorporated the knowledge of all stakeholders provided in the process of group decision making, and the process concluded with a recommendation for applying the most suitable option for implementing e-learning. The results of every participant’s decision-making model and the results of the group decision-making model were available after the meeting. The results of the TeamEC2000 group decision-making session were a Hierarchic model, the objectives’ relative significance, and the priorities of the alternatives, obtained by synthesizing the judgments of the active participants shown in Figure 2. The Legal and formal readiness for e-learning implementation criterion has the highest relative significance – 0,351, which makes it the most important for reaching the goal. The reasons for the enhanced significance of this criterion are efforts in the HE system in Croatia to establish an academic staff promotion system for implementing e-learning, and for setting and implementing evaluation and quality control at universities and faculties in Croatia. The Strategic readiness for e-learning implementation criterion was also recognized as very important and was relatively significant – 0,253. The lowest relative significance – 0,092 was for the criterion Specific ICT infrastructure for e-learning. The alternative Blended learning has the highest priority of 0,429, which means the recommendation is to apply a blended-learning (hybrid) model, i.e. to integrate e-learning IJAHP ARTICLE: Begičević, Divjak & Hunjak: Comparing AHP and ANP in a Case Study of E-learning Implementation International Journal of the Vol. 1 Issue 1 2009 ISSN 1936-6744 Analytic Hierarchy Process 39 and traditional classroom lectures as the best way to implement e-learning in Mathematics in the Faculty of Organization and Informatics. It is interesting that alternative Fully online learning has a higher priority (0,140) than the alternative Face to face learning (0,108). Figure 2 Results of the group decision-making exercise. 7. An ANP based model for decision making at the institutional level We developed an ANP based model which incorporated feedback for strategic planning and decision making on e-learning implementation at the institutional level. The model is based on the reduced and restructured theoretical model by means of factor analysis, and the connections, interdependences and outer dependences have been reviewed by experts in this domain. IJAHP ARTICLE: Begičević, Divjak & Hunjak: Comparing AHP and ANP in a Case Study of E-learning Implementation International Journal of the Vol. 1 Issue 1 2009 ISSN 1936-6744 Analytic Hierarchy Process 40 We built the ANP model in Super Decisions software and the domain expert evaluated the model. In Figure 3 are shown the clusters and nodes of a model for decision making on e-learning implementation at the institutional level. The results of the validation are presented in Figure 4. We developed an ANP model to use for structuring discussions on strategic decisions on e-learning implementation and the decision making process, and for designing the strategy to implement e-learning at the faculty and/or university level. In our study of “Strategic planning and decision making on e-learning implementation on an institutional level,” the domain expert evaluated the model using the SuperDecisions ANP software in the numerical judgment mode. The domain expert is a member of the committee for developing e-learning strategy at the University of Zagreb and a member of the committee for e-learning strategy of the Faculty of Organization and Informatics. The results of the decision making process, based on the developed ANP model, were used for defining a structure of these strategies for e-learning at the institutional level. There has been a crucial difference in the way we used the AHP and ANP models. The AHP model was used for carrying out the recommendation for applying the most suitable option for implementing e-learning at the course or department level, but the ANP model has been primarily exploited for defining the structure of strategies for e-learning on the institutional level. These strategies, based upon the ANP model we developed, will be analysed in the implementation phase. Figure 4 presents the clusters and elements with their priorities, which have been crucial for structuring e-learning strategies, and it can be said that these priorities have been a guideline to create an acceptable e-learning strategy. The evaluation and quality control at universities and faculties are recognized as being very influential elements in planning e-learning implementation. Furthermore, the variables Exam management system, Library management system and Systems for simulation and virtual environment were identified as the most important in the framework of a Specific ICT infrastructure for e- learning, and Managed learning environment and Teachers and students equipped with computers within the framework of a Basic ICT infrastructure for e-learning. The crucial points which must be taken into account in the process of strategic planning of e-learning implementation are the establishment of the Specialized e-learning centres and Continuous training of academic staff. The alternative Blended learning has the highest priority and the alternative ICT supported face-to-face learning follows. Finally, there is the alternative Fully online learning, and this ranking can be understood if we take into account the fact that the University of Zagreb is an old and traditional university which appreciates the face-to-face approach and pedagogies, and considers e-learning as merely one way to improve the quality of teaching and learning. IJAHP ARTICLE: Begičević, Divjak & Hunjak: Comparing AHP and ANP in a Case Study of E-learning Implementation International Journal of the Vol. 1 Issue 1 2009 ISSN 1936-6744 Analytic Hierarchy Process 41 Figure 3 Overview of ANP network “Strategic planning and decision making on e- learning implementation on an institutional level.” IJAHP ARTICLE: Begičević, Divjak & Hunjak: Comparing AHP and ANP in a Case Study of E-learning Implementation International Journal of the Vol. 1 Issue 1 2009 ISSN 1936-6744 Analytic Hierarchy Process 42 Alternatives Total Normal Ideal Ranking 1 ICT supported face to face learning 0.0314 0.2554 0.4446 2 2 Blended learning 0.0706 0.5745 1.0000 1 3 Fully online learning 0.0209 0.1701 0.2960 3 Figure 4 Priorities of the elements and the alternatives. IJAHP ARTICLE: Begičević, Divjak & Hunjak: Comparing AHP and ANP in a Case Study of E-learning Implementation International Journal of the Vol. 1 Issue 1 2009 ISSN 1936-6744 Analytic Hierarchy Process 43 8. Conclusion The results of the survey performed by the group of experts on e-learning in HE were used as input for two mathematical models. This modeling contributed significantly to institutional planning, management and quality development for online distance education and e-learning. The problem of prioritizing e-learning options was solved with the help of multi-criteria decision making modelling. The AHP model was developed and validated during a group decision-making process. The ANP model was developed by the team and then restructured by the domain expert before giving judgments. The results of the decision-making process, based on the developed ANP model, have been used for defining and prioritizing the factors and strategies for e-learning on the institutional level. Our experience shows that such models for decision making strongly motivate all participants in the process, speed up decision-making, make the process more effective, and indicate a need for systematic e-learning usage in our educational institutions. More details about the AHP model can be found on http://www.projekti.hr/, and the model can be used and tested in new situations just by acknowledging the authors. REFERENCES Bates, T. (2005). Strategy and Visions of e-learning in Higher Education. Zagreb, Croatia, URL: http://eqibelt.srce.hr/lectures/bates.html, 10(3).2006. Begicevic, N., Divjak, B. (2006). Validation of theoretical model for decision making about e-learning implementation. Journal of Information and Organizational Sciences, Faculty of Organization and Informatics. University of Zagreb, 30 (2), 171-185. Begicevic, N., Divjak, B., Hunjak, T. (2006). Development of AHP based model for decision making on e-learning implementation. 17th International Conference on Information and Intelligent Systems IIS 2006, Varaždin, Faculty of Organization and Informatics, University of Zagreb, 375-383. Brace, N., Kemp, R., Snelgar, R. (2000). SPSS for Psychologists: A guide to data analysis using SPSS for Windows. New York, N.Y: Palgrave. Divjak, B., Begicevic, N. (2006). Imaginative acquisition of knowledge —strategic planning of e-learning. ITI 2006. Proceedings of the 28th International Conference on Information Technology Interfaces. Cavtat, Croatia, 47-53. IJAHP ARTICLE: Begičević, Divjak & Hunjak: Comparing AHP and ANP in a Case Study of E-learning Implementation International Journal of the Vol. 1 Issue 1 2009 ISSN 1936-6744 Analytic Hierarchy Process 44 EQIBELT, Education Quality Improvement by E-Learning Technology,. 16(01)2006, URL:. http://eqibelt.srce.hr/ Rummel, R. J. (1967). Understanding factor analysis. The Journal of Conflict Resolution, 20(04)2006 URL: www.hawaii.edu/powerkills/UFA.HTM