IJAHP Article: de Castro-Pardo, de la Fuente-Cabrero, Laguna-Sánchez, Pérez-Rodriguez/ Combining AHP and goal programming in the context of the assessment of e-learning International Journal of the Analytic Hierarchy Process 301 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.630 COMBINING AHP AND GOAL PROGRAMMING IN THE CONTEXT OF THE ASSESSMENT OF E-LEARNING Mónica de Castro-Pardo* 1 Universidad Rey Juan Carlos monica.decastro@urjc.es Concepción de la Fuente-Cabrero Universidad Rey Juan Carlos concepcion.delafuente@urjc.es Pilar Laguna-Sánchez Universidad Rey Juan Carlos pilar.laguna@urjc.es Fernando Pérez-Rodríguez fora forest technologies SLL fernando.perez@fora.es *Corresponding Author ABSTRACT The Analytical Hierarchy Process is a very common method used in Multi-Criteria Decision Making (MCDM) to analyze participative assessments. However, due to the qualitative nature of this methodology, a high percentage of inconsistencies need to be addressed when analyzing user preferences. This work analyzes the efficiency of the Goal Programming model in order to reduce inconsistencies with pairwise comparisons when working with inexpert participants and time limitations. A case study has been carried out that assesses online courses in higher education with the Analytical Hierarchy Process in order to understand the usefulness and feasibility of the method. Evaluation of four e-learning tools (collaboration tools, content tools, tutorial sessions and evaluation tools) used in an online business degree were collected from 72 students through a ‘Saaty-type’ survey, and the model was applied to improve the consistency of these results. This model has been able to minimize the inconsistencies of individual preferences while avoiding the loss of primary information. Keywords: Goal Programming; Analytical Hierarchy Process; inconsistencies; e- learning; participative decision making 1. Introduction Effective quality measures for e-learning have been described as being “urgently required” (Martínez-Caro et al., 2014). In this sense, it is important to remark on the This work was supported by the Government of Spain, Department of Economy, Industry and Competitiveness under the Torres Quevedo Contract PTQ-16-08633. mailto:monica.decastro@urjc.es mailto:concepcion.delafuente@urjc.es mailto:pilar.laguna@urjc.es mailto:fernando.perez@fora.es IJAHP Article: de Castro-Pardo, de la Fuente-Cabrero, Laguna-Sánchez, Pérez-Rodriguez/ Combining AHP and goal programming in the context of the assessment of e-learning International Journal of the Analytic Hierarchy Process 302 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.630 importance of assessment in e-learning environments (Strother, 2002; García-Peñalvo & Seoane-Pardo, 2015). The participative processes oriented to distance learning assessment have been broadly studied (Bozkurt et al., 2016). Nevertheless, participating stakeholders could show inconsistencies, which could be circular or undefined preferences (Brunelli, 2017). In such a situation, special attention must be given to the methodology of these processes in order to maintain objectivity and representation without losing usefulness or efficacy. For some participants, such as students, it is difficult to define individual preferences in the early stages of the decision-making process which makes it necessary to incorporate a high degree of iteration in certain phases of the evaluation process (Owen, 2015). This can be tedious and adds complexity to the process, consuming additional resources (Belton & Steward, 2002). Moreover, reviewing responses or asking for a repetition of responses from the same participant does not guarantee the reduction of inconsistencies. Furthermore, on many occasions it is not possible to do this because of time limitations. Some studies have utilized pairwise comparisons to assess e-learning systems (Jeong & Yeo, 2014; de Castro et al., 2017). In fact, one of most used methods when designing participative processes is a methodology based on paired comparisons, the Analytical Hierarchy Process (AHP) (Saaty, 1990). The AHP has been applied to educational environments and has been applied to participative processes with users in order to assess e-learning (Ho, 2008; Lin, Ho & Chang, 2014). Recently, studies have been published by Anggrainingsih et al. (2018) that use AHP to evaluate e- learning criteria such as “Quality of Design and Material”, and by Mohammed et al. (2018) that apply the same method with more technical criteria. Nevertheless, it is common to obtain a high number of inconsistent primary observations because of the subjective nature of the human mind. To solve this problem, inconsistent responses are generally removed, or the valuations are repeated until they generate results with an acceptable consistency level (Shee & Wang, 2008; Li & Ma, 2007; Lin et al., 2014). The first option, discarding inconsistent results, causes valuable information to be lost and may negatively impact the reliability of the result as it reduces the sample size. The second option, iterating the evaluation process, requires more resources and increases complexity. Some studies evidence positive results for iteration in the participative process as it reduces conflicts and increases consensus. On the other hand, this option is only practical for small groups that are easily managed, with plenty of time, and an intimate knowledge of the evaluation process. However, this iterative process is generally unpractical due to the limited availability of some students. In this study, we propose the use of weighted goal programming to correct inconsistencies in the primary results and thus avoid information loss without modifying the data collection process (Chen, Kou & Li, 2018). This model allows researchers to obtain consistent results that are as similar as possible to the original results. Therefore, the objective of this work is to evaluate the applicability of using a goal programming model to reduce inconsistencies in participative evaluation systems which collect the preferences of higher education students that study business administration through online courses. 2. Methodology “AHP is a multi-criteria decision making approach in which factors are arranged in a hierarchic structure” (Saaty, 1990). AHP can measure preferences through pairwise IJAHP Article: de Castro-Pardo, de la Fuente-Cabrero, Laguna-Sánchez, Pérez-Rodriguez/ Combining AHP and goal programming in the context of the assessment of e-learning International Journal of the Analytic Hierarchy Process 303 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.630 comparisons to derive priority scales. It is these scales that measure intangibles in relative terms (Saaty, 2001). For these reasons, it has a wide area of applicability and has been successfully used to solve a wide variety of public and private sector decision making problems that require group consensus (Belton & Steward, 2002). AHP measures individual preferences through judgment evaluations on the relative importance of different paired criteria that are being considered. The decision maker can express the intensity of their preference on a 9-point scale. If two criteria are equally important, they receive a score of 1. A score of 9 indicates that a criterion is extremely preferable over another (Saaty, 2001). Through it, pairwise comparison scores are used to build reciprocal matrices. From these, the relative weights of each attribute are measured. Based on these weights, the different alternatives are ranked. Pairwise comparison matrices must be reciprocal, homogeneous and consistent (Saaty, 2001). Let M=(𝑚𝑖𝑗 )𝑖𝑗 a pairwise comparison matrix, M verifies the reciprocity condition when 𝑚𝑖𝑗 𝑚𝑗𝑖 = 1 ∀𝑖 ≠ 𝑗, and verifies the consistency condition when 𝑚𝑖𝑗 𝑚𝑗𝑘 − 𝑚𝑖𝑘 = 0 ∀𝑖 ≠ 𝑗 𝑗 ≠ 𝑖 (González-Pachón & Romero, 2004). Notwithstanding, in the decision making processes the consistency condition is usually not accomplished. Participants in a decision-making process usually provide inconsistent results because the judgment calls have innate subjectivity. The level of consistency can be measured with the Consistency Index (CI), the cumulative average of matrix inconsistencies. The Consistency Ratio (CR) is the comparison between the CI and the Random Consistency Index (RI). An acceptable CR is equal to or less than 0.10 (Saaty, 2001). AHP inconsistency reduction has been studied in depth using different approaches (Kulakowski, 2018). Khatwani and Kumar (2017) used a stochastic method to define the Cosine Consistency Index. This method is based on a cosine maximization that uses an iterative basis to achieve the most consistent solution. In that case, AHP can be used iteratively until it achieves a consistent ratio. Some studies have been oriented to deal with inconsistencies to improve the group decision making processes. Fuhua et al. (2010) used two qualitative strategies allocating a weight vector based on the “expert’s experience value”. The main limitations of these methods are twofold: first, the loss of information is important and second, the participants may not feel the decision-making process and final result is their own. Srdjevic et al. (2013) proposed a model to assign the weights to the users in order to obtain consistent results. However, the problem related to the lost information remained. Moreover, the qualitative approach involves subjectivity and bias on the part of the user who is determining the weights. Ivanco et al. (2017) used sensitivity analysis to improve the consistency of the AHP matrices, taking into account the consensus of the group solution. This method presents more flexibility in order to obtain a consensual solution, however it proposes to address users’ disparities without quantifying them. Benitez et al. (2014) proposed a linear approach to obtain the closest consistent matrix through a suitable orthogonal projection expressed in terms of a Fourier-like expansion. This method achieves the proposed goal, however, modeling the problem is very complex. IJAHP Article: de Castro-Pardo, de la Fuente-Cabrero, Laguna-Sánchez, Pérez-Rodriguez/ Combining AHP and goal programming in the context of the assessment of e-learning International Journal of the Analytic Hierarchy Process 304 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.630 In this paper, we apply a simpler linear approach to optimize the consistency of the AHP matrices based on the Goal Programing method in order to improve the group decision making processes with inexpert participants. Goal Programming (GP) is a versatile multi-criteria technique used to resolve complex problems. In addition, it has been applied in other management science techniques (Tamiz et al., 1998). GP finds compromise solutions that may not fully satisfy all the goals but do reach certain satisfaction levels set by the decision-maker. For this, an objective function and some constraints are defined. The constraints of the model are formed by the relationship between the objectives of the achievement level for each attribute with these attributes linking themselves through negative and positive deviations. GP can be modelled with different approaches: MinMax GP, Lexicographic GP and Weighted GP. Weighted GP is a linear model that minimizes the weighted sum of the deviations from each goal and provides the most balanced solution. MinMax GP minimizes the maximum deviation between all possible deviations. Lexicographic GP seeks to minimize an achievement function based on a pre-emptive or non-Archimedean priorities approach (Romero, 2014). In this specific case, we applied an Archimedean GP model as laid out by Gónzalez-Pachón and Romero (2004). With a n=4 matrix the model is as follows: Min ∑ (𝑛𝑙 (1) + 𝑝𝑙 (1) )𝑝𝑙 + ∑ (𝑛𝑠 (2) + 𝑝𝑠 (2) )𝑝𝑠 + ∑ (𝑛𝑡 (2) + 𝑝𝑡 (2) )𝑝𝑡 (1) s.t. 𝑤𝑖𝑗 − 𝑚𝑖𝑗 + 𝑛𝑙 (1) − 𝑝𝑙 (1) = 0, l=1, 2,…, n(n-1), (2) 𝑤𝑖𝑗 𝑤𝑗𝑖 + 𝑛𝑠 (2) − 𝑝𝑠 (2) =1, 𝑠 = 1,2, … , 𝑛(𝑛−1) 2 , (3) 𝑤𝑖𝑗 𝑤𝑗𝑘 − 𝑤𝑖𝑘 + 𝑛𝑡 (3) − 𝑝𝑡 (3) =0, t=1, 2,…., n(n-1)(n-2), (4) 0.11≤ 𝑤𝑖𝑗 ≤ 9 ∀ 𝑖, 𝑗. (5) Where: 𝑛𝑙 (1) and 𝑝𝑙 (1) are the negative and positive deviations of the goal, respectively, for constraints that ensure the condition of similarity in the position l, 𝑛𝑠 (2) and 𝑝𝑠 (2) are the negative and positive deviations of the goal, respectively, for constraints that ensure the condition of reciprocity in the position s, and 𝑛𝑡 (3) and 𝑝𝑡 (3) are the negative and positive deviations of the goal, respectively, for constraints that ensure the condition of consistency in the position t. 𝑚𝑖𝑗 are the components of the matrix M for each pair of criteria. 𝑤𝑖𝑗 are the components of the matrix W, formed by the weights that represent the most similar weights to the components of the original matrix M for each pair of criteria ij. These are the results of the model. Let M=(𝑚𝑖𝑗 )𝑖𝑗 a general matrix given by a student, there exists a set of positive numbers, (𝑤𝑙 … 𝑤𝑛 ), such that 𝑚𝑖𝑗 = 𝑤𝑖 𝑤𝑗 for every 𝑖, 𝑗 = 1, … , 𝑛 This model uses a distance-based framework approach to inconsistencies in pairwise comparison matrices. The goal is to obtain a matrix that is as similar as possible to the IJAHP Article: de Castro-Pardo, de la Fuente-Cabrero, Laguna-Sánchez, Pérez-Rodriguez/ Combining AHP and goal programming in the context of the assessment of e-learning International Journal of the Analytic Hierarchy Process 305 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.630 one generated by the decision maker while meeting Saaty’s conditions of similarity, reciprocity, and consistency (González-Pachón & Romero, 2004). After correcting inconsistencies in individual pairwise comparison matrices, we aggregated their values by calculating their geometric mean. The resulting matrix represents the collective evaluation of all participants. From this aggregated matrix, we generated the matrix of weights indicating the priorities of each tool in the achievement of the competency under study using the eigenvector method. 3. Application The process was organized into two steps. First, we collected the evaluations obtained through a survey designed according to Saaty’s (2001) guidelines and corrected the inconsistent matrices. Second, conjoint results were assessed by the students in order to identify the consensus between individual preferences and aggregated results. The high levels of agreement suggest that this method was effective, which improves the entire process by making it more flexible, efficient, and practical. Students were asked to evaluate four e-learning tools (collaboration tools, content tools, tutorial sessions and evaluation tools) used in an online business degree course, based on how well they helped them acquire a specific competency (Table 1).The competency being evaluated was ‘Ability to work autonomously’, which is especially relevant in online courses. Table 1 Description of e-learning tools analyzed as criteria in an inquiry E-learning Tools Objective Collaborative Facilitate the interaction with the professor and among students through chat, messages and a forum. Contents Providing the courses theoretic and practical assignments. Evaluation Allow students to follow the continuous evaluation process through tasks and test type exams. Tutorial sessions To solve course doubts and questions at a one-to-one level. Online or in person. The GP model was applied in the inconsistent matrices collected, as shown in the example below. Example: Let P={𝑐𝑙, 𝑐𝑙, 𝑡, 𝑒} be a set of pairwise comparisons that represent the individual preferences of one student about the importance of each type of e-learning tool (collaboratives (cl), contents (ct), tutorials (t) and evaluations (e)), for the achievement of the competency “Ability to work autonomously” explained in the Table 1. All these preferences were collected using Saaty’s scale. Also, the matrix M formed by the cardinal pairwise comparisons over P is: M=( 1 1/7 1 7 1 7 1 1 1/7 1/7 1 1 7 7 1 1 ) IJAHP Article: de Castro-Pardo, de la Fuente-Cabrero, Laguna-Sánchez, Pérez-Rodriguez/ Combining AHP and goal programming in the context of the assessment of e-learning International Journal of the Analytic Hierarchy Process 306 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.630 So, the matrix M can be approximated by a reciprocal and consistent matrix, by using the following GP model: Min ∑ (𝑛𝑙 (1) + 𝑝𝑙 (1) ) + ∑ (𝑛𝑠 (2) + 𝑝𝑠 (2) ) + ∑ (𝑛𝑡 (3) + 𝑝𝑡 (3) ) ,24𝑡=1 6 𝑠=1 12 𝑙=1 s.t. 𝑤12 − 1 7 + 𝑛1 (1) − 𝑝1 (1) = 0, 𝑤13 − 1 + 𝑛2 (1) − 𝑝2 (1) = 0, 𝑤14 − 7 + 𝑛3 (1) − 𝑝3 (1) = 0, 𝑤21 − 7 + 𝑛4 (1) − 𝑝4 (1) = 0, 𝑤23 − 7 + 𝑛5 (1) − 𝑝5 (1) = 0, 𝑤24 − 7 + 𝑛6 (1) − 𝑝6 (1) = 0, 𝑤31 − 1 + 𝑛7 (1) − 𝑝7 (1) = 0, 𝑤32 − 1/7 + 𝑛8 (1) − 𝑝8 (1) = 0, 𝑤34 − 1 + 𝑛9 (1) − 𝑝9 (1) = 0, 𝑤41 − 1 + 𝑛10 (1) − 𝑝10 (1) = 0, 𝑤42 − 1 7 + 𝑛11 (1) − 𝑝11 (1) = 0, 𝑤43 − 1 + 𝑛12 (1) − 𝑝12 (1) = 0; 𝑤12𝑤21 + 𝑛1 (2) − 𝑝1 (2) =1, 𝑤13𝑤31 + 𝑛2 (2) − 𝑝2 (2) =1, 𝑤14𝑤41 + 𝑛3 (2) − 𝑝3 (2) =1, 𝑤23𝑤32 + 𝑛4 (2) − 𝑝4 (2) =1, 𝑤24𝑤42 + 𝑛5 (2) − 𝑝5 (2) =1, 𝑤34𝑤43 + 𝑛6 (2) − 𝑝6 (2) =1; 𝑤13𝑤32 − 𝑤12 + 𝑛1 (3) − 𝑝1 (3) =0, 𝑤14𝑤42 − 𝑤12 + 𝑛2 (3) − 𝑝2 (3) =0, 𝑤12𝑤23 − 𝑤13 + 𝑛3 (3) − 𝑝3 (3) =0, 𝑤14𝑤43 − 𝑤13 + 𝑛4 (3) − 𝑝4 (3) =0, 𝑤12𝑤24 − 𝑤14 + 𝑛5 (3) − 𝑝5 (3) =0, 𝑤13𝑤34 − 𝑤14 + 𝑛6 (3) − 𝑝6 (3) =0, 𝑤23𝑤31 − 𝑤21 + 𝑛7 (3) − 𝑝7 (3) =0, 𝑤24𝑤41 − 𝑤21 + 𝑛8 (3) − 𝑝8 (3) =0, 𝑤21𝑤13 − 𝑤23 + 𝑛9 (3) − 𝑝9 (3) =0, 𝑤24𝑤43 − 𝑤23 + 𝑛10 (3) − 𝑝10 (3) =0, 𝑤21𝑤14 − 𝑤24 + 𝑛11 (3) − 𝑝11 (3) =0, 𝑤23𝑤34 − 𝑤24 + 𝑛12 (3) − 𝑝12 (3) =0, 𝑤32𝑤21 − 𝑤31 + 𝑛13 (3) − 𝑝13 (3) =0, 𝑤34𝑤41 − 𝑤31 + 𝑛14 (3) − 𝑝14 (3) =0, 𝑤31𝑤12 − 𝑤32 + 𝑛15 (3) − 𝑝15 (3) =0, 𝑤34𝑤42 − 𝑤32 + 𝑛16 (3) − 𝑝16 (3) =0, 𝑤31𝑤14 − 𝑤34 + 𝑛17 (3) − 𝑝17 (3) =0, 𝑤32𝑤24 − 𝑤34 + 𝑛18 (3) − 𝑝18 (3) =0, 𝑤42𝑤21 − 𝑤41 + 𝑛19 (3) − 𝑝19 (3) =0, 𝑤43𝑤31 − 𝑤41 + 𝑛20 (3) − 𝑝20 (3) =0, 𝑤41𝑤12 − 𝑤42 + 𝑛21 (3) − 𝑝21 (3) =0, 𝑤43𝑤32 − 𝑤42 + 𝑛22 (3) − 𝑝22 (3) =0, 𝑤41𝑤13 − 𝑤43 + 𝑛23 (3) − 𝑝23 (3) =0, 𝑤42𝑤23 − 𝑤43 + 𝑛24 (3) − 𝑝24 (3) =0; 0.11≤ 𝑤𝑖𝑗 ≤ 9 ∀ 𝑖, 𝑗. As a result of the application of the model in the matrix M, we obtained a consistent matrix W: W=( 1 1/7 1 7 1 7 1 1 1/7 1/7 1 1 𝟏 7 1 1 ) , where the only corrected component of the original matrix was (𝑚14). This change only permitted one to obtain a matrix with a CR=0 when the index of the original matrix was CR=0.4192. This process was applied to improve the consistency of all the matrices that represented the students' preferences, that were obtained using the Saaty survey, with a CR>0.10. IJAHP Article: de Castro-Pardo, de la Fuente-Cabrero, Laguna-Sánchez, Pérez-Rodriguez/ Combining AHP and goal programming in the context of the assessment of e-learning International Journal of the Analytic Hierarchy Process 307 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.630 4. Results and discussion Individual opinions were collected from the 72 students through a ‘Saaty-type’ survey with a 1 to 9 scale. In it, they were asked to perform pairwise comparisons of collaboration, content, tutorial sessions, and evaluation tools as a means of acquiring the ‘Ability to work autonomously’ competency. From the resulting 72 pairwise comparison matrices, 7 were excluded for being incomplete or incorrectly completed. Of the remaining 65 valid matrices, 8 had a consistency ratio of less than 0.1. The inconsistent results were corrected by modeling a goal programming function using LINGO 17.0. We obtained improvements in the consistency of 57 matrices. As a result, 65 n=4 matrices were obtained with a consistency ratio under 0.1. Differences between the weights obtained with corrected inconsistencies and the weights obtained with the original results removing inconsistent answers were not relevant, but these differences changed the final priorities over each e-learning tool (Table 2). Table 2 Results and differences between results with corrected inconsistencies and results with original results removing inconsistent answers E-learning tools Consistent results without corrections Results with corrected inconsistencies (de Castro et al., 2017) Differences in percentages Collaborative 20.33% 18.91% 1.42% Contents 32.23% 32.53% -0.30% Evaluation 16.84% 21.89% -5.05% Tutorial sessions 30.59% 26.66% 3.93% Table 3 Results and differences between results with corrected inconsistencies and results with original results considering inconsistent answers E-learning tools Results with inconsistent and consistent matrices Results with corrected inconsistencies (de Castro et al., 2017) Differences in percentages Collaborative 19.05% 18.91% 0.14% Contents 34.67% 32.53% 2.14% Evaluation 25.39% 21.89% 3.5% Tutorial sessions 20.87% 26.66% -5.79% The results after correcting the inconsistencies give more weight to evaluation tools and less to collaborative tools. Collaborative tools are prioritized in the same order with just 1.42% less relative importance than the original results. Content tools are similarly ranked in the original and corrected versions (Table 2). IJAHP Article: de Castro-Pardo, de la Fuente-Cabrero, Laguna-Sánchez, Pérez-Rodriguez/ Combining AHP and goal programming in the context of the assessment of e-learning International Journal of the Analytic Hierarchy Process 308 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.630 Furthermore, when the original inconsistent and consistent results were compared with the results, after correcting the inconsistencies, differences were found. Tutorial sessions showed the highest divergences when the original inconsistencies remained (Table 3). This shows the effect of the inconsistent results over the group solution. Table 4 Ranking provided by the inconsistent and consistent original results, the original results removing inconsistent answers and the results with corrected inconsistencies E-learning tools Ranking inconsistent and consistent results Ranking only consistent results Ranking with corrected inconsistencies Collaborative 4 3 4 Contents 1 1 1 Evaluation 2 4 3 Tutorial sessions 3 2 2 These differing prioritizations between the results after discarding the inconsistent preferences and the results with corrected inconsistences, illustrate the effect of information loss on results (Table 4). A sensitivity analysis performed on the aggregated results shows the importance of evaluation tools in acquiring the competency under study, providing the most robust prioritization (Figure 1). Figure 1 Graphic of sensitivity analysis with variation of one positive point in each pair of aggregated comparison matrices 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 Sensitivity analisys Collaborative Contents Tutorial sessions Evaluation IJAHP Article: de Castro-Pardo, de la Fuente-Cabrero, Laguna-Sánchez, Pérez-Rodriguez/ Combining AHP and goal programming in the context of the assessment of e-learning International Journal of the Analytic Hierarchy Process 309 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.630 Furthermore, the differing prioritizations between the results considering inconsistencies, the results with corrected inconsistencies and the results after discarding inconsistent preferences show the effect of inconsistent responses on distorting global results (Table 4). The goal programming model proposed in this work consolidates the weights of content and evaluation tools, thus reducing the sensitivity of the overall results. The data suggests that information loss can distort evaluation results and diminish the quality of the process. After this phase, students were asked to complete an online, Likert-scale survey that focused on their level of agreement or disagreement with the results, where 1 represented the minimum agreement and 5 represented the maximum agreement. Here, 87.5% of respondents agreed highly or very highly with the priority ranking generated by the aggregated matrices corrected for inconsistencies. This high level of agreement demonstrated the effectiveness of the proposed model to treat inconsistencies in pairwise comparison matrices of e-learning tools for acquiring competencies. Further validation was provided by high levels of participant satisfaction with the aggregated results. This would seem to suggest that the changes carried out to diminish inconsistencies did not significantly alter the opinion of the group. Finally, the model allowed researchers to recover 90.47% of the missing information while maintaining the flexibility of the evaluation process; thus, making it more practical. In addition, the high level of agreement from the participants with the results validates the effectiveness of this method. 5. Conclusion Global results are different when consistency is improved using the proposed GP model. Thus, e-learning tools received different weights when inconsistencies were corrected. Both results, corrected and primary, agree with the assessment that content and tutorial sessions are the most important elements, even though tutorial sessions received a lower weight with the corrected matrices. Notably, the corrected model prioritizes evaluation tools over collaborative tools. GP is an effective technique when correcting inconsistencies in pairwise comparison matrices as applied to higher education evaluation systems. By correcting the primary results, both the quality and the agility of the evaluation process are improved. The GP model has improved the performance of the AHP method in order to reduce the inconsistencies of the pairwise comparison matrices, solving some of the limitations of previously proposed methods. First, the loss of information has been avoided. Second, the preferences of all the participants have been considered in the decision- making process. Third, the applicability of the process has remained, thus avoiding the use of iterations. Finally, rigor has been maintained throughout the process. Ultimately, the use of GP in the proposed model efficiently improved the Analytical Hierarchy Process in the context of working with inexpert users such as those who might evaluate an online business degree course. IJAHP Article: de Castro-Pardo, de la Fuente-Cabrero, Laguna-Sánchez, Pérez-Rodriguez/ Combining AHP and goal programming in the context of the assessment of e-learning International Journal of the Analytic Hierarchy Process 310 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.630 REFERENCES Anggrainingsih R, Umam MZ, Setiadi H. (2018). Determining e-learning success factor in higher education based on user perspective using Fuzzy AHP. MATEC Web of Conferences, 154. Doi: 10.1051/matecconf/201815403011 Belton, V., Stewart, T. (2002). Multiple criteria decision analysis: an integrated approach. Boston: Kluwer Springer Science and Business Media. Benítez, J., Izquierdo, J., Pérez-García, R., & Ramos-Martínez, E. (2014). A simple formula to find the closest consistent matrix to a reciprocal matrix. Applied Mathematical Modelling, 38(15-16), 3968-3974. Doi: https://doi.org/10.1016/j.apm.2014.01.007 Bozkurt, A., Akgun-Ozbek, E., Yilmazel, S. et al. 2015. Trends in distance education research: A content analysis of journals 2009-2013. The International Review of Research in Open and Distributed Learning, 16(1), 330-363. Doi: http://dx.doi.org/10.19173/irrodl.v16i1.1953 Brunelli, M. (2017). Studying a set of properties of inconsistency indices for pairwise comparisons. Annals of Operations Research, 248(1-2), 143-161. Doi: 10.1007/s10479-016-2166-8 Chen, K., Kou, G., Li, C. (2018). A linear programming model to reduce rank violations while eliciting preference from pairwise comparison matrix. Journal of the Operational Research Society, 69(1-12). Doi: 10.1080/01605682.2017.1409156 Cho, Y. G., & Cho, K. T. (2008). A loss function approach to group preference aggregation in the AHP. Computers & Operations Research, 35(3), 884-892. Doi: https://doi.org/10.1016/j.cor.2006.04.008 de Castro, M., de la Fuente-Cabrero, C., Laguna Sánchez, MDP (2017). Assessment of autonomous learning skill through multi-criteria analysis for online ADE students in Moodle. In: M. Peris-Ortiz, J. Gómez, J. Merigó-Lindahl, and C. Rueda-Armengot (Eds). Entrepreneurial universities: Exploring the academic and innovative dimensions of entrepreneurship in higher education. (197-2013). Washington, DC, USA: Springer. García-Peñalvo, FJ, Seoane-Pardo, AM (2015). Una revisión actualizada del concepto de eLearning. Décimo Aniversario Education in the Knowledge Society, 16(1), 119- 144. Doi: 10.14201/eks2015161119144 González-Pachón J, Romero C (2004) A method for dealing with inconsistencies in pairwise comparisons. European Journal of Operational Research, 158(2), 351-361. Doi: 10.1016/j.ejor.2003.06.009 Ho, W. (2008). Integrated analytic hierarchy process and its applications–A literature review. European Journal of Operational Research, 186(1), 211-228. Doi: 10.1016/j.ejor.2007.01.004 https://doi.org/10.1016/j.apm.2014.01.007 http://dx.doi.org/10.19173/irrodl.v16i1.1953 https://doi.org/10.1016/j.cor.2006.04.008 IJAHP Article: de Castro-Pardo, de la Fuente-Cabrero, Laguna-Sánchez, Pérez-Rodriguez/ Combining AHP and goal programming in the context of the assessment of e-learning International Journal of the Analytic Hierarchy Process 311 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.630 Ivanco, M., Hou, G., & Michaeli, J. (2017). Sensitivity analysis method to address user disparities in the analytic hierarchy process. Expert Systems with Applications, 90, 111-126. Doi: https://doi.org/10.1016/j.eswa.2017.08.003 Jeong, HY, Yeo, SS (2014). The quality model for e-learning system with multimedia contents: a pairwise comparison approach. Multimedia Tools and Applications, 73(2), 887-900. Doi: 10.1007/s11042-013-1445-5 Khatwani, G., & Kar, A. K. (2017). Improving the cosine consistency index for the analytic hierarchy process for solving multi-criteria decision making problems. Applied Computing and Informatics, 13(2), 118-129. Doi: https://doi.org/10.1016/j.aci.2016.05.001 Kułakowski, K. (2018). Inconsistency in the ordinal pairwise comparisons method with and without ties. European Journal of Operational Research, 270(1), 314-327. Doi: https://doi.org/10.1016/j.ejor.2018.03.024 Li, HL, Ma, LC (2007). Detecting and adjusting ordinal and cardinal inconsistences through a graphical and optimal approach in AHP models. Computers and Operations Research, 34(3), 780-798. Doi: 10.1016/j.cor.2005.05.010 Lin, TC, Ho, HP, Chang, CT (2014). Evaluation model for applying an e-learning system in a course: an analytic hierarchy process-Multi-Choice Goal programming approach. Journal of Educational Computing Research, 50(1), 135-157. Doi: https://doi.org/10.2190/EC.50.1.g Martínez-Caro, E., Cegarra-Navarro, JG, Cepeda-Carrion, G. (2015). An application of the performance-evaluation model for e-learning quality in higher education. Total Quality Management and Business Excellence, 26(5-6), 632-647. Doi: 10.1080/14783363.2013.867607 Mohammed, HJ, Kasim, MM, Shaharanee, IN (2018). Evaluation of e-learning approaches using AHP-TOPSIS technique. Journal of Telecommunication, Electronic and Computer Engineering (JTEC,) 10(1-10), 7-10. Owen, D. (2015). Collaborative decision making. Decision Analysis, 12(1), 29-45. Doi: 10.1287/deca.2014.0307 Romero, C. (2014). Handbook of critical issues in goal programming. Elsevier. Saaty, TL (1990). How to make a decision: the analytic hierarchy process. European Journal of Operational Research, 48(1), 9-26. Doi: 10.1016/0377-2217(90)90057-I Saaty, TL, Vargas, L. (2001). Models, methods, concepts and applications of the Analytic Hierarchy Process. London, United Kingdom: Kluwer Academic Publishers. Shee, DY, Wang, YS (2008). Multi-criteria evaluation of the web-based e-learning system: A methodology based on learner satisfaction and its applications. Computers and Education, 50(3), 894-905. Doi: 10.1016/j.compedu.2006.09.005 Srdjevic, B., Srdjevic, Z., Blagojevic, B., & Suvocarev, K. (2013). A two-phase algorithm for consensus building in AHP-group decision making. Applied https://doi.org/10.1016/j.eswa.2017.08.003 https://doi.org/10.1016/j.aci.2016.05.001 https://doi.org/10.1016/j.ejor.2018.03.024 https://doi.org/10.2190/EC.50.1.g https://doi.org/10.1287/deca.2014.0307 https://doi.org/10.1016/0377-2217(90)90057-I IJAHP Article: de Castro-Pardo, de la Fuente-Cabrero, Laguna-Sánchez, Pérez-Rodriguez/ Combining AHP and goal programming in the context of the assessment of e-learning International Journal of the Analytic Hierarchy Process 312 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.630 Mathematical Modelling, 37(10-11), 6670-6682. Doi: https://doi.org/10.1016/j.apm.2013.01.028 Strother, JB (2002). An assessment of the effectiveness of e-learning in corporate training programs. The International Review of Research in Open and Distributed Learning, 3(1). Tamiz, M., Jones, D., Romero, C. (1998). Goal programming for decision making: An overview of the current state-of-the-art. European Journal of Operational Research, 111(3), 569-581. Doi: 10.1016/S0377-2217(97)00317-2 https://doi.org/10.1016/j.apm.2013.01.028