IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 117 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 EXTENDED CONSISTENCY ANALYSIS FOR PAIRWISE COMPARISON METHOD Sahika Koyun YΔ±lmaz Yildiz Technical University Turkey skoyun@yildiz.edu.tr Vidan Ozkir Yildiz Technical University Turkey cvildan@yildiz.edu.tr ABSTRACT Pairwise comparison (PC) is a widely used scientific technique to compare criteria or alternatives in pairs in order to express the decision maker’s judgments without the need for a unique common measurement unit between criteria. The method constructs a PC matrix by requesting the assessments of the decision maker(s) in the judgment acquisition phase and calculates an inconsistency measure to determine whether the judgments are adequately consistent with each other before subsequent phases. Although the method requires the decision maker to make all judgments in a PC matrix, it does not force him/her to make a judgment for each element of the matrix. If any judgment in a PC matrix is absent, for this reason, the judgment acquisition phase yields an incomplete PC matrix rather than a complete one. Missing judgments are calculated by multiplication of the judgments made by the decision maker. If the judgements of the decision maker are transitive and well-proportioned, missing judgments will not disturb the consistency of the resulting PC matrix. In other words, consistency of a PC matrix relies on the judgments made by the decision maker. Since the current consistency analysis procedure is designed for complete PC matrices, the suitability for evaluating the inconsistency of incomplete PC matrices is questionable. Probability density functions of random PC matrices with altering numbers of missing judgments show distinct features, indicating an incomplete PC matrix and a complete PC matrix do not come from the same probability function, and their mean consistency index (RI) is different. Consequently, we propose an extended consistency analysis procedure to evaluate the consistency of incomplete PC matrices. Keywords: consistency analysis; decision support systems; pairwise comparison; random index 1. Introduction Multiple attribute decision-making problems are frequently encountered in real life decision making, and they search for the best alternative among a set of feasible alternatives regarding a set of predefined criteria. These problems require examining a number of feasible alternatives with respect to a number of criteria in order to determine IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 118 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 the best decision. As the number of criteria increases, the problem becomes more complex. The construction of the attribute scales and their associated weights challenge decision makers because of the absence of a common unit of measurement. Multiple criteria decision making often requires tradeoffs between money, environmental quality, health and similar entities (Thurstone, 1994). In order to get relative weights of importance, the PC method processes both objective assessments and subjective judgments of decision makers together without the need of a common unit of measurement. The consistency analysis procedure is used to determine whether the judgments in a PC matrix are appropriate for use or not. If the inconsistency of judgments is unacceptable, the decision maker is asked to revise his/her judgments. The PC method cannot automatically eliminate inconsistency caused by the initial judgments. Therefore, the viability of the final decision substantially depends on the judgments of the decision maker. Current consistency analysis procedure consists of two steps as follows: the first step evaluates if the given judgments are consistent within, and the second step evaluates if the matrix is consistent in comparison to randomly generated matrices. If a PC matrix satisfies both steps, then it is accepted as consistent. Moreover, the procedure does not address the case of missing information. It assumes that the decision maker can provide all initial judgments in the PC matrix. Saaty (1980) explained a computational procedure which preserves the mathematical consistency of a PC matrix; acquired judgments can be utilized in order to determine any of these missing judgments. Regarding the judgment acquisition process, PCs can be classified into two groups: Direct Pairwise Comparisons (direct PCs) and Indirect Pairwise Comparisons (indirect PCs). Direct PCs are directly given by decision maker(s), while indirect PCs are derived through direct PCs. Indirect PCs enable the decision maker to not evaluate some of the pairwise comparisons (Harker, 1987a, 1987b). Considering the successive judgment capability of a human being, it is difficult to obtain consistent PC matrices when a large set of comparisons is present. The use of indirect PCs assists decision makers by allowing less comparison among pairs of PC matrices with higher dimensions. However, indirect PCs may cause false negatives and false positives during consistency analysis. Since indirect PCs are calculated based on at least two distinct direct PCs, the mathematical consistency of PC matrix increases. The PC method is generally used with the Analytical Hierarchy Process technique which is a systematized method for handling multiple criteria in a hierarchy structure. The AHP takes advantage of using the PC method while transforming subjective judgments into analytical information. The PC method preserves the inherent characteristics of judgments; the conclusion of Analytical Hierarchy Process technique is highly dependent on initial judgments and their consistency levels. Therefore, completeness and consistency of initial judgments have a fundamental role in the formation of the final decision. This study proposes a two-dimensional consistency analysis for evaluating the consistency of an incomplete PC matrix. IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 119 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 2. Pairwise Comparison Method Thurstone (1994) formulated the law of comparative judgment, defined a psychological scale and introduced the PC method in 1927. The method has been adapted for assisting several decision-making methods to assess the relative importance of criteria and alternatives (Siraj, Mikhailov, & Keane, 2015). Despite the fact that studies on PCs are increasing rapidly, the PC literature is still intact. Dede, Kamalakis et al (2016) present a methodology to extend the pairwise comparison framework in order to provide some information on the credibility of its outcome. Elliot (2010) investigates how the final decision is effected when different numerical scales are utilized in PC. The use of different numerical scales in the PC method yields significant effects on the attribute weights that are calculated from the judgments supplied and potentially on the preferred decisions that these judgments imply (Elliott, 2010). Let us consider n criteria Ci, (i = 1,2,3,…,n). A PC matrix is A = [aij]nΓ—n where aij reflects the relative importance of criterion i over criterion j. A is an n th order square matrix including aii = 1 for all self-comparisons and aji = 1/ aij for all reciprocal PCs. The acquired set of PCs constructs the PC matrix. Before the extraction of the weights from a PC matrix, it should be confirmed by consistency analysis. Further explanation of the consistency analysis is given in Section 3.1. πΆπ‘œπ‘šπ‘π‘™π‘’π‘‘π‘’ 𝑃𝐢𝑀 = 𝑏1 𝑏2 𝑏3 𝑏4 𝑏1 𝑏2 𝑏3 𝑏4 [ 1 1 2⁄ 1 4⁄ 2 1 1 2⁄ 4 8 2 4 1 2 1 8⁄ 1 4⁄ 1 2⁄ 1 ] (1) Figure 1 Graph of complete pairwise comparison matrix (Eq. 1) A PC matrix can be depicted as a graph (Figure 1). In such a graph, vertices represent the criteria and edges represent the comparative judgments between criteria. Any pairwise comparison can be computed straightforwardly if the graph of the incomplete PC matrix satisfies the minimum requirement of being a spanning tree. Figure 1 illustrates the graph representation of the PC matrix given in Equation 1. Here bi (i={1,2,3,4}) are arbitrary criteria and aij (i,j = {1,2,3,4}) are the pairwise comparison of criterion bi over bj. Figure 1a is the representation of the upper triangle of the PC matrix while Figure 1b is a representation of the lower triangle. The reciprocity property allows us to represent the PC matrix as two distinct graphs whether it is complete or not. IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 120 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 Considering existing judgments constitutes at least a sub-tree of the PC matrix then missing judgments can be calculated as illustrated in Figure 2. Let m be the number of missing judgments in the lower or upper triangle of a n th order PC matrix. If existing judgments constitute at least a spanning tree of the PC matrix then (n-2) number of missing judgments can be calculated through a single node as in Equation 2 while the remaining can be calculated through multiple nodes. Figure 2 illustrates the graph representation of calculating a missing judgment through a single node. Therefore, an incomplete PC matrix with the order of n should at least be a connected graph containing no loop with (𝑛 βˆ’ 1) direct PCs. This condition verifies that at least two direct PCs are required in order to determine an associated indirect PC. Eq. 1 represents the calculation of an indirect PC by using two direct PCs, which are the associated two edges in the path. If these direct PCs are consistent with each other, the path, used for determining the value of indirect PC, does not yield any variation. If not, the value of indirect PC can be assigned to the geometric mean of corresponding links. π‘Žπ‘–π‘— = 𝑀𝑖 𝑀𝑗 , π‘Žπ‘—π‘˜ = 𝑀𝑗 π‘€π‘˜ β†’ π‘Žπ‘–π‘˜ = π‘Žπ‘–π‘— Γ— π‘Žπ‘—π‘˜ (2) Figure 2 Graph representation of calculation process for a14 and a41 3. Consistency analysis In real life decision problems, PC matrices are rarely consistent. Nevertheless, decision makers are interested in the level of consistency of the judgments, which somehow expresses the goodness or harmony of pairwise comparisons totally, because inconsistent judgments may lead to senseless decisions (BozΓ³ki & RapcsΓ‘k, 2008). In the literature, several approaches can be found for evaluating the consistency of PC matrices such as consistency ratio using an eigenvector method, the least squares method, the Ξ§ squares method, the singular value decomposition method, Koczkodaj's inconsistency index, the logarithmic least squares method and geometric consistency index (Saaty, 1980; Chu, Kalaba, & Spingarn, 1979; Jensen, 1983; Gass & RapcsΓ‘k, 2004; W.W. Koczkodaj, 1993; Waldemar W. Koczkodaj, Herman, & OrΕ‚owski, 1997; Crawford & Williams, 1985; AguarΓ³n & Moreno-JimΓ©nez, 2003; BozΓ³ki & RapcsΓ‘k, 2008). Rahmami et al. (2009) propose a method that will improve consistency with less cost and better results than other methods in practice. A generalization of the Purcell IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 121 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 method is used to solve a system of homogeneous linear equations. Dadkhah et al. (1993) assume inconsistency is a measurement error, and propose a revision method similar to Saaty’s. The proposed method is also based on locating the elements that will contribute the most to increasing consistency. The difference from Saaty’s revision method is that the elements which will decrease the maximum eigenvalue are investigated; in order to do so, the first derivation of Ξ»A of aij is used. The element with the largest absolute value for δλA Ξ΄aij is revised; if it is negative the decision maker is instructed to increase aij, if it is positive the decision maker is instructed to decrease aij. Benitez et al. (2011) use a linearization technique that provides the closest consistent matrix to a given inconsistent matrix using orthogonal projection in a linear space. In order to measure the closeness of two given matrices, the Frobenius norm is utilized. The proposed method improves consistency in a direct and straightforward way differently from iterative methods. Gomez et al. (2009) propose a multi-layer perceptron-based model to improve the consistency of a given PCM. The given model is capable of computing missing values while also improving consistency In this study, to investigate the consistency of incomplete PC matrices, we analyzed Saaty's (1980) consistency analysis which defines the inconsistency ratio as an index for the deviation from randomness. 3.1 Consistency of complete PC matrices in AHP Consistency analysis presents a systematic approach to assess the consistency of judgments in a PC matrix. A pairwise comparison matrix A is consistent if it satisfies the transitivity property: π‘Žπ‘–π‘— Γ— π‘Žπ‘—π‘˜ = π‘Žπ‘–π‘˜ 𝑖, 𝑗, π‘˜ = 1,2, … , 𝑛 It was shown by Saaty (1980) that a pairwise comparison matrix is consistent if and only if it is of rank one. When a pairwise comparison matrix A is consistent, the normalized weights computed from A are unique. Otherwise, an approximation of A by a consistent matrix (determined by a vector) is needed (BozΓ³ki & RapcsΓ‘k, 2008). A PC matrix is said to be perfectly consistent if its maximum eigenvalue is equal to its order n. On the contrary, the deviation of Ξ»max from n is utilized as a measure of inconsistency. When a PC matrix deviates from perfect consistency with an acceptable degree, it is said to be consistent. If the deviation exceeds the acceptable degree, then it is an inconsistent PC matrix. If there are small variations of aij and the transitivity condition holds, then the maximum eigenvalue of A become close to n. This results in a CI value close to 0, and the matrix is accepted as consistent (Harker, 1987a, 1987b). There are some reasons for inconsistency, namely, a decision maker’s mistakes while using the 9-point scale, the scale is unsuitable for the specific criteria, transitivity issues, psychological dependence, etc (Kwiesielewicz & van Uden, 2004). When dealing with an increasing number of PCs the possibility of consistency error also increases (Franek & Kresta, 2014). Saaty implied that psychological dependence between comparisons causes higher inconsistency; hence, higher levels of consistency can mostly be achieved for a smaller number of objects (Ishizaka & Lusti, n.d.; Saaty, 1980; Wedley, Schoner, & IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 122 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 Tang, 1993). Since decreasing the number of criteria (or alternatives) to a relatively small number is not always possible, the use of incomplete pairwise matrices can help eliminate the side effects of higher dimensions. Additionally, with lack of a common unit of measurement among the elements of A, deviation from the exact PC is inevitable. Subjective judgments are a kind of estimation of reality and are expected to converge to the exact PC. Small variations of aij keep the maximum eigenvalue close to n, and remaining eigenvalues close to zero (Saaty, 1980). 3.2 Consistency of incomplete PC matrices in AHP Missing information is encountered due to various reasons. The criteria pair may not be suitable for pairwise comparison, the decision maker may not be eligible for assessing this pair, or the decision maker may not have a reasonable time or adequate motivation for evaluating all PCs. In such cases, decreasing the number of PCs assists the decision maker by providing flexibility and preserving motivation (Harker, 1987a, 1987b). Any PC matrix is investigated in terms of consistency and reliability before it is integrated into the decision-making process. A direct PC is a representation of human judgment, which is collected directly from the decision maker(s). Therefore, the number of direct PCs in a PC matrix affects the consistency of the matrix. As the number of direct PCs in a PC matrix increases, the consistency index also generally increases. Let us consider a two-level analytical hierarchy model with n number of criteria and n number of alternatives and suppose that each criterion has n number of sub-criteria. Eq. 3 represents the total number of PCs, which increases rapidly as n increases n(n2 + n + 1)(n βˆ’ 1) 2 (3) Figure 3 Total number of PCs vs the size of PCMs With the increase in the number of direct PC judgments, the time required to complete the evaluation process also increases rapidly. This leads to negative effects due to IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 123 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 psychological dependence, and therefore higher CI values. Regarding Saaty's 9-point scale, we generated random PC matrices to examine the relationship between CI values and the number of direct PCs. The consistency of an incomplete PC matrix relies on the consistency of the existing judgements. If the judgments of the decision maker do not create loops in the graph i.e. do not violate transitivity of judgments and are mathematically consistent, consistency of the PC matrix does not deteriorate. In Section 3.3 the influence of indirect judgments on the consistency of a PC matrix is further investigated. 3.3 An experimental research for understanding consistency behavior This section presents a numerical study to illustrate that the presence of indirect PC(s) affects the consistency of a PC matrix. The following example also implies that a change in the number of indirect PCs causes a deviation in the consistency of the PC matrix. Let R be the randomly generated PC matrix being investigated. In order to generate the PC matrix R, the following steps are applied, correspondingly. The right upper half of the matrix is filled by random evaluations using Saaty's 9-point scale, the main diagonal of R is filled with only values of 1, and finally, the lower left half of the matrix can thus be completed with the corresponding reciprocals. The generated complete PC matrix R and corresponding consistency measures are given in Eq. 4. 𝐑 = [ 1 1/4 1 4 1 7 1 1/7 1 6 9 1 1/4 1 7 3 1 1 1/6 4 1/3 1/9 1 1 1 1/7 1 1 1 3 1 1 1/5 1/3 5 1 ] πœ†_ max = 9.5499 π‚πˆ = 0.7099 𝐂𝐑 = 0.5726 (4) Suppose R1 is an incomplete PC matrix, which is derived from R by removing any 4 judgments. The PC matrix R1, calculated indirect PCs (a12, a16, a25, a35) and corresponding consistency measures are given in Eq 5. π‘πŸ = [ 1 βˆ’ 1 βˆ’ 1 7 1 1/7 1 6 9 βˆ’ 1/4 βˆ’ 7 3 1 1 1/6 4 1/3 1/9 βˆ’ 1 βˆ’ 1/7 1 1 βˆ’ 3 βˆ’ 1 1/5 1/3 5 1 ] π‘Ž12 = 1 6 7 π‘Ž16 = 3 1 5 π‘Ž25 = 15 2 3 π‘Ž35 = 15 πœ†_ max = 8.6643 π‚πˆ = 0.5329 𝐂𝐑 = 0.4297 (5) Now, suppose R2 is also an incomplete PC matrix, which is derived from R by removing any 6 judgments. The PC matrix R2, calculated indirect PCs (a13, a15, a23, a36, a45, a56) and corresponding consistency measures are given in Eq 6. IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 124 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 π‘πŸ = [ 1 1/4 βˆ’ 4 1 βˆ’ βˆ’ βˆ’ 1 6 βˆ’ 1 1/4 1 7 3 1 βˆ’ 1/6 4 1/3 βˆ’ 1 1 1 1/7 βˆ’ 1 βˆ’ 3 βˆ’ 1 1 1/3 βˆ’ 1] π‘Ž13 = 2 π‘Ž15 = 1 4 π‘Ž23 = 2 7 π‘Ž36 = 9 π‘Ž45 = 1 1 6 π‘Ž56 = 7 πœ†_ max = 8.5358 π‚πˆ = 0.5072 𝐂𝐑 = 0.4090 (6) The maximum eigenvalue of PC matrices R, R1 and R2 are 9.5499, 8.6643 and 8.5358 respectively. Consistency index (CI) and consistency ratio (CR) are dependent on the maximum eigenvalue of a PC matrix since RI is the expected value of a consistency index and is constant. Figure 4 illustrates the decrease in consistency indicators, i.e the PC matrix becomes more consistent while the number of indirect judgments increases. As a result, the consistency of a PC matrix does not deteriorate with the number of indirect judgments instead it becomes more consistent. Figure 4 Changes in consistency indicators while the number of indirect judgments increases This example straightforwardly indicates that any increase in the number of indirect PCs corresponds to higher consistency in judgments. The differences between the original judgments and the calculated indirect PCs for removed elements can be considered as the divergence of the decision maker's judgments from mathematical consistency. An increase in the number of indirect PCs corresponds to a decrease in the number of direct PCs; R2 is arithmetically more consistent than others are. For investigating the effects of direct PCs on consistency in detail, we generated 10,000 samples of random PC matrices in a similar manner. Let m denote the number of direct PCs. Figure 5a shows the mean and standard deviation of CI for an increasing number of direct PCs, and Figure 5b shows probability density functions of CI for three different m values. Probability density functions for different m values show that the random index (RI) e.g the expected value of the consistency index of random PC matrices is dependent on the number of indirect judgments. In other words, the probability distribution changes with the number of indirect judgments. IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 125 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 Simulated results imply that the mean of the CI is robust, but the variance is clearly unstable to the changes in the number of direct PCs. (a) Mean and standard deviation (b) Probability density function Figure 5 CI behaviour while the number of direct PCs changes 4. Material and methods The main purpose of this section is to analyze Random Index (RI) values in case of missing information. RIs are the consistency indices of a randomly generated PC matrix from scale 1 to 9, with reciprocals forced (Saaty, 1980). Let A be an incomplete PC matrix with the order of n and m denotes the number of direct pairwise comparisons. In order to attain missing elements in a PC matrix, the existing direct PCs must construct a spanning tree. Eq.7 represents the interval for m. [(𝑛 βˆ’ 1), 𝑛(𝑛 βˆ’ 1) 2 ) = {π‘š ∈ β„€+|(𝑛 βˆ’ 1) ≀ π‘š > 𝑛(𝑛 βˆ’ 1) 2 } (7) Similar to the instance given in Section 3.3, complete and incomplete PC matrices are generated randomly and the corresponding consistency levels are investigated. For PC matrices with the order of 4 through 11, we investigate the mean and the standard deviation of RIs for each level of m using the sample size of 10,000 simulated data. Figure 6 shows the types and the ranges of simulated PC data. IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 126 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 Figure 6 Simulated dataset The pseudo code for construction of random PC matrices and calculation of corresponding RIs is given as in Figure 7. IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 127 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 Figure 7 Pseudo code for Random Index generation 4.1 Computational experiments and results For each order of n, we generated a set of 10,000 random PC matrices for each m number of direct PCs. Next, calculated CI values are accumulated for each set and corresponding RIs are examined. The aim of our simulation is to analyze the empirical distributions of the maximum eigenvalues Ξ»max of randomly generated PC matrices for varying direct PC levels. The elements aij were chosen randomly from the scale: [ 1 9 , 1 8 , 1 7 , … , 1,2, … ,7,8,9] and aji is defined as 1/aij. By investigating probability density functions of CI values, we find that the number of direct PCs plays a major role in the consistency analysis. The results of the computational analysis can be summarized as follows: 1. While the number of direct PCs increases, the expected value of random index also increases and the variance decreases. 2. While the number of direct PCs increases, the consistency index (CI) of a random PC matrix converges to the expected value of the random index. 3. While the number of direct PCs increases, the probability density function of random index converges to a normal distribution. Let CIn m and RIn m denote the corresponding CIs and RIs of an n th order PC matrix with m number of direct PCs, respectively. For distinctive threshold levels, Table 1 shows the number of consistent PC matrices comparatively for m-specific and non-specific RI values. IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 128 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 Table 1 The number of PC matrices below different levels of CR threshold For any CR threshold, the total number of acceptable matrices decreases as the number of direct PCs increases, as in Table 1. When Saaty's (1980) proposal CR≀10% is employed to evaluate whether the judgments are deemed to have the acceptable consistency or not, the decrease in the total number of acceptable PC matrices is more drastic comparatively to other thresholds. When non-specific RI values are utilized in the accept/reject decision process, only three of 10,000 complete PC matrices (n=15) can be accepted as consistent, while 4,353 of 10,000 incomplete PC matrices with six direct PCs can be accepted as consistent. If m-specific RI value is utilized for assessing the consistency of 10,000 incomplete PC matrices with six direct PCs, only 2,844 of them will be accepted. As mentioned earlier, the number of indirect PCs can generate false positives and false negatives in the decision making process. For any incomplete n th order PC matrix, false positives are mostly encountered in the matrices that have a lower number of direct PCs, and the false negatives are mostly encountered in the matrices that have a higher number of direct PCs. Here, Table 1 illustrates the case for sixth order PC matrices. The difference between 4,353 and 2,844 implies that false positives exist in the consistency analysis. About 1,500 PC matrices could be inadvertently accepted during the consistency evaluation process. Conversely, for the case with 11 direct PCs and 0.5 threshold level, the acceptable number of PC matrices increases to 2,118 when m-specific RI value is utilized, whereas it is 2,050 for nonspecific RI computation. This instance illustrates the case for false negatives. Depending on the sensitivity of the problem, the decision maker can determine particular threshold level(s) regarding the number of direct PCs. For example, a lower threshold level and a higher number of PC judgments can be required for the problems with high sensitivity and vice versa. IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 129 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 5. Experimental study and results We conducted an experimental study to investigate the effects of incomplete PC matrix on consistency. The survey experiment was carried out in cooperation with 104 participants. Each participant was given six differently sized zeros shown in Figure 8 and asked to judge their sizes relatively. Participants are required to make 10 randomly chosen PCs and observed the durations during the judgment process. Average completion time was about 12 minutes, and participants spent about one minute for each evaluation. Additionally, we noticed that the average duration of a previous judgment is generally lower than that of the subsequent one. As we have not previously set up a hypothesis on this resulting observation, we have not logged a separate data for each of the evaluation processes. However, this observation can be tested by a more extensive experiment in the future. After the data was collected from participants and transferred to a data file, we removed four randomly chosen PCs from each matrix in order to construct new incomplete PC matrices. Therefore, this manipulation provided us two different sets of incomplete PC matrices for each participant. We have analyzed their consistency with non-specific and m-specific RI values (Table 1). Figure 8 Survey example The consistency measures, CI and CR, were investigated for two sets of incomplete PC matrices. We depict the frequency distribution of consistency measures, which are uniformly grouped by 10% percentiles in Figure 9. In Figure 9a, the frequencies of CR values change for two RI values. Assuming that the duration needed for any PC judgment is uniformly distributed, the time needed for 15 PCs is nearly 2.5 times greater than the time needed for six PCs. For the first percentile, only six of 102 incomplete PC matrices fall under 0.10 when a non-specific RI value is used for analysis. In Figure 9b, the frequency distributions of CR values are identical for m-specific and nonspecific RIs. However, the time needed for 10 direct PCs is expected to be shorter than that of 15 direct PCs. IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 130 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 (a) n=6, m=6 (b) n=6, m=10 Figure 9 CR intervals for RI6 6 , RI6 10 and RI6 15 Consequently, the required time for the acquisition process of PC judgment can be reduced by the use of m-specific RIs without incurring a loss of information. 6. Conclusions and discussions In decision-making problems, experts generally express the information, evaluations, preferences and weights linguistically. Since the data is inherently non-numeric, decision- making methods require a pre-procedure to transform the non-numerical data into numeric values. The PC method is one of the popular methods for transforming subjective judgments into analytical information. It plays a significant role in multiple criteria decision-making methods, especially with the Analytic Hierarchy Process and Analytic Network Process methods. The PC method is mathematically capable of dealing with large sets of criteria and helps experts focus on only two elements one at a time. However, successive judgment capability of a human being is limited. Hence, it can be puzzling for experts to provide consistent pairwise judgments when a large set of comparisons is available. In these cases, the judgment matrix fails in conformity to the transitivity requirement and exhibits some inconsistency. The use of incomplete PC matrices assists decision makers by enabling a substantial decrease in the number of PCs when a higher number of criteria are available. However, the use of indirect PCs can cause false positives or false negatives throughout consistency analysis. Namely, for an incomplete PC matrix, the use of non-specific (original) RI with a consistency test can indicate the presence of consistency, when it is not consistent in reality. The obtained weights from a PC matrix are highly dependent on initial judgments and its level of consistency and completeness has a fundamental role in the formation of the final decision. In this paper, a two-dimensional consistency analysis approach was presented for evaluating the consistency of incomplete PC matrices. The computational experiment reveals that the number of direct PCs is the significant factor while evaluating IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 131 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 consistency. Moreover, indirect judgments do not deteriorate the consistency ratio given that the judgments of the decision maker are mathematically consistent and transitive. Our study demonstrates that if there are missing judgments in a PC matrix the probability distribution is different from the case of a complete PC matrix. As a result, the expected value of the consistency index (RI) is changed. In order to avoid false-positive results for a consistency ratio, an incomplete PC matrices consistency ratio should be calculated with the respective probability distribution. For future study, the approach presented could be extended to fuzzy approaches and group decision making. A more extensive experiment could be conducted for examining the required time for successive evaluations in the future. Furthermore, the relationship between the sensitivity of decision problem and the number of direct PCs can be investigated. IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 132 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 APPENDIX IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 133 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 REFERENCES AguarΓ³n, J., & Moreno-JimΓ©nez, J. M. M. (2003). The geometric consistency index: Approximated thresholds. European Journal of Operational Research, 147(1), 137–145. Doi: https://doi.org/10.1016/S0377-2217(02)00255-2 BenΓ­tez, J., Delgado-GalvΓ‘n, X., Izquierdo, J., & PΓ©rez-GarcΓ­a, R. (2011). Achieving matrix consistency in AHP through linearization. Applied Mathematical Modelling, 35(9), 4449–4457. Doi: https://doi.org/10.1016/j.apm.2011.03.013 BozΓ³ki, S., & RapcsΓ‘k, T. (2008). On Saaty’s and Koczkodaj’s inconsistencies of pairwise comparison matrices. Journal of Global Optimization, 42(2), 157–175. Doi: https://doi.org/10.1007/s10898-007-9236-z Chu, A. T. W., Kalaba, R. E., & Spingarn, K. (1979). A comparison of two methods for determining the weights of belonging to fuzzy sets. Journal of Optimization Theory and Applications, 27(4), 531–538. Doi: https://doi.org/10.1007/BF00933438 Crawford, G., & Williams, C. (1985). A note on the analysis of subjective judgment matrices. Journal of Mathematical Psychology, 29(4), 387–405. Doi: https://doi.org/10.1016/0022-2496(85)90002-1 Dadkhah, K. M., & Zahedi, F. (1993). A mathematical treatment of inconsistency in the analytic hierarchy process. Mathematical and Computer Modelling, 17(4–5), 111–122. Doi: https://doi.org/10.1016/0895-7177(93)90180-7 Dede, G., Kamalakis, T., & Sphicopoulos, T. (2016). Theoretical estimation of the probability of weight rank reversal in pairwise comparisons. European Journal of Operational Research, 252(2), 587–600. Doi: https://doi.org/10.1016/j.ejor.2016.01.059 Elliott, M. A. (2010). Selecting numerical scales for pairwise comparisons. Reliability Engineering & System Safety, 95(7), 750–763. Doi: https://doi.org/10.1016/j.ress.2010.02.013 Franek, J., & Kresta, A. (2014). Judgment scales and consistency measure in AHP. Procedia Economics and Finance, 12, 164–173. Doi: https://doi.org/10.1016/S2212- 5671(14)00332-3 Gass, S. I., & RapcsΓ‘k, T. (2004). Singular value decomposition in AHP. European Journal of Operational Research, 154(3), 573–584. Doi: https://doi.org/10.1016/S0377- 2217(02)00755-5 Gomez-Ruiz, J. A., Karanik, M., & PelΓ‘ez, J. I. (2009). Improving the consistency of AHP matrices using a multi-layer perceptron-based mode. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 5517 LNCS(PART 1), 41–48. Doi: https://doi.org/10.1007/978-3-642- 02478-8_6 Harker, P. T. (1987a). Alternative modes of questioning in the analytic hierarchy process. IJAHP Article: Koyun Yilmaz, Ozkir /Extended consistency analysis for pairwise comparison method International Journal of the Analytic Hierarchy Process 134 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.506 Mathematical Modelling, 9(3–5), 353–360. Doi: https://doi.org/10.1016/0270- 0255(87)90492-1 Harker, P. T. (1987b). Incomplete pairwise comparisons in the analytic hierarchy process. Mathematical Modelling, 9(11), 837–848. Doi: https://doi.org/10.1016/0270- 0255(87)90503-3 Ishizaka, A., & Lusti, M. (n.d.). An expert module to improve the consistency of AHP matrices. Retrieved from http://www.ishizaka-dev.myweb.port.ac.uk/ITOR.pdf Jensen, R. (1983). Comparison of eigenvector, Least squares, Chi square and Logarithmic least square methods of scaling a reciprocal matrix. Retrieved from https://scholar.googleusercontent.com/scholar.bib?q=info:BB3LRd5maOkJ:scholar.googl e.com/&output=citation&scisig=AAGBfm0AAAAAWXdVkq5u779Tdq1ABrGuPB7Cu oZwcF61&scisf=4&ct=citation&cd=-1&hl=tr&scfhb=1 Koczkodaj, W. W. (1993). A new definition of consistency of pairwise comparisons. Mathematical and Computer Modelling, 18(7), 79–84. Doi: https://doi.org/10.1016/0895- 7177(93)90059-8 Koczkodaj, W. W., Herman, M. W., & OrΕ‚owski, M. (1997). Using consistency-driven pairwise comparisons in knowledge-based systems. In Proceedings of the sixth international conference on Information and knowledge management - CIKM ’97 (pp. 91–96). New York, New York, USA: ACM Press. Doi: https://doi.org/10.1145/266714.266867 Kwiesielewicz, M., & van Uden, E. (2004). Inconsistent and contradictory judgements in pairwise comparison method in the AHP. Computers & Operations Research, 31(5), 713–719. Doi: https://doi.org/10.1016/S0305-0548(03)00022-4 Rahmami, M., & Navidi, H. (2009). A new approach to improve inconsistency in the analytical hierarchy process. Applications and Applied Mathematics: An International Journal, 4(1), 40–51. Saaty, T. L. (1980). The Analytic Hierarchy Process. New York: McGraw-Hill. Siraj, S., Mikhailov, L., & Keane, J. A. (2015). Contribution of individual judgments toward inconsistency in pairwise comparisons. European Journal of Operational Research, 242(2), 557–567. Doi: https://doi.org/10.1016/j.ejor.2014.10.024 Thurstone, L. L. (1994). A law of comparative judgment. Psychological Review, 101(2), 266–270. Retrieved from https://brocku.ca/MeadProject/Thurstone/Thurstone_1927f.html Wedley, W. C., Schoner, B., & Tang, T. S. (1993). Starting rules for incomplete comparisons in the analytic hierarchy process. Mathematical and Computer Modelling, 17(4–5), 93–100. Doi: https://doi.org/10.1016/0895-7177(93)90178-2