IJAHP Article: Repetski, Sarkani, Mazzuchi / Applying the Analytic Hierarchy Process (AHP) to expert documents International Journal of the Analytic Hierarchy Process 1 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.919 APPLYING THE ANALYTIC HIERARCHY PROCESS (AHP) TO EXPERT DOCUMENTS Edward Repetski George Washington University edward_repetski@gwu.edu Shahram Sarkani George Washington University sarkani@gwu.edu Thomas Mazzuchi George Washington University mazzu@gwu.edu ABSTRACT This paper explores an innovative technique to elicit prioritization from expert documents using the Analytic Hierarchy Process (AHP). Practitioners within domains that have a large quantity of expert literature can utilize these prior efforts to answer new questions. This method can solve some of the challenges of the AHP which include dealing with inconsistencies amplified by large numbers of options, locating experts with the availability and commitment to undergo an iterative opinion refinement process, and the speed of obtaining priorities. By extracting comparisons through rules applied to expert literature, this paper shows how prioritization for a number of options larger than 15 can be achieved using the AHP. Using the literature creation process as a means to gain consistency from a panel of experts, the extracted priorities are absolutely consistent. Further, the process can be accomplished without iterative engagement of experts which helps reduce time and cost. Overall, this unique application of the AHP provides those with access to a set of expert literature a method to quantify expert opinions in a consistent and cost-effective manner. Keywords: expert opinion; opinion extraction 1. Introduction This paper details a technique to elicit expert opinions from documents. While this is a defense-related example, this technique could be used with any well-documented domain. Various domains require prioritization of lists of requirements, constraints, conditions, etc. While this technique was used to prioritize eighteen linkages for operational integration optimization, the method could easily prioritize a large number of requirements for any project. The authors wished to find a way to weight the eighteen linkages as priorities for integration efforts in a separate model. The Analytic Hierarchy Process (AHP) has been mailto:edward_repetski@gwu.edu mailto:sarkani@gwu.edu mailto:mazzu@gwu.edu IJAHP Article: Repetski, Sarkani, Mazzuchi / Applying the Analytic Hierarchy Process (AHP) to expert documents International Journal of the Analytic Hierarchy Process 2 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.919 used successfully to weight priorities of sets; however, using the tool, in this case, presented a few challenges including the number of elements to prioritize, access to a large field of experts and the time needed to solicit opinions in the traditional manner from those experts. Commercially available AHP software typically addresses cases with fifteen or fewer options, so the authors built an AHP tool that could accommodate a larger number of options instead of adding more layers to the hierarchy. Further, access to experts for the extended amount of time required to extract these many paired comparisons and to iteratively engage until the opinions about priorities were complete and adequately consistent was not available. The authors decided to address this challenge by using the doctrinal literature as experts. Fortunately, the area of this research was well covered by detailed documents. The authors treated the doctrinal manuals as experts and used metrics to elicit the opinions from those manuals. This technique proved useful in three ways; it was effectively broader and faster than the typical AHP elicitation process, and provided absolutely consistent responses. This method is broader as the study used five doctrinal manuals, each created, edited and approved by an extended set of experts. The manuals could be evaluated in a few days once a set of metrics was determined and thus this is faster than any survey tool. Lastly, the use of metrics provided absolutely consistent results. Systems thinking, through Model-Based Systems Engineering (MBSE) is expanding into broader areas of application. One such area of potential application is military readiness. The case study involved the operational integration and testing of unstabilized gunnery crews composed of US Army soldiers. This type of integration has long been the domain of expert opinion. The authors were attempting to model the various integration tools and the efficacy of their application to the integration of humans and equipment sets into effective teams. Such a model would advance the application of MBSE into the operational life of systems and incorporate humans into those models. The potential impact is significant given the sheer quantity of teams that fall into this category. A part of that model, shown in Figure 1, illustrates a Combined Integration Matrix (CIM). This matrix shows all the potential linkages between crew members and with each crew member and parts of the kit of a mounted crew. The integration matrix is derived from a Multi-Domain Architecture Design Structure Matrix (MDA-DSM) of a three-soldier crew. The mechanics of that model aside, this matrix needed to be weighted for that model by prioritizing the 18 linkages. The left side of Figure 1 shows those 18 cells as they are abstracted from a larger DSM with the linkages of interest labeled A-R. On the right side of Figure 1 is the model with the weighted values (in percentages) that were determined by this proposed technique applied to the AHP. Figure 1 Left: Combined Integration Matrix (CIM); letters identify the 18 linkages to be prioritized. Right: weighting determined by this method IJAHP Article: Repetski, Sarkani, Mazzuchi / Applying the Analytic Hierarchy Process (AHP) to expert documents International Journal of the Analytic Hierarchy Process 3 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.919 To validate this method, a group of qualified US Army Master Gunners were engaged for their expert opinions to determine if these were consistent with the results of the doctrinal survey process. Four Master Gunners were engaged and provided their input to the model. This paper illustrates the extraction of priorities from a doctrinal literature variant of the AHP and validates that method using the AHP by traditional means. 2. Literature review In this increasingly complex world, governments and industry have pursued ways to combine large multitudes of expert opinions to find optimal recommendations for complex situations. The AHP, created by Saaty, is a recognized tool for decision making in complex situations (Saaty, 1988). His essential text on this subject even mentions โ€œresource allocationโ€ in its title. The goal of this work is indeed seeking to best allocate resources. The AHP has been described and examined in the literature and a brief synopsis is provided below. The major components of the AHP involve determining a hierarchy and then prioritizing the elements at any given level using a method to translate the opinions of experts into numeric values. Early work in AHP involved the use of additive and multiplicative formats which were logarithmically related (Lootsma, 1999). Multiplicative AHP achieves the same results mathematically and more intuitively as fewer practitioners work directly with logarithms with the growth of available computing power. Therefore, this work uses multiplicative AHP. In either processing method, the practitioner needs to extract opinions from experts, iteratively refine those opinions to deal with inconsistencies, and combine those opinions to reach a recommended decision or prioritization. We commend Lootsmaโ€™s paper on Saaty for a strong synopsis of the mathematics of this process (Lootsma, 1980). Eliciting opinions in multiplicative AHP involves using adjectives or a small range of numbers to quantify opinions by doing pairwise comparisons (Saaty, 2013). For example, if A is more important than B, then A is 2B; if much more important than B, then A is 4B. By focusing on each pair, the expert isolates themselves from the multitude of judgements and makes a determination. This can be a daunting process. In the given example with 18 options, a staggering 306 opinions are required to complete a table. This assumes the diagonal of ones where each element is as important as itself. This is a monotonous task that often discourages experts from participating. The many comparisons can be daunting, but also unnecessary. If A is 2B, then B is ยฝ A. Two cells can be completed with one judgement. Further, if A is 2B and B is 2C, then A is 4C in a perfectly consistent situation. These assumptions of consistency reduce the number of comparisons that need to be elicited from users. Reciprocity alone reduces the 306 opinions needed in the example to 153. Another third or more opinions can be eliminated with assumptions of consistency. This balancing act is complicated. The fewer pairwise comparisons elicited, the more pronounced the impact of inconsistencies will be. Also, the selection of the pairs could vary between experts. The practitioner doing the iterative work can complete the entire matrix and determine the inconsistencies during further engagements; however, garnering data iteratively is challenging. Lootsma provides an interesting example as he works to extract data from practitioners in one of his papers IJAHP Article: Repetski, Sarkani, Mazzuchi / Applying the Analytic Hierarchy Process (AHP) to expert documents International Journal of the Analytic Hierarchy Process 4 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.919 (Lootsma, 1999a). In this case, he found it challenging to obtain voluntary responses to his questions from experts even in the very field of applying the AHP. Multiplicative AHP is based on absolute ratio scales; therefore, zeroes are not necessary (Garuti, 2018). The opinions gathered here are comparisons and a ratio scale is essential to this process. Experts may be constrained in their judgements depending on how the opinion-gathering instrument is constructed (for example, only able to use adjectives, numbers or even intermediate values). Even so, challenges will be faced with consistency. The two areas of consistency are in terms of intransivity and cyclic judgements (Lootsma, 1999b). Intransivity is the idea that aij is the inverse of aji or if A is 2B and B is 2C then A should be 4C. If this is violated, consistency suffers. Cyclic judgements where A>B and B>C, but C>A severely compromises consistency. Relative comparisons require a ratio scale. With a ratio scale, geometric means rather than arithmetic means are preferred for averaging to obtain results that have mean the same thing for each opinion. This is shown in Equation 1. The n th root of the n multiples provides a means of combining the n opinions while maintaining the relationship of aij always equals 1/aji (Lootsma, 1999b). Critical relationships between cells are lost if this is not observed. ๐บ๐‘’๐‘œ๐‘š๐‘’๐‘ก๐‘Ÿ๐‘–๐‘ ๐‘€๐‘’๐‘Ž๐‘› = โˆš(๐ด1๐ด2 โ€ฆ ๐ด๐‘›) 1/๐‘› (1) Even after a cyclic process of opinion solicitation is complete, the opinions must be evaluated for consistency. The intransitives and logical breaks inside an expertโ€™s opinion or across the set of expertsโ€™ opinions must be sufficiently small to allow the consolidated opinion to be seen as acceptable. Here, the standard is the Consistency Ratio (Saaty, 1988). The Consistency Ratio (CR) is the Consistent Index (CI) divided by the Random Consistency Index (RI) as shown in Equation 2. ๐žดmax is the maximum Eigenvalue, n the number of options in the set, all divided by (n-1). RI is obtained from Saatyโ€™s text and shown in Figure 2. Note that RI was created by comparing consistency against randomly created matrices. The scale ranges from 3 to 15 for values of n. A CR of less than 0.1 is acceptably consistent with 0 representing perfectly consistent and higher values being less and less consistent. RI is not defined for situations like the one at hand when n = 18. ๐ถ๐‘… = ๐ถ๐ผ ๐‘…๐ผ = ๐žด๐’Ž๐’‚๐’™โˆ’๐’ ๐’โˆ’๐Ÿ ๐‘น๐‘ฐ (2) Figure 2 Random Consistency Index (RI) (Saaty, 1988) To find, or at least approximate, the Maximum Eigenvalue, ๐žดmax the practitioner must define the consolidated opinion matrix and use the right Eigenvector. The right Eigenvector can be computed in a number of ways. All the methods agree in the case where the matrix is absolutely consistent; therefore, the simplest method works as well as n 3 4 5 6 7 8 9 10 11 12 13 14 15 โ€ฆ RI 0.58 0.9 1.12 1.24 1.32 1.41 1.45 1.49 1.51 1.53 1.55 1.57 1.59 โ€ฆ IJAHP Article: Repetski, Sarkani, Mazzuchi / Applying the Analytic Hierarchy Process (AHP) to expert documents International Journal of the Analytic Hierarchy Process 5 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.919 any other. The combined opinion matrix is summed by rows, then the values are normalized to one. That is, the sum of each row is divided by the total of row sums. These represent the priority ranking of the options in each row. To find the Eigenvalue, the opinion matrix is multiplied by the Eigenvector. This produces a matrix of the same form as the Eigenvector. This new matrix is now divided by n times the Eigenvector to achieve a new matrix that will be close to a column of ones (a column of ones in the perfectly consistent case) and their sum will be the maximum Eigenvalue. We recommend Lootsmaโ€™s text to understand the calculation in detail (Lootsma, 1999b). Another benefit of the Eigenvalue calculations is that the practitioner now has a ready means to check their work. The maximum Eigenvalue cannot be less than n. As this process allows for a spread of resources over a large range of options, the matrices involved grow very difficult to manually check for errors. Therefore, the assumption of consistency must be checked both to confirm consistency and to check for errors in logic. A large amount of literature has focused on the level of inconsistency that can be tolerated and how to extract sufficiently consistent data from available experts (Russo & Camanho, 2015). There are challenges involved with identifying the experts, soliciting their opinions, and ensuring that those opinions are consistent. First, the decision maker or their agents must identify the experts. This is not a trivial decision. The practitioner must define the requirements for experts and define what mix of experts is needed from differing domains (Garuti & Sandoval, 2006). Then, the experts need to be engaged. Large organizations with internal experts can task those experts to participate in the gathering of opinions, but this is rarely the case. Most often, the practitioner must solicit voluntary participation from the experts. Several mechanisms exist to address inconsistencies. As expected, iteratively engaging experts to conform their opinions into consistent matrices is time and cost prohibitive (Basak, 2020). Inconsistency grows as the number of options increase (Saaty, 1988). As the decision set increases, inconsistencies grow faster. A solution to this problem is to add a level to the hierarchy to create more pools with fewer decisions (Lootsma, 1999b). Saaty and Sagir (2009) make a case for this layered approach. If such a bifurcation of the problem is not practical, the reality of the slow increase of the RI has greater impact. As shown in Figure 2, the value of RI increases very slowly after n = 8. Excessive inconsistency invalidates the use of precious data; therefore, various techniques have been proposed to deal with this challenge. Some have recommended using bootstrapping to improve inconsistent data (Basak, 2020). Others have applied fuzzy comparisons to refine the data (Boender, de Graan, & Lootsma, 1989; Liu, Xu, & Liao, 2016). Lastly, at least one practitioner has used rules to apply the AHP in cloud computing (Ergu, Kou, Peng, Shi, & Shi, 2013). It is valuable to discuss the scale that is used. Saaty firmly supported the idea that a scale of 1 to 9 was ideal (Saaty, 1988). Others have postulated different scales (Lootsma, 1999b). Saaty was looking for numbers that could be easily interpreted by the experts providing the opinions and he found, through trial and error, that using 1-9 was most effective. Lootsma shows both logarithmic and multiplicative scales as options. He further posits that any ratio scale works mathematically (Lootsma, 1999b). The fundamental issue with a relatively short scale is that intransitivity can quickly appear IJAHP Article: Repetski, Sarkani, Mazzuchi / Applying the Analytic Hierarchy Process (AHP) to expert documents International Journal of the Analytic Hierarchy Process 6 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.919 when A is four times as important as B and B is three times as important as C and C is more important than D. While not a cyclic intransitive case, the math quickly explodes the range of 1 to 9. Saaty shows how these issues can be returned to experts for refined opinions. In this example, because the numbers are not linked to adjectival ratings by experts, the option was used to apply a larger scale than normally used for the AHP. The validation tool, however, does use a 1 to 9 scale. In summary, the AHP is a recognized tool that uses experts to weight priorities and combine their opinions. The shortcomings of this method, especially when accounting for a large value of n, the undefined value of RI for n greater than 15, include the difficulty of gathering expert opinions, the risk of losing those opinions through iteration and the increasing inconsistencies that must be addressed. The methodology applied addresses those issues successfully. 3. Methodology The proposed technique addresses all of the challenges of the AHP including large n values and limited access to iteratively work with experts. Successful application of this technique requires that the subject be addressed in a significant body of technical literature. Our case study meets this requirement with multiple US Army doctrinal publications that directly deal with the topic of integrating and testing three-person crews of vehicle-mounted, unstabilized weapons. The technique treats each of the documents as an expert or a panel of experts. This approach addresses the challenges of the AHP which include a dearth of experts, the time-consuming impacts of iteration, the potential for increased inconsistencies, and problems resulting from a large value of n. It shifts the benefit of opinion refinement from the AHP interaction with experts to those opinions being refined in the writing of doctrine. Each Army doctrinal publication is a summary of the work of multiple experts, often the most qualified, across a very large organization. Each document represents not a single expert, but a panel of selected experts. Further, since the manuals are written at different times, these writing teams effectively represent a different pool of experts each time. The metrics that are used must clearly relate to the topic of the document. In this example the manuals deal with the process of crew integration and the chosen metrics are focused on that area. Once the metrics are chosen, they are drawn from the documents by a procedure. Perhaps most importantly, once the metrics are selected, because they are drawn by a set of rules, they provide absolutely consistent results. These panels of experts, in the writing and editing of these publications, act as the iterative event which drives the panel of experts to be consistent. Thus, the prior event, the publication writing process, acts as the iterative, consistency-improving, time-consuming process from which the practitioner can profit. In this case, the Armyโ€™s Doctrinal Library was consulted for related technical publications. Five manuals were found that addressed the case study. Then, a process was followed whereby the relevant section or sections of the documents were examined and a count was taken of each reference to each linkage as shown in Figure 3. Each sentence that addressed the interaction between two elements and thus representing a linkage to be integrated was counted against that linkage. For example, the sentence โ€œThe Gunner IJAHP Article: Repetski, Sarkani, Mazzuchi / Applying the Analytic Hierarchy Process (AHP) to expert documents International Journal of the Analytic Hierarchy Process 7 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.919 orients the weaponโ€ฆโ€ would be counted against letter I. Similarly, in diagrams and illustrations, bullet comments that addressed a linkage were counted. Note, in Figure 3 the variance between sources both in the volume of comments and in their dispersion across topics. Figure 3 Raw data drawn from five doctrinal publications For each expert document, a matrix was created with the 18 elements (A-R) along the left and top margins. Before simply dividing the row by the column value, some pre- processing was necessary to ensure that the matrices were compatible, consistent and easily usable. First, the decision was made to set all five to a common scale to ensure that the weights of each opinion were the same. Because the data was integer in nature it was easier to convert all ratios to a common base. Each value was multiplied by 256 and divided by the strongest opinion in that source. Therefore, the strongest opinion always had a value of one. This was not an opinion scale such as 1 to 9, but a way to balance the opinions between each other. We could have converted to any base number, but settled on 256. Adding one to each value removed zeroes which simplified the use of ratios while keeping the distance between the counts consistent. Some variance in the opinions was seen when the common factor was small. This rapidly smoothed out at a larger scale and a very stable set of opinions was achieved with a common factor of 256. IJAHP Article: Repetski, Sarkani, Mazzuchi / Applying the Analytic Hierarchy Process (AHP) to expert documents International Journal of the Analytic Hierarchy Process 8 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.919 Figure 4 shows the details of one of the opinion matrices; specifically, the top left corner. Imagine the columns running from A to R as well as the rows. Focusing on this corner of one matrix nicely illustrates the process. The raw data is shown in the column and row labeled CNT or count. The row and column labeled 1 shows the values as one is added to each count to convert to a ratio scale. The column labeled NORM has the largest value converted to 256, that is multiplied by 256 and divided by the largest value. The largest data point is 256 while all others are scaled similarly. Next, the opinions are created by dividing row by column. Note that the appearance of the expected diagonal of ones as items are compared to themselves. Also, the relationships are as expected as aij always equal 1/aji; for example, the two circled cells are the inverse of each other. Of course, these numbers are truncated to two decimal places for legibility. Figure 4 Matrix of opinion comparisons for one of the expert documents More verbose manuals had more points to distribute to more linkages and addressed more detail while less wordy texts often omitted some linkages of lesser importance. The normalization to a common number allowed the five expert documents to be weighted the same in the ensuing combination of tables and allowed for easier comparison between opinions. Further, adding unity before normalizing allowed for less impact of the omitted factors in the less wordy texts, where omission had more impact on ranking in longer writings. Because they were built by a rule, aij always equals 1/aji and A > 2B and B > 2C always means A > 4C. This is, absolutely, the definition of consistency. Once the five opinions were moved into these standard matrices, the five matrices were combined into a unified opinion matrix (see Figure 5). This is the matching highlight of the combined opinions to the single opinion in Figure 4. This was combined using the geometric mean, the 5 th root of the product of the five ratios. Note again the inverse relationships across the main diagonal and highlighted in the circled squares with 2.11 equal to 1/0.47. This is the main reason to use the geometric mean as it retains reciprocity. From here, two essential functions are possible. First, the Eigenvector can be extracted to see the consolidated opinion of priorities for the 18 linkages. As discussed, this is most easily obtained from a consistent matrix by summing the rows into a column vector on the right of the matrix and then normalizing the values by dividing each by the IJAHP Article: Repetski, Sarkani, Mazzuchi / Applying the Analytic Hierarchy Process (AHP) to expert documents International Journal of the Analytic Hierarchy Process 9 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.919 sum of the column matrix providing a new matrix that sums to unity. This priority matrix was the answer to the question that had prompted the effort. Figure 5 Matrix of combined opinions The second function is to use this Eigenvector to find the maximum Eigenvalue and test for consistency. ๐žดmax was extracted as described in the literature review and found to be equal to 18 which is the definition of a consistent 18 x 18 matrix. This reduces the consistency index and thus the consistency ratio to zero meaning absolutely consistent data. 4. Results Figure 6 shows the right Eigenvector (transposed to horizontal) which is the prioritization of the 18 linkages. Further, note how they loop back to Figure 1 where they are shown as percentages. This was a way to extract the initial weighting of the larger model by using readily available data in the form of a large body of expert documentation. From here, the larger model was successfully tested. This is as expected with a perfectly consistent matrix. Figure 6 Right Eigenvector (transposed) or the list of the priority for each linkage 5. Validation This method applied to the work at hand demands to be validated. A sample of experts was surveyed using traditional AHP processes to confirm the variation as described. An AHP survey of four US Army Master Gunners was conducted using a pair-wise comparison tool. The tool asked them to respond to 153 pair-wise comparisons allowing responses from 1 to 9. These pairs were randomized in order to prevent long sections focusing on only one linkage. The total matrix has 324 cells. The diagonal of all ones representing items compared to themselves were automatically valued at one. Because aij = 1/aji, the remaining 306 cells were divided in half. The 153 pairs were, perhaps Ranking I A K B H C E O G F P D J M R N L Q Priority 0.215 0.197 0.097 0.093 0.048 0.043 0.040 0.040 0.032 0.027 0.024 0.024 0.022 0.020 0.020 0.019 0.018 0.018 IJAHP Article: Repetski, Sarkani, Mazzuchi / Applying the Analytic Hierarchy Process (AHP) to expert documents International Journal of the Analytic Hierarchy Process 10 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.919 obviously, the most daunting number of comparisons required. This created a longer tool that attempted to reduce iteration by being more intense at the beginning. These pairwise matrices were entered into the same tool as the literature described earlier with their comparisons being entered into the tables directly as ratios. Each was independently exposed to a consistency test and all four achieved CR values below 0.1. Further, these used a RI value of 1.59 which is the value of RI for n = 15. Given the slowly increasing value of RI, this was reasonably close but still a conservative approach. All four expert opinions proved sufficiently consistent on the first iteration to be usable. The value of the CR for the four sources and the combined opinion matrix is shown in Figure 7. Figure 7 CR values in validation The combined results are shown in Figure 8. The percentage values and ordinals show variance between the two surveys, but the graph in Figure 9 tells the complete story. The most important connection, linkage I, is the same in both surveys confirming that the model for the larger study remained unchanged. However, some values make large ordinal changes; B drops from 4 to 11, H drops from 5 to 14, O climbs from 8 to 2, while Q climbs from 18 to 12. These appear to be wild swings; however, the graph helps to clarify the data. Figure 9 graphs the two sets of results; the dashed line represents the doctrinal AHP results and the solid line represents the validation survey results. Notice how flat the AHP survey results are compared to the doctrinal extract AHP. The range is more than doubled. The graphs suggest an amplified signal. The similar shapes of the two validate the doctrinal method, even endorse it. The issue with the survey results is that they are much flatter, a small fluctuation in a value moves it wildly up or down the rank ordering. This may be due to the difference between using a 1 to 9 ratio scale for the survey instead of the 1 to 256 scale for the doctrinal extraction method. This appears to have amplified the results producing greater clarity. The smaller scale allowed for many elements to be in a very narrow band in the middle of the rankings. The larger scale spread those elements out while tightening the values toward the bottom. The consistent shape of the two curves allows the doctrinal extraction method to be used for this model. IJAHP Article: Repetski, Sarkani, Mazzuchi / Applying the Analytic Hierarchy Process (AHP) to expert documents International Journal of the Analytic Hierarchy Process 11 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.919 Figure 8 Table of results for the doctrinal tool and the validation surveys IJAHP Article: Repetski, Sarkani, Mazzuchi / Applying the Analytic Hierarchy Process (AHP) to expert documents International Journal of the Analytic Hierarchy Process 12 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.919 Figure 9 Graph of AHP values for doctrinal literature and validation surveys 6. Conclusions As systems thinking moves into domains where a large body of expert documentation exists, the AHP can be usefully applied to literature to extract embedded expert opinions. While the practitioner must be careful to select appropriate methods to measure opinions, well-selected metrics can readily convert those opinions into priorities or quantitative recommendations. A formal proof of this process should be pursued. Applications in the area of this study, operational integration and operational integration testing of human and equipment organizations, are legion. These are domains where expert opinion has long held sway and those opinions have been memorialized in texts, reviews, studies, etc. The ability to quantify the opinions on the most important issues in after-action reports or prioritizing steps in a process provides the ability to better focus time and other resources to more effectively, even optimally, address issues. The proposed process enables studies to exploit the corpus of existing documentation to inform more reasoned applications to solving problems with a labor and time saving tool. Further, the validation process suggests that the literature extraction method could be used in other domains where large bodies of documents written by experts can be put to use. IJAHP Article: Repetski, Sarkani, Mazzuchi / Applying the Analytic Hierarchy Process (AHP) to expert documents International Journal of the Analytic Hierarchy Process 13 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.919 REFERENCES Basak, I. (2020). Estimation of priority weights based on a resampling technique and a ranking method in analytic hierarchy process. Journal of Multi-Criteria Decision Analysis, 27(1โ€“2), 61โ€“64. Doi: https://doi.org/10.1002/mcda.1664 Boender, C. G. E., de Graan, J. G., & Lootsma, F. A. (1989). Multi-criteria decision analysis with fuzzy pairwise comparisons. Fuzzy Sets and Systems, 29(2), 133โ€“143. Doi: https://doi.org/10.1016/0165-0114(89)90187-5 Ergu, D., Kou, G., Peng, Y., Shi, Y., & Shi, Y. (2013). The analytic hierarchy process: Task scheduling and resource allocation in cloud computing environment. Journal of Supercomputing, 64(3), 835โ€“848. Doi: https://doi.org/10.1007/s11227-011-0625-1 Garuti, C. (2018). Reflections on common misunderstanding when using AHP and a response to criticism of Saatyโ€™s consistency index. International Journal of the Analytic Hierarchy Process, 10(3), 488โ€“501. Doi: https://doi.org/10.13033/ijahp.v10i3.573 Garuti, C., & Sandoval, M. (2006). The AHP: A multicriteria decision making methodology for shiftwork prioritizing. Journal of Systems Science and Systems Engineering, 15(2), 189โ€“200. Doi: https://doi.org/10.1007/s11518-006-5007-5 Liu, H., Xu, Z., & Liao, H. (2016). The multiplicative consistency index of hesitant fuzzy preference relation. IEEE Transactions on Fuzzy Systems, 24(1), 82โ€“93. Doi: https://doi.org/10.1109/TFUZZ.2015.2426315 Lootsma, F. A. (1980). Saatyโ€™s priority theory and the nomination of a senior professor in operations research. European Journal of Operational Research, 4(6), 380โ€“388. Doi: https://doi.org/10.1016/0377-2217(80)90189-7 Lootsma, F. A. (1999a). Letter to the editor: The expected future of MCDA. Journal of Multi-Criteria Decision Analysis, (8), 59โ€“60. Retrieved from https://search.proquest.com/docview/220297257?accountid=12834 Lootsma, F. A. (1999b). Multi-Criteria Decision Analysis via ratio and difference judgement. Springer. Russo, R. D. F. S. M., & Camanho, R. (2015). Criteria in AHP: A systematic review of literature. Procedia Computer Science, 55(Itqm), 1123โ€“1132. Doi: https://doi.org/10.1016/j.procs.2015.07.081 Saaty, T. L. (1988). The Analytic Hierarchy Process: Planning, priority setting and resource allocation (2nd ed.). Doi: https://doi.org/0-07-054371-2 Saaty, T. L. (2013). Fundamentals of decision making and priority theory with the Analytic Hierarchy Process (ebook Seco). Pittsburgh, PA, USA: RWS Pubications. Saaty, T. L., & Sagir, M. (2009). Extending the measurement of tangibles to intangibles. IJAHP Article: Repetski, Sarkani, Mazzuchi / Applying the Analytic Hierarchy Process (AHP) to expert documents International Journal of the Analytic Hierarchy Process 14 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.919 International Journal of Information Technology and Decision Making, 8(1), 7โ€“27. Doi: https://doi.org/10.1142/S0219622009003247