IJAHP Article: Vachnadze/Prioritization of performance measures using Analytic Hierarchy Process International Journal of the Analytic Hierarchy Process 490 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.442 PRIORITIZATION OF PERFORMANCE MEASURES USING ANALYTIC HIERARCHY PROCESS Revaz Vachnadze Full Professor, ESM Business School, Free University of Tbilisi; Visiting Professor, Georgian Institute of Public Affairs; Tbilisi, Georgia r.vachnadze@freeuni.edu.ge, revaz.vachnadze@gipa.ge ABSTRACT The purpose of this paper is to provide an integrated approach that prioritizes organizational performance measures and critical success factors towards the strategic objectives and initiatives of a firm. The Analytic Hierarchy Process (AHP) pair-wise comparisons and hierarchic composition technique is used to prioritize the key performance indicators (KPIs) and the key result indicators (KRIs) as well as the critical success factors (CSFs) of the organization within the frame of a single hierarchy. The new model presented in the paper will be more suitable for dealing with the problem than the others which are available. The application of the suggested model will enable staff to more closely align their daily activities to the strategic objectives of the firm. The suggested approach allows for a wide applicability to different types of organizations (business, nonprofit, public) and its use could significantly improve resource allocation and the overall performance in organizations. Keywords: Performance measures; Analytic Hierarchy Process; key performance indicators; key result indicators; critical success factors 1. Introduction Organizational performance measurements and measures have attracted a great deal of attention among scholars and practitioners in different spheres of human activities. Of particular note is that nonfinancial factors have received more importance in recent years in measuring the overall performance of any firm. Therefore, the inclusion of nonfinancial factors such as higher customer satisfaction (delight), effective management and leadership, using more advanced technology in operations, etc., makes valuable contributions to the measurement of the overall performance of organizations rather than limiting the measurement to financials alone. In spite of the wide attention given to performance measures, there is a certain misunderstanding and mixing up of the measures. Therefore, to avoid these drawbacks this paper uses the approach based on Parmenter (2015) where performance measures are clearly classified, thereby considerably improving their applicability. In the presence of a scarcity of resources, which is a common phenomenon, it is necessary to obtain more relevant measures and pay attention to those having a higher priority. So, along with the development and establishment of relevant characteristics, mailto:r.vachnadze@freeuni.edu.ge mailto:revaz.vachnadze@gipa.ge IJAHP Article: Vachnadze/Prioritization of performance measures using Analytic Hierarchy Process International Journal of the Analytic Hierarchy Process 491 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.442 such as key performance indicators (KPIs), key result indicators (KRIs), and critical success factors (CSFs), the prioritization of these characteristics should be strongly emphasized. Performance measures and their prioritization have been the subject of interest of many studies (e.g.,Chen, 1999; Ho & Zhu, 2004; Salmeron & Herrero, 2005; Shahin & Mahbod, 2007; Survadi, 2007; Teker et al., 2011; etc.). Inspired by Kaplan and Norton’s Balanced Scorecard (Kaplan & Norton, 1996), these studies paid attention not only to financial but to nonfinancial perspectives (Customer, Internal Processes, and Learning and Growth) as well. The prioritization of KPIs, KRIs and CSFs should be viewed as a multi-criteria decision making problem which may be solved using Thomas Saaty’s Analytic Hierarchy Process (AHP) which is referred to as one of the most powerful and widely used techniques for decision making. The applicability of the AHP here is especially worthy due to the presence of tangible as well as intangible measures. In this paper, the elaboration of the model and a corresponding method are suggested which will enable staff to more closely align their daily activities to the strategic objectives and initiatives of the firm. This will be achieved by properly selected and prioritized characteristics. The suggested approach is preceded by an example of a football team’s preparation for a championship. 2. Critical success factors and performance measures According to performance improvement thought leader Dean Spitzer, measurement is fundamental to high performance, improvement, and, ultimately, success in business or in any other area of human endeavor and it is potentially one of the highest leverage activities any organization can perform (Spitzer, 2007).Today the most popular measures of organizational performance for many organizations worldwide are deployed using the term: key performance indicator (KPI). Unfortunately, in many practical cases performance measurement is failing. This is due to the fact that these organizations are working with the wrong measures, many of which are incorrectly termed key performance indicators (KPIs). The measures that have usually been adopted have no link to the critical success factors (CSFs) of the organizations (Parmenter, 2015). Critical success factors can be defined as the list of issues or aspects of organizational performance that on the whole determine ongoing health, vitality and well-being. We should emphasize that the main purpose of performance measures is to ensure that staff members spend their working hours focused primarily on the organization’s CSFs. It is the CSFs, and performance measures within them, that link daily activities to the organization’s strategies/goals. Being aware of the significance of a well thought through and executed strategy is the responsibility of a selected group of senior executives in the organization whereas the critical success factors should be the daily focus of all of the staff in the organization as this will positively impact the strategic initiatives (Parmenter, 2015). One of the most important roles of management is to communicate expectations to the workforce. In fact, people will do what management inspects (measures) and not necessarily what management expects (Spitzer, 2007). Thus, the right measures need to be put in place. KPIs are the main things that truly link day-to-day performance in the workplace to the organization’s CSFs. IJAHP Article: Vachnadze/Prioritization of performance measures using Analytic Hierarchy Process International Journal of the Analytic Hierarchy Process 492 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.442 In representing performance measures as a set we can pick out two subsets from within: result indicators (RIs) and performance indicators (PIs). Parmenter (2015) calls these subsets groups. When we use the term result indicators, we emphasize the fact that these measures are a summation or a result of more than one team’s efforts. Unfortunately, these indicators usually do not help management fix a problem as it is difficult to bring out which teams were responsible for high or low performance. As an example, financial indicators are a result of different activities and so financial performance measures are result indicators. Financial indicators are useful, but they do not reveal the real drivers of performance. To fully understand what to change, we need to look at the activities that created the financial indicator. Performance indicators, on the other hand, are measures that can be linked to a team or a cluster of teams working closely together for a common purpose. In this case, high or low performance is now the responsibility of one team. Obviously, some measures are more important and so picking out corresponding subsets from RIs and PIs and using the extra word “key” we come to: Key result indicators (KRIs) - measures which give an overall summary of how the organization is performing. KRIs display a clear picture of whether or not your organization is moving in the right direction and at the right speed. They provide the board of a governing body with a good overview as to progress with regard to the organization’s strategy. Separating KRIs from other measures has an important impact on reporting. This results in a separation of performance measures into those impacting governance (KRIs) and those impacting management (KPIs, PIs, RIs). Key performance indicators (KPIs) – measures which show management how the organization is performing on their critical success factors (CSFs) and by monitoring them management is able to increase performance dramatically. KPIs focus on the aspects of organizational performance that are the most critical for the current and future success of the organization (Parmenter, 2015). 3. Analytic Hierarchy Process (AHP) In the 1970s, Thomas L. Saaty developed an elegant approach to help decision makers in modeling complex problems in a simple way. His textbook, entitled “The Analytic Hierarchy Process” and the PC-based software, entitled “Expert Choice” helped popularize the process amongst operations research practitioners (Saaty, 1980; Expert Choice, 2002). In the 40 years since the publication of the first papers, books and software, AHP has been used by decision makers all over the world to model problems in diverse areas including resource allocation, strategic planning, public policy, etc. It has been used to rank, select, evaluate and benchmark a wide variety of decision alternatives (Saaty, 2008). AHP has been used by organizations in both the public and private sectors to deal with complex problems, and it has had a significant impact on the practice of decision making. Operations research practitioners around the world have repeatedly embraced AHP as a methodology that can produce insightful results for difficult, real-world decision problems. For more than two decades, AHP has been taught as an important part of the curriculum covering decision making techniques at business and engineering schools worldwide. It has been incorporated into mainstream operations research college- level textbooks (e.g.,Anderson et al., 2012; Liberatore and Nydick, 2003) and commercial software packages (e.g.,Criterion Decision Plus available at www.infoharvest.com). A wide range of AHP applications and software packages have been catalogued, categorized, and annotated in edited volumes and books http://www.infoharvest.com/ IJAHP Article: Vachnadze/Prioritization of performance measures using Analytic Hierarchy Process International Journal of the Analytic Hierarchy Process 493 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.442 (e.g.,Golden et al., 1989; Saaty and Vargas, 2000), and in journal articles (e.g., Forman and Gass, 2001; Vaidya and Kumar, 2006; Ishizaka and Labib, 2009, 2011; McGinley, 2012). AHP is based on the following three principles: decomposition, comparative judgment, and a synthesis of priorities. It is a theory of measurement for dealing with quantifiable and intangible criteria that has been applied to numerous areas. It generally involves three steps. The first step is to structure the problem into a hierarchical framework with successive levels of goal, criteria and alternatives. The alternatives are placed at the bottom level. Such structuring requires some experience with AHP techniques, but the following guidelines are helpful: 1) Start structuring top down – Specify an overall goal first and then criteria and the alternatives that have an impact on the goal or which will help to achieve that goal. 2) Comparison analysis – Once the hierarchy has been structured, the second step is to establish ratio priorities for each node of the hierarchy. This is done through pair- wise comparisons of the child items below a parent node. The comparisons are done with respect to the importance or contribution of the item to the parent node. Hence, this comparison analysis is generally conducted from bottom to top. Once sufficient comparisons have been made for a node, the principal eigenvector of the comparison matrix is standardized so that it sums to one and becomes the ratio measure of the relative importance of each item. Since these priorities reflect the relative importance of only the items below a parent node, they are called local weights. 3) Aggregate the local weights into a composite priority – This is the AHP’s final step and is done through the principle of hierarchic composition that first multiplies local weights by the product of all higher-level priorities. Within the hierarchy, this process transforms the local weights into global weights that measure the importance of each node in the total hierarchy (Figure 1). Figure1. AHP’s three-level hierarchy (level 1 - Goal, level 2 - Criteria, level 3 - Alternatives). Here, n and m describe number of criteria and alternatives accordingly. 4. Suggested approach Let us examine a hypothetical example of the hierarchy with the focus/goal being a football team’s preparation for a championship in order to achieve a better place than in the previous year (Figure 2). . . . Goal Criterion 1 Criterion 2 Criterion n Alternative 1 Alternative 2 Alternative 3 Alternative m . . . IJAHP Article: Vachnadze/Prioritization of performance measures using Analytic Hierarchy Process International Journal of the Analytic Hierarchy Process 494 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.442 Figure 2. Four-level hierarchy of the football team’s preparation for a championship Critical success factors (criteria) for this focus/goal could be physical (s1), technical (s2), tactical (s3) and psychological (s4) preparation/training. Using the eigenvector method of AHP we obtain CSF’s local weights towards the goal. It is important to note that the pair-wise comparison matrix should be filled by the manager (head coach) of the team. These judgments reflect his/her professional and personal conception of football and it is not surprising that calculated weights could differ for different coaches. Suppose that our manager has the following priorities: s1=0.2, s2=0.2, s3=0.3, s4=0.3. The next level down the hierarchy consists of the two elements which are Generalized Key Result Indicator (GKRI) and Generalized Key Performance Indicator (GKPI).The GKRI shows how good or bad the condition of the team is as a result of its pre-season preparation, whereas the GKPI shows the consequences of the preparation later - in the course of the championship. The appearance of the GKRI and the GKPI here is caused by this specific example. Their presence in some cases may cause certain difficulties during the judgment process and so they may not be necessary in these cases. Naturally, comparative judgments on the GKRI and the GKPI in each specific case can be different. For each CSF the manager together with his/her assistants form pair-wise comparison matrices and calculate weights prij of the j-th indicator(j=1,2) towards i-th factor (i=1,2,3,4). Suppose that prij–s are as follows: pr11= 0.6, pr12=0.4, pr21=0.4, pr22=0.6, pr31=0.3, pr32=0.7, pr41=0.5, pr42=0.5. Taking into account weights of the CSFs, for the GKPI with respect to the focus we obtain weight P=∑sj*pr1j=0.44 and for the GKRI we obtain R=∑sj*pr2j=0.56. Focus/Goal: Preparation for a championship CSF1 Physical Training GKPI - Generalized Key Performance Indicator KPI1– Fitness KPI2– Trainings with the Ball, Training Games KPI3 – Theoretical Lessons KPI4– Creation of the Mutual Respect and Trust in The Team. KRI1 – Mobility, Endurance KRI2- “Intimacy” With the Football, Sight of the Playing Field, Improvisation KRI3 – Discipline, Carrying out Tasks Given by the Coach KRI4 – Purposefulness GKRI - Generalized Key Result Indicator CSF2 Technical Training CSF3 Tactical Training CSF4 Psychological Training IJAHP Article: Vachnadze/Prioritization of performance measures using Analytic Hierarchy Process International Journal of the Analytic Hierarchy Process 495 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.442 At the bottom level there are different KPIs and KRIs, which separately affect the GKPI and the GKRI. Suppose that they are as follows: For KPIs: KPI1(u1) – fitness; KPI2 (u2) – trainings with the ball, training games; KPI3 (u3) – theoretical lessons; KPI4 (u4) –creation of the mutual respect and trust in the team. For KRIs: KRI1(v1) – mobility, endurance; KRI2 (v2) –“intimacy” with the ball, sight of the playing field, improvisation; KRI3 (v3) – discipline, carrying out tasks given by the coach; KRI4 (v4) – purposefulness. Both pair-wise comparison matrices are to be filled by the concerted judgments of the manager, his/her assistants, trainer responsible for the team’s physical preparation, and the consultant - psychologist. Here, the requirement for consensus should be obligatory; you cannot require coordinated actions from the players when the trainers themselves cannot come to unanimity. Suppose that we have the following weights for the KPIs and the KRIs: u1=0.15, u2=0.35, u3=0.3, u4=0.2and v1=0.2, v2=0.25, v3=0.3, v4=0.25. Finally, we calculate the global weights (composite priorities) wi of the key indicators with respect to the focus. w1=P*u1=0.44*0.15=0.066, w2=P*u2=0.44*0.35=0.154, w3=P*u3=0.44*0.3=0.132, w4=P*u4=0.44*0.2=0.088; w5=R*v1=0.56*0.2=0.112, w6=R*v2=0.56*0.25=0.14, w7=R*v3=0.56*0.3=0.168, w8=R*v4=0.56*0.25=0.14. Note that ∑wi=1, and the average is 1/8=0.125 Calculated weights allow the manager to realize how the strategy for the team’s preparation plan for the championship corresponds to his/her conception (which is reflected in his/her weights of the CSFs) and to check its effectiveness. It should be emphasized that the weight vector w may present the necessary condition for reaching success, but it is far from being sufficient. To fully execute this task, it is necessary to have a more comprehensive model, which would consider such aspects as the financial resources of the team/club, the roster, individual fitness, operative control, medical control, etc. These aspects are beyond the scope of this paper and should be the object of special research. All the numbers obtained above are conditional, as they might be allowable within the framework of an illustrative example. But even here something must be noted; if we settle on calling the sums of the similar elements of the vector w “actual weights”, then we obtain: Actual weight of the physical training (process+result): w1+w5=0.066+0.112=0.178; Actual weight of the training in technical skills (process+result): w2+w6=0.154+0.14= 0.294; Actual weight of the training in tactics (process+result): w3+w7=0.132+0.168=0.3; Actual weight of the psychological training (process+result): w4+w8=0.088+0.14=0.228 IJAHP Article: Vachnadze/Prioritization of performance measures using Analytic Hierarchy Process International Journal of the Analytic Hierarchy Process 496 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.442 Naturally, the numbers obtained differ from the CSF’s weights, which can be viewed as global towards the whole process of the team’s management, whereas “actual weights” relate only to the stage of the team’s preparation for the season and its actual performance. Note that if the results obtained by the team do not meet the manager’s expectations, then he/she should make appropriate corrections in the process of the team’s preparation for the next season. The hierarchy of our example is shown with the obtained local weights in Figure 3. Figure 3. Four-level hierarchy of the football team’s preparation for a championship, with obtained local weights Let us turn to the general case of the prioritization of the performance measures. Similar to Shahin and Mahbod (2007), the approach in this paper is based on using Saaty’s AHP. Shahin and Mahbod use the three-level hierarchy with the SMART conception’s components (Specific, Measurable, Achievable, Relevant and Time- based) as criteria and KPIs as alternatives. Taking into consideration the above mentioned importance of CSFs in establishing key performance measures, this paper suggested the addition of a CSFs level to form a four-level hierarchy of the following type (see Figure 4). Focus/Goal: Preparation for a championship CSF1 Physical Training 0.2 GKPI - Generalized Key Performance Indicator 0.44 KPI1– Fitness 0.15 KPI2– Trainings with the Ball, Training Games 0.35 KPI3 – Theoretical Lessons 0.3 KPI4– Creation of the Mutual Respect and Trust in the Team. 0.2 KRI1 – Mobility, Endurance 0.2 KRI2- “Intimacy” With the Football, Sight of the Playing Field, Improvisation 0.25 KRI3 – Discipline, Carrying out Tasks Given by the Coach 0.3 KRI4 – Purposefulness 0.25 GKRI - Generalized Key Result Indicator 0.56 CSF2 Technical Training 0.2 CSF3 Tactical Training 0.3 CSF4 Psychological Training 0.3 IJAHP Article: Vachnadze/Prioritization of performance measures using Analytic Hierarchy Process International Journal of the Analytic Hierarchy Process 497 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.442 Figure 4. Suggested four-level hierarchy(level 1 - Goals/Strategies, level 2 - SMART criteria, level 3 – CSFs, level 4 - KPIs and KRIs) In the suggested model each of the KPI and KRI subsets is to be prioritized separately so, essentially we will have a hierarchy with two different sets of alternatives. The CSFs, KPIs and KRIs can be found by using the procedures suggested in the references (Barr, 2014; Parmenter, 2015). Best practice recommends limiting the number of CSFs, KPIs and KRIs to 10 each (Parmenter, 2015). In Figure 4, the numbers of CSFs, KPIs and KRIs are taken willfully. For any type of organization, each KPI or KRI should not necessarily affect every CSF and so not every child item (alternative) of the hierarchy should be connected to every parent node in this case. The hierarchy of Figure 4 is not complete which is fine because AHP does not require a complete hierarchy. The presence of several connections of each KPI and KRI with different CSFs reflects the fact that these measures should have a significant impact on the organization (e.g., KPI or KRI impacts on more than one of the top CSFs). Due to the positive impact on performance, KPIs encourage appropriate action, whereas poorly thought through measures can lead to dysfunctional behavior. The model of Figure 4 can be substantially simplified. In practice, well formulated organizational goals/strategies are usually a priori stated in a SMART mode; therefore, the hierarchy of Figure 4 can be reduced by the elimination of the level 2 - SMART criteria, thus transforming to the following three-level hierarchy (Figure 5). Goals/Strategies Specific Measurable Achievable Relevant Time-based CSF 1 CSF 2 CSF 3 CSF 4 KRI-1 KRI-2 KRI-3 KPI-1 KPI-2 KPI-3 KPI-4 IJAHP Article: Vachnadze/Prioritization of performance measures using Analytic Hierarchy Process International Journal of the Analytic Hierarchy Process 498 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.442 Figure 5.Suggested three-level hierarchy. In this case, the willfully taken number of CSFs is 4 whereas numbers of KRIs and KPIs are 3 and 4 accordingly. The suggested approach is rather easily operationally applicable for different types of problems, as one could notice in the example shown at the beginning of this section. This approach is now applied to the prioritization of performance measures of a higher education institution in Georgia. 5. Discussions and conclusions In this paper a novel integrated approach is proposed using Thomas Saaty’s Analytic Hierarchy Process (AHP) to prioritize key performance measures of organizations; namely, key performance indicators (KPIs) and key result indicators (KRIs), as well as critical success factors (CSFs) towards the strategic objectives and initiatives of the firm. The prioritization of KPIs, KRIs and CSFs allows the organization’s governing bodies and management to focus scarce resources on the top priorities/issues. The separation of KRIs and KPIs might reflect the degree of interest of the governing body (board) and the management in them so that a governance report should consist of a list of prioritized KRIs whereas KPIs are to be of prime interest for CEOs, management and staff. The advantages of the proposed approach are listed below. • The proposed approach is practically applicable to any type of organization of any size in any particular area. • The evaluation of KPIs and KRIs by the integration of AHP and CSFs goal setting can take both quantitative and qualitative factors into consideration. • KPIs and KRIs can be arbitrary and subjective. They have significant impact on more than one of the top CSFs. • The proposed approach makes it possible to involve all informed persons in establishing priorities for the indicators and reaching a dynamic group decision to obtain the final weights. CSF 1 CSF 2 CSF 3 CSF 4 KRI-1 KRI-2 KRI-3 KPI-1 KPI-2 KPI-3 KPI-4 Goals/Strategies IJAHP Article: Vachnadze/Prioritization of performance measures using Analytic Hierarchy Process International Journal of the Analytic Hierarchy Process 499 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.442 • The AHP approach is well structured and holistic. The pair-wise comparisons make multi criteria decision making possible and ensure the analyzer that comprehensive decision making has been undertaken based on prioritized CSFs. The decision making process is rational and consistent and aids objectivity and understanding. • Flexibility of the proposed approach does not require a repeat of all of the judgments when a change in the model, such as the addition or removal of the KPI, KRI, or CSF, is made. At the same time, there are some limitations and subjective factors by which the results obtained using this approach might be influenced. They are as follows: • A variation in the views of the people participating in the judgment process of the prioritization procedure (e.g., consistency ratios) might lead to differing results. • The accuracy of the suggested approach is limited by estimates obtained in the processes of judgments in the AHP and the actual finding of the right performance measures. • It seems that the categorization of the performance measures most likely has to be carried out within the framework of Zadeh’s fuzzy sets (Dubois &Prade, 1988). • The proposed approach helps to determine which dimensions require improvement, but it does not provide guidance on the actions to be taken. These last two issues might be objects for further studies. IJAHP Article: Vachnadze/Prioritization of performance measures using Analytic Hierarchy Process International Journal of the Analytic Hierarchy Process 500 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.442 REFERENCES Anderson, D., Sweeney, D., Williams, T., Camm, J., and Martin, K. (2012). An introduction to management science: Quantitative approaches to decision making, revised 13 th edition. Cincinnati, OH: South-Western Cengage Learning. Barr, S. (2014).Practical performance measurement: Using the PuMP blueprint for fast, easy and engaging KPIs. The PuMP Press. Chen, T. (1999). Critical success factors for various strategies in the banking industry. International Journal of Bank Marketing, 17(2), 83-91. Doi: http://dx.doi.org/10.1108/02652329910258943 Chiu, C.R., et al. (2014). The performance of commercial banks based on a context- dependent range-adjusted measure model. International Transactions in Operational Research, 21(5), 761-778. Doi: 10.1111/itor.12069 Dubois, D. and Prade, H. (1988). Fuzzy sets and systems. New York: Academic Press. Expert Choice (2002). Expert Choice 2000.Pittsburgh, PA: Expert Choice Inc. Forman, E. and Gass, S. (2001). The Analytic Hierarchy Process: An exposition. Operations Research, 49, 469-486. Doi: http://dx.doi.org/10.1287/opre.49.4.469.11231 Golden, B., Wasil, E. and Harker, P. (1989).The Analytic Hierarchy Process: Applications and studies. Berlin: Springer. Ho, C, and Zhu, D. (2004). Performance measurement of Taiwan’s commercial banks. International Journal of Productivity and Performance Management, 53(5), 425-434. Doi: http://dx.doi.org/10.1108/17410400410545897 Ishizaka, A, and Labib, A. (2009). Analytic Hierarchy Process and Expert Choice: Benefits and limitations. OR Insight, 22, 201-220.Doi: 10.1057/ori.2009.10 Ishizaka, A, and Labib, A. (2011). Review of the main developments in the Analytic Hierarchy Process. Expert Systems with Applications, 38, 14336-14345. Doi: http://dx.doi.org/10.1016/j.eswa.2011.04.143 Kaplan, R.S. and Norton D.P. (1996).The balanced scorecard: Translating strategy into action. Cambridge, MA: Harvard Business Press. Doi:10.1016/S0737-6782(97)89410-6 Liberatore, M. and Nydick, R. (2003). Decision technology: Modeling, software, and applications. New York, NY: Wiley. McGinley, P. (2012).Decision analysis software survey. OR/MS Today, 39. Parmenter, D., (2015). Key Performance Indicators: Developing, implementing, and using winning KPIs, Third Edition. Wiley. Doi: 10.1002/9781119019855 Saaty, T.L. (1980).The Analytic Hierarchy Process. McGraw-Hill, New York, NY. Doi: http://dx.doi.org/10.1080/00137918308956077 http://dx.doi.org/10.1287/opre.49.4.469.11231 http://dx.doi.org/10.1108/17410400410545897 http://dx.doi.org/10.1016/j.eswa.2011.04.143 http://dx.doi.org/10.1016/S0737-6782(97)89410-6 http://dx.doi.org/10.1080/00137918308956077 IJAHP Article: Vachnadze/Prioritization of performance measures using Analytic Hierarchy Process International Journal of the Analytic Hierarchy Process 501 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.442 Saaty, T.L. (2008), Decision-making with the analytic hierarchy process. International Journal of Services Science, 1(1), 83-98. Doi: 10.1504/IJSSCI.2008.017590 Saaty, T.L. and Vargas, L. (2000).Models, methods, concepts and applications of the Analytic Hierarchy Process. Boston, MA: Kluwer. Salmeron,J.L. and Herrero,I. (2005). An AHP-based methodology to rank critical success factors of executive information systems. Journal of Computer Standards & Interfaces, 28(1), 1-12. Doi: http://dx.doi.org/10.1016/j.csi.2004.09.002 Shahin,A. and Mahbod,M. (2007). Prioritization of key performance indicators: An integration of analytical hierarchy process and goal setting. International Journal of Productivity and Performance Management, 56(3), 226-240. Doi: http://dx.doi.org/10.1108/17410400710731437 Spitzer, D. (2007).Transforming performance measurement: Rethinking the way we measure and drive organizational success. New York: AMACOM. Suryadi, K. (2007). Key performance indicators measurement model based on AHP and trend-comparative dimension in higher education institution, ISAHP 2007, Vina Del Mar, Chile. Teker S., Teker D. and Kent O. (2011).Measuring commercial banks performances in Turkey: A proposed model. Journal of Applied Finance & Banking, 1(3), 97-112. Vaidya, O. and Kumar, S. (2006). Analytic hierarchy process: An overview of applications. European Journal of Operational Research, 169, 1-29. Doi: http://dx.doi.org/10.1016/j.ejor.2004.04.028 http://dx.doi.org/10.1504/IJSSCI.2008.017590 http://dx.doi.org/10.1016/j.csi.2004.09.002 http://dx.doi.org/10.1016/j.ejor.2004.04.028