IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 1 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 PRIORITIZING CRITERIA TO EVALUATE PROJECT SUCCESS: MODELING WITH THE ANALYTIC HIERARCHY PROCESS (AHP) Luciano Azevedo de Souza Universidade Federal Fluminense (UFF) Brazil lucianos@id.uff.br Helder Gomes Costa Universidade Federal Fluminense (UFF) Brazil Helder.hgc@gmail.com Hernando Oliveira de Araujo Universidade Federal Fluminense (UFF) Brazil fernandoaraujo@id.uff.br ABSTRACT Despite numerous attempts to systematize the evaluation of project success, the topic remains unaddressed, mainly because of the lack of appropriate models for dealing with the subjectivity associated with evaluation. This paper aims to contribute to this discussion by proposing a model for determining the relative importance of the criteria based on a multi-criteria technique (AHP). A core feature of the AHP is determining the relative weights of the criteria, considering the subjectivity associated with the problem. The proposed model was applied to a set of data collected through structured interviews from a sample of 54 respondents consisting of managers and project professionals in a given organization. The criteria with the highest priorities were 'learning opportunities' (20.4%), 'scope' (15.8%) and 'innovation' (14.1%). Unexpectedly, the criteria ‘cost’, ‘schedule’, and ‘scope’, although widely used in evaluating success, did not rank as most important. This proposed prioritization can be useful to top management when making decisions about the application of resources that contribute to the success of the projects in the organization, as well as to guide project managers as they decide what actions are necessary to address the most relevant aspects in the context of the organization. Keywords: project performance; project success criteria; project success; multicriteria decision analysis (MCDA); Analytic Hierarchy Process (AHP) 1. Introduction One of the problems faced by organizational leaders is how to evaluate the success of their projects. It is necessary for leaders to be able to determine the relative importance of the factors or criteria that influence the success of projects. Despite existing efforts to mailto:lucianos@id.uff.br mailto:Helder.hgc@gmail.com mailto:fernandoaraujo@id.uff.br IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 2 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 address this problem, this question remains, mainly due to the lack of adequate models to deal with the subjectivity present in evaluation. There are several published works that discuss establishing criteria weights in the context of project evaluation including da Silva et al. (2021), Roy, Das, Kar and Pamučar (2019), Šenitková, Burdová and Vilčeková (2010), Yi, Li and Zhang (2019) and Zagonari (2016). However, despite the great advances achieved in these studies, most of them do not use modeling based on trade-offs for the treatment of the data collected regarding the importance of criteria. It also appears that these works are not based on specific techniques that recognize and address subjectivity, despite the existence of multicriteria decision making (MCDM) which is specifically developed for decisions in subjective environments. According to Keeney and Raiffa (1993) and Saaty (1980), eliciting weights for the importance of criteria involves the use of perception discovery methods based on trade-off administration. This work fills the gap in the literature by proposing and applying the Analytic Hierarchy Process (AHP), which is a MCDM method supported by a trade-off algorithm to obtain the relative importance of the weights of the project evaluation criteria. This is based on a multicriteria technique that has at its core the attribution of relative weights of criteria considering the subjectivity inherent to the problem (Rocha, Barros, Silva & Costa, 2016). In general, this study answers the following research questions:  What are the criteria for evaluating success in projects reported in the scientific literature?  What is the relative importance attributed to each criterion found in the literature from the perspective of project management specialists, executives and others involved in the projects? In order to answer these questions, some procedures were performed, and the methods applied are described in section 2. In section 3, we present the application of the AHP approach. In section 4, we discuss the results, the practical applications, and limitations of the study, and finally, section 5 presents the conclusion with suggestions of further studies. 2. Methods This study was organized in the following stages: literature review, (project success criteria), construction of the data collection instrument, data collection, construction of criteria, hierarchy of criteria, pairwise judgment of value from the data obtained in the survey, evaluation of the consistency of the pairwise evaluations, and prioritization analysis (from eigenvectors). These stages are presented in Figure 1. IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 3 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 Figure 1 Procedure flow 2.1 Literature review To further understand project evaluation criteria, bibliographic research was carried out through a literature review of papers published in journals indexed in Scopus or in Web of Sciences (WOS). According to Rodriguez, Costa and Carmo (2013), these selection criteria avoid the use of non-peer reviewed articles and reduce the probability of taking into account the so called “gray literature” (Rothstein & Hopewell, 2009) or even predatory sources. 2.2 Construction of the data collection instrument To understand the degree of application of each of the criteria identified in the literature, respondents were asked to choose the alternative that would best represent their opinion regarding the use of these criteria in the organization according to the following scale: (1) Strongly disagree; (2) Disagree; (3) Neither agree / disagree; (4) Agree; (5) Strongly agree. 2.3 Data collection Based on the theoretical findings, the survey was carried out by direct questioning of those involved in project management in the organization by asking about their experience in the organization related to project performance. The questionnaires contained objective questions about their experience with projects in the organization, specifically about the criteria for evaluating their performance in the projects. 2.4 Prioritizing the criteria The AHP was applied to assign weights to each criterion using the eigenvector. As reported in Costa and Correa (2010) and Rocha, Barros, Silva and Costa (2016), the 1.Literature review 2.(Project success criteria) 1.Construction of the data collection instrument Data collection 1.Construction of criteria hierarchy of criteria 1.Pairwise judgment of value from the data obtained in the survey 1.Evaluation the degree of consistency 1.Prioritization analysis IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 4 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 choice of this method is justified when the problem of weight assignment is a tradeoff problem. The AHP algorithm is adequate to deal with this type of problem, where the preference for one criterion leads to the substitution of another with lesser preference. As reported in Saaty (1980), when building a model based on AHP, the following steps are applied:  Build the hierarchy of criteria  Perform the pairwise comparison of criteria according to their relevance regarding the goal  Run the AHP prioritization algorithm  Evaluate the consistency of the pairwise evaluations  Perform a sensitivity analysis 3. An AHP approach to evaluate the criteria prioritization This section shows the outcome of the steps described in the methods section. For better understanding, it has been organized according to the same structure as the methods section. 3.1 Criteria definitions In this section, the criteria collected during the literature review and data collection procedure are presented. 3.1.1 Criteria collected from the literature review The theme “project success evaluation criteria” was searched in Scopus and Web of Science databases from April 10, 2018 to April 16, 2018. Table 1 presents the criteria, a short description of each criterion and references where each criterion was cited. IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 5 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 Table 1 Criteria for evaluating success in projects Criterion Short description References  Schedule  Carried out within the initially established period  Pinto and Slevin (1988), Shenhar et al. (1997), De Wit (1988), Baccarini (1999), Judge and Müller (2005), Abdulah et al. (2010), Serrador and Turner (2015), Albert et al. (2017), Koops et al. (2017), Pollack et al. (2018), Redda and Turner (2018)  Cost  Does not exceed the established budget limits  Pinto and Slevin (1988), Shenhar et al. (1997), De Wit (1988), Baccarini (1999), Judge and Müller (2005), Andersen et al. (2006), Abdulah et al. (2010), Serrador and Turner (2015), Albert et al. (2017), Koops et al. (2017), Pollack et al. (2018), Redda and Turner (2018)  Scope  Carried out in the scope, technical performance and expected quality, according to prescribed requirements and unstated expectations.  Pinto and Slevin (1988), Shenhar et al. (1997), De Wit (1988), Atkinson (1999), Baccarini (1999), Judge and Müller (2005), Abdulah et al. (2010), Serrador and Turner (2015), Albert et al. (2017), Koops et al. (2017), Pollack et al. (2018), Redda and Turner (2018)  Efficiency  Resources rationally applied  De Wit (1988), Atkinson (1999), Baccarini (1999), Judge and Müller (2005), Andersen et al. (2006), Serrador and Turner (2015)  Mission/purpose  The output of the project fulfilled the desired purpose that motivated the realization of the project  Pinto and Slevin (1988), Shenhar et al. (1997), Atkinson (1999), Baccarini (1999), Judge and Müller (2005), Andersen et al. (2006), Serrador and Turner (2015), Montes-Guerra et al. (2015)  Organizational benefits  Contributes strategically to the organization  De Wit (1988), Atkinson (1999), Baccarini (1999), Judge and Müller (2005), Andersen et al. (2006), Serrador and Turner (2015), Redda and Turner (2018)  Preparing for the future  Creates conditions for continuous organizational development.  Shenhar et al. (1997), Serrador and Turner (2015), Redda and Turner (2018)  Stakeholders satisfaction  Meets stakeholders expectations, including  Pinto and Slevin (1988), Shenhar et al. (1997), De Wit (1988), Atkinson IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 6 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 Criterion Short description References sponsor, end-user and project team, suppliers, etc. (1999), Baccarini (1999), Judge and Müller (2005), Andersen et al. (2006), Serrador and Turner (2015), Albert et al. (2017), Koops et al. (2017)  Political and social aspects  Meets motivating political and social expectations  Montes-Guerra et al. (2015), Koops et al. (2017), Redda and Turner (2018)  Legality and compliance  Carried out respecting governance rules, observing legality and compliance requirements.  Montes-Guerra et al. (2015), Koops et al. (2017), Redda and Turner (2018)  Safety  Carried out safely and offers security to the end user  Koops et al. (2017), Acheamfour et al. (2019)  Sustainability  Positively impacts the environment, society and finances.  Koops et al. (2017), Acheamfour et al. (2019), Mansell and Philbin (2020) 3.1.2 Data collection All criteria identified in the bibliometric study were included in the form used by the researcher for direct contact with the participants of the survey. Thus, the participant who was interviewed was asked to give information about the gradation used by the studied organization to evaluate projects using the scale (1) Strongly disagree; (2) Disagree; (3) Neither agree / disagree; (4) Agree and (5) Strongly agree. The interview was done using a structured form printed for further typing by the researcher using Google forms. The researcher contacted potential participants and then performed the interviews with those who agreed to participate during a personal meeting, where it was possible to discuss their questions. The questionnaire was applied over a sample of 54 employees of an organization from the health sector in Brazil from August 27- October 20, 2018. Table 2 shows how the respondents were classified into three different groups according to their organizational role. This table also shows the estimated size of the population and the percentage of respondents. IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 7 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 Table 2 Respondents Group Estimated Population Interviewed % A – Executives 10 10 100 B - Project Management Specialists 6 5 83.3 C - Project managers or team members 100 39 39.0 ∑ 116 54 46.5 Potential participants were contacted by phone, and the objective was explained and a contact with the researcher was scheduled when the terms of confidentiality of the data and purpose of the research were presented. During the interviews, which were performed by direct contact of the researcher with the respondents, some participants spontaneously reported that the organization did not have formal and clear criteria for determining the performance evaluation of projects. The information collected regarding the criteria used by the organization in a non-systematic and informal way can be found in Figure 2. According to the respondents interviewed, the organization’s “Legality and Compliance”, “Purpose and Mission”, “Organizational Benefits” and “Political and Social Factors” were perceived by the respondents as being more likely to be considered by the organization for evaluation; on the other hand, “Efficiency, “Project Team Satisfaction”, “Stakeholder’s Satisfaction” and “Sustainability” were not perceived by the respondents to be considered. However, the aim of this paper is to determine the relative importance of each criterion using a method to organize the balance of trade-offs, and for this purpose, we employed the AHP method. IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 8 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 Figure 2 Project evaluation criteria 3.2 Construction of criteria hierarchy of criteria The AHP method considers pairwise comparisons; however, when there are a large number of criteria it is more difficult to evaluate the nuances between two criteria. Thus, a hierarchy was created aggregating criteria in a structure with macro criteria and sub- criteria. Therefore, the 16 criteria identified in the literature were aggregated into four macro criteria (C1-Iron Triangle, C2-Mean-related, C3-End-related and C4-Related to the future). This proposed structure is shown in the Figure 3. IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 9 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 Figure 3 Project success criteria hierarchy Next, a pairwise comparison was performed between the sub-criteria and then between the criteria. 3.3 Pairwise judgment of value from the data obtained in the survey and evaluation of the degree of consistency The first judgment was performed by comparing the answers of the interviewed participants about Cost, Schedule, and Scope. The results of this group of traditional project success criteria also named the “Iron Triangle” in the related professional literature are shown in the Figure 4. Figure 4 Agreement grade of Iron Triangle criteria IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 10 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 The importance between the facets of success in projects was evaluated in pairs from the data obtained in the empirical survey beginning with the sub-criteria that make up the so- called Iron Triangle, (cost, schedule and scope) using the scale of preferences (Saaty, 1980). The obtained results are shown in the pairwise comparison matrix in Table 3. Table 3 Pairwise comparison matrix: Iron Triangle The consistency ratio (CR) was 0.01, which is less than the recommended limit of 0.10 or 10% (Saaty, 1980). The same procedure was used to evaluate the criteria that compose the group mean- related criteria, which aggregates criteria that describe the way the project was done (Efficiency, Legality and Compliance, Safety, and Sustainability). Figure 5 shows the results from the interview related to this group. Figure 5 Agreement grade of Mean-related criteria Then, a comparison between the sub-criteria was performed and is shown in Table 4. Table 4 Pairwise comparison matrix: Mean-related criteria IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 11 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 The consistency ratio (CR) was 0.09, which is less than the recommended limit of 0.10 or 10% (Saaty, 1980). The end-related criteria are used as an aggregation of “Meeting end user expectations”, “Meeting political and social factors”, “Meeting project team expectations”, “Meeting purpose and mission of the project”, “Meeting stakeholders’ expectations” and “Production of organizational benefits”. This group of criteria are related to the efficacy of the project. The results can be seen in Figure 6. Figure 6 Agreement grade End-related criteria The pairwise comparison of this group of criteria is shown in Table 5. Table 5 Pairwise comparison matrix: End-related criteria The last group of criteria named “related to the future” groups criteria related to non- immediate impact (Learning opportunity, Preparing for the future and Promotion of innovation). The results are shown in Figure 7. Figure 7 Agreement grade criteria related to the future IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 12 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 The evaluation of the data collected from the interviewed participants about this variable is organized in Table 6. IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 13 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 Table 6 Pairwise comparison matrix: Criteria related to the future The consistency ratio (CR) is 0.09, less than the recommended limit of 0.10 or 10% (Saaty, 1980). After all the sub-criteria were evaluated, an evaluation of the criteria was performed. The results from the sub-criteria were summed to compose each top-level criterion as detailed in Table 7. The plot compares the percentage frequencies in each criterion by using the Likert scale, and the results are presented in Figure 8. Table 7 Consolidation of sub-criteria for each criterion Criteria / Sub-Criteria Answers given C1 Iron triangle 1 2 3 4 5 sc1.1 Schedule 2 2 13 18 19 sc1.2 Cost 1 4 10 26 13 sc1.3 Scope 0 4 8 20 22  3 10 31 64 54 C2 Mean-related 1 2 3 4 5 sc2.1 Safety 0 5 13 19 17 sc2.2 Legality and compliance 0 0 3 21 30 sc2.3 Sustainability 0 8 13 20 13 sc2.4 Efficiency 0 11 17 12 14  0 24 46 72 74 C3 End-related 1 2 3 4 5 sc3.1 Purpose and Mission 0 0 7 29 18 sc3.2 Political and social factors 0 1 9 34 10 sc3.5 Stakeholders’ satisfaction 1 6 15 26 6 sc3.6 Organizational benefits 0 1 6 23 24  1 8 37 112 58 C4 Related to the future 1 2 3 4 5 sc.4.1 Preparing for the future 0 3 15 20 16 sc.4.2 Innovation 0 2 11 21 20 sc.4.3 Learning opportunity 1 2 10 24 17  1 7 36 65 53 IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 14 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 Figure 8 Agreement grade of success of project Finally, the upper level of the AHP hierarchy was evaluated by aggregating all the responses for each component as follows in Table 8. Table 8 Criteria and sub-criteria Criteria (upper level) Sub-criteria (lower level) Traditional criteria (Iron Triangle) Cost Schedule Scope Mean-related Efficiency Legality and compliance Safety Sustainability End-related Meeting political and social factors Meeting purpose and mission of the project Meeting stakeholders’ expectations Production of organizational benefits Future-related Learning opportunity Preparing for the future Promotion of innovation These data were used to construct the last pairwise comparison matrix shown in Table 9. Table 9 Pairwise comparison matrix: Project success The consistency ratio (CR) is 0.03, less than the recommended limit of 0.10 or 10% (Saaty, 1980). IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 15 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 The pairwise comparisons are now completed, and it is possible to identify the weights supplied by the AHP method. 3.4 Prioritization analysis The results are consolidated in Table 10 which shows the relative importance (weight) assigned using the eigenvector of the previous pairwise matrices, and the global weight is obtained by the multiplication of each sub criteria by the upper-level criteria, thus providing the relative importance ranking of each. Table 10 Relative importance (weights) of criteria Sub criteria Weight Global Weight C1 Iron triangle 29.2% sc1.1 Schedule 16.3% 4.8% sc1.2 Cost 29.7% 8.7% sc1.3 Scope 54.0% 15.8% C2 Mean-related 10.8% sc2.1 Safety 16.0% 1.7% sc2.2 Legality and compliance 65.8% 7.1% sc2.3 Sustainability 12.3% 1.3% sc2.4 Efficiency 5.9% 0.6% C3 End-related 18.7% sc3.1 Purpose and Mission 45.1% 8.4% sc3.2 Political and social factors 17.3% 3.2% sc3.5 Stakeholders satisfaction 3.6% 0.7% sc3.6 Organizational benefits 27.0% 5.1% C4 Related to the future 41.3% sc.4.1 Preparing for the future 16.4% 6.8% sc.4.2 Innovation 34.2% 14.1% sc.4.3 Learning opportunity 49.3% 20.4% The data in Table 10 communicates the goal of this paper which is to elicit the relative importance of each criterion through the application of the AHP, a method that is able to deal with subjectivity, from the results of the empirical study at the selected organization. Table 11 shows a list of criteria with their respective relative importance between parenthesis in descending order. IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 16 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 Table 11 Project success criteria ordered by relative importance 4. Discussion, practical applications, and limitations One important consequence of this method is that the results could be affected because the amount of sub-criteria for each criterion was not similar. This is because the relative importance of one criterion could be reduced at the product of the eigenvalues of the criteria and sub-criteria depending on the number of sub-criteria that compose the criterion in the hierarchy. Thus, it is recommended that the structure be equilibrated. Another observation of this study performed at a public organization is that the criterion “Legality and Compliance” shown in Table 1 was the top ranked criterion; however, by using this method it was reduced in importance to the sixth position when the criterion “Mean-related” received a lower relative importance because the other sub-criteria that compose it were rated low (Efficiency, Safety and Sustainability). The practical implication of this work lies in the possibility of offering support for the most used criteria for the evaluation of project performance. In this way, it is possible to direct resources towards the criteria with greater sensitivity to project success. Even though the study was carried out with a good sample of respondents, they all belong to the same organization that has a specific organizational culture, and may not, therefore, represent all organizations and their individual cultures and environments. It is also important to note that the researcher is connected to institution that was analyzed. 5. Conclusion The main contribution of this work is to provide the relative weights of project success criteria that have been identified in the scientific literature using a method that allows the consistency of subjective evaluations to be validated. This contribution fulfills an existing Sub criteria Weight Criterion Weight Final Weight Classification sc.4.3 Learning opportunity 49.3% C4 Related to the f uture 41.3% 20.4% 1 sc1.3 Scope 54.0% C1 Iron triangle 29.2% 15.8% 2 sc.4.2 Innovation 34.2% C4 Related to the f uture 41.3% 14.1% 3 sc1.2 Cost 29.7% C1 Iron triangle 29.2% 8.7% 4 sc3.1 Purpose and Mission 45.1% C3 End-related 18.7% 8.4% 5 sc2.2 Legality and compliance 65.8% C2 Mean-related 10.8% 7.1% 6 sc.4.1 Preparing f or the f uture 16.4% C4 Related to the f uture 41.3% 6.8% 7 sc3.6 Organizational benef its 27.0% C3 End-related 18.7% 5.1% 8 sc1.1 Schedule 16.3% C1 Iron triangle 29.2% 4.0% 9 sc3.2 Political and social f actors 17.3% C3 End-related 18.7% 3.2% 10 sc2.1 Saf ety 16.0% C2 Mean-related 10.8% 1.7% 11 sc2.3 Sustainability 12.3% C2 Mean-related 10.8% 1.3% 12 sc3.5 Stakeholders satisf action 3.6% C3 End-related 18.7% 0.7% 13 sc2.4 Ef f iciency 5.9% C2 Mean-related 10.8% 0.6% 14 IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 17 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 gap in the field of a lack of adequate models to deal with the present subjectivity in this evaluation. We determined the relative importance of the criteria for evaluating success in projects using the AHP method considering the responses of the 54 participants who were interviewed as follows: Learning opportunity (20.4%), Scope (15.8%), Innovation (14.1%), Cost (8.7%), Purpose and Mission (8.4%), Legality and compliance (7.1%), Preparing for the future (6.8%), Organizational benefits (5.1%), Schedule (4.0%), Political and social factors (3.2%), Safety (1.7%), Sustainability (1.3%), Stakeholders’ satisfaction (0.7%), and Efficiency (0.6%). We highlight the unexpected finding that despite the fact that the aspects of cost, schedule and scope are widely used in performance evaluation, they did not rank highly in the study. It is also important to highlight that the consistency ratio (CR) of all the evaluations was less than 10% which indicates a consistent result using the AHP method. For further study, we propose comparative studies with similar organizations in order to identify possible differences in success criteria prioritization between them. We also recommend extending this work by using the ANP to see the relationships among the criteria. IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 18 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 REFERENCES Abdullah, A.A., Rahman, H.A., Harun, Z., Alashwal, A.M., & Beksin, A.M. (2010). Literature mapping: A bird’s eye view on classification of factors influencing project success. African Journal of Business Management, 4(19), 4174-4182. Acheamfour, V.K., Kissi, E., & Adjei-Kumi, T. (2019). Ascertaining the impact of contractors pre-qualification criteria on project success criteria. Engineering, Construction and Architectural Management, 26(4), 618-632. Doi: https://doi.org/10.1108/ECAM-03-2018-0110 Albert, M., Balve, P., & Spang, K. (2017). Evaluation of project success: a structured literature review. International Journal of Managing Projects in Business, 10(4), 796- 821. Doi: https://doi.org/10.1108/IJMPB-01-2017-0004 Andersen, E.S., Birchall, D., Jessen, S.A., & Money, A.H. (2006). Exploring project success. Baltic Journal of Management, 1(2), 127-147. Doi: https://doi.org/10.1108/17465260610663854 Atkinson, R. (1999). Project management: cost, time and quality, two best guesses and a phenomenon, its time to accept other success criteria. International Journal of Project Management, 17(6), 337-342. Baccarini, D. (1999). The logical framework method for defining project success. The Project Management Journal, 30(4), 25-32. Costa, H.G., & Correa, P. de S. (2010). Construction of an AHP-Based model to catch criteria weights in post-occupancy evaluation. International Journal of the Analytic Hierarchy Process, 2(1), 30-43. Doi: 10.13033/ijahp.v2i1.71 da Silva, F. F., Souza, C. L. M., Silva, F. F., Costa, H. G., da Hora, H. R. M., & Erthal Junior, M. (2021). Elicitation of criteria weights for multicriteria models: Bibliometrics, typologies, characteristics and applications. Brazilian Journal of Operations and Production Management, 18(4). Doi: https://doi.org/10.14488/BJOPM.2021.014 De Wit, A. de. (1988). Measurement of project success. Project Management, 6(3), 165- 170. Jugdev, K., & Müller, R. (2005). A retrospective look at our evolving understanding of project success. Project Management Journal, 36(4), 19–31. Doi: https://doi.org/10.1177/875697280503600403 Keeney, R.L., & Raiffa, H. (1993). Decisions with multiple objectives : preferences and value tradeoffs. Cambridge University Press. Koops, L., van Loenhout, C., Bosch-Rekveldt, M., Hertogh, M., & Bakker, H. (2017). Different perspectives of public project managers on project success. Engineering, https://doi.org/10.1108/ECAM-03-2018-0110 https://doi.org/10.1108/IJMPB-01-2017-0004 https://doi.org/10.1108/17465260610663854 https://doi.org/10.14488/BJOPM.2021.014 https://doi.org/10.1177/875697280503600403 IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 19 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 Construction and Architectural Management, 24(6), 1294-1318. Doi: https://doi.org/10.1108/ECAM-01-2015-0007 Mansell, P., & Philbin, S.P. (2020). Measuring sustainable development goal targets on infrastructure projects. Journal of Modern Project Management, 8(1), 1-22. Montes-Guerra, M.I., De-Miguel, A.R., Pérez-Ezcurdia, M.A., Gimena Ramos, F.N., & Díez-Silva, H.M. 2015. Project management in development cooperation. Non- governmental organizations. Rev. Innovar, 25(56), 52-67. Doi: http://dx.doi.org/10.15446/innovar.v25n56.48989 Pinto, J. K., & Slevin, D.P. (1988). Project success: definitions and measurement techniques. Newton Square, PA: Project Management Institute. Pollack, J., Helm, J., & Adler, D. (2018). What is the Iron Triangle, and how has it changed?. International Journal of Managing Projects in Business, 11(2), 527-547. Doi: https://doi.org/10.1108/ijmpb-09-2017-0107 Redda, Y., & Turner, R. (2018). Mapping the success dimensions of large oil and gas projects. In SPE Annual Technical Conference and Exhibition. Society of Petroleum Engineers. Rocha, P.M.P.M., Barros, A.P.D.A.P., Silva, G.B.D.G.B., & Costa, H.G. (2016). Analysis of the operational performance of Brazilian airport terminals: A multicriteria approach with De Borda-AHP integration. Journal of Air Transport Management, 51, 19–26. Doi: https://doi.org/10.1016/j.jairtraman.2015.11.003 Rodriguez D.S.S., Costa H.G., & Carmo L.F.R.R.S. (2013). Métodos de auxílio multicritério à decisão aplicados a problemas de PCP: Mapeamento da produção em periódicos publicados no Brasil. Gest e Prod, 20, 134–146. Doi: https://doi.org/10.1590/S0104-530X2013000100010 Rothstein, H.R., & Hopewell, S. (2009). Grey literature. In H. Cooper et al. (Eds.), The Handbook of Research Synthesis and Meta-Analysis (pp. 103-125). New York: Russel Sage Foundation. Roy, J., Das, S., Kar, S., & Pamučar, D. (2019). An extension of the CODAS approach using interval-valued intuitionistic fuzzy set for sustainable material selection in construction projects with incomplete weight information. Symmetry, 11(3), 1-24. Saaty, T.L. (1980). The analytic hierarchy process: planning, priority setting, resource allocation. McGraw-Hill International Book. Šenitková, I., Burdová, E.K., & Vilčeková, S. (2010). Building environmental assessment system-weight determination. Proceedings of CESB 2010 Prague - Central Europe towards Sustainable Building “From Theory to Practice”. Prague: Czech Technical University. https://doi.org/10.1108/ECAM-01-2015-0007 https://doi.org/10.1108/ijmpb-09-2017-0107 https://doi.org/10.1016/j.jairtraman.2015.11.003 https://doi.org/10.1590/S0104-530X2013000100010 IJAHP Article: de Souza, Costa, de Aruajo/Prioritizing criteria to evaluate project success: modeling with the Analytic Hierarchy Process (AHP) International Journal of the Analytic Hierarchy Process 20 Vol. 14 Issue 1 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i1.913 Serrador, P., & Turner, R. (2015). The relationship between project success and project efficiency. Project management journal, 46(1), 30-39. Doi: https://doi.org/10.1002/pmj Shenhar, A.J., Levy, O., & Dvir, D. (1997). Mapping the dimensions of Project Success. Project Management Journal, 28(2), 5-13. Yi, P., Li, W., Zhang, D. (2019). Assessment of city sustainability using MCDM with interdependent criteria weight. Sustainability (Switzerland), 11(6), 1-20. Doi: https://doi.org/10.3390/su11061632 Zagonari, F. (2016). Choosing among weight-estimation methods for multi-criterion analysis: A case study for the design of multi-purpose offshore platforms. Applied Soft Computing Journal, 39, 1–10. Doi: https://doi.org/10.1016/j.asoc.2015.11.003 https://doi.org/10.1002/pmj https://doi.org/10.3390/su11061632 https://doi.org/10.1016/j.asoc.2015.11.003