IJAHP News and Events: Saaty/Best practices in analytic network studies: interview with Enrique Mu and Orrin Cooper International Journal of the Analytic Hierarchy Process 357 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.779 Best Practices in Analytic Network Studies: Interview with Enrique Mu and Orrin Cooper Rozann Saaty IJAHP Managing Editor Enrique Mu, Gabriela Sava and Orrin Cooper at the ANP Best Practice Presentation at INFORMS2017 Enrique Mu, IJAHP Editor-in-chief, Orrin Cooper, IJAHP editorial board member and Michael Peasley, AHP/ANP researcher, have recently published the paper “Best Practices in Analytic Network Studies” (Mu, Cooper, & Peasley, 2020). This study provides a set of guidelines to improve the validity of the ANP model and reporting. Given the importance of this study, I decided to interview the leading authors about this vital subject for the benefit of IJAHP readers. 1. How did you get the idea to write a paper about ANP best practices? Enrique: I think the first thought about this came to me during the panel discussion “Publishing AHP/ANP Papers” at the 12 th International Symposium of the Analytic Hierarchy Process, ISAHP2013, in Kuala Lumpur, Malaysia. There, I gave a presentation titled, “What makes an AHP/ANP paper publishable?” and started to list some of the criteria that, in my editorial and reviewer experience, made a paper publishable such as: Who developed the model?, How were the group judgments aggregated?, How was group consistency addressed?, and so forth (DeFelice et al., IJAHP News and Events: Saaty/Best practices in analytic network studies: interview with Enrique Mu and Orrin Cooper International Journal of the Analytic Hierarchy Process 358 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.779 2013). Over the years, I began to compile a checklist of these criteria that I would use as a pre-screening tool for the articles submitted to our IJAHP journal. In 2016, when I was discussing a study that would require a literature review of ANP papers published the previous year with Orrin Cooper and Michael Peasley, the idea came up about taking advantage of this literature review to check how ANP studies were reported. The initial set of publishing criteria was corrected and improved in discussions with Orrin and Michael and as a result of the ANP literature review. We called these criteria “Checklist of Best Practices in ANP Reporting” and started to test them on ANP paper submissions in our editorial and reviewers’ activities. Orrin: Enrique approached me in 2016 while we were working with each other on other projects. I could tell right away that something was bothering him. I could see it weighing on his mind. He talked generally about some of the papers he saw as the editor of IJAHP and the inconsistency in what was reported which in some cases led him to doubt the validity of the results in the submitted papers. He suggested the idea of developing a set of best practices for AHP/ANP. Personal experiences from reviewing journal submissions and working with MBA students who were taking Dr. Saaty’s class about the ANP immediately came to my mind. Enrique was on to something! I knew this would both help users understand the ANP better and improve the quality of ANP reports which in the end would further the recognition and future application of the ANP. This would be a challenging but influential project to tackle. And so the journey started. 2. I understand you collected a great deal of papers for this study, what were your main findings with respect to what the best practices in ANP reporting are? Enrique: For just the year 2015, there were more than one hundred studies; however, in the end we focused on 84 relevant studies that were published that year in Web of Science journals. The preliminary results were presented at INFORMS2017, and we basically found that most papers that were published were not consistent in what they reported. Those who did a good job reporting how the ANP model had been developed would sometimes not report if and how group inconsistency had been addressed and so forth. These findings further confirmed the need for some tentative standards that would constitute best practices to report ANP studies. Based on what we found, we further refined our proposed best practices, and a preliminary version of the study and new checklist was presented at ISAHP2018 in Hong Kong (Mu, Cooper, & Peasley, 2018). Orrin: I will admit I was a little nervous to meet with Enrique to review our initial results from the literature review. I even doubted my reviews because of how many papers had not addressed what we determined as crucial information to include. Some authors said nothing about consistency in the pairwise comparison matrices – not even something like “we checked the consistency/inconsistency.” Few papers presented any sensitivity analysis, and others only provided a part of the final priority vector without any of the supermatrices. Many of the weighted supermatrices were not column stochastic and so sparse that there weren’t enough connections to raise the supermatrix to powers. I was very surprised at how inconsistent the reporting was. When our results were the same, it underscored the need to have best practices. Our list of best practices went through another revision based on what we had found. IJAHP News and Events: Saaty/Best practices in analytic network studies: interview with Enrique Mu and Orrin Cooper International Journal of the Analytic Hierarchy Process 359 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.779 Speaking of revisions, I am grateful for the feedback that others provided during the conference presentations. We reorganized the best practices and checklists and it was interesting at that point to see how the current checklist came together until we finally felt like we got it right, so then we returned to evaluating the papers. 3. What are the main contributions of your study? Enrique: The most important contribution is that for the first time, tentative standards for reporting ANP studies have been suggested. This allows the overall validity of the ANP study to be assessed. These standards are provided in the form of validity checklists for both AHP and ANP. Orrin: I think it is pretty simple, best practices improve the quality of work. The regression analogy we use in the paper comes to mind where not calculating and reporting a p-value would raise red flags. The standards serve as a guide: first, to make sure the appropriate analysis was done; and, second, that enough information is reported about the study. These two factors increase the validity of the study and exponentially increase the ability to replicate the results and use them in other settings. 4. Where can we get these AHP/ANP best practices checklists? Enrique: The pre-print versions of these checklists are included here as appendices, for the reader’s convenience. The final printed version of the study for which these checklists were developed is available at: https://doi.org/10.1016/j.eswa.2020.113536. We strongly recommend reading the study to understand the validity criteria definitions and their examples. Speaking of examples, the ANP validation criteria proposed in our study were applied, almost in totality, in an ANP study about merging IT functions for the City of Pittsburgh, which was simultaneously being conducted (Mu & Stern, 2018). It also shows how to deal with a major concern about reporting with respect to length of the article and using appendices. Therefore, we strongly recommend reading this study as a practical example of how to implement the checklist criteria in an actual published paper. 5. Do you think these best ANP practices will evolve over time? Orrin: We definitely hope this remains a living document and is updated as the field continues to mature. We listed a few potential areas that over time may become more advanced and recognized as best practices. Best practices must be updated and kept current so that ANP researchers do the best research possible. There may even become sub-groupings like ordinal and cardinal consistency, dealing with group decision making, i.e., gaining consensus, dealing with “outliers” in the group, etc. Each of these (sub-groupings) is important, but may not apply to a majority of models. This is what we felt was a minimum that applied to ANP models in general. It is also important to emphasize that there is definitely more that can be included in reports and analysis. The best practices are not the comprehensive list of everything that can be done; again, I think authors need to recognize it as a minimum bar and a living document. https://doi.org/10.1016/j.eswa.2020.113536 IJAHP News and Events: Saaty/Best practices in analytic network studies: interview with Enrique Mu and Orrin Cooper International Journal of the Analytic Hierarchy Process 360 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.779 6. Why is the model development explanation so important? Enrique: In any research study, the first step is always ensuring the face validity of the study which in practical terms can be defined as “the degree to which a procedure, especially a psychological test or assessment, appears effective in terms of its stated aims” (Lexico.com, 2020). If ANP authors want their readers to spend the next hour of their lives reading what should be done about COVID-19, they must start by providing face validity for their study. This develops trust between the authors and their readers. During my years as a doctoral student, I had a professor who used to tell me that the face validity phase of research was actually the Aha! test of the study. Indicating up front who developed the model, why, and how the model (e.g. factors) came to be provides face validity to the overall study and creates the motivation to dig deeper into the details. The greatest contribution of proposing an ANP model to address a decision issue is that the model can be adopted, used or adapted by other members of the community with similar decision issues. For this purpose, trust in the validity of the proposed ANP model is very important. Thank you, Enrique and Orrin, for an illuminating discussion about best practices for what to include in ANP articles and about writing research papers in general. IJAHP News and Events: Saaty/Best practices in analytic network studies: interview with Enrique Mu and Orrin Cooper International Journal of the Analytic Hierarchy Process 361 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.779 REFERENCES DeFelice, F., Karpak, B., Mu, E., Pecchia, L., Petrillo, A., Saaty, R. & Vargas, L. (2013). Publishing AHP/ANP Papers. International Journal for the Analytic Hierarchy Process, 5(1), 116-174. doi:https://doi.org/10.13033/ijahp.v5i1.187 Lexico.com. (Ed.) (2020) Dictionary.com. UK: Lexico.com. Mu, E., Cooper, O. & Peasley, M. (2018). Improving Analytic Network Process Reporting. Paper presented at the International Symposium of the Analytic Hierarchy Process, Hong Kong. Mu, E., Cooper, O. & Peasley, M. (2020). Best practices in Analytic Network Process studies. Expert Systems with Applications, 159(30). doi:https://doi.org/10.1016/j.eswa.2020.113536 Mu, E. & Stern, H. (2018). A contingent/assimilation framework for public IS inter- organizational decisions. International Journal of Information Technology and Decision Making, 17(6), 1611-1658. doi: 10.1142/S0219622018420014 https://doi.org/10.13033/ijahp.v5i1.187 https://doi.org/10.1016/j.eswa.2020.113536 IJAHP News and Events: Saaty/Best practices in analytic network studies: interview with Enrique Mu and Orrin Cooper International Journal of the Analytic Hierarchy Process 362 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.779 APPENDIX ANP Best Practices Preliminary Checklist Version: 03-21-2020 Source: Mu, E., Cooper, O., & Peasley, M. (2020). Best Practices in Analytic Network Process Studies. Expert Systems with Applications, 159(30). doi:https://doi.org/10.1016/j.eswa.2020.113536 ITEM VARIABLE DESCRIPTION IDEAL YOURS QUALITATIVE 1 MODEL Model Development Explanation Who developed the Model? How was the model developed? Decisions models can be developed by the authors, experts and participation of stakeholders. Authors should indicate which approach was used, indicating number of participants and their qualifications to participate as appropriate. 2 FACTOR Clusters and Nodes 2a FACTOR1 Clusters and nodes must be clearly identified/labeled. 2b FACTOR2 They must be clearly defined, indicating how they will be measured and used. 2c FACTOR3 The source (e.g. extant literature, expert opinion, stakeholders) for the cluster/node must be specifically cited. 3 EXCOMPQ Example of Comparison Questions An example for each of the different comparison questions should be provided, including questions related to inner relationship comparisons. If surveys were used, either provide example questions or -even better- include the survey as an appendix. 4 GAGGREG Group Decision Aggregation If group decision making: How were the group decisions aggregated? Was there any additional examination of the aggregation such as group dispersion analysis? 5 GCONSIST Group Consistency If group decision making: How was group consistency addressed? (e.g. Did you discard respondents that were too inconsistent?) 6 SUBNET Report on each of the sub-networks used in the study For each sub-network (e.g. B, O, C & R) report on the items in the checklist. QUANTITATIVE 7 INFLUENCEM Influence Matrix Report the influence matrix showing the node interactions. 8 CONSIST Consistency 8a CONSIST1 Consistency threshold (e.g. CR <=0.1) explicitly stated. 8b CONSIST2 How inconsistency situations were addressed. 9 CLUSTERCOMP Cluster Comparison Matrix Report the cluster comparison matrix 10 LIMITMAX Limit Comparison Matrix 10a FINALCRITPRIOR Global Priority for Criteria From Limit Supermatrix 10b FINALTERPRIOR Global Priority for Alternatives From Limit Supermatrix 11 WEIGHTSUPMAX Weighted Supermatrix Report the weighted Supermatrix 12 SENSITIVITY Sensitivity Analysis In complex models, sensitivity analysis is possible at different levels (e.g. individual subnets, strategic criteria). Indicate which sensitivity analysis was chosen and why. 13 RATINGSCALE Ratings Scales If applicable, both the development of the ratings scales and the scales must be reported. 14 SUBNETAGGREG Relationship among sub-networks How were sub-networks combined? For example, in BOCR analysis there are different ways to combine the subnetworks (e.g. additive, multiplicative). https://doi.org/10.1016/j.eswa.2020.113536 IJAHP News and Events: Saaty/Best practices in analytic network studies: interview with Enrique Mu and Orrin Cooper International Journal of the Analytic Hierarchy Process 363 Vol. 12 Issue 2 2020 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v12i2.779 AHP Best Practices Preliminary Checklist Version: 03-21-2020 Source: Mu, E., Cooper, O., & Peasley, M. (2020). Best Practices in Analytic Network Process Studies. Expert Systems with Applications, 159(30). doi:https://doi.org/10.1016/j.eswa.2020.113536 ITEM VARIABLE DESCRIPTION IDEAL YOURS QUALITATIVE 1 MODEL Model Development Explanation Who developed the Model? How was the model developed? Decisions models can be developed by the authors, experts and participation of stakeholders. Authors should indicate which approach was used, indicating number of participants and their qualifications to participate as appropriate. 2 FACTOR Clusters and Nodes 2 a FACTOR1 Clusters and nodes must be clearly identified/labeled 2 b FACTOR2 They must be clearly defined, indicating how they will be measured and used. 2 c FACTOR3 The source (e.g. extant literature, expert opinion, stakeholders) for the cluster/node must be specifically cited. 3 EXCOMPQ Example of Comparison Questions An example for each of the different comparison questions should be provided, including questions related to inner relationship comparisons. If surveys were used, either provide example questions or -even better- include the survey as an appendix. 4 GAGGREG Group Decision Aggregation If group decision making: How were the group decisions aggregated? Was there any additional examination of the aggregation such as group dispersion analysis? 5 GCONSIST Group Consistency If group decision making: How was group consistency addressed? (e.g. Did you discard respondents that were too inconsistent?) 6 SUBNET Report on each of the sub-networks used in the study For each sub-network (e.g. B, O, C & R) report on the items in the checklist QUANTITATIVE 8 CONSIST Consistency 8 a CONSIST1 Consistency threshold (e.g. CR <=0.1) explicitly stated. 8 b CONSIST2 How inconsistency situations were addressed. 10 LIMITMAX Limit Comparison Matrix 1 0 a FINALCRITPRIOR Global Priority for Criteria From Limit Supermatrix 1 0 b FINALTERPRIOR Global Priority for Alternatives From Limit Supermatrix 12 SENSITIVITY Sensitivity Analysis In complex models, sensitivity analysis is possible at different levels (e.g. individual subnets, strategic criteria). Indicate which sensitivity analysis was chosen and why. 13 RATINGSCALE Ratings Scales If applicable, both the development of the ratings scales and the scales must be reported. 14 SUBNETAGGREG Relationship among sub-networks How were sub-networks combined? For example, in BOCR analysis there are different ways to combine the subnetworks (e.g. additive, multiplicative). https://doi.org/10.1016/j.eswa.2020.113536