IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 331 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 THE DESIGN OF A PERFORMANCE MEASUREMENT SYSTEM FOR A FRUIT PROCESSING EQUIPMENT MANUFACTURER Bruna Pachecho bruna_scarduelli@yahoo.com.br Claudio Piratelli clpiratelli@uniara.edu.br Ethel Silva e-chiari@uol.com.br Mischel Carmen Belderrain carmen@ita.br University of Araraquara-Uniara Brazil ABSTRACT Several Performance Measurement Systems (PMS) are referred to in the literature, but there is little mention of the design stage in the construction of a PMS. The purpose of this paper is to design a performance measurement system capable of assisting management and continuously improving the production process of a manufacturer of food production equipment and maintenance service provider headquartered in the state of São Paulo. It is based on the method proposed by Piratelli (2010) called The Performance Prism (TPP) framework. It has two phases, and the first is based on the Strategic Options Development and Analysis (SODA) methodology; the second uses the Analytic Network Process (ANP), a Multiple Criteria Decision Making (MCDM) method. Nine performance indicators were created to meet the needs presented by the stakeholders: the directors, quality coordinator, and production manager, given the faces of The Performance Prism. The construction of a PMS allowed the company being studied to draw up strategy and management plans for the production process. The proposed method was shown to be effective when applied. Keywords: The Performance Prism; SODA; ANP; intermediate goods industry 1. Introduction Performance Measurement Systems (PMS) are addressed in the literature as an efficient way to seek continuous improvement and help organizations be competitive in the market. However, there is little in the literature on how to design a PMS to meet the particular needs of each company. Measuring performance is not a trivial task since it involves considering specific characteristics of each unique company or sector. Choong (2013; 2014a) conducted a scientific study using ABI/Inform ProQuest, Emerald Full mailto:bruna_scarduelli@yahoo.com.br mailto:clpiratelli@uniara.edu.br mailto:e-chiari@uol.com.br mailto:carmen@ita.br IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 332 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 Text, Science@Direct and EBSCO 1990-2012, and noted a lack of articles about the attributes needed for a PMS. Choong's (2014b) research recommends the need for a new paradigm in the advancement of PMS studies. He recommends a shift in the emphasis on the measurement of a more theoretical approach to the adoption of a more scientific approach to improve the measurement of organizational activities and performance. The multi-methodology used in this work aims to integrate the Problems Structuring Method (PSM) and Multiple-Criteria Decision-Making (MCDM) which belong to different, soft and hard paradigms of Operational Research. Marttunen, Lienert and Belton (2017) conducted a study to explore how the PSMs and MCDM methods are applied together and concluded that they are complementary methods and when applied together result in many synergies and mutual benefits because the combination provides a richer picture of the decision situation and a methodology that can better handle the various phases of decision making. The PSM used is the Strategic Options and Development Analysis (SODA) and the MCDM used is the Analytic Network Process (ANP). For Eden and Simpson (1989), the forerunners of SODA, it is an approach to complex organizational problems that uses the constructivist paradigm to help decision-makers structure their ideas, facilitating the construction of a model that brings the objectives of each stakeholder involved in the decision-making process, and integrates the participants into a collective reflection on the problem. According to Saaty (1999; 2008), the ANP although built on the Analytic Hierarchy Process (AHP), goes beyond it because it includes the dependence between the criteria. The objective of this study is to design a performance measurement system for the production process management at a company that manufacturers fruit processing equipment, based in the state of São Paulo, Brazil. The specific objectives can be divided into the following steps: 1 - Identifying the performance criteria for stakeholders through the SODA methodology, bearing in mind the faces of TPP framework. 2 - Faithful modeling of the criteria's dependency relationships and prioritizing them using the ANP. 3 - Applying these frameworks in the production area at the food equipment manufacturing company in the state of São Paulo. The second section of this paper provides a brief review of the literature on the main performance measurement frameworks, focusing on presenting TPP framework. The third section presents the Operational Research methods used in the proposed method, in particular the SODA methodology and the ANP. The fourth section presents the operational procedures for the proposed method. The fifth section presents the results obtained by applying the method to the study object, and the sixth section presents the final considerations. IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 333 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 2. Performance Measurement Systems "If you can't measure it, you can't manage it" (Kaplan & Norton, 1997). According to Sobreira Netto (2007) and Neely (1999; 2005), there is a recurring concern about monitoring the performance of organizations. Performance measurement is nothing new, and there is a significant number of frameworks available (Neely et al., 1997; Neely & Bourne, 2000). The definition of performance management should be clearly related to the achievement of strategic priorities, as a management tool, and the language should be simple and understandable (Atkinson, 2012; Bititci et al., 1997; Bititci et al., 2000; Hanson, Melnyk & Calantone, 2010). 2.1 Frameworks and views of Performance Measurement Systems Choong (2013) surveyed the literature on performance measurement systems (published from January 1990 to November 2012) and found 479 articles on ABI/Inform ProQuest, Emerald Full Text, Science@Direct and EBSCO. Many of the articles made a brief reference to PMS and were excluded from the survey, as were articles that used methodologies such as case studies or studies/reviews that addressed the subject too narrowly. After these exclusions, the number of relevant articles in the survey, with more than five citations within two years of each other, fell to 67. The number of articles that address measuring attributes for PMS is low showing it is poorly researched. In the survey, Kaplan and Norton are the most commonly cited authors in reference to the Balanced Scorecard, followed by Neely, who created The Performance Prism (Choong, 2013). This research was updated in 2014 by the same author. 2.2 The Performance Prism framework According to Neely et al. (2002), The Performance Prism (TPP) attempts to integrate five related perspectives in a three-dimensional framework as follows: a) Stakeholder satisfaction - Who are the company's stakeholders and what do they want and need?; b) Strategies - What strategies does the company need to implement in order to meet the stakeholders' needs?; c) Processes - What critical processes are required to carry out these strategies?; d) Capabilities - What capacities does the company need to operate and improve these processes?; e) Stakeholders' contribution - What contributions does the company demand of stakeholders to maintain and develop these capacities? For Neely et al. (2002), TPP complements the Balanced Scorecard by analyzing end users, employees, suppliers, regulators, pressure groups or local communities. It reinforces the thesis that these stakeholders can have a significant impact on the organization and the conduction of its activities in addition to building the strategy that is, how the objectives will be achieved. 2.3 PMS design Neely et al. (2000) proposed a method for the development of a PMS in six phases, presented in a workbook about which indicators are needed, the cost-benefits of the process, the purpose of performance measurement for the application, functional testing, IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 334 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 environmental analysis and periodic maintenance of the system. According to Rentes, Carpinetti and Van Aken (2002), steps can be taken to design a PMS, in other words, to create a system of construction or revision of the organizational performance measurement that allows incorporating improvements in the critical points or failures of the development process and implementation of performance measurement. The steps are as follows: - Identify the strategic objectives; - Identify critical factors/areas for success; - Define high-level metrics related to critical factors/areas for success and goals; Implement performance metrics and goals at other levels; - Plan the operation of the system. Note that the above present steps are common to the PMS design process, but do not detail or exemplify how to put them into practice. Based on the use of the Multi-Criteria Decision Aid Constructivist (MCDA-C) methodology, Ensslin et al. (2010) proposed the systematic building of a framework for performance evaluation that is systematically structured into three groups of activities: structuring, evaluation and recommendations. The structuring phase is to identify, organize and ordinally measure the concerns that the decision-maker considers necessary and sufficient for assessment of the context. The second phase, assessment, is an instrument to improve understanding by building cardinal scales and replacement rates to represent their local and global preferences. Recommendations comprise the third phase and aim to continue the process of expanding the understanding of the context by trying to understand the consequences of possible decisions in representative criteria for the dimensions considered relevant and in the context as a whole. Piratelli (2010) proposed a method to design a PMS based on the TPP framework, combining the constructivist SODA methodology to identify the performance criteria identified by the stakeholders with the ANP method with a rationalist nature, which will be detailed in section 4. 3. Operational research methods This section contextualizes the Operational Research (OR) methods used in combination in this study. According to Amaral and Araujo (1998), the ORs essentially assist decision makers in troubleshooting. According to Mingers and Brocklesby (1997), a very common use of PSMs is through multi-methodology or combining methods according to each phase of a problem. Marttunen, Lienert and Belton (2017) corroborate this view. Mingers and Rosenhead (2004) state that each problem is unique, and it is not possible to compare the effectiveness of different methodologies for a specific situation. Ackermann et al. (2014) defend the mixture of methods, as this allows the field of research to remain fresh and vibrant, and provides the extensions/adaptations needed to provide the means to manage a wider range of problems. Also, the very act of mixing IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 335 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 methods helps broaden awareness in the field of research since many of the combinations incorporate the research of structuring problems in different disciplines. Complex decisions must be made in the design phase of a PMS, both at the time of structuring the PMS (what to measure) and at the time of modeling (how to measure it). The SODA methodology will be used to structure the PMS and capture how the stakeholders think about organizational performance in order to select the evaluation criteria. The modeling will use the ANP, which enables the PMS to evaluate the identified criteria while maintaining the same relationships (dependency and/or feedback) surveyed in the phase by those involved in the structuring. 3.1 Strategic Options Development and Analysis According to Rosenhead (1989), soft OR has been developed since 1980 through methodologies such as Soft Systems Methodology (SSM), Strategic Options Development and Analysis (SODA), and Strategic Choice Approach (SCA). For Eden and Simpson (1989), SODA is a methodological approach to complex organizational problems using the constructivist paradigm to assist decision makers in structuring their ideas through a cognitive map. In SODA, the objectives for each stakeholder involved in the decision-making process are presented and interrelated in order to get a collective reflection on the problem. Hence, software such as Banxia's Decision Explorer® is indispensable (Westcombe, 2002). The procedures for preparing and analyzing a cognitive map can be found in Eden (1989), Ensslin et al. (2001), and Piratelli (2010), and are also described in Section 4. The constructivist methodology has been applied in several decision situations (Ensslin, Dutra & Ensslin, 2000; Ensslin, Ensslin & Lacerda, 2011; Rosa et al., 2012; Ensslin et al. 2013). 3.1.1 Cognitive map applications in PMS Ensslin et al. (2010) made use of the cognitive mapping technique to propose a performance evaluation model for outsourced companies in telecommunications with operations in southern Brazil. The application identified the criteria deemed important by the Project Manager and made it possible to build a decision support tool to approve subcontractors. Madeira Junior, Gonçalves and Belderrain (2011) presented the structuring of a quality performance evaluation for container terminals using cognitive maps. The authors concluded that the structuring of the evaluation system was consistent with the literature. Azevedo et al. (2011) made use of the cognitive mapping technique to evaluate the performance of the budgeting process at a construction company. Meetings were held with decision makers and facilitators, lasting 40 and 200 hours, respectively. Subsequently, the value hierarchy was developed, identifying Fundamental Points of View (FPV) and Elementary Points of View (EPV). Bortoluzzi et al. (2011) built an evaluation model for economic-financial performance in a case study at Seprol Computadores e Sistemas Ltda., considering financial indicators, while using cognitive mapping. The stakeholders were identified and maps were drawn from Primary Assessment Elements (PAEs) and defined concepts, and the clusters were defined. The hierarchical value structure was then drawn up, identifying the three most prominent dimensions: Financial Management, Human Resources and the Market. The IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 336 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 financial management performance descriptors identified below-par performance and corrective action was decided on. Ensslin et al. (2016) used cognitive maps to build a performance evaluation model for customer service and business performance at a bank, based on the team manager's values and preferences. The objectives were to identify the criteria that demonstrate process performance and identify the status quo with regard to them. The mapping process identified 91 PAEs and 7 FPVs, allowing the decision maker to determine performance criteria and identify points to be improved. The involvement of the decision maker meant the study was aligned with their perception. However, it limited the research regarding their vision of the context described. Furthermore, it does not show whether the suggestions for improvement were successful. 3.2 Multicriteria Decision Making methods Multicriteria Decision Making (MCDM) comprises methods and techniques to assist or support people and organizations in making decisions that are influenced by multiple criteria, bearing in mind the need for prior specification of the objectives intended by the decision maker compared against the alternatives (Bana e Costa, 1992; Rosenhead, 1989; Salgado, Belderrain & Silva, 2009). 3.2.1 The Analytic Network Process The ANP method is a generalization of the AHP which allows dependencies between the criteria and influences between the alternatives to be analyzed. The ANP does not follow the axiom of independence and if there is dependency between the criteria or influence among the alternatives it judges how dependent a criterion is on another and how much an alternative is influenced by or influences others (Hernández, Marins & Salomon, 2011; Kravchenko & Seredenko, 2011; Mostaf et al. 2015). The possibility of negotiating tangible values with intangible ones is one of the great attractions of the AHP, according to Salomon and Montevechi (1998). The possibility of negotiating values that do not need to be totally independent is the great attraction of the ANP. Silva et al. (2009) proposed the application of the ANP in 3 steps as follows: 1- Formulation of the Problem, 2-Judgments and 3-Structuring of the supermatrix and obtaining the results. Saaty and Vargas (2006) suggest using the Super Decisions software to perform comparisons and algebraic calculations, referring to the matrix of judgments between related nodes and clusters that present interrelated elements. 3.2.1.1 Applications of the ANP in PMS design Carlucci (2010) used the ANP to select key performance indicators in a case study at a sofa maker. The model assessed the importance of the existing performance indicators in the manufacturing process and prioritized a set of indicators to provide adequate information with which to evaluate and make decisions. The method proved to be efficient in prioritizing performance criteria, and assessing the influence among them. Piratelli and Belderrain (2010) addressed the design of a PMS based on TPP using the ANP method for modeling and rating performance indicators for the Production Engineering course at an institution of higher education. The model and its results represent the needs of stakeholders (students, organizations, the educational institution and society/government) through 58 performance indicators divided into four groups: IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 337 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 satisfaction, processes, skills and contribution (TPP faces). The model was shown to robustly and accurately reflect the strengths and weaknesses of the course. Hu, Wang and Wang (2012) built a tailor-made homestay PMS by reviewing the literature and interviewing experts to make the evaluation framework more comprehensive and more practical. They adopted the ANP to obtain the weights and verify the performance of the homestay business through diffuse theory. As far as the fundamentals are concerned, owners and customer groups both weigh the surroundings of the site as well as the characteristics, quality of service, operation and management. As far as the overall performance of the homestay is concerned, customer groups consider it to have reached a satisfactory level. Guimarães and Salomon (2015) presented an assessment of the priorities of reverse logistics indicators in a small footwear factory in the state of Ceará, through the ANP. They found that there is a similarity to the study in a sample of Brazilian companies for the first four indicators found, which is mainly explained by the priority level of economic drivers 55.7% and the image 24.6%, and the influences that these indicators receive from other indicators. For the other indicators, there were changes in priorities in relation to the sample survey with Brazilian companies. They considered the influence of the type of enterprise included in the research and reverse logistics programs. Kucukaltan, Irani and Aktas (2016) built a PMS for the logistics sector in Turkey, combining the Balanced Scorecard to identify the performance criteria and since the indicators are not independent of each other, the ANP method to analyze the interrelationships. 43 indicators were found, and 15 of them were identified as the most important in the logistics sector. In contrast to common expectations regarding the importance of some specific metrics (eg. timely delivery), research indicates that the educated employee is the most important performance indicator for competitiveness. During the research, a lack of studies that use the ANP was noted, with most articles admitting a lack of dependency between the criteria and applying the Analytic Hierarchy Process (AHP). Ho and Ma (2018) conducted a review of the literature on integrated AHP approaches and applications published between 2007 and 2016 and compared those studies with articles published during the previous decade (1997-2006). Performance evaluation, according to this study, is the second most commonly used problem, second only to vendor selection and evaluation. 4. The proposed method – operational procedures The objective of this section is to present the method proposed by Piratelli (2010) applied in a higher education institution of the state of São Paulo, and to apply it in the company that manufactures equipment for the food industry, as well as provides maintenance services. Figure 1 shows the sequence of this method. The first phase of the method is constructivist and qualitative and includes two stages. Its goal is to structure the PMS using the SODA methodology. The second phase is rationalist and quantitative and includes two stages. It models the relationship between IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 338 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 the performance criteria (identified in the structuring stage) to order them for evaluation of the organization's overall performance. Hence, it uses the ANP method. In phase 1, the first stage is divided into two steps: identifying the stakeholders in the organization, those who are concerned with the problem, and then identifying the decision makers, those who have decision-making authority, the designers of the PMS. The second stage has seven steps. The first step is to define the label or briefly describe the problem to be solved, which in this case is a PMS for the organization studied. In the second step there will be a brainstorming session with decision makers and stakeholders at the company (Primary Assessment Elements (PAE) are generated from the label). In the third step, the individual cognitive maps are drawn. The fourth step is the aggregation of individual maps into a single map. The construction of the aggregate map, the fifth step, is done at a meeting between the facilitator and decision makers to arrive at a consensus to validate the collective learning map. The analysis of the aggregate map by the facilitator is the sixth step, and is essential to the success of the rest of the process. It must identify the concepts in each line of reasoning (tails, means and ends), branches and clusters of the map. The seventh step is the identification of Fundamental Points of View (FPVs) which are the concepts classified as essential, controllable and measurable. Some FPVs need to be broken down into more than one concept to evaluate the same performance, the Elementary Points of View (EPVs). The FPVs and EPVs form the Family of Fundamental Points of View (points that decision makers want to measure). The seventh step also classifies aggregated MC concepts in regard to TPP-satisfaction (stakeholders, satisfaction or value delivery processes, the organization's capacities and the contribution from stakeholders). The third stage (phase 2) has four steps and starts with modeling (decision problem structuring) by defining the model's objectives, clusters and network elements (performance criteria and sub-criteria). The criteria and sub-criteria are the FPVs and EPVs, respectively which were identified in the seventh step of stage 2 of the first phase. It defines the overall performance of the organization in terms of satisfaction criteria for its various stakeholders. The modeling of the PMS follows the four or five faces of the prism (clusters: satisfaction, value delivery processes, capacities and contributions of the stakeholders. Strategy is not a measurable face and, therefore, should not be incorporated into the model). The construction of the network starts in the second step, where the clusters' inter- and intra-relations of dependence are indicated, and those for each network element with the others. Dependence relations and feedback between the elements must be in accordance with the concepts identified in the MC. In the third step, the performance criteria and sub-criteria judgments for strategic definition are determined through pairwise comparisons using the Saaty scale (1980). The weighting for the performance criteria and sub-criteria determines what is strategic for the organization's performance through an array of priorities. A consistency analysis should be performed in this step. The fourth step presents the model's results, ordering the criteria priorities for the organization's performance. IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 339 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 Figure 1 Detailed sequence of the method proposed Source: Piratelli (2010) The fourth stage has five steps. In the first step, indicators are built (a performance criterion must have a descriptor or metric and a value function). The descriptor or metric is a group of impact levels associated with a value function constructed by the decision makers. In the second step, an evaluation of organizational performance is conducted. The organization is evaluated (by the decision makers) in each performance indicator, according to the level at which this indicator best represents its current situation (attributed to its respective value level, according to the value function obtained in the previous step). This evaluation provides the management priorities (third step), calculating the impact of each indicator on the organization's final performance. The impact of each indicator can IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 340 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 be defined as the potential contribution that it would make to the organization's maximum performance. In the fourth step the PMS's response sensitivity is analyzed to changes in the weights of the clusters (TPP faces). Its goal is to verify the organization's performance and management priorities with regard to the prioritization of each cluster at the expense of the others. Finally, in the fifth step, the PMS is verified by the group of decision makers. 5. The study The company being studied is classified as an intermediary goods company, and operates in the food equipment manufacturing sector performing custom designing engineering, sales and industrial maintenance projects. It employs 18 people and is ISO 9001 certified. One of the recommendations made by ISO auditors was the implementation of a PMS for the production system since it is its core business and there was not a system in place to analyze its production processes, cost, reworking and so on. The application of the proposed method follows the sequence presented in the operational procedures (section 4). Phase 1 - Constructivist Stage 1: (Steps 1 and 2) Identification of stakeholders and decision makers involved in the PMS design. Stakeholders identified in the study subject company with regard to the construction of a PMS for the production process are the board/quality coordinator and production manager. Since the coordinator of quality is a board member management and coordination of quality is considered as a single stakeholder. The identified decision maker is the director/coordinator of quality. Stage 2: Structuring the PMS using SODA. Step 1: Defining the problem label The problem label is "Building a PMS for the company's production process." Step 2: Individual brainstorming with decision makers and stakeholders Four visits were made from January to July 2014 to discuss the current production process and the difficulties and needs identified by the board. The stakeholders were worked with individually, and an example of an PAE was obtained from the board: Have indicators (number 1). Step 3: Drawing up of individual cognitive maps From May to September 2014, two cognitive maps were drawn up and transcribed by January 2015 with the help of Banxia's Decision Explorer® software. Step 4: Aggregation of individual cognitive maps After the construction of the individual maps, they were aggregated to gather all the concepts and lines of reasoning in a single map. Similar concepts were added in one single branch; distinct concepts created new lines of reasoning. In the aggregation of the IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 341 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 maps, several concepts were repeated, showing that the stakeholders have similar views on what is important for the production process and they share the same ideal about what is necessary to the rollout of the PMS. In the process of aggregating the MC, it was noted that among the aspects that differentiate the individual MCs was the concern among directors and coordinators about quality regarding ISO requirements in order to ensure certification, and the pursuit of continuous improvement in the production process. Step 5: Construction of the Aggregated MC The aggregated MC was conceived of at the meeting with stakeholders where the aggregated cognitive map was presented. The parties involved got to know the whole and learned from the other concepts and views presented. The stakeholders validated 31 concepts in 38 lines of reasoning on the congregate map. Figure 2 shows the (congregate) collective maps . Figure 2 Congregate map Step 6: Analysis of the congregate MC The congregate map allowed the identification of lines of reasoning and, therefore, the aspects most valued by the company that must be taken into account for composition of the PMS. In this study, the clusters found manually were classified according to the four faces of TPP: Process satisfaction of stakeholders, Contribution by stakeholders and Capacities. IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 342 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 Step 7: Determination of the Family of Fundamental Points of View (FPVs) and classification of concepts of the congregate MC regarding the faces of The Performance Prism. The concepts of the congregate map's 38 lines of reasoning were classified by the decision makers as Essential, Controllable and Measurable to identify the family of FPVs, whose properties are: essential, manageable, complete, measurable, operational, non-redundant, concise and comprehensible. The isolable property does not apply to the family since dependency was observed between some concepts (potential performance criteria), which justifies the choice of the ANP method for the next phase of this research. The decision makers also classified the concepts according to TPP faces. Given that the focus of the design is the construction of a PMS for the production area, the processes are the face of TPP with the highest density of lines of reasoning on the map. Chart 1 shows the FPVs and EPVs and their respective performance criteria. Measurable EPVs give rise to nine performance criteria that can measure them. IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 343 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 Chart 1 FPVs and EPVs (constructivist phase) and their performance criteria (rationalist phase) Concepts (Constructive phase) FPV EPV TPP face Performance criteria (Rationalist phase) 4 Meet deadlines ... do not meet them X Processes Deadline 5 Comply with production scheduling...delay X Processes Stock turnover, Defect index, Stoppage index, Scrap index, Rework index, Percentage of C items. 11 Track material ... do not track X Processes Stock turnover, Defect index, Stoppage index, Scrap index, Rework index, Percentage of C items. 12 Identify / evaluate raw material quality ... do not evaluate X Processes / Contribution by stakeholders Defect / scrap index / Supplier qualification index 13 Monitor supplier performance ... do not monitor X Contribution by stakeholders Supplier qualification index 15 Identify production bottlenecks ... do not identify X Processes Stock turnover, Defect index, Stoppage index, Scrap index, Rework index, Percentage of C items. 16 Monitor information on scrap... do not monitor X Processes Scrap index 17 Identify human error in the production process ... keep as is X Capacities Training 18 Monitor costs ... do not monitor X Processes Stock turnover, Defect index, Stoppage index, Scrap index, Rework index, Percentage of C items. 19 Monitor information on reworking... do not monitor X Processes Rework index 20 Controlling stock ... do not control X Processes Percentage items C / Stock turnover 21 Identify / classify items of greater and lesser importance ... do not classify X Processes Percentage items C 22 Identify / Classify supplier lead times ... keep as is X Contribution by stakeholders Supplier qualification index 24 Monitor information on the types of product defects ... do not monitor X Processes Defect index 25 Train employees ... do not train X Capacities Training 27 Perform maintenance on equipment... do not perform X Processes Defect, stoppage, scrap and rework indexes 28 Prevent unnecessary stoppages ... stop X Processes Defect, stoppage, scrap and rework indexes 29 Comply with an ISO 9001 requirement.. does not comply X Processes / Stakeholder satisfaction Working stock index defects, stop index, index scrap, rework Index Percentage of items C, delivery time. IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 344 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 Phase 2 – Rationalist Stage 3: Multi-criteria modeling of the PMS This step includes the modeling of the PMS in a network and the organization of the ordering of performance criteria and sub-criteria occur. Super Decisions software is used as a tool to facilitate the implementation of the ANP. Step 1: Decision problem structuring - definition of the model's objectives, clusters, and the network elements (performance criteria and sub-criteria). The first step in the rationalist phase begins with problem structuring. The model's objective is the organization's performance and the performance depends on the performance criteria distributed in the prism faces (clusters). Figure 3 Modeling with the PMS's ANP (production process) The performance criteria relations must be faithful to the relations identified in the congregate MC. The modeling must be based on four of the five TPP faces (satisfaction, value delivery processes, capacities and contributions from stakeholders) as strategy, not being measurable, will not be included in the model. Strategic direction can only be built after knowing the relative importance of each performance criterion. Super Decisions software was used to build the model shown in Figure 3, which shows relationships of dependence and feedback between the elements of the clusters. Step 2: Construction of the network: indication of dependency relations and feedback between the elements of the clusters The dependency relationships among the elements of the model in Figure 3 are presented in the global reach matrix, Table 1, and the local reach matrix, Table 2 (both binary). The relationships are extracted and faithful to the congregate map. For example, with regard to the global reach matrix, the processes depend on the objectives clusters, the cluster process itself that influences it and the satisfaction cluster. In the local reach matrix, for example, the defect index depends on training and the supplier qualification index. IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 345 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 Table 1 Global reach matrix Objective Processes Satisfaction Capacity Contribution Objective 0 1 1 1 1 Process 0 1 0 1 1 Satisfaction 0 1 0 0 0 Capacity 0 0 0 0 0 Contribution 0 0 0 0 0 Table 2 Local reach matrix Training Supplier Qualification Index. Stock Turnover Defect Index Stoppage Index Scrap index Rework index % of C items Delivery deadline Training 0 0 0 1 0 1 1 0 1 Supplier Qualification Index 0 0 1 1 0 1 1 0 1 Stock Turnover 0 0 0 0 0 0 0 0 1 Defect Index 0 0 0 0 0 1 1 0 1 Stoppage Index 0 0 0 0 0 0 0 0 1 Scrap index 0 0 0 0 0 0 0 0 0 Rework index 0 0 0 0 0 0 0 0 1 % of C items 0 0 0 0 0 0 0 0 0 Delivery deadline 0 0 0 0 0 0 0 0 0 Step 3: Performance criteria and sub-criteria judgments for strategic definition (and respective consistency analysis). In a meeting with the decision maker, the relative importance of the clusters and performance criteria of the PMS were judged through pairwise comparisons. The definition of the weights is strategic to improve the production process at the company, considering the priority of performance criteria. The consistency of judgment matrices was also assessed at this stage, where the facilitator presented the judgments' inconsistencies to make sure that the decision maker was aware of and agreed with them. The inconsistencies found presented a consistency ratio of <0.1 and were validated by the decision maker. In the comparison between the clusters, the weight for the cluster 'process' was greater than for the others, justified by the decision maker by the importance of that cluster to the production process. Step 4: Getting results from the model. The weights found for the performance criteria were: supplier qualification index (26%), followed by training criteria (20%) and the defect index (18%). These three criteria have a prominent position in the PMS, as they make up 64% of the total weight. The delivery IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 346 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 time has a weight of 11%, followed by the rework index, with 7%, and stock turnover and stoppages with 5%. The scrap index and the percentage of C items have the lowest weight with 4%. Stage 4: Performance evaluation, analysis of the results and validation of the PMS model. The PMS was subjected to an initial performance evaluation, in order for the decision makers to verify its robustness. Step 1: Construction of performance indicators (descriptors or metrics for performance criteria and sub-criteria) The decision maker built the performance indicators, which have to have a descriptor, whose objective is to evaluate the performance of possible action in regard to each criterion or sub-criterion (FPV or EPV). The value function (VF) is a quantitative instrument to assist the decision maker in ordering the intensity of their preferences between pairs of levels, and the Saaty scale was used for this construction. Frameworks were constructed for each of the performance criteria listed within the clusters. Step 2: Performance evaluation (identification of strengths and weaknesses of the performance) and Step 3: Obtaining management priorities In the evaluation stage of the production process, the decision-maker attributed the value of PV for each criterion corresponding to the level of impact that best represents the performance of the production process, according to built descriptors (Table 3). As there is only one decision maker, the overall performance is equivalent to the evaluation of the same for the PMS built. The defect index requires the most attention, followed by the supplier qualification index and training index. The prioritization of performance criteria allows the company to evaluate and define its strategy, improving the processes and meeting the ISO 9001 requirements that led to this study. Table 3 Levels indicators assigned by the decider and Performance priorities Indicator Weight Level indicator Performance Priorities % Performance Indicator Defect Score 18% 35% 12% 6% Supplier Qualification Index 26% 63% 10% 16% Training 20% 63% 7% 13% Inventory turnover 5% 9% 5% 1% Rework Index 7% 35% 5% 3% Deadline 11% 63% 4% 7% Stop Index 5% 33% 3% 2% Percentage C items 4% 15% 3% 1% Scrap Index 4% 33% 2% 1% Overall performance 48% Step 4: Analysis of the PMS's response sensitivity to variations in cluster weights Given that there is only one decision maker, the analysis of the PMS's response sensitivity was performed by varying the cluster weights, in order to check if changing IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 347 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 them varies the PMS's performance. Changes in the weight values were made systematically by changing the weight of a given cluster to 9 and keeping the others at 1. These disturbances took the PMS from 48% in the evaluation of the decision maker to 57% in the highest index found (contribution cluster), which did not result in a significant change to its performance. There were only changes in the weights of indicators and their orders. Step 5: Validation of the PMS In a meeting with the decision maker to validate the PMS, the PMS was considered representative and valid, and will be used as a continuous improvement tool for production processes. The meetings and the process as a whole were considered to have been opportunities to analyze the production process and its importance to the company. 5.1 Comments on the results The certification of ISO 9001 presented the need to implement the SMD for the company's productive process, and its construction allowed a comprehensive vision for the management of the productive process of this company to be developed. The opportunity to listen to the stakeholders involved in the construction of the cognitive maps was of great value to the organization, considering that the routine running and execution of the activities of the company did not allow those involved to perceive the importance of the performance measures to better guide the strategies and actions of the company. Through the use of the ANP, it was possible to visualize the performance criteria under levels of importance within the constructed model, to evaluate the current performance and to identify the management priorities that will guide the company with regard to the productive process. The management priorities obtained are highly representative, guiding where corrective actions should be aimed at improving the production process. In addition, with such priorities in place, it is possible to return to the CM and check whether the processes continue to follow the concepts mentioned, or whether they need to be changed. In the case of the defect index, for example, which is in the first position in the ranking of management priorities, as well as having significant weight for the SMD, the current level assigned by the decision maker shows the need to return to the MC, reassess the processes, and propose actions for improvement, such as: presenting the indicator to the employees involved, demonstrating the importance of the processes carried out, providing training to improve their qualification, creating an award policy for employees to encourage better results, as well as actions that the company can evaluate and implement. In second place, among the management priorities, is the supplier qualification index, which highlights the need to create a policy for supplier selection, evaluation and monitoring, which will assist both the management process itself and the fulfillment of ISO requirements 9001. The training index, which occupies the third place in management priorities and is also a requirement of the standard, demonstrates a special need for the company's human resources and will also assist in the improvement of the index of defects that occupies the first place. The other indicators in the priority list IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 348 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 should also be analyzed and actions should be planned to improve performance and assist the company in the search for continuous improvement. 6. Final considerations The proposal to design a performance indicator capable of assisting management and continuously improving the production process at a food equipment manufacturing company was achieved. The Performance Prism model was adequate to the management of the productive system of the company under study since it was able to identify the needs of the stakeholders involved and contributed to the alignment of the company's strategy to the production area. The Processes face was the one that appeared more in the concepts listed and therefore had the greatest prominence in the study. The SODA methodology proved to be a great contribution to the work, enabling learning in the construction of the MC and evolving the stakeholders in collective reflections. The construction of MCs by the facilitator proved to be a complex and difficult task, considering the particularities of the stakeholders involved in the elaboration. The use of the Decision Explorer® software favored the construction of the maps, as well as the aggregation and analysis of them. The application of the ANP as a MCDM method to aid decision making was fundamental to modeling SMD as a function of TPP. The use of the Super Decisions® software facilitated the modeling, judgments according to the Saaty scale and the inconsistencies with the decision maker. The company that was analyzed felt that the cohesion and the commitment of those involved, even though it has a very small family team, was one of the strengths to the process of construction and implementation of the SMD for the productive process. This has achieved satisfactory results with customers, and will certainly improve its strategy in the production process through SMD, which can optimize both the process itself and profitability since one of the weaknesses of the company was the absence of performance measures capable of assisting the production management process. It was verified that this process of construction of an SMD should not be static. Within a period to be defined by the company it should be reassessed in order to verify if the needs of the stakeholders presented in the MCs remain the same, or if changes in the process are necessary for some reason which would result in an update of the SMD. 6.1 Contributions and search limitations Piratelli (2010) conceived a method for the construction of an SMD and applied it in a Production Engineering undergraduate course of a private higher education institution in the interior of the state of São Paulo, Brazil. The present study used the same method to build an SMD for the production process of a company that manufactures equipment for fruit processing. Thus, the progress/contribution consists of the replication of the method in another sector, with different needs and numbers involved. The method proved to be able to support the SMD construction process as valid for the company under study. As a IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 349 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 result, we highlight the contribution to help the quality management process requested by the company to verify its management priorities, as well as to provide better knowledge of its processes and, consequently, to promote their continuous improvement. Using a method of proven effectiveness for one sector in another is to make use of scientific studies to enable improvements, promote interdisciplinary cooperation and contribute to the knowledge of the area as well as, in this case, to solve an industry problem. Limitations in the study are related to the fact that the company studied has only one decision maker, and group decisions would certainly broaden the points of view and enrich the study. For future studies, it is suggested the method proposed be applied to other types of companies in other sectors. IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 350 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 REFERENCES Amaral, D. C. & Araújo Filho, T. (1998). Aplicação da metodologia SODA no processo de desenvolvimento de produto. In: Encontro Nacional de Engenharia de Produção, 18. Niterói. Anais. Niterói:Universidade Federal Fluminense. Doi: https://doi.org/10.24873/j.rpemd.2018.05.216 Ackermann, F., Franco, L. A., Rouwette, E., & White, L. (2014). Special issue on problem structuring research and practice. EURO Journal on Decision Processes, 2 (3/4), 165–172. Doi: https://doi.org/10.1007/s40070-014-0037-6 Atkinson, M. (2012). Developing and using a performance management framework: a case study. Measuring Business Excellence, 16(3), 47-56. Doi: https://doi.org/10.1108/13683041211257402 Azevedo, R. C., Ensslin, L., Lacerda, R. D. O., França, L. A.,, Gonzalez, C. J. I., Jungles, A. E. & Ensslin, S. R. (2011). Avaliação de desempenho do processo de orçamento: estudo de caso em uma obra de construção civil. Ambiente Construido, 11(1). Doi: https://doi.org/10.1590/s1678-86212011000100007 Bana e Costa, C. A. (1992). Structuration, construction et exploitation d'un modèle multicritère d'aide à la décision. Tese (Doutorado em Engenharia de Sistemas) – Instituto Superior Técnico, Universidade Técnica de Lisboa, Lisboa, Portugal. Doi: https://doi.org/10.3934/krm.2018053 Bititci, U. S., Carrie, A.S. & Mcdevitt, L. (1997). Integrated performance measurement systems: a development guide. International Journal of Operations & Production Management, 17(5), 522-534. Doi: https://doi.org/10.1108/01443579710167230 Bititci, U. S., Turner, T. & Begemann, C. (2000). Dynamics of performance measurement systems. International Journal of Operations & Production Management, 20(6), 692-704. Doi: https://doi.org/10.1108/01443570010321676 Bortoluzzi, S. C., Ensslin, S. R. & Ensslin, L. (2011). Avaliação de desempenho multicritério como apoio à gestão de empresas: Aplicação em uma empresa de serviços. Gestão & Produção, 18(3), 633-650. Doi: https://doi.org/10.1590/s0104- 530x2011000300014 Carlucci, D. (2010). Evaluating and selecting key performance indicators: an ANP-based model. Measuring Business Excellence, 14( 2), 66-76. Doi: https://doi.org/10.1108/13683041011047876 Choong, K. K. (2013). Understanding the features of performance measurement system: a literature review. Measuring Business Excellence, 17(4), 102-121. Doi: https://doi.org/10.1108/mbe-05-2012-0031 Choong, K. K. (2014a). Has this large number of performance measurement publications contributed to its better understanding? A systematic review for research and IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 351 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 applications. International Journal of Production Research, 52(14), 4174–4197. Doi: https://doi.org/10.1108/ijppm-01-2013-0015 Choong, K. K. (2014b). The Fundamentals of Performance measurement systems: A systematic approach to theory and a research agenda. International Journal of Productivity & Performance Management, 63(7), 879-922. Doi: https://doi.org/10.1108/ijppm-01-2013-0015 Coughlan, P. & Coghlan, D. (2002). Action research for operations management. International Journal of Operations & Production Management, 22(2), 220-240. Eden, C. & Simpson, P. (1989). Using cognitive mapping for strategic options development and analysis. In: Rosenhead, J. (Ed) Rational analysis for a problematic world. London: WiIey. Ensslin, L., Dutra, A. & Ensslin, S. R. (2000). MCDA: a constructivist approach to the management of human resources at a governmental agency. International Transactions in Operational Research, 7, 79-100. Doi: https://doi.org/10.1111/j.1475- 3995.2000.tb00186.x Ensslin, S.R., Ensslin, L., Back, F. & Lacerda, R.T.O. (2013). Improved decision aiding in human resource management A case using constructivist multi-criteria decision aiding, International Journal of Productivity and Performance Management, 62(7), 735–757. Doi: https://doi.org/10.1108/ijppm-04-2012-0039 Ensslin, L., Montibeller Neto, G. & Noronha, S. M. (2001). Apoio à decisão: metodologias para estruturação de problemas e avaliação multicritério de alternativas. Florianópolis: Insular. Ensslin, L., Giffhorn, E., Ensslin, S. R., Petri, S. M. & Vianna, W. B. (2010). Avaliação do desempenho de empresas terceirizadas com o uso da metodologia multicritério de apoio à decisão. Construtivista. Pesquisa Operacional, 30(1), 125-152. Doi: https://doi.org/10.1590/s0101-74382010000100007 Ensslin, S.R., Ensslin, L. & Lacerda, R.T.O. (2011). A performance measurement framework in portfolio management, Management Decision, 49(4), 648-668. Doi: https://doi.org/10.1108/00251741111126530 Ensslin, L., Dutra, A., Ensslin, S. R., Longaray, A.A. & Dezem, V. (2016). Modelo construtivista de apoio à gestão bancária. Espacios, 37(9), 12. Doi: https://doi.org/10.14488/enegep2017_tn_sto_243_408_31385 Guimarães, J. L. S. & Salomon, V. A. P. (2015). ANP applied to the evaluation of performance indicators of reverse logistics in footwear industry. Procedia Computer Science, 55, 139–148. Doi: https://doi.org/10.1016/j.procs.2015.07.021 Hanson, J.D., Melnyk, S.A. & Calantone, R.A. (2010). Defining and measuring alignment in performance management. International Journal of Operations & http://onlinelibrary.wiley.com/doi/10.1111/j.1475-3995.2000.tb00186.x/full http://onlinelibrary.wiley.com/doi/10.1111/j.1475-3995.2000.tb00186.x/full IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 352 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 Production Management, 31(10), 1089-1114. Doi: https://doi.org/10.1108/01443571111172444 Hernández, C.T., Marins, F.A.S. & Salomon, V.A.P. (2011) Análise da importância dos indicadores de desempenho da logística reversa mediante a utilização do analytic network process. XLIII Simpósio de Pesquisa Operacional. Ubatuba/SP. Doi: https://doi.org/10.5151/marine-spolm2015-140144 Ho, W. & Ma, X. (2018). The state-of-the-art integrations and applications of the analytic hierarchy process. European Journal of Operational Research, 267(2), 399–414. Doi: https://doi.org/10.1016/j.ejor.2017.09.007 Holmberg, S. (2000). A systems perspective on supply chain measurements. International Journal of Physical Distribution & Logistics Management, 30(10), 847- 868. Doi: https://doi.org/10.1108/09600030010351246 Hu, Y. C., Wang, J-H. & Wang, R-Y. (2012). Evaluating the performance of Taiwan homestay using Analytic Network Process. Mathematical Problems in Engineering, 2012, 1-24. Doi: https://doi.org/10.1155/2012/827193 Kaplan, R. & Norton, D. P. (1997). Balanced scorecard: a estratégia em ação. Rio de Janeiro: Campus. Kravchenko, T. & Seredenko, N. (2011) Decision-making with modeling of problem situations using the analytic network hierarchy process. International Journal of the Analytic Hierarchy Process, 3(1), 28-45. Doi: https://doi.org/10.13033/ijahp.v3i1.81 Kucukaltan, B.; Irani, Z.; Aktas, E. (2016). A decision support model for identification and prioritization of key performance indicators in the logistics industry. Computers in Human Behavior, 65, 346–358. Doi: https://doi.org/10.1016/j.chb.2016.08.045 Madeira Junior, A. G., Gonçalves, T.J.M. & Belderrain, M.C.N. (2011). Estruturação do problema de avaliação da qualidade dos terminais de contêineres por meio de mapas cognitivos. SOBRAPO - Revista Eletrônica Pesquisa Operacional para Desenvolvimento. Rio de Janeiro, 3(3), 230-241, set. – dez., 2011 Doi: https://doi.org/10.14488/1676- 1901.v11i4.643 Marins, F.A.S., Pereira, M. S., Belderrain, M.C.N. & Urbina, L. M. S. (2010). Métodos de Tomada de Decisão com Múltiplos Critérios - aplicações na indústria aeroespacial. São Paulo: Edgard Blucher Ltda., p.248. Marttunen, M., Lienert, J. & Belton, V. (2017). Structuring problems for Multi-Criteria Decision Analysis in practice: A literature review of method combinations. European Journal of Operational Research, 263, 1-17. Doi: https://doi.org/10.1016/j.ejor.2017.04.041 IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 353 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 Mingers, J. & Rosenhead, J. (2004). Problem structuring methods in action. European Journal of Operational Research, 152, 530-554. Doi: https://doi.org/10.1016/s0305- 0483(97)00018-2 Mingers, J. & Brocklesby, J. (1997). Multimethodology: towards a framework for mixing methodologies. The International Journal of Management Science, 25(5), 489-509. Doi: https://doi.org/10.1016/s0305-0483(97)00018-2 Mostafa, S., Abdelhamid, T., Chileshe, N. & Dumrak, J. (2015). Decision support model using ANP to align leagile strategies to off-site manufacturing in Australia, International Journal of the Analytic Hierarchy Process, 7(3), 374-404. Doi: https://doi.org/10.13033/ijahp.v7i3.340 Neely, A. (1999). The performance measurement revolution: why now and what next. International Journal of Operations & Production Management. v. 19. n. 2, p. 205-228. Neely, A. (2005). The evolution of performance measurement research: Developments in the last decade and a research agenda for the next. International Journal of Operations & Production Management, 25(12), 1264-1277. Doi: https://doi.org/10.1108/01443579910247437 Neely, A., Gregory, M. & Platts, K. (2005). Performance measurement system design: developing a literature review and research agenda. International Journal of Operations & Production Management, 25(12), 1228-1263. Doi: https://doi.org/10.1108/01443579510083622 Neely, A., Gregory, M. J. & Platts, K. W. (1995). Performance measurement system design – a literature review and research agenda. International Journal of Operations & Production Management, 15(4), 80-116. Doi: https://doi.org/10.1108/01443579510083622 Neely, A., Richards,H., Mills, J., Platts, K. & Bourne, M. (1997). Designing performance measures: a structured approach. International Journal of Operations & Production Management, 17(11), 1131-1152. Doi: https://doi.org/10.1108/01443579710177888 Neely , A., Mills, J., Platts, K., Richards,H., Gregory, M., Bourne, M. & Kennerley, M. (2000). Performance measurement system design: developing and testing a process based approach. International Journal of Operations & Production Management, 20(10), 1119- 1145. Doi: https://doi.org/10.1108/01443570010343708 Neely, A., Adams, C. & Kennerley, M. (2002). The Performance Prism: the scorecard for measuring and managing business success. London, Great Britain: Prentice Hall, Pearson Education Limited. Piratelli, C.L. (2010). Medidas de avaliação de desempenho de uma instituição de ensino superior: uma abordagem de pesquisa operacional. 300f. Tese (Doutorado em Engenharia Aeronáutica e Mecânica. Área Produção) – Instituto Tecnológico de Aeronáutica, São José dos Campos. Doi: https://doi.org/10.14393/19834071.2014.26671 IJAHP Article:Pacheco, Piratelli, Silva, Belderrain /The design of a performance measurement system for a fruit processing equipment manufacturer International Journal of the Analytic Hierarchy Process 354 Vol. 11 Issue 3 2019 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v11i3.640 Piratelli, C. L. & Belderrain, M. C. N. (2010). Supporting the design of a performance measurement system with the analytic network process. International Journal of the Analytic Hierarchy Process, 2(1), 44-65. Doi: https://doi.org/10.13033/ijahp.v2i1.72 Rentes, A.F., Carpinetti, L.C.R. & Van Aken, E.M. (2002). Measurement system development process: a pilot application and recommendations. In: Performance Measurement And Management Conference, World Trade Centre. Boston: Performance Measurement Association. Rosa, F. S., Ensslin, S. R., Ensslin, L. & Lunkes, R. J. (2012). Environmental disclosure management: a constructivist case. Management Decision, 50(6), 1117-1136. Doi: 10.1108/00251741211238364 Rosenhead, J. (1989). Rational analysis for a problematic world. London: WiIey. Saaty, T. L. (1980). The Analytic Hierarchy Process. New York: McGraw-Hill, 1980. Saaty, T. L. (1999). Fundamentals of the analytic network process. ISAHP1999, Kobe, Japan, August 12-14. Saaty, T. & Vargas, L. G. (2006). Decision making with the analytic network process. Berlin: Springer Science Business Media. Saaty, T. L. (2008). The Analytic Hierarchy and Analytic Network Measurement Process: Applications to decisions under risk. European Journal of Pure and Applied Mathematics, 1(1), 122–196. Salgado, M. C. V., Belderrain, M. C. N. & Silva, A. C. S. (2009). Avaliação do vôo tecnológico XVT02 do Veículo Lançador de Satélites VLS-1 por meio de decisão em grupo. Journal of Aerospace Technology Management, 1(1), 79-90. Doi: https://doi.org/10.5028/jatm.2009.01017990 Salomon, V. A.P. & Montevecchi, J. (1998). Método de análise em redes: sucessor do método de análise hierárquica. Produto e Produção, 2(3), 107-117. Doi: https://doi.org/10.17771/pucrio.acad.29536 Sobreira Netto, F. (2007). Medição de Desempenho Organizacional: um estudo das vantagens e desvantagens dos principais sistemas sob as óticas teórico-acadêmica e de práticas de mercado. In: XXXI Encontro da ANPAD, Rio de Janeiro/ RJ. Doi: https://doi.org/10.5753/ersirj.2018.4656 Westcombe, M. (2002). Problem structuring: the process of SODA modelling. In: Workshop on facilitating hypertext-augmented collaborative modeling. College Park: University of Maryland. http://scholar.google.com.br/citations?view_op=view_citation&hl=pt-BR&user=Z5Kmcr0AAAAJ&citation_for_view=Z5Kmcr0AAAAJ:d1gkVwhDpl0C http://scholar.google.com.br/citations?view_op=view_citation&hl=pt-BR&user=Z5Kmcr0AAAAJ&citation_for_view=Z5Kmcr0AAAAJ:d1gkVwhDpl0C