Microsoft Word - 6B_Piratelli_Performance_Vol2_Issue1.docx IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 44 Vol. 2, Issue 1, 2010 ISSN 1936-6744 SUPPORTING THE DESIGN OF A PERFORMANCE MEASUREMENT SYSTEM WITH THE ANALYTIC NETWORK PROCESS Claudio Luis Piratelli University Center of Araraquara (UNIARA) Araraquara, SP, Brazil Email: clpiratelli@gmail.com Mischel Carmen N. Belderrain Aeronautics Institute of Technology (ITA) São José dos Campos, SP, Brazil Email: carmen@ita.br ABSTRACT The development process for a Performance Measurement System (PMS) can be split into four phases: (1) design; (2) planning and construction; (3) implementation, and; (4) operation and updating. The design phase focuses on the choice of performance indicators and is crucial to the success both of the PMS and the organization. This paper deals with the design phase for a PMS based on the Performance Prism using the Analytic Network Process (ANP) for modeling and ranking of the performance indicators. The application of the ANP as support for the PMS design was executed in the higher education sector with a view to the management of an undergraduate course in Production Engineering. The model and its results assured the representation of the various stakeholders´ objectives – in a significant and balanced manner – through 58 performance indicators distributed in four clusters: satisfaction, processes, capabilities and contribution. Keywords: Performance Measurement System, The Performance Prism, ANP, Undergraduate Course 1. Introduction According to Fernandes (2004), organizational Performance Measurement Systems (PMS) have been used for more than fifty years, when the Tableau de Board came about in France. Currently, the Balanced Scorecard (BSC) from Kaplan and Norton (1990) is the most commonly used PMS in corporations and its creators have been the most referenced in the literature over the last two decades – Akkermans and Oorschot, 2005). Following Kaplan and Norton in the ranking of references in performance measurement are the proposers of The Performance Prism (Neely et al., 2002). There are different approaches to the subdivision of the construction process into phases. The construction process for a PMS (BSC, The Performance Prism, or other) can be subdivided into three large phases (Bourne et al., 2000): design (construction), implementation and use of the performance measurements. Neely et al. (2002) proposed another subdivision for the process in four phases: design, plan & build, implement & operate and refresh (update). The first, design, focuses on the choice of measurements and their metrics. The second, plan & build, plans the construction of the PMS (type of system, form of data access, data distribution configurations and manipulation, etc.), in addition to communicating its goals to the organization. The third, implement & operate, is concerned with the operation of the PMS (use of data for management). Finally, the fourth phase, refresh, revises the PMS and refines it. Rob Typewritten Text http://dx.doi.org/10.13033/ijahp.v2i1.72 Rob Typewritten Text Rob Typewritten Text IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 45 Vol. 2, Issue 1, 2010 ISSN 1936-6744 In any of the two classifications, the design phase is crucial to the success of the PMS and of the organization – unfeasible strategies and visions and badly-planned performance indicators (PI) are the leading factors causing the failure of PMSs (Bourne et al., 2002). Neglecting this stage can result in the construction of a set of inappropriate measurements and metrics and lead to more serious consequences for an organization. According to Neely et al. (2002), organizations usually choose measurements that are easily obtained – with the focus on alternatives instead of studying appropriate measurements for the fundamental goals — value-focused thinking (Keeney, 1992). According to Suwignjo et al. (2000), organizations do not dedicate time to structuring their performance measurements and understanding their interconnections in a logical manner. This could be decisive in the success of a performance measurement system because: (1) measurements must relate to the organization’s strategy; (2) performance measurements vary from organization to organization, and; (3) performance measurements are dynamic (changing with time). Bourne et al. (2002) and Smith (2005) corroborate the paragraphs above, stating that between 40% and 60% of large companies in the USA tried to implement the BSC at the end of the last century, and 70% failed, mainly due to: ● The wrong decision about what the measure. Many companies identified their performance criteria through diverse techniques (such as Brainstorming) without critical analysis of what really is important. In failing to identify a causal relationship between the performance indicators, it is not possible to establish a strategic map and, therefore, the measurements make no sense and are unfocused. ● Failing during implementation for diverse reasons, chief among them: internal difficulties, such as boycotts by people who feel threatened, inadequate infrastructure (especially in information technology, which demands heavy investment), and a loss of focus (mainly due to the implementation time that takes from 18 to 24 months on average). The previous discussion highlights the critical importance of the design or construction phase. For this reason, this article focuses on the use of the Analytic Network Process (ANP) as a support method for the design phase of a PMS. To this end, a multi-criteria decision model will be conceived of in the form of a network, based on the framework for The Performance Prism in order to ordinate the performance indicators identified as important by the stakeholders on an undergraduate course in Production Engineering. Experimental performance evaluation will be used to validate the model. The article is structured in the following way: section 2 introduces The Performance Prism concepts; section 3 describes the ANP steps; section 4 introduces the justification for choosing the application in the education sector; section 5 introduces the model and discusses the results achieved; finally, section 6 introduces the paper conclusion. 2. The Performance Prism model In the transition from the 20th to the 21st century, The Performance Prism model came about as a more flexible proposal in regard to the BSC, capable of being applied to any kind of organization/business. The result of various workshops on performance measurement run by researchers at Andersen Consulting and the British Universities of Cranfield and Cambridge, the new model is based on three premises: (1) organizations must not center their efforts on satisfying only the expectations and needs of their shareholders and clients, but rather on all the stakeholders involved; IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 46 Vol. 2, Issue 1, 2010 ISSN 1936-6744 (2) an organization’s strategies, processes and capabilities must be well-integrated and aligned with the aim of delivering value to its stakeholders, and; (3) organizations and their stakeholders must understand their reciprocal relationships – stakeholders must contribute to organizations in order get value out of them. Such premises can be represented in the five faces of a prism, as in Figure 1. Handy (2002) defines The Performance Prism (Figure 2) as a model that helps identify the critical components of the strategies, processes and capabilities that need to be developed – from a managerial and performance control standpoint – as prerequisites for the satisfaction of stakeholders’ needs and expectations, as well as those of the organization itself. The analogy is to a prism, which, in refracting white light, illustrates the complexity of an apparently simple phenomenon (the same happens when thinking about an organization from the multifaceted standpoint of performance and management). The main difference between The Performance Prism and the BSC is the premise that, in the former, the strategies are not defined, but must be constructed by the identification of stakeholders’ needs and expectations. This affirmation is corroborated by Handy (2002), who points out the main advantage of The Performance Prism in regard to the BSC: through application of the model in an organization, following the five perspectives in Figure 2, the elements that must be approached by the managers become evident. Figure 1. The five facets of the Performance Prism (Source: Neely, 2005). Figure 2. The Performance Prism model (Source: Handy, 2002). 2.1 The first face of the prism: Stakeholders The first face of the prism aims to reflect on who the fundamental stakeholders in the organization are (investors, employees, consumers, intermediaries, suppliers, regulators, and the community) and what their needs and expectations are. According to Handy (2002), the concept “derive its measurements from strategy” is an error committed by nine out of ten citations related to the theme of performance measurement. Performance measurements must help the managers to move in the direction desired and the strategy represents only one among many routes to achieving these goals, and may therefore be wrong. Hence, instead of identifying the strategies of an organization, its stakeholders and their needs and expectations must be defined so that consistent strategies can be decided on. IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 47 Vol. 2, Issue 1, 2010 ISSN 1936-6744 An organization’s strategy must transmit its goals and a plan to achieve them. Any and every action plan seeks to create value for its multiple stakeholders. So, a Performance Measurement System must begin from the perspective of all the stakeholders involved (Neely et al., 2002). 2.2 The second face of the prism: Contribution by Stakeholders The second face of the prism aims to understand what the organization needs and wants from its stakeholders, for example: capital and credit from investors, loyalty and profitability from its clients, ideas and competencies from employees, quality materials and services from suppliers, and so on. This perspective is based on the premise that the organization needs contributions from its stakeholders to better play its role, just as they want to have their needs and expectations satisfied by it. For Handy (2002), organizations need loyal and profitable consumers, good suppliers, loyal and satisfied employees, to in return deliver valuable products and services to clients, pay promptly for supplies and reward their employees, respectively. 2.3 The third face of the prism: Strategies Based on the previous faces, the third face of the prism seeks to reflect on which of the organization's strategies it must conceive of to satisfy the stakeholders. In other words, having defined the main stakeholders, their needs and expectations, and their contributions to the organization, strategies that will be adopted so that the organization can satisfy them must be defined. In this perspective, measurements must be established, the roles of which are: (1) to identify whether the strategies defined are being implemented; (2) to make communication of the strategy within the organization clear; (3) to encourage and incentivize the implementation of the strategies, and (4) to identify whether the strategies are working as planned. Different authors have stated that, within an organization, people perform their functions better when they are evaluated by measurements. Handy, in Neely et al. (2002), says that when measurements are coherent with strategies, human behavior consistent with the strategies is achieved. According to Neely et al. (2002), 90% of managers fail in implementing their strategies, because: (1) they assume hypotheses about the organization’s performance drivers – if such hypotheses are not true, the goals will not be achieved; (2) they do not develop “capabilities” for the internal processes and/or they plan processes that are not designed to execute the strategies in practice. In this regard, the authors corroborate Kaplan and Norton (1992), ratifying that correct measurement of indicators is crucial to the development of capabilities and processes. 2.4 The fourth face of the prism: Processes Once the strategies have been defined, the fourth face of the prism aims to identify which processes the organization needs to perfect to put the strategies into practice. A process should be understood as a set of operations, stages, and events, which are necessary to the execution of a certain job. Within an organization it must be described where, when, and how the work will be done. Conceptually, these are easier to understand through representation of the system: inputs- actions-outputs-results. According to the authors of The Performance Prism, the entire process needs macro and micro measurements in order to provide an overview and identify critical details, such as the existence of bottlenecks. The whole process, then, must have someone in charge of identifying what performance measurements and metrics must be taken and by whom. Such aspects can be classified, in turn, as measurements of efficiency and measurements of effectiveness. In general, measurements of efficiency are more closely related to process inputs and actions, and IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 48 Vol. 2, Issue 1, 2010 ISSN 1936-6744 measurements of effectiveness to outputs and results. Measurements of inefficiency and variability are also important performance indicators, such as: defects, waiting time, time without adding value, overproduction, unnecessary movements, excessive stock, space wasting, pollution generated, oversizing, excessive complexity, etc. A compilation of various performance criteria common to a wide range of industrial processes can be found in the appendix of Neely et al. (2002) and in the article by Neely et al. (2005). 2.5 The fifth face of the prism – Capabilities Finally, the fifth face of the prism seeks to reflect on what capabilities need to be developed to conduct such processes. Behind an efficient and effective process there must be capabilities. Handy (2002) defines organizational capabilities as those formed by competent people, practices, technology and infrastructure capable of creating value for the stakeholders through distinct processes and operations. According to the authors of The Performance Prism, even the well-known capabilities of an organization – those that support the differentiated processes – must be constantly measured to guarantee their sustainability. This section succinctly presented The Performance Prism, a framework for organizational performance measurement that is based on performance indicators according to the various stakeholders. For additional information on the subject, reading of the authors referred to herein is recommended. 3. The Analytic Network Process The section introduces the ANP and its operation steps. The Analytic Network Process (ANP) is a multi-criteria decision-making support method from the American School, originating in Graph Theory, which allows the modeling of a decision-making problem in the form of a network, in order to achieve priorities as regards its elements (criteria and alternatives) (Saaty, 2005). In the context of the ANP, a network can be defined as a set of clusters, each one with its nodes, which can present dependency relations between each other (intra- and inter-clusters) in any direction (including feedback). If the elements of a certain network only present dependency relations in one single direction there is a hierarchical structure. In other words, a hierarchical or tree structure may be understood as a particular network case (Silva et. al., 2009). Figure 3 illustrates the representation of a decision-making problem in network form. In Figure 3, the clusters are represented by an ellipse, and the nodes belonging to a cluster are represented by full circles. The arrows indicate the relations of influence (dependency) between the elements. Figure 3. Network structure Saaty (2005) classifies the nodes in a cluster as: (1) source component: that which exercises an influence on the other elements and is not influenced; (2) intermediary component: that which exercises an Goal Criteria Alternatives IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 49 Vol. 2, Issue 1, 2010 ISSN 1936-6744 influence and is influenced by other elements, and (3) absorbing component: that which is only influenced by others. All three types of nodes are included in the example of a network structure portrayed in Figure 3. In modeling a decision-making problem in a network, the hypothesis of independence between its elements (criteria and/or alternatives), necessary for the use of the Analytic Hierarchy Process (AHP) by Saaty (1980) – one of the widest-used methods in dealing with multi-criteria problems – is left aside. According to Saaty (2005), the main advantage of the ANP over the AHP is the possibility of working with problems whose criteria, sub-criteria and/or alternatives have interdependencies, which is very common in practice. Hence, the results tend to be more effective than for the AHP as a cost to efficiency – greater analytical effort as the number of pairwise comparisons increases between the elements (Paula and Salomon, 2008). Silva et al. (2009) present three stages for the application of the ANP to a decision-making problem: 1) Formulation of the problem, 2) Judgments, and 3) Algebraic development. The procedures contained in these steps are presented in brief in Figure 4. Figure 4. Stages for the application of the ANP to a decision-making problem (Source: Silva et al., 2009). Stage 1 models the decision-making problem in two steps: the structuring of the problem and the construction of the network. IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 50 Vol. 2, Issue 1, 2010 ISSN 1936-6744 Step 1.1: In structuring of the problem, the objective of the decision-making context is defined, the clusters, elements and alternatives for decision-making. Step 1.2: The construction of the network indicates the dependency relations between the elements of the clusters. According to Silva et al. (2009), the dependency relations are described in the matrixes of global and local reach, both of which are binary. The first indicates whether there are dependency relations (1. Relation or 0. no relation) between intra- or inter-clusters (any elements). The second describes the relation of dependency for each element of the network with the rest (1 or 0) elements. Stage 2 can be summed up by the key word judgment. According to Silva et al. (2009) ,it is executed in one step and a verification sub-step. Step 2.1: Pairwise comparisons. For all the connections established in Step 1.2, pairwise comparisons must be made according to Saaty’s Fundamental Scale (1980), Chart 1. According to Silva et al. (2009), two kinds of comparison are made in the ANP: (1) between two or more elements when they influence another element in conjunction, and; (2) between two or more clusters (whenever there is at least one relation of dependency between any of its elements). Chart 1. Saaty’s Fundamental Scale (Source: Saaty, 1980). Intensity of importance Definition Description 1 Equal importance The two elements contribute equally to the goals 2 Intermediate value 3 Moderate importance Experience and judgment favors one element in relation to the other 4 Intermediate value 5 Great importance Experience and judgment strongly favors one element in relation to the other 6 Intermediate value 7 Very great importance One element is very strongly favored in relation to the other 8 Intermediate value 9 Absolute importance One element is absolutely prioritized in relation to the other Figure 5 illustrates the use of Saaty’s fundamental scale (chart 2) for pairwise comparison between two elements (X and Y) – cluster or node – in a network. importance of X in regard to Y X=Y importance of Y in regard to X 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Figure 5. Illustration of Saaty’s fundamental scale to compare two elements (Source: Silva et al., 2009). The judgments made in comparisons (1) and (2) described above are computed in decision matrixes of order n, reciprocal and positive (where n is the number of elements compared). For each decision matrix A, the eigenvector and maximum eigenvalue are calculated ( which express the priority value (W) of the elements compared. IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 51 Vol. 2, Issue 1, 2010 ISSN 1936-6744 According to Gomes (2004), W e  can be obtained, respectively, by (1) and (4) or (5). ninAWAW m j jii ,...,1/)()( 1    , (1) Where: )2( ,...,1)( 1 nj a a AW m i ij ij ji    Such that: 1       1,…,                                                    3                                                                          4   Or  1                                                               5   Another way to get the priority vector is by calculating the normalized geometric average for each line of matrix A (Saaty, 2001). Silva et al. (2009) also present the theoretical foundation of the numerical power method to get W and  , used by Saaty (2005). This method is relatively simpler for matrixes of large dimensions and seeks convergence for the eigenvalue and eigenvector through the iteration of vectors (Oliveira and Belderrain, 2008). As the number of comparisons to be made in the ANP depends on the number of judgment matrixes between related nodes and between clusters that present inter-related elements, Saaty proposes the use of SuperDecisions software to make comparisons and the respective algebraic calculations in Stage 3. Equation (6) presents the number of comparisons necessary for the N judgment matrixes ni a decision- making problem, where ni is the order of the i-th matrix. ∙ 1 2                                                                          6 Step 2.1.1 verifies the consistency of the comparison judgments made in 2.1. The decision matrix A is said to be consistent when all the value judgments are perfect, which means to say that aij x ajk = aik, for any i, j, k (Gomes, 2004). In other words, the eigenvector for A ( must be the closest to n. Nevertheless, Saaty (1980) admits a certain degree of inconsistency in human judgments, above all in quadratic matrixes with n>3, through the indicator IC defined in (7):  1                                                                    7    IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 52 Vol. 2, Issue 1, 2010 ISSN 1936-6744 Hence, he proposes the calculation Consistency Ratio (CR), obtained by (8), where IR (Index Random) are randomly tabled values in function of n, presented in chart 2. According to Gomes (2004), when n=2, CR must be zero; when n=3, CR must be less than 0.05; when n=4, CR must be less than 0.09 and; for n>4 CR must be less than or equal to 0.10.                                                                                   8 Saaty (1994) observes that the inconsistency indicator must be used to alert the decision maker to the need for a possible revision of their judgments. In other words, the rectification of judgments is not compulsory. Chart 2. IR values for squared matrixes of order n, according to the Oak Ridge National Laboratory, USA (Source: Gomes, 2004). n 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 IR 0 0 0.58 0.9 1.12 1.24 1.32 1.41 1.45 1.49 1.51 1.48 1.56 1.57 1.59 Stage 3, according to Silva et al. (2009), comprises the construction of the supermatrices and getting results. The authors subdivide it into 5 steps: Step 3.1: Construction of the supermatrix without weight W (generically represented in Figure 6). In W, the network clusters are defined by Ch (h=1, 2, ..., N) and the respective nodes by hnn, in the following form: Nhnhh eee ,,, 21  . The components Whh of the supermatrix represent the matrixes obtained by aggregating the eigenvectors obtained in the pairwised comparisons between the elements through step 2.1 (Silva et al., 2009). Figure 6. Standard structure of a supermatrix (Source: Saaty, 2005). Step 3.2: Obtaining the weighted supermatrix through the multiplication of each matrix Whh by the corresponding weight of the cluster Ch. Step 3.3: Verification of the weighted supermatrix. According to Saaty (2005), the weighted supermatrix obtained in Step 3.2 must be stochastic in regard to the columns. Otherwise, it must be normalized by the sum in regard to the columns. Step 3.4: Calculation of the limit matrix through the power method described by Oliveira and Belderrain (2008). The limit matrix must also be stochastic in regard to the columns. Step 3.5: Obtaining the results for ranking of the alternatives and criteria, according to the limit matrix obtained in Step 3.4. IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 53 Vol. 2, Issue 1, 2010 ISSN 1936-6744 For a better illustration of the ANP method, reading of Saaty (2005), Figueira et al. (2005), Saaty and Vargas (2006) and Silva et al. (2009) is recommended. For an understanding of the functioning of the SuperDecisions software, reading of Saaty (2003) is recommended 4. Application in higher education This section briefly explains why a PMS should be constructed for educational institutions, more specifically, for undergraduate courses. In Brazil, most educational institutions still work without a control and management system using indicator measurement (Bressiani et al.,2001). The managers of Higher Educational Institutes (HEI) and programs, for the most part, do not have a management system that includes performance indicators for their business units (courses) with the level of detail and scope necessary for effective management. Course and program coordinators are generally aware of the performance indicators used by the Brazilian Ministry of Education (MEC) – government – for course accreditation processes, but often do not have access to other important indicators such as the financial impact of their courses and the satisfaction of those benefited directly and indirectly by the service provided. Many private HEI managers have financial control through indicators that often do not describe the true cost/benefit relations of the programs/courses in their departments/institutions. A review of the literature shows the increase over time of work proposing the use of the PMS as a strategic management system for HEIs. Nevertheless, most of them suggest performance indicators for the institutions, but none deals with the application of the BSC in its full conception, according to the proposal by Kaplan and Norton (1992). Higher Educational Institutions may have dozens of separate business units, focused on diverse areas of knowledge with their own goals, targets and operational strategies. Their corporate strategies and missions, however, tend to be generic. Porter (1998) suggests that competition in a given sector is at the level of business units and not between corporations. It makes sense, then, that PMSs be molded to the business units, as their strategies must support the corporation’s strategy. 4.1 Evaluation by Ministry of Education (MEC) Currently, Brazilian higher education is evaluated by two agents: one internal to the HEI itself, called the Self-Evaluation Commission, whose main instrument is institutional self-evaluation; the other is carried out by external agencies linked to the Ministry of Education (MEC), which carry out inspections: registration and re-registration of institutions, authorization, accreditation and renewal of accreditation for courses, and examining student performance. These agents and their instruments comprise SINAES – the National Higher Education Evaluation System – created by Law n° 10,861 (2004). The three main instruments used by SINAES are:  Institutional evaluation that aims identify the profile, vocation and operation of the HEI, through its activities, courses, programs, projects and sectors, respecting the diversity and specifications of the different academic organizations;  Evaluation of the undergraduate courses, with a view to conceptualizing the teaching conditions offered through three main categories: didactic-pedagogical organization, the academic and technical-administrative staff, and the physical installations;  Evaluation of student performance on undergraduate courses, via the National Student Performance Examination (ENADE), to verify student performance in terms of general and specific knowledge acquired, besides skills and competencies required of the career chosen. IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 54 Vol. 2, Issue 1, 2010 ISSN 1936-6744 However, what is noted in practice is that the mechanisms of evaluation created by MEC are not being carried out as planned by SINAES. The very expansion of higher education in the face of the limited capacity of the Ministry to evaluate the universe of higher courses has become one of the major stumbling blocks in the system. The focus in evaluation now on diagnosing the quality of higher education is centered on ENADE and its indicators. According to the concepts obtained from students on a course that was assessed by this exam, MEC calculates a Preliminary Course Concept (CPC), which can dispense it from a renewal of accreditation process (CPC ≥ 3) (Normative Ordinance n. 4, 2008). For Macedo et al. (2005), even if the full range of SINAES evaluation were fully implemented, it would not be enough to contemplate the size and heterogeneity of current higher education. In the scope of evaluation, other authors have suggested that an effective project to reform higher education must conceive of an evaluation system that can handle the different educational segments and institutions that comprise the educational system. In other words, it must be able to identify the strengths and weaknesses of each HEI in order to be able to improve them. This makes it necessary for the instruments of evaluation to be able to describe the trajectory followed by the institution and, mainly, compliance with its mission, through careful measurement of pertinent performance indicators. Piratelli et al. (2009) present the current Evaluation Instrument that supports the government accreditation processes for undergraduate courses and some of their potential deficiencies. The results of evaluation process simulations have identified that potential injustices could occur when using the instrument without the subjective intervention of an evaluation commission. The authors particularly show that it is easily possible to approve a course that is failing to comply with the demands of the labor market and pedagogically poorly structured because it has a good physical structure and good people. They also show that a course committed to pedagogical and professional aspects may not have a minimum concept of accreditation because of some deficiencies in the way it hires academic staff and promotes them. In addition, from the Balanced Scorecard or The Performance Prism standpoint, MEC’s diagnostic and evaluation instruments focus on internal process and learning/growth indicators, and ignore performance from the point of view of the other stakeholders – and mainly for the direct and indirect clients, who chiefly benefit from their processes and products. The arguments presented in this section justify the application of The Performance Prism in higher education, more specifically for the management of university courses. IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 55 Vol. 2, Issue 1, 2010 ISSN 1936-6744 Figure 7. The PMS modeled with ANP in the SuperDecisions software. 5. Modeling of The Performance Prism in the ANP To introduce the model and its results, this section follows the same stages described in section 3. The results are subdivided in sections 5.1 (model and results), 5.2 (evaluation) and 5.3 (model sensitivity analysis). 5. The model and its results Stage 1: Formulation of the problem. Step 1.1: The objective of the problem is to order the performance indicators (PI) for an undergraduate course in Production Engineering. The PI were identified as relevant by its stakeholders (students, academic staff, educational institute (manager), organizations, and society). The ranking will serve to evaluate the course performance from the standpoint of the various stakeholders, allowing better strategic management (focusing on critical points). Step 1.2: The PMS was modeled on 4 of the 5 faces of the prism: satisfaction, value delivery processes, capabilities and stakeholder contributions. Each face of the prism is represented by a cluster in the ANP. So the clusters are Satisfaction, Processes, Capabilities and Contribution. The course performance is measured through satisfaction indicators for the stakeholders: students, academic staff, educational institution, society and organizations. Satisfaction indicators for each stakeholder, in turn, depend on the nodes belonging to each of the other clusters. As in section 2, the face “strategy” is not measurable, and so is not incorporated in the model. The decision-makers (collegiate board) understand that strategic direction can only be conceived after knowing the importance of each indicator from the stakeholders’ standpoint. Figure 7 presents the PMS model. Chart 3 has the key for the clusters and nodes in the PMS network. Chart 3: Model key Code Indicator or cluster name Code Indicator or cluster name Ab1 laboratories basic scope Cu3 activities content integration Ab2 laboratories specific scope Cu4 teaching multi-methodology Ab3 professional training laboratories scope D global performance Bb1 collection update (library) Dc1 focused complementary activities Bb2 number of copies (library) Dc2 guideline percentages IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 56 Vol. 2, Issue 1, 2010 ISSN 1936-6744 Code Indicator or cluster name Code Indicator or cluster name C1 economic competencies to engineering Dc3 students intern supervision C2 general competencies to engineering En1 assisting pupils C3 human competencies to engineering En2 technical visits C4 socio-political competencies to engineering Eq1 laboratories basic equipment C5 technical competencies to engineering Eq2 laboratories specific equipment CAP cluster: Capabilities Eq3 laboratories professional equipment Cd1 teaching didactics Fi1 classroom climate Cd2 teaching experience Fi2 classroom space Cd3 professional experience (teacher) Fi3 classroom furniture Cd4 degree (teacher) GP cluster: Global Performance Co1 collegiate board performance I cluster: main indicator for each subnet Co2 structuring core faculty performance IE1 library Co3 time to coordinate IE1 cluster: Library Cp1 level of entering students IE2 laboratories Cp2 secretarial IE2AB cluster: Laboratories : Scope CP2 cluster: Secretarial IE2EQ cluster: Laboratories : Equipment/Material Cp3 working infrastructure IE3 classrooms CP3 cluster: working infrastructure IE3FI cluster: Classrooms – physical aspects Cp4 coordination IE3RD cluster: Classrooms – didactic resources CP4 cluster: Coordination IF1 percentage of occupation Cp5 capacity of academic staff IF2 percentage revenue /revenue potential CP5 cluster: Capacity of academic staff IO1 ENADE concept Cp6 teaching infrastructure IT1 scientific bases CP6 cluster: Teaching infrastructure IT2 work rooms Cp7 pedagogical policy project P1 publications CP7co cluster: project consistency P2 service provision (to society) CP7cu cluster: Curriculum indicators P3 social projects CP7dc cluster: compliance by DCs P4 solving organizations problems Cs1 curriculum-goal P4 cluster: Resolution of problems Cs2 curriculum-intended egress profile P5 competencies to engineering Ct1 contribution academic staff P5 cluster: Competencies CT1EN cluster: teaching commitment PE cluster: Processes CT1RE cluster: rule compliance Rd1 internet access Ct2 employability Rd2 multimedia (availability) Ct3 contribution HEI-staff Re1 staff punctuality CT3 cluster: Contribution HEI-staff Re2 staff deadlines Ct4 contribution students and society Rp1 quality of interns CT4FI cluster: Financial Rp2 research CT4IO cluster: Others (ENADE) Rp3 monographies applied Ct5 contribution HEI –students (scholarships P-E) S1 students (satisfaction) Cti1 research incentive S2 academic staff (satisfaction) Cti2 working hours S3 HEI (satisfaction) Cti3 career plan S4 organizations (satisfaction) Cti4 remuneration S5 society (satisfaction) IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 57 Vol. 2, Issue 1, 2010 ISSN 1936-6744 Code Indicator or cluster name Code Indicator or cluster name CTRB cluster: Contribution Sa1 number of terminals Cu1 professional attributions Sa2 employee qualification Cu2 regional focus SAT cluster: Satisfaction The dependency relations between the elements of the network in Figure 7 are presented in the global reach matrixes (charts 4 and 6) and in the local reach matrixes (charts 5 and 7). Stage 2: Judgment matrixes and verification of consistencies Step 2.1: The judgment matrixes were filled in, mostly, by members of the collegiate board (5 academic staff and 2 students). Each group of stakeholders judged, in a consensual manner, the pertinent indicators and clusters. The indicators referring to the HEI stakeholders (director board) and organizations were judged by members external to the collegiate board in identical questionnaires shown in Figure 5. In these cases the geometrical average was used to get the priority vectors (Saaty and Peniwati, 2007). The judgments of the indicators for the satisfaction cluster were made in a consensual manner by all members of the collegiate board, resulting in the priority vector in Table 1. Chart 4. Global reach matrix. GP SAT PE CAP CTRB GP 0 1 0 0 0 SAT 0 0 1 1 1 PE 0 0 1 1 1 CAP 0 0 0 0 0 CTRB 0 1 0 0 0 Chart 5. Main network local reach matrix. CAP CTRB G P PE SAT Cp1 Cp2 Cp3 Cp4 Cp5 Cp6 Cp7 Ct1 Ct2 Ct3 Ct4 Ct5 D P1 P2 P3 P4 P5 S1 S2 S3 S4 S5 C A P Cp1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Cp2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Cp3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Cp4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Cp5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Cp6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Cp7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 C T R B Ct1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 Ct2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 Ct3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Ct4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 Ct5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 G P D 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 P E P1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 P2 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 P3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 P4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 P5 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 S A T S1 1 1 0 0 1 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 S2 0 0 1 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 58 Vol. 2, Issue 1, 2010 ISSN 1936-6744 S3 0 0 0 1 0 0 0 0 0 0 1 0 0 1 1 1 0 0 0 0 0 0 0 S4 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 S5 0 0 0 0 1 1 1 0 0 0 0 0 0 1 1 1 0 1 0 0 0 0 0 Chart 6. Subnet global reach matrices. Chart 6. Subnet global reach matrices (continuance) Chart 7. Subnet local reach matrices. I P4 I P5 I CP2 I 0 1 I 0 1 I 0 1 P4 0 0 P5 0 0 CP2 0 0 I CP3 I CP4 I CP5 I 0 1 I 0 1 I 0 1 CP3 0 0 CP4 0 0 CP5 0 0 I CP6 I Ct3 I IE1 I 0 1 I 0 1 I 0 1 CP6 0 0 Ct3 0 0 IE1 0 0 I CT4IF CT4IO I CT1EN CT1RE I IE2AB IE2EQ I 0 1 1 I 0 1 1 I 0 1 1 CT4IF 0 1 0 CT1EN 0 0 0 IE2AB 0 0 0 CT4IO 0 0 0 CT1RE 0 0 0 IE2EQ 0 0 0 I CP7co CP7cu CP7dc I IE3FI IEFRD I 0 1 1 1 I 0 1 1 CP7co 0 0 0 0 IE3FI 0 0 0 CP7cu 0 0 0 0 IEFRD 0 0 0 CP7dc 0 0 0 0 IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 59 Vol. 2, Issue 1, 2010 ISSN 1936-6744 Table 1. Priority vector for the elements of the satisfaction cluster (stakeholders). Stakeholder Priority Stakeholder Priority S1 23.81% S4 19.05% S2 19.05% S5 19.05% S3 19.05% Step 2.2. The consistency of the judgments was guaranteed in a satisfactory manner in all the comparison matrixes. Stage 3 corresponds to the construction of the supermatrices and obtaining results from the model. Steps 3.1 to 3.4 have been omitted for reasons of space. Step 3.5: Results. The results for the model (weight of indicators in the evaluation) are in Table 2. I I P4 Rp1 Rp2 Rp3 P5 C1 C2 C3 C4 C5 I P4 0 1 1 1 I P5 0 1 1 1 1 1 Rp1 0 0 0 0 C1 0 0 0 0 0 0 Rp2 0 0 0 0 C2 0 0 0 0 0 0 Rp3 0 0 0 0 C3 0 0 0 0 0 0 C4 0 0 0 0 0 0 C5 0 0 0 0 0 0 I I I S1 S2 Cp2 It1 It2 Cp3 Co1 Co2 Co3 Cp4 S1 0 0 0 It1 0 0 0 Co1 0 0 0 0 S2 0 0 0 It2 0 0 0 Co2 0 0 0 0 I Cp2 1 1 0 I Cp3 1 1 0 Co3 0 0 0 0 I Cp4 1 1 1 0 I I Cd1 Cd2 Cd3 Cd4 Cp5 IE1 IE2 IE3 Cp6 Cd1 0 1 0 0 0 IE1 0 0 0 0 Cd2 0 0 0 0 0 IE2 0 0 0 0 Cd3 0 0 0 0 0 IE3 0 0 0 0 Cd4 0 0 0 0 0 I Cp6 1 1 1 0 I Cp5 1 1 1 1 0 CP7 I Cp7 Cs1 Cs2 Cu1 Cu2 Cu3 Cu4 Dc1 Dc2 Dc3 En1 En2 Re1 Re2 Ct1 CP7 Cp7 0 1 1 1 1 1 1 1 1 1 En1 0 0 0 0 0 Cs1 0 0 0 0 0 0 0 0 0 0 En2 0 0 0 0 0 Cs2 0 0 0 0 0 0 0 0 0 0 Re1 0 0 0 0 0 Cu1 0 0 0 0 0 0 0 0 0 0 Re2 0 0 0 0 0 Cu2 0 0 0 0 0 0 0 0 0 0 I Ct1 1 1 1 1 0 Cu3 0 0 0 0 0 0 0 0 0 0 Cu4 0 0 0 0 0 0 0 0 0 0 I Dc1 0 0 0 0 0 0 0 0 0 0 IE1 Bb1 Bb2 Dc2 0 0 0 0 0 0 0 0 0 0 I IE1 0 1 1 Dc3 0 0 0 0 0 0 0 0 0 0 Bb1 0 0 0 Bb2 0 0 0 I I IE2 Ab1 Ab2 Ab3 Eq1 Eq2 Eq3 IE3 Fi1 Fi2 Fi3 Rd1 Rd2 I IE2 0 1 1 1 1 1 1 I IE3 0 1 1 1 1 1 Ab1 0 0 0 0 0 0 0 Fi1 0 0 0 0 0 0 Ab2 0 0 0 0 0 0 0 Fi2 0 0 0 0 0 0 Ab3 0 0 0 0 0 0 0 Fi3 0 0 0 0 0 0 Eq1 0 0 0 0 0 0 0 Rd1 0 0 0 0 0 0 Eq2 0 0 0 0 0 0 0 Rd2 0 0 0 0 0 0 Eq3 0 0 0 0 0 0 0 I CT4IF CT4IO I Cti1 Cti2 Cti3 Cti4 Ct3 IF1 IF2 IO1 Ct4 Cti1 0 0 0 0 0 CT4IF IF1 0 0 0 0 Cti2 0 0 0 0 0 IF2 1 0 0 0 Cti3 0 0 0 0 0 CT4IO IO1 0 0 0 0 Cti4 0 0 0 0 0 I Ct4 1 1 1 0 I Ct3 1 1 1 1 0 Subnet Ct3 CT3 CT3 Subnet Ct4 IE3RD Subnet IE2 IE2AB IE2EQ IE2AB IE2EQ IE1 Subnet IE3 IE3FI IE3RD IE3FI CT1RE CT1EN CT1RE Subnet IE1 IE1 CP6 CP6 CP4 Subnet Ct1 CT1ENCP7dcCP7cuCP7co CP7co CP7cu CP7dc P4 P5 CP2 CP3 CP5 Subnet CP4 CP4 Subnet CP6 P4 P5 CP2 CP3 CP5 Subnet P4 Subnet P5 Subnet CP2 Subnet CP3 Subnet CP5 Subnet CP7 IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 60 Vol. 2, Issue 1, 2010 ISSN 1936-6744 Table 2. Weight of indicators in the evaluation of course performance. Indicators Weight in evaluation Indicators Weight in evaluation Indicators Weight in evaluation Indicators Weight in evaluation Cd1 3.58% Eq2 2.15% C4 1.44% Sa1 1.08% Cd2 3.58% Eq3 2.15% C5 1.44% Ab1 1.07% P2 3.31% P1 2.01% IT1 1.39% Fi1 1.06% Ab2 3.22% Cu1 1.98% Rp2 1.38% Fi3 1.06% Ab3 3.22% Cu2 1.98% Sa2 1.33% Bb1 1.03% P3 3.14% Cu3 1.98% IF1 1.31% IT2 1.00% Co2 2.92% Cu4 1.98% IO1 1.31% Dc1 0.99% Co3 2.92% Rp3 1.70% Cp1 1.27% Dc2 0.99% Cd3 2.79% Cd4 1.64% Bb2 1.25% Dc3 0.99% Fi2 2.66% Ct5 1.61% Cti4 1.25% Rd1 0.98% Rp1 2.37% Ct2 1.52% Cti2 1.25% Eq1 0.72% Cs1 2.34% Co1 1.46% Cti3 1.25% Re2 0.58% Cs2 2.34% C1 1.44% En2 1.16% IF2 0.44% En1 2.31% C2 1.44% Re1 1.16% Cti1 0.42% Rd2 2.21% C3 1.44% 5.2 Evaluation of Course Performance Through the ranking of indicator priorities obtained (Table 2), it was possible to measure course performance individually, as well as globally. For each indicator the course collegiate board described five levels of impact. For each descriptor, a function value was constructed, varying between 0 and 100 points with a corresponding one for each level of impact. Course evaluation was done by the collegiate board. Table 3 presents the satisfaction percentage for each stakeholder (performance). The global course indicator was 58.62%. Figure 8 details the performance indicators pertinent to each stakeholder, illustrating chromatically what must be prioritized by the manager: red-orange: worst performance (urgent action required); yellow- green: average performance (attention: improvement is required) and; green-Dark green: good performance – no action required, keep monitoring). IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 61 Vol. 2, Issue 1, 2010 ISSN 1936-6744 Figure 8. The PMS: its indicators and evaluation of the course for management. Table 3. Course performance by satisfaction of the stakeholders. Final Performance % S1 57.10 S2 38.48 S3 54.74 S4 65.81 S5 59.07 D 58.62 Table 4. Course performance by stakeholder satisfaction (with equal weights). Final Performance % S1 57.89 S2 41.25 S3 54.88 S4 64.58 S5 58.99 D 58.50 5.3 Analysis of the model’s sensitivity As each judgment was made consensually by the stakeholder groups, analysis of the model’s sensitivity was carried out by varying the priority vector in Table 1. Considering all the stakeholders to have equal importance in judging the indicators, there were no significant changes in the evaluation of the course (despite small variations in the priority vectors). Table 4 presents the satisfaction percentage for each stakeholder (performance) for this configuration. Table 5 presents the new ranking of the indicators. The new global performance indicator was 58.50%. Two other sensitivity analyses were carried out: (1) considering internal stakeholders to be twice as important as the external ones, and (2) considering the external stakeholders to be twice as important as the internal ones. The results of the model are in Tables 6 and 7, respectively. Indicadores % Indicadores % Indicadores % Indicador % Indicador % Cd1 50.00% Fi2 0.00% P2 0.00% Rp1 20.00% Cd1 50.00% Cd2 100.00% Rd2 55.00% P3 37.50% Cs1 100.00% Cd2 100.00% Ab2 75.00% IT1 50.00% Co2 80.00% Cs2 100.00% P2 0.00% Ab3 65.00% Sa2 50.00% Co3 100.00% Cu1 75.00% Ab2 75.00% Cd3 95.00% Cti4 100.00% P1 100.00% Cu2 75.00% Ab3 65.00% Fi2 0.00% Cti2 50.00% Ct5 0.00% Cu3 10.00% P3 37.50% En1 35.00% Cti3 25.00% Co1 80.00% Cu4 30.00% Cd3 95.00% Rd2 55.00% Fi1 50.00% IF1 50.00% Rp3 100.00% Fi2 0.00% Eq2 90.00% IT2 30.00% IO1 50.00% C1 75.00% Cs1 100.00% Eq3 20.00% Rd1 0.00% IF2 60.00% C2 80.00% Cs2 100.00% Cd4 100.00% Cti1 25.00% 54.74% C3 90.00% Rd2 55.00% Ct2 70.00% 38.48% C4 90.00% Eq2 90.00% Sa2 50.00% C5 80.00% Eq3 20.00% Cp1 25.00% Rp2 0.00% Cu1 75.00% Bb2 50.00% Dc1 50.00% Cu2 75.00% En2 60.00% Dc2 100.00% Cu3 10.00% Re1 60.00% Dc3 50.00% Cu4 30.00% Sa1 0.00% 65.81% Cd4 100.00% Ab1 50.00% Ct5 0.00% Fi1 50.00% C1 75.00% Fi3 70.00% C2 80.00% Bb1 25.00% C3 90.00% Rd1 0.00% C4 90.00% Eq1 55.00% C5 80.00% Re2 85.00% Bb2 50.00% 57.10% Ab1 50.00% Fi3 70.00% Bb1 25.00% Dc1 50.00% Dc2 100.00% Dc3 50.00% Rd1 0.00% Eq1 55.00% 59.07% S1 S5S2 S3 S4 IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 62 Vol. 2, Issue 1, 2010 ISSN 1936-6744 Table 5. Weights for performance indicators when all the stakeholders have equal weights. Indicators Weight in evaluation Indicators Weight in evaluation Indicators Weight in evaluation Indicators Weight in evaluation Cd1 3.49% Rp3 2.08% Cd4 1.60% Dc2 1.01% Cd2 3.49% Cu1 2.02% Cp1 1.52% Dc3 1.01% Rp1 2.91% Cu2 2.02% Co1 1.45% Sa1 0.98% Co2 2.89% Cu3 2.02% IT2 1.42% En2 0.90% Co3 2.89% Cu4 2.02% C1 1.39% Re1 0.90% P2 2.80% IT1 1.97% C2 1.39% Ab1 0.90% Ct5 2.80% Rd2 1.85% C3 1.39% Fi1 0.89% P1 2.76% Ct2 1.81% C4 1.39% Fi3 0.89% Cd3 2.72% En1 1.81% C5 1.39% Bb1 0.86% Ab2 2.70% Eq2 1.80% IF1 1.29% Rd1 0.82% Ab3 2.70% Eq3 1.80% IO1 1.29% Eq1 0.60% P3 2.64% Cti4 1.76% Sa2 1.20% Cti1 0.59% Cs1 2.39% Cti2 1.76% Bb2 1.05% Re2 0.45% Cs2 2.39% Cti3 1.76% Dc1 1.01% IF2 0.43% Fi2 2.23% Rp2 1.70% Through sensitivity analysis, no significant changes were observed in the course evaluation results. 6. Conclusion The design phase, in focusing on the choice of measurements and their metrics, is crucial to the success of the PMS and the organization. This article approached the design phase for a PMS bases on The Performance Prism using the Analytic Network Process (ANP) for modeling and ranking of the performance indicators. Application of the ANP as support for the PMS design was carried out in the higher education sector to aid management for an undergraduate Production Engineering course. The model and its results assured the representation of diverse stakeholders “voices” – in a significant and balanced manner – through 58 performance indicators distributed in 4 clusters: satisfaction, processes, capabilities and contribution. The results of the model – ranking of the indicators and measurement of performance – were useful for the course’s collegiate board to be able to reflect on issues that were lacking in information and to discuss action plans make improvements. The group was satisfied with the representative nature and robustness of the model, as sensitivity analysis did not significantly impact the performance evaluation results. It can be stated that the model is legitimate in accurately reflecting strengths and weaknesses of the course. Confidence in the model was guaranteed not only by the participation of diverse stakeholders (structuring of indicators and judgments), but also because the collegiate board was willing to construct value Table 6. Course performance by stakeholder satisfaction (1). Table 7. Course performance by stakeholder satisfaction (2). Final Performance % Final Performance % S1 54.88 S1 60.52 S2 39.55 S2 37.94 S3 56.58 S3 51.89 S4 64.49 S4 66.59 S5 54.67 S5 62.63 D 56.68 D 61.13 IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 63 Vol. 2, Issue 1, 2010 ISSN 1936-6744 descriptors and functions for each performance indicator which were useful in operating a subjective evaluation by the group. Although this model does not stipulate the MEC stakeholder (government), many performance indicators include this agent’s interest. The next step in this work will be to guarantee, through Monte Carlo simulation, that if the course performance is rated as good by the PMS constructed, the government stakeholder (MEC) will be satisfied (with a score sufficient for accreditation). REFERENCES Akkermans, H.A. & van Oorschot, K.E. (2005). Relevance assumed: a case study of Balanced Scorecard development using system dynamics. Journal of the Operational Research Society, 56(8), 931-941. Bourne, M., Mills, J., Wilcox, M., Neely, A. & Platts, K. (2000). Designing, implementing and updating performance measurement systems. International Journal of Operations & Production Management, 20(7), 754-771. Bourne, M., Neely, A., Platts, K. & Mills, J. (2002). The success and failure of performance measurement initiatives. International Journal of Operations & Production Management, 22(11), 1288-1310. Bressiani, F., Alt, P.R.C. & Massote, A.A. (2001). Using the Balanced Scorecard as an Instrument of Improving Performance of an Institution of Higher Education. Proceedings of the Brazilian Congress on Engineering Education (COBENGE), Porto Alegre, RS, Brazil, 29. [in Portuguese] Fernandes, B.H. (2004). Competencies and organizational performance: an empirical study. Doctoral dissertation, Faculdade de Economia e Administração, USP, Brazil. [in Portuguese] Figueira, J., Salvatore, G. & Ehrgott, M. (2005). Multiple Criteria Decision Analysis. State of the art surveys. Boston: Springer Science+Business Media, LLC. Gomes, L.F.A.M., Araya, M.C.G. & Carignano, C. (2004). Decision making in complex scenarios: Introduction to Discrete Methods of Multicriteria Decision Support. São Paulo: Pioneira Thomson Learning. [in Portuguese] Handy, C. (2002). The Performance Prism: Measuring more is easy, measuring better is hard. In: Neely, A., Adams, C. & Kennerley, M (2002). The Performance Prism: The Scorecard for Measuring and Managing Business Success. Great Britain: Prentice Hall Financial Times. Kaplan, R. & Norton, D. (1992). The Balanced Scorecard – measures that drive performance. Harvard Business Review, 70(1), 71-79. Keeney, R.L. (1992). Value-Focused Thinking: a path to creative decisionmaking, Cambridge (USA): Harvard University. Kennerley, M.P. & Neely, A.D. (2000). Performance Measurement Frameworks – A Review. Proceedings of the 2nd International Conference on Performance Measurement, 291-298. Law n. 10,861 of April 14th, 2004. (2004). Brasília, Federal District, Brazil. Retrieved Jun 26, 2005, from http://www.planalto.gov.br. [in Portuguese] IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 64 Vol. 2, Issue 1, 2010 ISSN 1936-6744 Lee, M.C. (2007). A Method of Performance Evaluation by Using The Analytic Network Process and Balanced Scorecard. Proceedings of The International Conference on Methods and Applications of Multicriteria Decision Making, Gyeongju, Republic of Korea. Macedo, A.R., Trevisan, L.M.V., Trevisan, P & Macedo, C.S. (2005). Higher Education in the XXI Century and the Brazilian university reform. Revista Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, 13(47), 127-148. [in Portuguese] Neely, A., Adams, C & Kennerley, M. (2002). The Performance Prism: The Scorecard for Measuring and Managing Business Success. Great Britain: Prentice Hall Financial Times. Neely, A. & Bourne, M. (2000). Why measurement initiatives fail. Measuring Business Excellence, 4(4), 3-6. Neely, A., Gregory, M. & Platts, K. (2005). Performance measurement system design: developing a literature review and research agenda. International Journal of Operations & Production Management, 25(12), 1228-1263. Normative Ordinance n. 4, of August 5th, 2008. (2008). Brasília, Federal District, Brazil. Retrieved August 7, 2008 from http://www.anaceu.org.br/conteudo/legislacao/portarias. [in Portuguese] Oliveira, C.A. & Belderrain, M.C.N. (2008). Considerations on finding the AHP priority vectors. In: Encuentro Nacional de Docentes de Investigación Operativa. Posadas, Argentina. [in Portuguese] Paula, D.C. & Salomon, V.A.P. (2008). Using indicators to analyze the application of methods for decision making with multiple criteria. Proceedings of XL Simpósio Brasileiro de Pesquisa Operacional “A Pesquisa Operacional e o uso racional dos recursos hídricos”, João Pessoa, PE. [in Portuguese] Piratelli, C.L., Belderrain, M.C.N & Azzolini Jr., W.A. (2009). Using Monte Carlo Simulation for Analyze potentials shortcomings of the Evaluation Instrument for Course Accreditation. Proceedings of 16nd Simpósio de Engenharia de Produção: “Ensino de Engenharia de Produção”, Bauru, SP, 16. [in Portuguese] Porter, M.E. (1998). Competitive Advantage to Corporate Strategy. In: Montgomery, C. A. & Porter, M. E. (1998). Strategy: Searching for competitive advantage. Rio de Janeiro-RJ: Campus. [in Portuguese] Saaty, R.W. (2003). Decision Making in Complex. The Analytic Hierarchy Process for Decision Making and The Analytic Network Process for Decision Making with Dependence and Feedback [Superdecisions Tutorial]. Retrieved Jun 01, 2008, from http://www.superdecisions.com. Saaty, T.L. (1980). The Analytic Hierarchy Process. New York (USA): McGraw-Hill. Saaty, T.L. (2001). Decision Making with Dependence and Feedback: The Analytic Network Process, Pittsburgh, PA: RWS Publisher. Saaty, T.L. (1994). Fundamentals of Decision Making and Priority Theory – With The Analytic Hierarchy Process. Pittsburgh, PA: RWS Publications. Saaty, T.L. (2005). Theory and applications of the analytic network process: decision making with benefits, opportunities, costs, and risks, Pittsburgh, PA: RWS Publications. IJAHP ARTICLE: Piratelli, Belderrain / Supporting the Design of a Performance Measurement System with the Analytic Network Process International Journal of the Analytic Hierarchy Process 65 Vol. 2, Issue 1, 2010 ISSN 1936-6744 Saaty, T.L. & Peniwati, K. (2007). Group decision-making: Drawing out and reconciling differences. Pittsburgh, PA: RWS Publications. Saaty, T. & Vargas, L.G. (2006). Decision Making with the Analytic Network Process. Economic, Political, Social and Technological Applications with Benefits, Opportunities, Costs and Risks, New York: NY: Springer Science+Business Media, LLC. Silva, A.C., Nascimento, L.P., Ribeiro, J.R & Belderrain, C. N. (2009). ANP and Ratings model applied to SSP. Proceedings of the International Symposium on the Analytic Hierarchy Process, Pittsburgh, PA, , 10, 1-11. Smith, M. (2005). The Balanced Scorecard. Financial Management, Feb 2005, 27-28. Suwignjo, P, Bitici, U.S. & Carrie, A.S. (2000). Quantitative models for performance measurement system. International Journal of Production Economics, 64, 231-241.