IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 372 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 A FRAMEWORK USING THE ANALYTIC HIERARCHY PROCESS FOR LOCAL GOVERNMENTS IN JAPAN TO EVALUATE PROJECTS BASED ON OUTCOMES Yoichi Iida 1 School of General Education and Management Studies, Suwa University of Science Nagano, Japan iida@rs.sus.ac.jp Ryo Koizumi School of General Education and Management Studies, Suwa University of Science Nagano, Japan su2.cha.n8.gmail.com ABSTRACT Currently, in Japan, women are required to participate and advance in the workplace because of the decline in the labor population. However, it is very difficult for them to do so because of the patriarchal system within Japanese history. In the 1990s, local governments began various projects to encourage women to participate in society. At the same time, local government administrations were introduced to project evaluation to boost the efficiency of their projects. However, project evaluation did not function well because project evaluation needs to be linked to outcomes of the program, and the outcome step was often performed independently. The purpose of this paper is to show how to perform project evaluation to promote women’s active participation in society in Japan. Using the Analytic Hierarchy Process, we show how to evaluate projects in a way that can connect to the outcomes of the program. This framework can also be used to calculate the degree of contribution of projects in the program from two perspectives. First, the decision makers in a local government evaluate the outcomes of the program and second, the people in charge of the projects that make up the program do the evaluation for their own projects. In this paper, we evaluated actual projects of a city in Japan and were able to show the direction they should take, such as reduction or expansion for each project in the next fiscal year, by using numerical values obtained in the process of calculation by this framework. 1 Acknowledgements: The first author wishes to express his deepest gratitude to late Professor Thomas L. Saaty for his very useful advice and warm encouragement. The authors wish to thank the anonymous referees for their helpful and useful suggestions. IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 373 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 Keywords: Analytic Hierarchy Process (AHP); local government; gender equality in Japan; project evaluation; contribution degree of projects. 1. Introduction Currently, the labor population is decreasing in Japan so women's participation and advancement in the workplace is required. However, it is very difficult for women to work outside of the home because of the patriarchal history of Japan. To overcome these difficulties, Japan has passed legislation such as the Basic Act for Gender-Equal Society established in 1999. Also, the local governments in Japan have focused on programs and projects to promote gender equality using project evaluation. On the other hand, about twenty years have passed since evaluation was introduced in Japan’s government administrations. At first, this evaluation was introduced as part of decentralization reform. It then changed to administrative evaluation for budget cuts to overcome financial crises in local government (Yamaya, 2016). After that, evaluation evolved with the Plan-Do-Check-Act cycle and Key Performance Indicator (KPI) which is like evaluating a company under the influence of New Public Management (NPM). In the midst of these changes, the original purpose of administrative evaluation by local governments became ambiguous. As a result, local government administration evaluation has begun to be regarded as useless work and officials who had to draw up the evaluation sheets have come to feel “evaluation tiredness”. The work is tedious and complicated because several programs are planned to achieve each policy (and have to be evaluated), and several projects are planned to achieve each program (and have to be evaluated) and actual administrative activities have to be implemented for these projects, and they too have to be evaluated. So, in this mass of activity, it is not easy to find the actual evaluation of projects that should be linked to the program. Most Japanese researchers in policy evaluation point out that if projects are not evaluated relative to each other with respect to the program, it is impossible to use administrative evaluation budgeting for the following fiscal year or to decide the future direction of the project (like whether or not it should be reduced). This is an ongoing problem of administrative evaluation in local government. The purpose of this paper is to promote women’s active participation in society in Japan using the Analytic Hierarchy Process (AHP) to perform evaluation projects that can be connected to the outcomes of the program to which they belong. It is difficult to measure how much each project affects the program if the people who evaluate outcomes must adhere to quantitative evaluation. With the AHP, it is possible to include intangibles in the evaluation. In this framework, outcomes of a program are represented as viewpoints and the projects that make up the program are divided into their functions. Eventually, the degree to which projects contribute to the program and their relative value can be calculated. By “contribution degree” we mean the relative value a project contributes to the program. This framework has two other perspectives to calculate contribution degrees of projects. IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 374 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 The decision makers in a local government make the judgments to evaluate the outcomes of the program, and the people in charge of the projects make the judgments related to their specific projects. In Section 4, we show how to evaluate the projects using the AHP and link these evaluations to the outcomes of the program. In Section 5, we give an example of real projects to promote women’s active participation in Chino city in Japan. This framework differs from the AHP because zero is accepted as an evaluation value. On the other hand, we did not use the Analytic Network Process to calculate the contribution degrees because of the need to keep the process simple so users could perform the calculations themselves. 2. Literature review First, we reviewed papers about project evaluation. There are many about policy evaluation and administrative evaluation by Japanese researchers (for example, Ishihara, 2005; Yamaya, 2016). Generally, an administrative evaluation in a local government is almost a project evaluation (Tanaka, 2014). Mie prefecture in Japan was the first local government to conduct administrative evaluations in 1996. Thereafter, many researchers studied how project evaluations were done in Mie, and many local administrative bodies in Japan imitated that method; for example, they drew up project evaluation sheets themselves. As is well-known, policies, programs and projects are often represented as the hierarchy shown in Figure 1. On the other hand, as mentioned above, an administrative evaluation is often a project evaluation and the hierarchy may be elaborated to include several criteria as shown in Figure 2. In many articles, the most famous criteria in Figure 2 are the 3E, namely economy, efficiency and effectiveness. Other typical criteria that are used are legality, regularity, security, resilience, necessity, urgency, priority, adequacy, satisfaction, transparency, quickness and so on (Tanaka, 2014). Figure 1 Hierarchy of administrative activities in a local government Program Project 1 Project 2 Project 3 Policy IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 375 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 Figure 2 Simple hierarchy for the relative evaluation of projects in a program Thus, the relationship between policies, programs and projects is described as a hierarchy and there are many articles using the AHP that do this in administrative policy evaluation. Miyagawa (1994) picked the AHP up in his book about policy science. Kinoshita published books collecting practical use examples of the AHP in local governments (Kinoshita & Taji, 2005; Kinoshita & Ohya, 2007). However, they did not do project evaluation linked to outcomes of the program. To do that, we need to investigate a logic model for project evaluation in evaluation theory. Administrative evaluation in Japan initially referred to the way it was done in the United States of America or the United Kingdom. Stipak pointed out the importance of subjective measures in program evaluation (Stipak, 1987). Hatry (2007) wrote “These objectives normally should be stated in general, not quantitative, terms and should remain reasonably stable”. His words “not quantitative” are very suggestive, although his performance measurement was actually a part of project evaluation related to efficiency. The AHP can deal with evaluation qualitatively without losing reasonable stability. As is well-known, Saaty applied the BOCR model (ANP) to local governments (Saaty & Özdemir, 2005). The problems of selection of projects, project evaluation, risk assessment in projects and so on have been studied using the AHP (Dey, 2006; Mahmoodzadeh et al, 2007; Huang et al, 2008; Zayed et al, 2008). Furthermore, there were often political or social problems to solve using the AHP (Saaty, 1982, 1983; Vargas, 1990). Next, we reviewed papers about gender equality in Japan. There are a lot of articles and books about women's social advancement written by Japanese researchers. For example, Naito, who is a very famous researcher of the Japanese gender equality issue, classified viewpoints on the topic in three ways including the division of labor by gender, the wage gap by gender and the job promotion gap by gender (Naito, 2015). We used these viewpoints as criteria in our hierarchy in Figure 5. In their book, Naito & Yamaya (2015) showed some functions promoting a society with gender equality. We used their results in the hierarchy in Figure 5. 3. Objectives of the study Most local governments do not use relative project evaluation because it is difficult to evaluate them relatively with respect to a program. In fact, it is difficult to elaborate outcome indicators of the program and the projects, particularly if the evaluators are restricted to quantitative evaluations as is prevalent in Japanese administrative evaluation. Program Project 1 Project 2 Project 3 Criterion 1 Criterion 2 Criterion 3 Criterion 4 Criterion 5 IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 376 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 As a result of what they perceive to be useless administrative evaluations for the purpose of decision-making, officials begin to feel “evaluation tiredness.” AHP has the following strong points:  It is possible to incorporate qualitative matters like human feelings and experiences into the evaluation, while at the end the results are comprehensive evaluation values similar to those obtained with quantitative evaluations.  It is possible to divide key items of evaluation into several viewpoints using a hierarchy, while different individuals can perform the relative evaluations that directly concern the alternatives.  Evaluators can argue the weight of each item during the process of evaluation and confirm the results of the process of evaluation later. This means that the AHP is suitable for solving the above-mentioned problem of administrative evaluation. Furthermore, as shown in Section 2, we already have the viewpoints related to outcomes of the program to promote women’s active participation in society in Japan which are approved by most people (Naito & Yamaya, 2015). The purpose of this paper is to show how to perform a project evaluation so the evaluation of projects can be connected to the outcomes of the program. We show how to use the AHP to easily perform relative evaluation of projects. 4. A framework of project evaluation linked to the program There are three points in this AHP framework: (1) What is the relative evaluation value of a project? (2) What is a hierarchy to calculate the relative evaluation values? and (3) Who does the evaluations by performing the pairwise comparisons of items in the hierarchy? By answering these three questions, we can perform relative evaluation of projects similar to the AHP. We constructed this framework under the following conditions:  The hierarchy in the AHP incorporates outcomes of the program in order to perform relative evaluation of projects.  We use only the existing numerical values on administrative evaluation documents such as project evaluation sheets.  The people in charge of the projects evaluate only their projects regardless of the outcome of the program, while the decision makers in a local government evaluate outcomes of the program and how much each project contributes to the program. First, we defined relative evaluation values of projects as follows: Relative evaluation value = Contribution degree × Individual project value Most local governments have already performed administrative evaluations and calculated the numerical values of projects. In many cases, they are calculating a target achievement rate based on numerical targets and actual results as shown on the left side of Figure 3. Several numerical targets are set for each project. So, in the above IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 377 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 formulation we defined an individual project value by averaging those target achievement rates. This allows us to use the already existing numerical values from administrative evaluation documents. Figure 3 The relationship between projects and the program Furthermore, to calculate the above relative evaluation values we need to decide how much each project contributes to the program (Figure 3). We simply called this “contribution degree of a project”. This contribution degree of each project is calculated using the AHP. Thus, the relative evaluation value of each project is calculated by multiplying the average of target achievement rates of it, which is the individual project value, by its contribution degree. Next, we constructed the hierarchy shown in Figure 4 to calculate the relative evaluation values of the projects. From the definition of relative evaluation values, it is sufficient to calculate the contribution degrees of the projects with the AHP. Firstly, the program is divided into some viewpoints which are the criteria used to evaluate projects in the AHP. In general, the outcome of the program is evaluated by these numerical indicators obtained from the viewpoints of the program. If adequate indicators are found for these outcomes, we can evaluate the program. Second, projects are divided into some functions to distinguish the program from the combined viewpoints. It is difficult for the evaluators to directly compare two projects with respect to each viewpoint because their judgement may include their bias. It is fairer and easier to compare functions of projects instead of projects with respect to each viewpoint. It is better to select viewpoints and functions based on academic results of researchers in the field related to the program. In this way, it is easier to obtain approval from many citizens. Thus, we constructed the four-layer hierarchy with a viewpoints level and a function level as in Figure 4. This hierarchy has two parts, namely, the strictly AHP upper part and the judgments inserted through pairwise comparisons in the lower part. The upper part is calculated by standard AHP. The pairwise comparison part does not always use the AHP because the evaluation values are often zero. Contribution degree Individual project value = the average of target achievement rates Target achievement rate = Actual Result / Numerical Target Intermediate outcome Final outcome Program Project n Target Activity Result Target Activity Result Target Activity Result ... IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 378 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 Figure 4 Hierarchy consisting of two parts Finally, we decided on the people to perform pairwise comparisons of items in the hierarchy. There are two parts in the hierarchy in Figure 4. The decision makers in a local government performed pairwise comparisons of items in the AHP part, while the people in charge of the project performed them for the project performance. These people were chosen because items in the AHP part relate to the future direction and the present state of the city, and items about the projects connect to the actual activity. We will show how to synthesize weights of items which were obtained by these pairwise comparisons using a real project in the next section. 5. Actual project evaluation about gender equality in a city We performed a project evaluation for the real projects of Chino city in Japan using the proposed framework. Chino city has been individually evaluating the projects using the original project evaluation sheets for about ten years (Inazawa, 2012). These project sheets include many numerical values, but they did not perform a relative evaluation of projects well. We applied the framework to projects belonging to the program to promote gender equality in Chino city, whose name is "realizing a society promoting women’s participation and advance in the workplace" and whose number is 06-05. This program began with ten projects, however, by 2016 it only had four active projects whose numbers are 06-05-01, 02, 03 and 08 (Chino city, 2017). These projects are as follows: Project 01 Holding lectures to enlighten gender equality Project 02 Gender equality promotion for employment and in the workplace Project 03 Gender equality promotion in subregion Project 08 Gender equality promotion for family and at school Program Viewpoint 1 Function 1 Viewpoint 2 Function 2 Function 3 Function 4 Project 1 Project 2 Project 3 AHP part Pairwise comparison part IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 379 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 5.1 Confirming the individual project values of the projects We need individual project values to calculate relative evaluation values of projects. As mentioned in the previous section, we used the average of these target achievement rates of each project (Table 1). For example, it follows from the row of Project 01 that this project has four types of target achievement rates and the average of them is 93.750%. If the target achievement rate is larger than 100%, then we replaced it with 100% because it means that the target of the activities was not adequate. For example, we replaced 166.67% with 100% for Project 02. Table 1 Individual project values of projects on project evaluation sheets (Chino city, 2017) Unit (%) Projects\Items 1 2 3 4 Average Project 01 100 100 100 75 93.750 Project 02 53.33 166.67 100 140 100 - 84.443 Project 03 90 0 100 77.14 66.785 Project 08 300 100 40 - - 70.000 5.2 Constructing a hierarchy and evaluators We constructed a four-layer hierarchy (Figure 5). First, the name of the program was at the top of the hierarchy and those of the projects were at the bottom. Next, we decided on three viewpoints as outcomes of the program which are the division of labor by gender, the wage gap by gender and the job promotion gap by gender. These are the most important viewpoints of gender equality in Japan (Naito, 2015). The division of labor by gender: Viewpoint that women are domestic laborers at home, while men work outside. The wage gap by gender: Viewpoint that women’s wages are lower than men’s wages. The job promotion gap by gender: Viewpoint that women are excluded from decision making and therefore find it hard to be promoted in the workplace. IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 380 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 Figure 5 Hierarchy for calculating contribution degrees We decided on functions of projects to achieve those viewpoints referring to the book by Naito and Yamaya (2015). These functions are place for advice, opportunity of learning, environment for access to information and interaction with specialists. Furthermore, we added two functions with characteristics of administrative activities and gender equality that are grasp of current situation and support of promotion in the workplace. Place for advice: Function of consultation such as telephone and individual consultation. Opportunity of learning: Function of learning issues about gender equality such as courses and workshops. Environment for access to information: Function of offering information to solve issues related to gender equality such as library and website. Interaction with specialists: Function of connecting to specialists like psychological counselors and attorneys to solve issues related to gender equality such as a party after a lecture. Grasp of current situation: Function of surveying the current situation such as questionnaire survey and consciousness investigation. Promotion of gender equality society The job promotion gap by gender The wage gap by gender The division of labor by gender Project 01 Project 02 Project 03 Project 08 Environment for access to information Opportunity of learning Place of advice Support of promotion in the workspace Grasp of current situation Interaction with specialists IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 381 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 Support of promotion in the workplace: Function of supporting the promotion of women's participation in decisions in the workspace such as certifications for companies promoting women’s participation and advance in the workplace. The director and the assistant director in charge of the program who are decision makers in a local government were chosen to perform pairwise comparisons in the AHP part, and the chief person in charge of these projects was chosen to make those comparisons for the project evaluation part (Figure 4). We thought that two or more people should decide on the future direction of the program. 5.3 Calculating the contribution degrees of projects First, we calculated the weights of items in the hierarchy in Figure 5 and obtained synthesized evaluation values of functions. After that, we calculated the contribution degrees at the end of this section. 5.3.1 Calculating the synthesizing evaluation values of functions We interviewed the director and the assistant director in charge of the program separately to perform pairwise comparisons of the AHP. We combined these values using geometric means and calculated weights using the eigenvector method of the AHP (Table 2). The question is which viewpoint is more important for realizing a society promoting women’s participation and advance in the workplace? And how much more important? Table 2 Pairwise comparison table and weights of viewpoints with respect to the program Program Division Wage Job Weights Division of labor 1 4.583 3.873 0.665 Wage gap 0.218 1 0.378 0.111 Job promotion 0.258 2.646 1 0.224 C.I.= 0.036 Similarly, we combined their values using geometric means and calculated weights of six functions with respect to each viewpoint (Tables 3 to 5). The question is which function is more important (as functions of activities) to achieve the viewpoint? And how much more important? Table 3 Evaluation table of functions with respect to the first viewpoint of the division of labor Division Adv. Learn Info. Spec. Current Prom. Weights Advice 1 1.095 1.183 2.449 4.472 1.118 0.237 Learning 0.913 1 0.447 0.913 5 1 0.167 Information 0.845 2.236 1 0.913 5.292 1 0.219 Specialists 0.408 1.095 1.095 1 3.873 1 0.166 Current 0.224 0.2 0.189 0.258 1 0.775 0.055 Promotion 0.894 1 1 1 1.291 1 0.156 C.I.= 0.069 IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 382 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 Table 4 Evaluation table of functions with respect to the second viewpoint of the wage gap Wage gap Adv. Learn Info. Spec. Current Prom. Weights Advice 1 0.183 0.236 5 0.913 0.236 0.094 Learning 5.477 1 1.732 3.873 3.873 0.913 0.290 Information 4.243 0.577 1 3.873 3.873 0.775 0.223 Specialists 0.2 0.258 0.258 1 2 0.236 0.060 Current 1.095 0.258 0.258 0.5 1 0.236 0.059 Promotion 4.243 1.095 1.291 4.243 4.243 1 0.272 C.I.= 0.112 Table 5 Evaluation table of functions with respect to the third viewpoint of the job promotion gap Promotion Adv. Learn Info. Spec. Current Prom. Weights Advice 1 0.2 0.224 0.2 0.267 0.154 0.032 Learning 5 1 2 3.873 3.464 0.169 0.192 Information 4.472 0.5 1 1.732 1 0.183 0.102 Specialists 5 0.258 0.577 1 1.291 0.183 0.085 Current 3.742 0.289 1 0.775 1 0.183 0.079 Promotion 6.481 5.916 5.477 5.477 5.477 1 0.510 C.I.= 0.100 In summary, the following evaluation table of functions with respect to viewpoints was obtained (Table 6). Table 6 Evaluation table of functions with respect to each viewpoint Functions \ Viewpoints Division Wage gap Promotion gap Advice 0.237 0.094 0.032 Learning 0.167 0.290 0.192 Information 0.219 0.223 0.102 Specialists 0.166 0.060 0.085 Current 0.055 0.059 0.079 Promotion 0.156 0.272 0.510 Finally, we calculated synthesizing evaluation values of functions with respect to the program. We obtained Table 7 from Tables 2 and 6 by the following calculation: ( 0.237 0.094 0.032 0.167 0.290 0.192 0.219 0.223 0.102 0.166 0.060 0.085 0.055 0.059 0.079 0.156 0.272 0.510) ( 0.665 0.111 0.224 ) = ( 0.175 0.187 0.193 0.136 0.061 0.248) . IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 383 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 Table 7 Synthesizing evaluation values of functions with respect to the program Functions Advice Learn Info. Special Current Promotion Total Valuation 0.175 0.187 0.193 0.136 0.061 0.248 1.000 5.3.2 Calculating the weights of projects with respect to functions We interviewed the chief person in charge of these projects and then asked her to perform pairwise comparisons of the AHP using a questionnaire. In this way, because all the projects do not have all the functions, we needed to decide which projects had meaningful functions. Table 8 was made with the chief person in charge, the predecessor and the vice-chief person in charge before the chief person in charge performed pairwise comparisons. Table 8 Relationship between projects and functions Advice Learn Info. Special Current Promotion Project 01 NA NA Project 02 NA Project 03 Project 08 NA According to Table 8, the chief person in charge compared projects with respect to each function by pairwise comparison of the AHP obtaining Tables 9 to 14. The question is which project has more function? And how much more important? Table 9 Weights of projects with respect to function, place for advice Advice Project 01 Project 03 Weights Project 01 1 1/5 0.167 Project 03 5 1 0.833 C.I.= 0 Table 10 Weights of projects with respect to function, opportunity of learning Learn Project 01 Project 02 Project 03 Project 08 Weights Project 01 1 1 5 1/7 0.145 Project 02 1 1 3 1/7 0.121 Project 03 1/5 1/3 1 1/7 0.051 Project 08 7 7 7 1 0.682 C.I.= 0.088 IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 384 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 Table 11 Weights of projects with respect to function, environment for access to information Info. Project 01 Project 02 Project 03 Project 08 Weights Project 01 1 5 5 1 0.411 Project 02 1/5 1 3 1/5 0.113 Project 03 1/5 1 1/5 0.064 Project 08 1 5 5 1 0.411 C.I.= 0.052 Table 12 Weights of projects with respect to function, interaction with specialists Special Project 01 Project 02 Project 03 Project 08 Weights Project 01 1 7 3 1 0.368 Project 02 1/7 1 1/3 1/7 0.050 Project 03 1/3 3 1 1/7 0.109 Project 08 1 7 7 1 0.473 C.I.= 0.042 Table 13 Weights of projects with respect to function, grasp of current situation Current Project 02 Project 03 Project 08 Weights Project 02 1 1 5 0.455 Project 03 1 1 5 0.455 Project 08 1/5 1/5 1 0.091 C.I.= 0 Table 14 Weights of projects with respect to function, support of promotion in the workplace Promotion Project 02 Project 03 Project 08 Weights Project 02 1 5 5 0.701 Project 03 1/5 1 1/3 0.097 Project 08 1/5 3 1 0.202 C.I.= 0.068 IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 385 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 In summary, we obtained the following table about weights of functions included in projects. Table 15 Weight of each function included in projects Advice Learn Info. Special Current Promotion Project 01 0.167 0.145 0.411 0.368 0 0 Project 02 0 0.121 0.113 0.050 0.455 0.701 Project 03 0.833 0.051 0.064 0.109 0.455 0.097 Project 08 0 0.682 0.411 0.473 0.091 0.202 Total 1.000 1.000 1.000 1.000 1.000 1.000 5.3.3 Calculating the contribution degrees of projects Here, we calculated the contribution degree of each project to the program by combining Table 7 and Table 15. In fact, Table 16 was obtained by the following calculation: ( 0.167 0.145 0.411 0.368 0 0 0 0.121 0.113 0.050 0.455 0.701 0.833 0.051 0.064 0.109 0.455 0.097 0 0.682 0.411 0.473 0.091 0.202 ) ( 0.175 0.187 0.193 0.136 0.061 0.248) = ( 0.186 0.253 0.235 0.327 ). Table 16 Contribution degrees of the projects with respect to the program Projects Project 01 Project 02 Project 03 Project 08 Contribution degrees 0.186 0.253 0.235 0.327 5.4 Calculating the relative evaluation values of the projects and implication Finally, we calculated the relative evaluation values of projects with respect to the programs by the following formula defined in Section 4: Relative evaluation value = Contribution degree × Individual project value. As mentioned in Section 4, individual project values are the average of target achievement rates of the project and this is shown in Table 1. Consequently, we obtained relative evaluation values in Table 17. This was our goal for the framework. IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 386 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 Table 17 Contribution degrees and relative evaluation values of projects with respect to the program Project 01 Project 02 Project 03 Project 08 Contribution degrees 0.186 0.253 0.235 0.327 Individual project values 93.750 84.443 66.785 70.000 Relative evaluation values 17.421 21.356 15.663 22.872 Once all of the calculations were completed the following results were obtained from Table 17:  Project 01 has the lowest contribution degree and a low relative evaluation value, although it has the highest individual project values (Table 17). It does not have the functions “support of promotion in the workspace” (Table 8) which has high synthesizing evaluation values (Table 7). Consequently, Project 01 should be reduced due to deep cuts in the budget in the next fiscal year.  Project 02 has high contribution degree and high relative evaluation value (Table 17). It has the function “support of promotion in the workspace” in a high ratio (Table 15) which has high synthesizing evaluation values (Table 7). Consequently, Project 02 should be maintained at the status quo in the next fiscal year.  Project 03 has the lowest relative evaluation and low contribution degree (Table 17). It has fewer functions of “support of promotion in the workspace”, “environment for access to information” and “opportunity of learning” (Table 15). Consequently, Project 03 should be cut from the budget in the next fiscal year.  Project 08 has the highest relative evaluation value and the highest contribution degree (Table 17). It has almost all functions in the highest ratio (Table 15). Consequently, Project 08 should be expanded in the next fiscal year.  If both Projects 01 and 03 are reduced in the next fiscal year, then a new project will need to be started for the function “place of advice” (Table 8). 6. Implications and limitations The developed framework of project evaluation linked to outcomes of the program in this paper has the following strong points:  It is possible to clarify outcomes of the program which are the same as those of the projects (Figure 5 and Table 2).  It is possible to clarify the current situation of the local government with regards to the program (or the policy) by giving weights of importance to viewpoints of the program (Table 2). For example, Chino city is developing gender equality as far can be seen from the results in Section 5.  It is possible to clarify the role of each project because it is divided into several functions (Tables 8 and 15).  It is easier for evaluators to compare functions of projects than to directly compare projects (Tables 3 to 5).  It is possible to separate the evaluation of city officials in charge of a project from the evaluation of the project (Tables 1 and 17). For example, if the individual project IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 387 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 value, which is the average of target achievement rates of the project, is high, then the officials in charge should be rated highly. On the other hand, if the relative evaluation value and the contribution degree of the project are low, then the project is evaluated low and will be reduced in the next fiscal year.  It is possible to decide on the future direction of each project by obtaining the contribution degree of the project (see Section 5.3). This means that this framework includes effectiveness evaluation of the projects to the program. Furthermore, we found out the following through the process of evaluation in Section 5:  This was the first time that the chief person in charge considered the program related to their projects deeply.  The chief person in charge could understand the relationship between the projects belonging to the program well. For example, they did not need to pursue target achievement rate over 100% while they had projects of low target achievement rate (Table 1). As a result, they will be able to improve the quality of their projects.  The director had a strong opinion about the gender equality program, although the assistant director and the chief person in charge did not know it.  The chief person in charge can discuss their projects with the director and the assistant director using this framework. This may solve the previous issue of administrative evaluation like project evaluation being seen as useless for officials. There are a few limitations of our research. First, we supposed that local governments had already evaluated their projects and calculated several kinds of target achievement rates for each project. In fact, target management type evaluation is the mainstream in Japan. If the governments do not have adequate numerical values for individual evaluations, they would need to find these individual project values before applying the framework. This means they would need to improve the indicators to evaluate the effectiveness of projects. Next, historically, administrative evaluation is a technique used for budget cuts. The framework proposed in this paper does not directly contribute to making a budget cut. However, this framework can contribute to the clarification of the outcome of the project by not using quantitative terms. This is a strong point of the AHP, that qualitative as well as quantitative factors can be included. If evaluators do not know outcomes of the program at all, they cannot evaluate the effectiveness of the projects. Finally, it is difficult to decide on viewpoints of criteria in this framework with which a majority of the people agree. For the program with the numerical example in Section 5, it was easier to decide on viewpoints of criteria because the field of gender equality has a lot of results from Japanese researchers. This difficulty is like that of a logic model in policy evaluation. On the other hand, the five functions in Figure 5, except for support of promotion in the workspace, can be used for many other projects in future problems. 7. Conclusions We showed a framework of project evaluation to calculate the effectiveness of projects for local governments using the AHP. We called this value the contribution degree of the IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 388 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 project. The contribution degree is calculated based on viewpoints of the program and functions of the projects. This framework is a kind of logic model using the AHP because we constructed a hierarchy with viewpoints and functions with it. Finally, we used the AHP because of its simplicity for evaluators, but at the same time we assumed that all the items were mutually independent. However, those in the same level have dependence between each other. A future problem to address would be to soak up dependence with the ANP to the extent that it does not compromise the results. IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 389 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 REFERENCES Chino city. (2016/17). Project evaluation sheets of the program: gender equality (in Japanese). No. 06-05-01 to 03 and 08. Dey, P. K. (2006). Integrated project evaluation and selection using multiple-attribute decision-making technique. International Journal of Production Economics, 103(1), 90- 103. Doi: https://doi.org/10.1016/j.ijpe.2004.11.018 Hatry, P.H. (2007). Performance measurement: Getting results, 2nd edition. Washington, D.C.: The Urban Institute. Doi: https://doi.org/10.1080/01944360701755709 Huang, C. C., Chu, P. Y., & Chiang, Y. H. (2008). A fuzzy AHP application in government-sponsored R&D project selection. Omega, 36(6), 1038-1052. Doi: https://doi.org/10.1016/j.omega.2006.05.003 Inazawa, K. (2012). Gyosei-hyoka no donyu to katsuyo: yosan/kesan, sogokeikaku (zoho ban) (in Japanese) [Introduction and practical use of administrative evaluation: budget, settlement of accounts, comprehensive plan (enlarged edition)]. Japan: Imagine Syuppan. Ishihara, T. (Eds.). (2005). Jichitai-gyosei-hyoka case study (in Japanese) [Performance Measurement in Japanese local government]. Japan: Toyo Keizai. Kinoshita, E., & Ohya, T. (Eds.). (2007). Kigyo & gyosei notameno AHP jireisyu (in Japanese) [The collection of AHP practical use examples]. Japan: JUSE Press. Kinoshita, E., & Taji, K. (Eds.). (2005). Gyosei-keiei notameno ishiketteiho: AHP wo tsukatta nanmon-dakai no shinsyuho (in Japanese) [Decision making methods for administrative management: a new method by AHP to overcome difficult problems]. Japan: Gyosei. Mahmoodzadeh, S., Shahrabi, J., Pariazar, M., & Zaeri, M. S. (2007). Project selection by using fuzzy AHP and TOPSIS technique. World Academy of Science, Engineering and Technology, 30, 333-338. Miyagawa, K. (1994). Seisaku-kagaku no kiso (in Japanese) [Foundations of policy sciences]. Japan: Toyo Keizai. Naito, K. (2015). Danjyo-kyodosankaku-syakai-keisei seisaku (in Japanese) [Gender equality society formation policy]. In Naito, K., & Yamaya, K. (Eds.). Danjyo- kyodosankaku-seisaku: Gyosei-hyoka to shisetsu-hyoka (in Japanese) (1-12). Japan: Koyoshobo. Naito, K., & Yamaya, K. (Eds.). (2015). Danjyo-kyodosankaku-seisaku: Gyosei-hyoka to shisetsu-hyoka (in Japanese) [Gender equality policy: Administrative evaluation and facility evaluation]. Japan: Koyoshobo. IJAHP Article: Iida, Koizumi/A framework using the Analytic Hierarchy Process for local governments in Japan to evaluate projects based on outcomes International Journal of the Analytic Hierarchy Process 390 Vol. 10 Issue 3 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i3.593 Saaty, T.L. (1980). The Analytic Hierarchy Process: Planning, priority setting, resource allocatiom. New York: McGraw-Hill. Saaty, T.L. (1982). Decision making for leaders. New York: Lifetime Learning. Saaty, T.L. (1983). Priority setting in complex problems. IEEE Transaction on Engineering Management EM, 30, 140–155. Doi: https://doi.org/10.1109/TEM.1983.6448606 Saaty, T.L. (1996). Decision making with dependence and feedback: The Analytic Network Process. Pittsburgh, PA: RWS Publications. Saaty, T.L. (2001). Fundamentals of decision making and priority theory with the Analytic Hierarchy Process, 2nd Edition. Pittsburgh, PA: RWS Publications. Saaty, T.L., & Özdemir, M.S. (2005). The Encyclicon: A Dictionary of decisions with dependence and feedback based on the Analytic Network Process. Pittsburgh, PA: RWS Publications. Stipak, B. (1987). Using subjective measures in program evaluation. In Busson, T., & Coulter, P. (Eds.). Policy evaluation for local government (45-61). Westport, CT: Greenwood Press. Tanaka, K. (2014). Jichitai-hyoka no senryaku (in Japanese) [Strategy of local government evaluation]. Japan: Toyo Keizai. Vargas, L.G. (1990). An overview of the analytic hierarchy process and its applications. European Journal of Operational Research 48 (1), 2–8. Doi: https://doi.org/10.1016/0377-2217(90)90056-H Yamaya, K. (2016). Many faces of evaluation in Japanese local governments: 20 years of decentralization and financial crisis (in Japanese). Japanese Journal of Evaluation Studies, 16(1), 31-45. Zayed, T., Amer, M., & Pan, J. (2008). Assessing risk and uncertainty inherent in Chinese highway projects using AHP. International Journal of Project Management, 26(4), 408-419. Doi: https://doi.org/10.1016/j.ijproman.2007.05.012