1. Introduction Nowadays technical assistance programs occupy a significant position in the process of social and economic transformation of many countries. For- eign aid affects integration processes in the global economy, ensures national households’ sustainable development as well as implementation of projects that governments are not able to finance them- selves. The largest donors to Ukraine have been and remain the United States and the European Union. Other donors include Canada, Germany, the Neth- erlands, Sweden, Switzerland, Japan, Denmark, and the United Nations organisations, the World Bank, the EBRD and the OECD. Fairly limited number of Ukrainian scientists ex- plores technical assistance given to Ukraine by donor countries, as well as the problems and mechanisms for implementing and evaluating the effectiveness of current projects. It should be pointed out that there are rather fewer studies analyzing technical assistance effectiveness in terms of meeting or fail- ing to meet objectives within a foreign aid project implemented in Ukraine. Most research works are devoted to other issues, e.g. legislative environment and political consequences of donor country impact in the country. Journal of Geography, Politics and Society 2017, 7(1), 70–76 DOI 10.4467/24512249JG.17.009.6208 THE METHODOLOGICAL PRINCIPLES OF EVALUATING THE EFFECTIVENESS OF FOREIGN AID PROJECTS IN UKRAINE Polina Yurijivna Kulish Law and European Integration Department, Kharkiv Regional Institute of Public Administration of the National Academy of Public Administration under the Office of the President of Ukraine, Moskovsky Prospect 75, 61001, Kharkiv, Ukraine, e-mail: polina.kulish@gmail.com Citation Kulish P.Yu., 2017, The methodological principles of evaluating the effectiveness of foreign aid projects in Ukraine, Journal of Geography, Politics and Society, 7(1), 70–76. Abstract The article examines the major issues of methodology that evaluates the effectiveness of technical assistance project imple- mentation in Ukraine. It analyzes the organizational and legal assessment of the impact of international technical assistance upon regional development in the present context. The article puts forward some steps to define key performance indicators, principal evaluation and monitoring methods for implementing ITA projects. It is reasonable to create a unified national system for monitoring and auditing the effectiveness of projects being implemented at the expense of funds under the Ukrainian legislation. Key words international technical assistance, method, methodology, evaluation, performance evaluation, monitoring. The methodological principles of evaluating the effectiveness of foreign aid projects in Ukraine 71 In recent years, the term ‘effectiveness’ has in- creasingly arisen in discourses on aid. This responds to different tendencies: debates on how to combine the aid accountability and effectiveness principles of the Paris Declaration on Aid Effectiveness. From one hand, Rosemary McGee treats transpar- ency as a means to achieving greater ‘aid effective- ness’, by permitting the tracking and achievement of Paris Declaration commitments (McGee, 2013). From other perspective, the OECD has admitted that, “There is little data based analysis of the overall effectiveness of technical assistance as an aid instru- ment, whether in terms of cost/benefit, impact on growth, fiscal impact or other financial measures.” (Development…, 2005, p.119) The Institute of foreign aid is relatively a new phe- nomenon in the public administration of Ukraine. Thus, the research of this realm has revealed being insufficient. Such scientists as A. Bazyliuk, M. Vasi- lyeva, O. Palyuh, K. Ploskyj (Базилюк, 2010; Палюх, 2012; Васильєва, 2010; Плоский, 2013) study vari- ous aspects of the subject. An important contribu- tion to the development of the theory of technical assistance from European Community Member States was made by M. Koryukalov, N. Hnydyuk, I. Kravchuk (Корюкалов, 2012; Гнидюк, 2011; Крав- чук, 2015). Analysis of some Ukrainian scientists’ (Плоский, 2013; Кравчук, 2015) works suggests that both theoretical and methodological principles of attracting and use of technical assistance, particular- ly in terms of project effectiveness, should be stud- ied in detail. It also should be noted, that most literature has tended to focus on viewing technical assistance as an instrument of donor policy or fundrising strate- gies of recipient country, rather than focusing on the issue of how to find a ballance between inter- national and national markets of expertise services, and improving effectiveness and accountability of domestic procurement procedures. For purpose of research technical assistance was defined according to the OECD. It has three main el- ements (Williams et al., 2003, p.2): • study assistance through scholarships and train- eeships, including training of developing country nationals in donor countries; • personnel, including long term and short term experts (both expatriate and national); • research into the problems of developing coun- tries, including diseases. The article is aimed at methodology of evaluating the effectiveness of the international technical assistance provided to Ukraine as well as mechanisms for improving assessment and monitoring of foreign aid projects. This article focuses primarily on Ukraine bilateral assistance, not on the work of multilateral aid entities, such as the UN or World Bank, to which the Ukraine contributes. The absence of accurate data on multilateral aid means that its hard to analyse ‘real aid’ for multilateral donors (share of administration costs in multilateral technical assistance). Also we should stated on that effectiveness – the extent to which initiatives are effective in achieving stated goals. Results are compared to the initial program goals and expectations to determine whether the project or contract has been performed successfully (performance monitoring), and if the resulting data are analyzed in an effort to explain how and why a program meets or fails to meet strategic objec- tives, this is called performance evaluation (White, 2011, p.3). It needs to be distinguished from impact – the degree to which the initiative attains its fur- ther-reaching or ‘second-order’ goals. 2. Research discussions At present, Ukraine actively participates in ITA, most- ly, as the recipient. Foreign aid projects represent one of the important components of integration strategy to join the European Community, support economic development, and stimulate internal and external interests of the state in the economy and national security spheres. Within the years of co- operation with international organizations, Ukraine has received more than 9 billion US dollars of foreign aid. Major donors of Ukraine are USAID and the Eu- ropean Union. Also there are top 10 organizations that should be mentioned. They are UN agencies in Ukraine, the US State Department, the German gov- ernment, the OSCE, SIDA, the EBRD, the World Bank, the EU instrument – the European Neighbourhood Instrument (ENI), which has replaced the European Neighbourhood and Partnership Instrument (ENPI) from 2014. According to the website of the Ministry of Eco- nomic Development and Trade of Ukraine as of 1st April 2016 were registered 289 projects with total contract value of over 3.3 billion US dollars. Technical assistance areas include: nuclear safety improvements, protection of the state and citizens; Shelter object transformation into ecologically safe system and environment protection; improvement of regulatory policy, investment climate, develop- ment of entrepreneurship and competition in the domestic market; financial market and banking sec- tor; agricultural sector; healthcare and social welfare; education, science, culture and tourism; civil society, rule of law, fight against corruption. 72 Polina Yurijivna Kulish According to the resolution of the Cabinet of Min- isters of Ukraine (15.02.2002, #153) “On Establishing the Unified System of the Use and Monitoring of In- ternational Technical Assistance” (Про створення…, 2002), and the order of the Cabinet of Ministers of Ukraine (11.09.2013, #697-р) «On Approval of the Strategy for Attracting, Use, and Monitoring of In- ternational Technical Assistance and Cooperation with International Financial Organizations in 2013– 2016» (Про схвалення…, 2013), it is stated that in- ternational technical assistance is aimed at helping acquire knowledge and skills in relevant areas, usu- ally in economics and management. It (foreign aid) might be represented in the form of any property re- quired for projects (programs) imported or acquired in Ukraine; works or services; intellectual property rights; financial resources (grants) in national or for- eign currency; other resources under the Ukrainian legislation, including scholarships (Про створен- ня…, 2002). It should be noted that the Ukrainian scientists studied the role of international assistance (Базилюк, 2010; Палюх, 2012; Васильєва, 2010; Плоский, 2013). In their works foreign aid in Ukraine has been considered as inefficient, due to the deficiencies of the regulatory framework and public management systems, as well as insufficient coordination of for- eign aid projects. In 2007 the country joined the Paris Declaration on Aid Effectiveness that sets out the principles and measures for planning, monitoring, and assess- ment of ITA effect, with appropriate mechanisms at the national level (Паризька декларація…, 2007). Moreover, within the framework of the European in- tegration policy the requirements and changes oc- curred while signing additional initiatives on foreign aid during such high-level events as Accra Agenda for Action (2008), Busan Partnership for Effective De- velopment Cooperation (2011), Mexico High-Level Meeting Communiqué (2014) should be taken into account as well. However, a clearly defined mecha- nism for evaluating and monitoring implementation of ITA projects in Ukraine does not exist. It should be stated that ITA projects do not differ from the usual concept of a project in management. Robert Archibald, one of the acknowledged classics of project management defines a project as a “set of efforts undertaken to obtain specific unique results within the time limit and approved budget allocated to pay for resources used or consumed in the course of a project” (Арчибальд, 2010, p.57). Effectiveness can be defined not only as the ra- tio of the results and costs but also as the degree of results compliance with stakeholders’ expectations. Appropriate expectations depend not only on the cost of resources, time, capital, but no less, on results of functioning of similar objects under conditions that are compared. One of the key factors of increasing the effective- ness of ITA project implementation in Ukraine is to create a complex system of monitoring and foreign aid projects assessment. On the one hand, improve- ments to the assessment, require increased trans- parency of donor organizations, including the pro- vision of relevant information, on the other hand, it needs building government institutions capacity to conduct appropriate monitoring and evaluation. After all, according to researchers in the public sec- tor, the assessment of public policy and programs is fragmented and chaotic. There is a lack of both cul- ture and demand assessment, as well as qualified specialists, methods and reliable data (Васильєва, 2010; Плоский, 2013). The resolution of the Cabinet of Ministers of Ukraine (Про створення…, 2002) contains a general list of indicators in accordance with those ones it is proposed to carry out ongoing and/or final monitor- ing: information about the terms of seminars, num- ber of participants, theme, equipment, transferred (developed) technology, work advisers’ services, de- veloped documents, publications, as well as on edu- cation, training specialists, provided grants, scholar- ships, total amount of money spent on the technical assistance during the term of a project (program), problem areas. However, methods for measuring the impact of ITA projects, for comparing planned performance criteria with achieved ones, are not specified. Yet, the choice of evaluation methodology con- siderably influences the objectivity of the evalua- tion of project effectiveness. The big drawback of ITA projects might be their inconsistency within even one donor. Thus, these projects have neither unified performance indicators, nor overall assess- ment which would put together all the ITA projects (programs) within their complex analysis. Given all the above mentioned, there is no point in doing a comparative analysis of the impact of ITA projects implementation. It is known that the wider range of project partici- pants, the more difficult coordination of the interests of each of them as well as control of compliance with contract commitments. This question requires the use of special methods of project management such as network planning, project financing methods. A rational step to improve the assessment of ITA projects implementation would be the develop- ment of key performance indicators. This can be di- vided into several phases (Арчибальд, 2010): The methodological principles of evaluating the effectiveness of foreign aid projects in Ukraine 73 1. Objectives, principles, and scope of key perfor- mance indicators. It is reasonable to analyze the current situation of projects and to identify im- provements in project management. 2. Business processes in project management. 3. Development, coordination, approval of key per- formance indicators. It is important to develop and ensure the procedures for planning, moni- toring and control projects effectiveness. 4. Key performance indicators for project managers. The practical implementation of such systems in national and foreign companies has revealed a significant degree of rejecting the system at dif- ferent stages of its development and operation, which is connected with individual and group resistance of employees. Ignoring the need to adapt the system leads to serious consequences, such as reducing the motivation, failure of pro- jects, loss of key employees. The suggested sequence of steps will promote the formation and development of indicators to assess the effectiveness of ITA projects implementation. Understanding the need for analysis and evalua- tion of government programs in different countries led to the emergence of different methods and ap- proaches. The comparative analysis of approaches to the assessment of programs shows the following foreign and national methods of evaluating the ef- fectiveness of programs (see Weimer, Vining, 2005; Dunn, 2009). Cost-benefit analysis method. Effectively, this ap- proach is used in areas where it is difficult to only assess all of the effects throughout financial instru- ments: infrastructure, transport, environment, edu- cation, health, and culture. This method allows from different sides to assess the impact of a program on the level of social welfare during any period of time. The method is easy to use and understand, but at the same time, it has highly subjective calculations of costs and benefits, the high cost of gathering information. Cost-effectiveness analysis method. This method is not designed for assessment of projects with mul- tiple objectives. It is better used to evaluate projects in areas where one can monetize achieved bene- fits: health, education and other social sectors. The method covers performance, as well as productivity and profitability assessment. Cost-profit analysis method. This approach com- pares costs, estimated in monetary terms and ben- efits for the population, resulting in the implemen- tation of budgetary expenditure, measured in terms of profits. The method differs in the subjective as- sessment of programs profit in relevant units and in a complexity of processing the received material. It is used to assess an activity leading to a range of re- sults or to the results that may be different not only in quantity but also in quality, for example, health programs. Rating scale method is based on PART instrument (Program Assessment Rating Tool), developed in 2001 by the U.S. Office of Management and Budget. According to this method, the assessment of budget- ary programs is based on a number of issues which form a coherent scheme to evaluate the effective- ness of programs in a broad sense. The advantage of this model is that it allows you to assess performance based on a careful study of the most important as- pects including the process from preparation stages of program documents to control procedures, out- comes of the assessment of implemented programs. It has some inaccuracy that occurs during the evalu- ation of programs on several criteria. In the international practice, the evaluation of economic effectiveness programs is carried out with the use of three main methods: cost – benefit, cost – profit, cost – effectiveness. These methods can determine the level of rationality resources spent to achieve the planned goals, the level of program (project) implementation. Another central issue, which was raised by inter- national community, is that technical assistance has been used as an instrument of donor policy that is sometimes disconnected from recipient country de- mand (Williams et al., 2003). That’s why we need take into account donors’ effectiveness. One of the initia- tive which measures donors’ performance on 31 in- dicators of aid quality to which donors have made commitments was announced by Center for Global Development in 2010 (Birdsall, Kharas, 2014). The indicators are grouped into four dimensions associ- ated with effective aid: maximizing efficiency, fos- tering institutions, reducing the burden on partner countries, and transparency and learning. The 2014 edition finds that donors are overall becoming more transparent and better at fostering partner country institutions but that there has been little progress at maximizing efficiency or reducing the burden on partner countries. Trying to find a balance between effectiveness and accountability sometimes donors are not so flexible in supporting domestic initiatives. Good example of improving reforming Ukrainian procure- ment system, which try to be changed during last two years due to triangular co-operation and not to advises from EU to harmonised public procure- ment rules with EU law at initial stage of reform. But as the rest of developing countries Ukraine is failed to build long term capacity concerning given the high cost of technical advisers, especially expatriate 74 Polina Yurijivna Kulish experts (Twinning and other EU funded projects). Some technical advisers, while working in the gov- ernment, remain formally accountable to donors, as with the EU funded advisers being contracted at the Twinning projects. The analysis of national approaches to as- sessment of programs and projects effectiveness showed the absence of the methodological basis for assessing different kinds of ITA programs and projects in Ukraine from domestic and international perspectives. Another dimension that help to find answer on how improving aid effectiveness is analysis of trian- gular co-operation. Since 2009, the OECD has been promoting dialogue and encouraging greater trian- gular co-operation. This work has helped to clarify what triangular co-operation is, how it is implement- ed and how it can be effective in promoting devel- opment (Fordelone, 2009). There nevertheless remains an important knowl- edge gap on effectiveness of triangular co-opera- tion. Currently, actors involved, developing coun- tries and researchers are not able to find complete, comparable and up-to-date information on where triangular co-operation is taking place, who is in- volved, in what type of activities and in which sec- tors (Development..., 2016). Donors continue to play a key role in the identi- fication of needs, project design, terms of reference design, reporting and monitoring and evaluation. For example, according to the ENI Regulation, EU support shall be incentive-based and differentiated in form and amounts and external action partners, including civil society organizations, national and local authorities, are involved in preparing, imple- menting and monitoring EU support through con- sultation and timely access to relevant information allowing them to play a meaningful role in that pro- cess (REGULATION (EU) No 232/2014…, 2014, art. 4). Consultation mean that EU is not obliged to take into account all proposals from external players. In preamble p.13 we have a norm which partly conflict with incentive-based approach. Support should be coherent with the objectives and principles of the Union’s external policies and, in particular, its devel- opment policy and the common foreign and securi- ty policy. Coherence with the external dimensions of the Union’s internal policies and instruments should also be ensured (REGULATION (EU) No 232/2014…, 2014). Also ENI Regulation stipulates that the sup- port and the amounts given to partner countries will be based on their progress achieve (Commission de- cisions) and thus can be reconsidered. 3. Conclusions In the present context, the external evaluation of the effectiveness is considered as part and parcel of project life cycle and often is required by donors at the stage of project development. However, this is not only an instrument of control or some formal requirement. Analysis of project effectiveness pro- vides managers with the opportunity to correct the deficiencies, and most importantly, to determine the prospects for further development. During last decade many donors try to change methodological approach in aid effectiveness prog- ress-monitoring. Using performance evaluation and financial audits, which play an important part in project management, are tend to be dismissed by impact evaluations, which look not at the output of an activity, but rather at its impact on a develop- ment objective. Those changes is better answer on questions about foreign aid effectiveness. Today, Ukraine has taken progressive steps in national system formation for monitoring projects supported by international donors. The resolution of the Cabinet of Ministers of Ukraine (27.01.2016, #70) approved the Procedure for preparation, implemen- tation, monitoring and completion of projects of economic and social development in Ukraine, sup- ported by international financial institutions (Про порядок…, 2016). This document regulates the co- operation among all of the parties while implement- ing the system (support of reforms in Ukraine with financing the state budget) and investment projects, still does not define approaches and methods of monitoring indicators, but the list of documents re- quired for signing an agreement with international financial institutions on cooperation, auditing, finan- cial reporting etc. In addition, the projects, being carried out in Ukraine, are aimed at improving the regulatory framework that governs monitoring and evaluation of technical assistance. One of the biggest donors is the European Un- ion and the US, which policy is directed at reducing the gap in development between different regions and countries. But this two donors use different ap- proach for evaluating the effectiveness of own aid programs. The key spheres of improvements of ITA project management are the following ones: concepts and strategies for main areas of use, ITA monitoring and programming; improvement of control system of the government ITA programs; creating a commission on monitoring of international technical assistance; setting clear deadlines for technical projects and monitoring of contract commitments; sustainable The methodological principles of evaluating the effectiveness of foreign aid projects in Ukraine 75 criteria of project effectiveness; registration of com- pleted projects and money spent; public discussion of projects, problems of implementation and ways out. Monitoring projects provides donors, partners, governments and other stakeholders with access to detailed information on project progress and results. Unfortunately, the mechanisms of control and coor- dination of ITA are formed only partially and subject to be reviewed in the context of problems such as fraud, imperfection of the monitoring systems, inad- equate legislative framework, and other. Taking into account the abovementioned, it should be emphasized the need to build a unified national system for monitoring and auditing the effectiveness of programs and projects implement- ed with the help of funds under the legislation of Ukraine. It goes about the cooperation with inter- national donor organizations, State Regional Devel- opment Fund, business entities, NGOs and all stake- holders thereof. References Birdsall N., Kharas H., 2014, The Quality of Official Development Assistance (QuODA), Center for Global Development and Brookings Institution, Washington, http://www. cgdev.org/publication/quality-official-development- assistance-quoda-third-edition [20.11.2016] Development co-operation report 2005, 2006, OECD Publish- ing, Paris, http://www.oecd-ilibrary.org/development/ development-co-operation-report-2005_dcr-2005-en [23.11.2016] Development Co-operation Report 2016: The Sustainable Development Goals as Business opportunities, 2016, OECD Publishing, Paris, http://www.oecd-ilibrary.org/ development/development-co-operation-report-2016_ dcr-2016-en [20.11.2016]. Dunn W.N., 2009, Public Policy Analysis: An Introduction, Long- man, New York. Fordelone T.Y., 2009, Triangular Co-operation and Aid Effectiveness, Working Paper, OECD Development Co- operation Directorate, www.oecd.org/dac/46387212.pdf [20.11.2016]. McGee R., 2013, Aid Transparency and Accountability: ‘Build It and They’ll Come’? Development Policy Review, 31, 107– 124. REGULATION (EU) No 232/2014 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 11 March 2014 establishing a European Neighbourhood Instrument, 2014, Official Jour- nal of the European Union, I.77/29. Weimer D.; Vining A., 2005, Policy Analysis: Concepts and Prac- tice. Upper Saddle River, Pearson Prentice Hall, New York. White H., 2011, An introduction to the use of randomized control trials to evaluate development interventions, Working Paper  9, International Initiative for Impact Evaluation, New Delhi, http://www.3ieimpact.org/media/ filer/2012/05/07/Working_Paper_9.pdf [23.11.2016] Williams G., Jones  S., Imber  V., Cox  A., 2003, A Vision for the Future of Technical Assistance in the International Development System – Final report, Oxford Policy Management, http://www.opml.co.uk/sites/default/files/ ACF54001_0.pdf [23.11.2016] Арчибальд Р., 2010, Управление высокотехнологичными программами и проектами, Компания АйТи, ДМК Пресс, Москва. Базилюк А.В., 2010, Економічні наслідки міжнародної технічної допомоги в організації агробізнесу України, Актуальні проблеми економіки, 9(111), 24–34. Васильєва М.О., 2010, Нова парадигма ефективності зарубіжної допомоги розвитку, Актуальні проблеми міжнародних відносин, 93(1), 103–107. Гнидюк Н.А., 2011, Механізм координації політики європейської інтеграції в Україні, abstract of thesis for obtaining the scientific degree of Candidate of Pub- lic Administartion on specialty 25.00.0, Національна академія державного управління при Президентові України, Одеський регіональний інститут державного управління, Одеса. Корюкалов М.В., 2012, Політика Європейського Союзу із надання зовнішньої допомоги для розвитку, abstract of thesis for obtaining the scientific degree of Candidate of Political Sciences on specialty 23.00.04, Київський націо- нальний університет імені Тараса Шевченка, Київ. Кравчук І.В., 2015, Теоретико-методологічні засади оцінювання державної політики в Україні, abstract of thesis for obtaining the scientific degree of Candidate of Public Administartion on specialty 25.00.02, Національ- на академія державного управління при Президентові України, Київ. Палюх О.О., 2012, Реформування системи використання міжнародної технічної допомоги в Україні, abstract of thesis for obtaining the scientific degree of Candidate of Economic Sciences on specialty 08.00.03, Міністерство економічного розвитку і торгівлі, Науково-дослідний економічний інститут, Київ. Паризька декларація щодо підвищення ефективності зовнішньої допомоги, 2007, http://zakon5.rada.gov.ua/ laws/show/998_271 [19.04.2007] Плоский К.В., 2013, Актуальні завдання розвитку державного управління щодо залучення міжнародної технічної допомоги в Україні, [in:] Зб. тез ХІІІ міжнар. наук. конгресу, Харків, 21–22 берез. 2013 р. – Х., Вид-во ХарРІ НАДУ «Магістр», Харків, 322–324. Про порядок підготовки, реалізації, проведення моніторингу та завершення реалізації проектів економічного і соціального розвитку України, що підтримуються міжнародними фінансовими організаціями : постанова Кабінету Міністрів України, 2016 http://zakon3.rada.gov.ua/laws/show/70- -2016-%D0%BF/page [27.01.2016] Про створення єдиної системи залучення, використання та моніторингу міжнародної технічної допомоги: постанова Кабінету Міністрів України, 2002, http:// 76 Polina Yurijivna Kulish zakon2.rada.gov.ua/laws/show/153-2002-%D0%BF [19.08.2015] Про схвалення Стратегії залучення, використання та моніторингу міжнародної технічної допомоги і співробітництва з міжнародними фінансовими організаціями на 2013-2016 роки: розпорядження Кабінету Міністрів України, 2013, http://zakon5.rada. gov.ua/laws/show/697-2013-%D1%80 [11.09.2013]