Plane Thermoelastic Waves in Infinite Half-Space Caused FACTA UNIVERSITATIS Series: Economics and Organization Vol. 11, N o 3, 2014, pp. 261 - 279 MULTI-CRITERIA METHODS AND MODELS FOR DECISION MAKING IN PUBLIC PROCUREMENT  UDC 658.71 Ivana Veselinović University of Niš, Faculty of Economics, Niš, Serbia Abstract. Selection of the best supplier in the public procurement process is a typical example of a multi-criteria decision making problem. The purpose of this paper is to present possible approaches for weights determination in order to facilitate decision making in the public procurement process. Considering the fact that criteria weights can affect the final ranking of the alternatives it is very important to access the process of weights assigning with seriousness and responsibility. Adequately estimated weights reduce the possibility of abuse and fraud in the public procurement system. Determination of weights is done based on a subjective approach (Analytic Hierarchy Process). Key Words: Supplier selection, Analytic Hierarchy Process, Multi-criteria decision making, Public procurement, Criteria Weights. INTRODUCTION One of the main features of modern business is the need to make a large number of decisions that are depending on a number of different criteria. The process of public procurement involves selection of the best supplier in the public procurement. One of the most important questions for local and national Governments is providing public facilities for their citizens in the time and in the amount they require. On the other hand, there is a question of choosing the right provider of those facilities, given the fact that there are a lot of private firms that are willing to offer their services. Procurer faces with the situation which demands that he makes a compromise between the available resources and the quality of required goods or services. Therefore, procurer usually considers only the price of the required goods and services, without paying any attention on the other aspects of the subject of procurement. However, selection of the offer based only on the criteria of Received September 9, 2014 / Accepted December 17, 2014 Corresponding author: Ivana Veselinović, PhD. student Faculty of Economics, University of Niš, Trg kralja Aleksandra 11, 18000 Niš, Serbia E-mail: ivana.veselinovic@eknfak.ni.ac.rs 262 I. VESELINOVIĆ the lowest price is not always the best possible solution, there are a lot of other criteria that should be taken into consideration. The quality of this process affects on the final efficiency of the organization. Application of multi-attribute decision making methods can be considered as a means to support decision making in public procurement. The question of weights determination in public procurement is very important. Procurer can favor a certain bidder by giving a high weight to a criterion that only that competitor can satisfy. Having in mind that public procurements are financed by the money of the tax payers that kind of situation can lead to unsatisfactory choice which will not fulfill the requires of the citizens. This is why the application of scientific methods is important in the weight determination process in public procurements. The basic premise of this paper is that the choice of the best offer is difficult in conditions where there are a number of criteria that can be used to assess them. The use of exact scientific methods for determination of the relative significance of each of the criteria and their use for ranking the offers can facilitate the decision making process. This paper will first emphasized the importance of the public procurement system, a brief overview of legislation in this area and the potential for misuse of public procurement will be presented. Furthermore we will explain the basic concepts of multi-criteria decision making and the way of forming the multi-criteria model. One of the key problems of multi-criteria decision making is to determine the relative significance or weights of different criteria. The process of determining the relative significance of attributes consists in defining and assigning weighting factor to each individual criterion. Then on an empirical example the calculation of weights using the AHP method as the subjective methods for determining the value of weights is shown. The subject of the analysis will be public procurement carried out by local authorities, and the ultimate goal is to determine relevant criteria and weights for this type of procurement. The main objective of this paper is to show the importance of the procurement process for the organization and the possibilities and advantages of multi-criteria analysis methods for the selection of the best offer. 1. BACKGROUND In the contemporary business conditions the selection of the best supplier becomes the problem from whose solution depend the business performance. Adequately organized tender procedure should simplify the selection of the best supplier. The organization of public procurement procedure and the selection of the supplier are prescribed by the Law on Public Procurement. Since the private sector does not have its own rules, the entities in the private sector also apply the guidelines given by the Law on Public Procurement. The problem of supplier selection has multi-criteria nature since it is characterized with number of criteria that should be examined. Therefore, it has been a subject of many different papers, books and case studies. A lot of authors have been dealing with the problem of public procurement. The main problem of procurement selection is related to objectivity of the selection process. Hence, the application of multi-attribute decision making methods is proposed. Cheung et al (2001) propose the use of an Analytical Hierarchy Process for procurement selection. They have developed a selection method that uses multi-attribute utility technology and Multi-Criteria Methods and Models for Decision Making in Public Procurement 263 the Analytical Hierarchy Process. Application of a model for supplier evaluation based on the Analytical Hierarchy Process can also be found in [11]. Haq and Kannan (2006) suggest a structured model for evaluating vendor selection using the analytical hierarchy process and fuzzy analytical hierarchy process. According to [20] fuzzy analytic hierarchy process can be used in selecting the best supplier firm, and it can satisfy the determined criteria to the greatest extent. Benyoucef and Canbolat (2007) propose the use of fuzzy AHP-based supplier selection, while Hsieha et al (2004) suggest fuzzy multi-criteria decision making approach for planning and design tenders selection. Contractor selection can be also performed by using the analytic network process [6]. However, great weakness of every model for procurement selection is disagreement among the experts about the importance of the criteria. Chan et al (2001) present the possibility of application of Delphi method in selection of procurement. According to them, the application of Delphi method leads to an objective opinion. Vendor selection can also be performed by integrated fuzzy multi-criteria decision making method [36]. Dulmin and Mininno (2003) suggest the use of a multi-criteria decision aid method for supplier selection. Other authors propose the use of outranking methods in support of supplier selection [12]. They show that an outranking approach can be applied as a decision making tool for initial purchasing decisions. De Boer, Labrob and Morlacchi (2001) give an extended review of decision methods for supplier selection support. Other authors propose the use of TOPSIS method combined with intuitionistic fuzzy set in selection of an appropriate supplier [5]. Few authors have shown that application of mathematical techniques (such as multi-objective programming or goal programming) is suitable for the solution of this decision problem [38]. Since these mathematical techniques have problems in inclusion of qualitative factors which are very important in supplier selection Ghodsypour and O'Brien (1998) suggest an integration of the Analytical Hierarchy Process and linear programming. 2. PUBLIC PROCUREMENT - DEFINITION AND IMPORTANCE The public procurement system is an area of public finances, which attracts a lot of attention of contemporary society. Through the effective functioning of this system fair and efficient allocation of public resources is carried out and the optimal quantity of goods is financed by public funds. It is necessary to have an adequate legal framework which will act as a prevention and which will inhibit the occurrence of corruption mechanisms. According to the Law on Public Procurement of the Republic of Serbia, which came into force on January 6, 2013 and is applicable from April 1, 2013, the term public procurement is defined as the procurement of goods, services or works by the purchaser in the manner and under the conditions prescribed by this Law. Bearing in mind that a prerequisite for the use of EU pre-accession funds, and other forms of international assistance is the existence of a regulated public procurement system, which includes effective monitoring and control of the probity of the public procurement system, it can be concluded that international aspects of public procurement is also very important. 264 I. VESELINOVIĆ Free competition, legal certainty and transparency in public procurements reduce overall business risk in Republic of Serbia, making it more attractive for foreign investors and companies, which ultimately have an impact on the dynamics of its economic development. Public procurements, through which the state determines what shall be bought, how and at what cost, are the key instrument through which the government achieve some of its strategic objectives, such as economic development, stimulation of employment and others [30]. For the achievement of this strategic role of public procurements, it is essential that the public procurement system is successful in carrying out its functions. 2.1. The role of the state in the public procurement system regulation The only way possible to ensure achievement of objectives stated in the law and other regulations relating to public procurements is an economical and efficient use of public funds, which will result in prevention of corruption in public procurements. Republic of Serbia until the 2002 did not have a law which has regulated public procurements area uniquely and comprehensively for all supplies made by public authorities and organizations, institutions and public companies. Far-reaching consequences of such a situation where the procurement of goods whose technology was outdated, and largely obsolete in developed countries, lack of interest of quality foreign bidders to offer their products and services in these circumstances, due to the lack of precise rules that would ensure healthy competition, and equality of all bidders and public in the process of choosing the best among them. In order to eliminate the negative consequences that may arise due to inadequate regulation of public procurements, the Republic of Serbia in May 2002 passed the first Law on Public Procurement (published in the Official Gazette of the Republic of Serbia, No. 39/02) After more than a year of implementation of this law certain deficiencies were identified in practice which led to dysfunction of purchasers, due to their commitment to carry out public procurement procedure under strictly defined rules even in the cases where that was not justified. In this regard, in the 2004 the Law on Amendments and Supplements to the Law on Public Procurement was passed (published in the Official Gazette of the Republic of Serbia, No. 55/04). With these changes, basic concept of the law was not violated. According to the first empirical indicators of long-term application of the Law on Public Procurement proper and consistent application of the basic principles underlying the implementation of public procurement was not fully ensured. To improve the regulation of the public procurement system, a new law was adopted on December 2008 (published in the Official Gazette of the Republic of Serbia, No. 116). Further improvement in this area has led to the adoption of the new Public Procurement Law (published in the Official Gazette of the Republic of Serbia, No. 124/2012) The goal of the normative regulation of an area is not to limit the area and to stop its development, but to provide a legislative framework for it to develop simultaneously with the development of these areas in neighboring countries and in other countries of democratic orientation. Only good legislative framework can provide unhindered exchange of goods and services, development and networking of the markets, the inflow of foreign capital, improvement of economic relations with other countries and the improvement of economic, and therefore the overall development of our society. Multi-Criteria Methods and Models for Decision Making in Public Procurement 265 2.2. Corruption mechanisms According to the Corruption Perception Index (CPI), published by the international non-governmental organization "Transparency International", Republic of Serbia in the 2013 was located at 72nd place out of 177 countries. As one of the main generators of corruption, the same report states the area of public procurement. Corruption in public procurement causes a loss of public funds, and also affects that the acquired goods, services and works do not suit the needs of the customer regarding to their characteristics, quality and delivery times. Inadequately implemented public procurement can enlarge the costs of purchaser on the one hand, while on the other hand, the quality of services provided to the citizens can be lower than expected. Various abuses may appear within the planning, implementation of the public procurement and realization of the contract. In this sense, different corruption mechanisms can be identified [37]:  Purchase of unnecessary items (in content, quantity or quality)  Deliberate determination of unrealistic estimated value  Illicit fragmentation of procurement in order to apply the procedure of procurement of low value  The formation of the procurement subject so that it can be provided only by a particular bidder  Frequent and unjustified use of exceptions  Conflict of interest  Discriminatory conditions for the participation of bidders  Discriminatory technical specifications  Discriminatory criteria for the selection of the best bid 3. METHODS AND MODELS OF MULTI-CRITERIA DECISION ANALYSIS The problem of making adequate decisions is the essence of every business and it affects the success and longevity of the business. The decision implies a choice between several different alternatives. In situations where the decision maker is faced with a choice between alternatives that can be evaluated on the basis of a single criterion issue boils down to a simple choice of alternative that is consistent with the objectives of the decision makers. However, in a situation where the decision maker is faced with the problem of choosing between different alternatives which are evaluated based on multiple criteria decision maker cannot make a simple comparison of alternatives. In order to perform the best choice the application of multi-criteria decision making is necessary. 3.1. Basic concepts of multi-criteria decision making and the formulation of multi-criteria model Multi-criteria decision making refers to the determination of the best alternative in the conditions where there is a larger number of, usually, mutually conflicting criteria. The real problems have some common characteristics, namely [10]:  A large number of criteria  Conflict among the criteria 266 I. VESELINOVIĆ  Not comparable units of measurement  Projection or selection. Solutions of this type of problem are either projection of the best action (alternative) or a selection of the best action from a set of pre- defined finite action. Methods of multi-criteria analysis are focused on the problem of choosing between one of m alternatives Ai (i = 1, 2, ..., m) based on the n criteria Xj (j = 1, 2, ..., n). Each of the alternatives is the vector Ai= (xi1, xi2, ..., xij, ..., xim). A common way of representing the problem of multi-criteria analysis is matrix form [18]. Alternatives in the model form set with a finite number of elements. Alternatives should be tested, evaluated, priorities should be established, and finally a choice should be made. The criteria in the model are represented by the corresponding function, and their importance is shown by the corresponding weights. Depending on the type of the extreme value of the criterion function, there are two types of criteria. The first group of criteria consists of those criteria where the interest of decision makers is to achieve the maximum value of the criterion function. The second group includes criteria where the interest of decision makers is to achieve the minimum value of the criterion function. The importance of the criteria in the model directly depends on the preferences of decision makers, more precisely, of the weights which are assigned to a specific criterion by the decision maker. Attributes are the relevant characteristics of each of the alternatives and they represent the means for evaluating achieved level of each of the criteria. 3.2. Methods of multi-criteria analysis Multi-criteria analysis methods are suitable for solving a large number of real problems of a different nature. Some of the most popular methods used for troubleshooting multi attribute decision making include the analytic hierarchy process (AHP) method, ELECTRE method, PROMETHEE method, TOPSIS method, the simple additive weighting (SAW) and many others. Examples of practical applications of these methods in the field of quantitative finance are given in Table 1-4 [28]. Categorization was done according to the data which Zopounidis (1999) introduced in his work "Multicriteria decision aid in financial management." Table 1 Application of multi-criteria analysis methods for the assessment of credit risk and the risk of bankruptcy Approach Method Study Multi-attribute utility theory AHP Srinivasan and Kim (1987) Srinivasan and Ruparel (1990) Jablonsky (1993) Comparison of alternatives ELECTRE Dimitras et al. (1995) Bergeron et al. (1996) Khalil et al. (2000) Multi-Criteria Methods and Models for Decision Making in Public Procurement 267 Table 2 Application of multi-criteria analysis methods in the portfolio selection and portfolio management Approach Method Study Multi-attribute utility theory AHP Saaty et al. (1980) Comparison of alternatives ELECTRE Martel et al. (1988, 1991) Szala (1990) Khoury et al. (1993) Hurson and Zopounidis (1995, 1997) Hurson and Ricci (1998) Comparison of alternatives PROMETHEE Khoury and Martel (1990) Martel et al. (1991) Hababou and Martel (1998) Table 3 Application of multi-criteria analysis method for assessing corporate performance Approach Method Study Multi-attribute utility theory AHP Lee et al. (1995) Babic and Plazibat (1998) Comparison of alternatives ELECTRE PROMETHEE Colson and Mbangala (1998) Mareschal and Mertens (1990, 1992, 1993) Mareschal and Brans (1991) Pardalos et al. (1997) Babic and Plazibat (1998) Colson and Mbangala (1998) Zmitri et al. (1998) Baourakis et al. (2002) Table 4 Application of multi-criteria analysis method for investment assessment Approach Method Study Multi-attribute utility theory AHP Kivijarvi and Tuominen (1992) Comparison of alternatives ELECTRE Danila (1980) Buchanan et al. (1999) Comparison of alternatives PROMETHEE Ribarovic and Mladineo (1987) Vranes et al. (1996) Examples of practical applications of TOPSIS method are given in Table 5. Categorization was done according to the data which Behzadian et al. (2012) introduced in their work “A state- of-art survey of TOPSIS applications”. 268 I. VESELINOVIĆ Table 5 Application of TOPSIS method Area of application Study Supply Chain Management and Logistics Alimoradi, Yussuf, and Zulkifli (2011) Cheng, Ye, and Yang (2009) Yang, Bonsall, and Wang (2011) Awasthi, Chauhan, and Omrani (2011) Design, construction and manufacturing Athanasopoulos, Riba, and Athanasopoulou (2009) Chang and Chen (2010) Lu, Yang, and Wang (2011) Li et al. (2009) Marketing Khademi-Zare, Zarei, Sadeghieh, and Saleh Owlia (2010) Secme et al. (2009) Yu, Guo, Guo, and Huang (2011) Human Resources Management Boran et al. (2011) Chen, Li, and Liu (2011) Wang, Liu, and Zhang (2005) Health and Safety Ekmekçioglu, Kaya, and Kahraman (2010) Krohling and Campanharo (2011) Wang, Fan, and Wang (2010) Energetics Azzam and Mousa (2010) Opricovic and Tzeng (2007) Yan et al. (2011) Water management Srđevic, Medeiros, and Faria (2004) Dai et al. (2010) Afshar et al. (2011) Simple additive weighting method has a wide range of applications. Most of the composite indicators are calculated by applying this method (e.g., GCI, The Global Competitiveness Index). Some well-known examples of the application of simple additive weighting method involve application of SAW method in the selection of staff [1], then for the selection of the best location of health facilities [21], as well as the application of SAW method in choosing the best location of factory plant [8]. 4. SIGNIFICANCE OF WEIGHTS Many methods for solving multi-attribute decision making problems require a clearly defined and expressed weights. However, in practice it is often difficult to determine the relative importance of the criteria, given the fact that the weights do not have clear economic significance, but they influence on the final result. Not all attributes have equal importance. The role of the weights is to reflect the relative importance of each of the attributes in relation to other attributes. One of the key problems of multi-criteria decision making is the determination of the relative significance of various criteria. The process of determining the relative significance of the attribute consists in defining and assigning a weight to each individual criterion. Particular weight should be as accurate as possible in order to show the contribution of each criterion to the overall result. Assigning weights in multi-criteria decision making is a critical phase of the entire Multi-Criteria Methods and Models for Decision Making in Public Procurement 269 decision making process. It is clear that the obtained result depends on the relative significance which has been assigned to each of the criteria. Therefore, evaluation and awarding of weights plays a key role in the multi-attribute decision making process Weights should be in accordance with the purpose of analysis. Further, the weights themselves are useful information for those who control the implementation of a specific project which is evaluated using the multi-attribute decision making methods, since they quantitatively show preferences of decision makers. One example may be the introduction of the Malcolm Baldrige National Quality Award in the United States. This award was introduced in order to stimulate American companies to improve quality and productivity. The relative importance of customer satisfaction is much higher than the weights of the other categories, which reflect the orientation of the Ministry of Trade to consumers [17]. 5. APPROACHES TO THE WEIGHTS DETERMINATION The impact of a particular criterion Cj (j = 1,..., n) on the final decision may have a different intensity, so it is necessary to determine the weights of each criterion, wj. There are several approaches for determining the relative significance of the attribute. Regardless of the applied approach, the value of the weights must be normalized, i.e., the sum of weight coefficients must be equal to one. In this regard, there are three approaches for determination of the relative significance of attributes: 1) The subjective approach 2) The objective approach 3) The combined approach, which combines the two previous approaches a) The subjective approach to the weights determination Subjective methods for determining the weights are based on the evaluation of experts. Their experience and knowledge are the most valuable information on contemplated criteria. Weights obtained from the subjective approach reflect the subjective judgment or intuition of the decision maker. Therefore, the results obtained by using weights established by this approach can be affected by the lack of knowledge or experience of the decision maker. Among the most famous subjective approaches are: 1) The Delphi Method 2) The Analytic Hierarchy Process Method (AHP method) 3) Additive Normalization Method 1) Delphi method Delphi method is one of the basic methods of forecasting and is the best known and most widely used method of expert evaluation. With this method direct discussion and confrontation of people and opinions is avoided, and that was something that made the classical method of obtaining the joint prediction from the expert group on the open meeting biased. The starting point of the method is the definition of the problem for which the forecast is required. After defining the problem, formation a group of experts who will participate 270 I. VESELINOVIĆ in the forecasting is the next step. Contacts with experts are carried out through the series of questionnaires. Through questionnaires forecasts and all the necessary information are required from them, while the anonymity of the experts and the obtained forecast is guaranteed. The first series of questionnaires which is being submitted to experts contain the necessary information, and they are asked to give their prediction that must be supported by appropriate arguments. Based on the obtained forecast, average forecast is being calculated, which represents the average of the individual forecasts, and also forecast variation around the mean value is determined, which is a measure of forecast accuracy. The second series of questionnaires sent to experts contain calculated average forecast, a measure of precision of forecasts and extreme forecasts with their reasons. Experts are then asked to reconsider their initial forecast, to do the correction if they want and to provide an opinion on the extreme forecasts, together with appropriate arguments. For processing the results Table 6 is used [24]. Table 6 Table for processing the results obtained by Delphi method Criteria Experts Mean value Standard deviation Coefficient of variation E1 E2 ... En C1 C2 ... Cn The process is being repeated until the mean value of the i+1 st circle does not show a slight deviation from the mean values of weight obtained in i th circle (usually no more than 5 rounds) or until the mean value of the coefficient of variation drops to a satisfactory level [24]. 2) The Analytic Hierarchy Process Method This method was developed by Thomas Saaty in the early seventies of the last century. AHP is a tool in the analysis of decision making, created in order to assist decision makers in solving complex problems involving a larger number of decision makers and a number of criteria. AHP is based on the concept of balance, which is used to determine the relative significance of the overall set of attributes, activities or criteria, and applies to analyzed decision problem [10]. “AHP allows the decision-maker to structure complicated problems in the form of a decision hierarchy” [2]. The hierarchy is structured in three levels: criteria, alternatives and goals. Bearing that in mind, the process starts from the lowest level in the hierarchy, and therefore the first step focuses on determination of relative importance of criteria. Main objective is to determine how much each of the criteria contributes to the goal. Next step consists in measuring the level of achievement of each criterion for alternatives. Finally, in the third step, the significance of the alternatives for the goal can be determined. The relative importance of alternatives demonstrates the relative importance of the criteria in achieving the goal of the hierarchy [23]. Multi-Criteria Methods and Models for Decision Making in Public Procurement 271 Determination of criteria weights is based on using pair-wise comparison of the criteria and calculation of weights by using a specific method of prioritization. The decision maker compares each criterion with the other and determines the level of preference for each pair of criteria. The process of creating the model requires four stages [18]: 1. Problem structuring 2. Data collection 3. Evaluation of the weights 4. Determination of the problem solution In the first stage the decomposition of decision making problem is carried out into a series of a hierarchy, where each level represents a smaller number of comparable attributes. In other words, the problem is viewed as a hierarchy where at the top is the goal of the problem, while the lower levels consist of attributes on the basis of which decisions are made. The lowest hierarchical level is composed of the range of alternatives from which the best is selected, i.e. m alternatives whose comparison is performed. The second phase of the AHP method involves collecting data and their measurement. Then, the comparison of the two pairs of attribute at a given level of the hierarchy, relative to an attribute of the higher level is performed. The essence of the comparisons by couples is to determine which of the two observed attributes better in relation to a given criterion. Obtained results from the corresponding comparison matrix. In the third phase corresponding weights are calculated. This phase gives a unique normalized eigenvectors of weights of all attributes at each level of the hierarchy. The process of weights determination will be shown through empirical research. The final step involves the determination of the final, overall priority vector at the level of the criteria. The relative importance of each criterion is expressed through weights. On the other hand, at the level of an alternative it is possible to determine the rank of alternatives for each of the monitored criteria. 3) Additive Normalization Method First step of this method consists of normalization of the columns. Normalization of columns is done by dividing each element of the column of the decision making matrix by the sum of that column. Then the obtained normalized values of elements are summed and divided by the number of elements in column. The method is simple and is often used in practice, although it may lead to a distortion of priorities in some specific cases. b) An objective approach to weights determination Taking into account the fact that the weights of criteria can significantly affect the outcome of the decision making process, it is clear that special attention must be paid to the objectivity of criteria. The methods of objective approach to weights determination focus on the analysis of decision making matrix. In the objective approach to the weights determination criteria are viewed as sources of information and the relative importance of the criteria reflects the amount of information contained in each of them. 272 I. VESELINOVIĆ The most known objective methods are: 1) Entropy method 2)Statistical methods 1) Entropy method Determination of objective weights of criteria according to the method of entropy is based on the measurement of indeterminacy of information contained in the decision matrix. Determination of criteria weights wj is carried out in four steps [22]. In the first step, the normalization of criterion values aij is performed. Obtained elements create normalized decision matrix. In the second step the value of entropy is determined. The amount of information contained in the normalized decision matrix and emitted by each criterion Cj can be measured as the value of entropy ej [31]. In the third step, the degree of diversification dj is determined. The greater the diversification of the initial criterion values aij of alternative Ai for a given criterion Cj, the value of dj for the given criteria is larger, and it can be concluded that the importance of criteria Cj for a given decision making problem is higher [25]. If all the values of the degree of diversification for a particular criterion are the same, observed criterion can be omitted because it does not give new information to decision makers [39]. In the fourth step the relative weights of the criteria can be obtained by the simple additive normalization. The method can be regarded as an objective because it generates criteria weights directly from the value of criterion for each alternative and it eliminates the problem of subjectivity, incompetence or lack of decision makers. 2) Statistical methods For the weights determination we can use a number of statistical methods. Two of them will be explain in detail. a) Method CRITIC (CRiteria Importance Through Intercriteria Correlation) is a method for the determination of objective weights of criteria that includes intensity of contrast and conflict contained in the structure of the decision problem. To determine the contrast of criteria the standard deviation of normalized criterion values in columns is used, as well as the correlation coefficients of all pairs of columns. Objective weights obtained in this way represent the amount of information contained in the decision matrix and provide unbiased information that decision maker uses in solving decision-making problems. b) Chi-square test is used to calculate whether there is a statistically significant correlation between the frequencies of the two attribute characteristics or between the obtained (observed) frequencies and the frequencies expected in a particular hypothesis. Chi-square is suitable in cases where the data have quantitative and qualitative nature. Chi-square test may include the following modalities:  the tests of goodness of fit which examines the difference between the distribution of obtained and expected frequency  the test of independence or test of equality (or difference) determines whether the examined independent samples belong to the same or are taken from different sets  the test of homogeneity which is used to test the correlation between the two characteristics of a set. Two independent samples are taken from one set, and the correlation between the two characteristics is tested. Determination of the intensity of Multi-Criteria Methods and Models for Decision Making in Public Procurement 273 the bonds between observed characteristics is done by calculating the coefficient of contingency. The value of Pearson's coefficient of contingency is positive and is located in the interval [0, 1]. 6. THE RESULTS OF EMPIRICAL RESEARCH In this section the advantages of the usage of multi-criteria decision making methods in public procurements will be presented. 6.1. Research methodology The basic premise of this paper is that the choice of the best offer is difficult in conditions where there are a number of criteria that can be used to assess them. The use of exact scientific methods for determination of the relative significance of each of the criteria and their use for ranking the offers can facilitate the decision making process. Therefore, the basic hypothesis (Ho) in this paper is that there is no difference in the perception of the decision makers of the importance of the criteria when the weights are determined using the exact scientific methods in relation to the empirical method. In order to prove that, a subjective method will be applied on the available data. Results will show that the use of the scientifically determined weights reduces the possibility of corruption in public procurements. Also, the use of the scientifically determined weights can provides higher welfare because procurer can obtain the object of the procurement that completely fulfills the requests of the citizens. The main objective of this section of the paper is determination of criteria weights on the basis of subjective approach. For that purpose the AHP method will be used. 6.2. Application of Analytic Hierarchy Process for weights determination Calculation of subjective preferences of decision maker will be performed based on data obtained from the Public Procurement Office of the City of Nis. Information about public procurements in the period from 2011 until 2013 shall be used. The purpose of the analysis is to assign weights to criteria relevant for the evaluation of the bids. In the given period Public Procurement Office of the City of Nis used empirical weights for bids ranking. They have performed two types of public procurement, one was based on the criterion of the lowest price, and other was most economically advantageous tender criterion. The offered price was the dominant criterion in this period. In 2011 lowest price criterion was applied in 162 tenders, in 2012 for 178 tenders and in 2013 for 130 tenders while other criteria were neglected. The most economically advantageous tender criterion was applied in 40 tenders-6 tenders in 2013, 16 tenders in 2012 and 18 tenders in 2011. In addition to the price, in this type of tenders other non-price criteria were also considered for the evaluation of the bids. Depending on the type of procurement different non-price criteria were applied. The list of all non-price criteria is given in the Table 7. 274 I. VESELINOVIĆ Table 7 Non-price criteria used for the evaluation of public procurements in the period from 2011 until 2013 Criterion Number of tenders in which the stated criterion was applied 2011 2012 2013 Deadline for works execution - 4 2 Deadline for construction works 1 - - References 5 1 1 Terms of payment 4 - - Delivery time 9 3 - Deadline for payment 9 3 - The conditions and terms of payment 1 1 - Economic characteristics - - 1 Technical characteristics - 1 - Quality of processing and manufacturing 1 2 - Esthetic and functional characteristics 1 2 - Number of gas stations in the City of Nis 2 1 1 Number of gas stations along the highway Belgrade-Nis 2 1 1 Loan period - 2 1 Effective interest rate - 2 1 In the further course of the research a specific public procurement related to a fuel supply for official vehicles shall be analyzed. In order to do that, a questionnaire was sent to the person responsible for the public procurements in the Public Procurement Office of the City of Nis. The questionnaire contained three criteria: offered price, number of gas stations in the City of Nis and number of gas stations along the highway Belgrade-Nis, that were supposed to be evaluated from 1 to 5. The following ratings were received: Table 8 Criteria rating Criterion Rating Offered price 5 Number of gas stations in the City of Nis 4 Number of gas stations along the highway Belgrade-Nis 2 Based on the obtained ratings the comparison of criteria was performed. Comparison of importance of the particular criteria regarding to the target was carried out based on the Saaty scale. The obtained values were entered into the comparison matrix. Table 9 Comparison matrix Offered price Number of gas stations in the City of Nis Number of gas stations along the highway Belgrade-Nis Offered price 1.000 3.000 9.000 Number of gas stations in the City of Nis 0.333 1.000 6.000 Number of gas stations along the highway Belgrade-Nis 0.111 0.167 1.000 Multi-Criteria Methods and Models for Decision Making in Public Procurement 275 To determine the vector of eigenvalues of the comparison matrix of comparison, a normalization of previously obtained values was performed. Normalization was performed by the following formula: 1 , , 1, 2, 3 ij ij m ij i a w i j a     The value of criteria weights was obtained by applying the formula: 3 1    n j ij j w w , i, j=1, 2, 3 Based on the obtained results it is possible to determine the weights of each criterion. Table 10 Criteria weights Criterion Weights Offered price 0.658 Number of gas stations in the City of Nis 0.282 Number of gas stations along the highway Belgrade-Nis 0.060 In order to calculate the Consistency Ratio (CR), the measure of consistency was calculated first. To calculate the consistency measures MMULT function in Microsoft Excel, MS Office package was applied. The obtained results are given in Table 11. Table 11 Calculation of the vector of priorities and consistency measure Offered price Number of gas stations in the City of Nis Number of gas stations along the highway Belgrade-Nis Wj Consistency Measure (λ) Offered price 0.692 0.720 0.563 0.658 3.103 Number of gas stations in the City of Nis 0.231 0.240 0.375 0.282 3.051 Number of gas stations along the highway Belgrade-Nis 0.077 0.040 0.063 .060 3.009 Based on the calculated consistency measure the calculation of consistency index is performed. Consistency ratio is the ratio between consistency measure λmax = 3.103 and random index (RI). Random index depends on the number of rows in the decision matrix, and its values are given in Table 12. Table 12 Random index value 1 2 3 4 5 6 7 8 9 10 0.00 0.00 0.580 0.900 1.120 1.240 1.320 1.410 1.450 1.490 276 I. VESELINOVIĆ If the consistency ratio is less than 0.10 result is accurate enough, and there is no need for further adjustment. If the level of consistency ratio is greater than 0.10 result should be re-analyze. Table 13 Consistency Ratio CI 0.051 RI (n=3) 0.580 CR=CI/RI 0.088 6.3. Comparative analysis A comparative analysis of the results of different approaches is aimed to assess the differences in the weights determined by different approaches. Also, the initial hypothesis assumes that there is a difference in the preferences of decision makers about the importance of the criteria when they are expressed through empirical weights and those preferences generated using the scientific method. The Public Procurement Office of the City of Nis had previously established certain weights based on experience. When it comes to the purchase of fuel for official vehicles, this procurement is evaluated by three criteria whose weights are given in Table 14. Table 14 Empirical weights Criterion Weights Offered price 0.700 Number of gas stations in the City of Nis 0.200 Number of gas stations along the highway Belgrade-Nis 0.100 In the further course of the analysis t-test will be applied to determine whether there are statistically significant differences between the empirically determined weights and weights determined by AHP method. The results are shown in Table 15. Table 15 Overview of the weights determined by different approaches and p-value Empirical approach AHP approach p-value Interpretation 0.7000 0.6583 0.0572 By conventional criteria, this difference is considered to be not quite statistically significant. 0.2000 0.2819 0.0067 By conventional criteria, this difference is considered to be very statistically significant. 0.1000 0.0598 0.0653 By conventional criteria, this difference is considered to be not quite statistically significant. For this purpose the GraphPad Software was used. The results show that there is no statistically significant difference between the empirical weights and weights determined by AHP method regarding to the first and third criteria, i.e. in these cases the p-value is greater than 0.05. Regarding the second criterion, p-value is less than 0.05, and it can be concluded that there is a statistically significant difference between empirical weights and weights obtained by AHP method. Multi-Criteria Methods and Models for Decision Making in Public Procurement 277 CONCLUSION Adequate legislation and the use of multi-criteria analysis methods in evaluating public procurements can greatly facilitate the process of decision making and reduce the abuse of the public procurement system, which is especially important considering the fact that the abuse of this system leads to inefficient allocation of public funds. The efficiency of the public procurement system leads to an increase in welfare due to the fact that public funds are used in a quantity that is sufficient to provide the required quantity and quality of the subject of procurement. Therefore, in this paper we have tried to prove that for an adequate, efficient and consistent decision making in the public procurement application of multi-criteria analysis methods is essential. Assigning criteria weights is an important step in the decision making process. This is because the value of weights largely determines the final decision of the decision maker. In this paper we have tried to prove that for a consistent and good decision making one of the most important issues that should be considered is determination of the appropriate weights. The first part of the paper defined the main terms in the public procurement system, gave a short review of the legislation in this area in the Republic of Serbia, and also showed potential corruption mechanisms which could arise in the public procurement system. The second part of this paper was committed to the methods and models of multi-criteria decision analysis, basic concepts of multi-criteria decision making and the formation multi- criteria model. Short review of some of the most important multi-criteria decision making methods was shown at the end of this section. The third part of this paper demonstrated the importance of the criteria weights and presented some of the basic approaches which could be used for weights determination. In addition, these approaches were classified into three groups, the subjective and objective approach, while the combined approach is actually a combination of subjective and objective approaches. Some of the most important subjective approaches such as Delphi method, AHP method, and the Additive Normalization Method were shown, and also some of the well- known objective methods such as the method of entropy and statistical methods. In the fourth part of the paper the results of an empirical study were presented. An overview of all possible criteria that Public Procurement Office of the City of Nis used in the period from 2011 to 2013 was given. Then, based on the score obtained from the person responsible for public procurements in the Public Procurement Office of the City of Nis the determination of weights was performed by the AHP method. Based on the obtained results it can be concluded that the initial hypothesis is not proven, i.e. there is a difference in the perception of the decision makers of the importance of the criteria when the weights are determined using the exact scientific methods in relation to the empirical method. Acknowledgement. The paper is a part of the research done within the projects 41010 and 44007, funded by the Ministry of Education, Science and Technology of the Republic of Serbia. 278 I. VESELINOVIĆ REFERENCES 1. Afshari, A., Mojahed, M., Yusuff, R. M. (2010) Simple Additive Weighting approach to Personnel Selection problem, International Journal of Innovation, Management and Technology, 1 (5): 511-515 2. Ali, A., Tahriri, F., Osman M. R., Yusuff R. M. (2008) A review of supplier selection methods in manufacturing industries, Suranaree Journal of Science and Technology 15 (3): 201-208 3. Behzadian M., Otaghsara, K., Yazdani, M., Ignatius, J. (2012) A state-of-art survey of TOPSIS applications, Expert Systems with Applications, An International Journal, 39 (17): 13051-13069 4. Benyoucef, M., Canbolat, M. (2007) Fuzzy AHP-based supplier selection in e-procurement, International Journal of Services and Operations Management, 3 (2): 172-192 5. Boran, F. E., Gença, S., Kurtb, M., Akay, D. (2009) A multi-criteria intuitionistic fuzzy group decision making for supplier selection with TOPSIS method, Expert Systems with Applications, 36 (8): 11363-11368 6. Cheng, E. W. L., Li, H. (2004) Contractor selection using the analytic network process, Construction Management and Economics, 22: 1021-1032. 7. Cheung, S.-O., Lam, T.-I., Leung, M.-Y., Wan, Y.-M. (2001) An analytical hierarchy process based procurement selection method, Construction Management and Economics, Volume 19 (4): 427-437 8. Chou, S.-Y., Chang, Y.-H. & Shenc, C.-Y. (2008) A fuzzy simple additive weighting system under group decision-making for facility location selection with objective/subjective attributes, European Journal of Operational Research, Volume 189 (1): 132-145 9. Cook, R. L., Stewart, T. R. (1975) A Comparison of Seven Methods for Obtaining Subjective Descriptions of Judgmental Policy, Organizational Behavior and Human Performance, Volume 13 (1): 31-45 10. Cupic, M., Tummala, R., Suknović, M. (2003), Decision making: a formal approach (in Serbian), Belgrade, Faculty of Organizational Sciences 11. De Boer L., Labrob E., Morlacchi P. (2001). A review of methods supporting supplier selection, European Journal of Purchasing & Supply Management, Volume 7 (2): 75-89 12. De Boer L., van der Wegen L., Telgen J. (1998) Outranking methods in support of supplier selection, European Journal of Purchasing & Supply Management, Volume 4 (2–3): 109-118 13. Dulmin, R., Mininno, V. (2003) Supplier selection using a multi-criteria decision aid method, Journal of Purchasing and Supply Management, Volume 9 (4): 177-187 14. Figueira, J., Greco, S., Ehrgott M. (2005) Multiple Criteria Decision Analysis: State of the Art Surveys, Boston: Springer Series: International Series in Operations Research & Management Science, Volume 78 (36): 3-24 15. Haq, N., & Kannan, G. (2006) Fuzzy analytical hierarchy process for evaluating and selecting a vendor in a supply chain model, The International Journal of Advanced Manufacturing Technology, Volume 29 (7-8): 826-835 16. Hsieha, T.-Y., Lub, S.-T., Tzeng, G.-H. (2004) Fuzzy MCDM approach for planning and design tenders selection in public office buildings, International Journal of Project Management, Volume 22 (7): 573-584 17. Hwang C.-L., Yoon K.P. (1995), Multiple Attribute Decision Making: An Introduction, Sage Publications, USA 18. Jankovic-Milic V., Stankovic J. (2010), Bayesian approach to multicriteria decision analysis in business decision making (in Serbian), Vrnjacka Spa 19. Jaskowski, P., Biruk, S., Bucon, R. (2010). Assessing contractor selection criteria weights with fuzzy AHP method application in group decision environment, Automation in Construction, Volume 19 (2): 120-126 20. Kahraman, C., Cebeci, U., Ulukan, Z., (2003) Multi-criteria supplier selection using fuzzy AHP, Logistics Information Management, Volume 16 (6): 382-394 21. Lin, H. Y., Liao, C. J. & Chang, Y. H. (2010), Applying fuzzy simple additive weighting system to health examination institution location selection, Industrial Engineering and Engineering Management (IE&EM), IEEE 17 th International Conference, 29-31 Oct. 2010, Xiamen: 646-650 22. Lofti, F.L., Fallahnejad, R. (2012) Imprecise Shannon's Entropy and Multi Attribute Decision Making, Entropy. 2010, Volume 12(1): 53-62 23. Maggie C.Y.T. & Tummala, V.M.R. (2001). An application of the AHP in vendor selection of a telecommunications system, Omega, Volume 29 (2): 171-182 24. Milicevic, M. R, Zupac G. Z. (2012), Subjective approach to determination of the criteria weights (in Serbian), Military Technical Courier (in Serbian), Volume 60 (2): 48-70 25. Milic, M.R., Zupac, G.Z. (2012) An objective approach to determination of the criteria weights (in Serbian), Military Technical Courier (in Serbian), Volume 60 (1): 39-56. 26. Haq, N., & Kannan, G. (2006) Fuzzy analytical hierarchy process for evaluating and selecting a vendor in a supply chain model, The International Journal of Advanced Manufacturing Technology, Volume 29 (7-8): 826-835 Multi-Criteria Methods and Models for Decision Making in Public Procurement 279 27. Pavlicic, D. (2002). Consistency of selection of multi-attribute analysis methods (in Serbian), Economic annals (in Serbian), 155: 59-82. 28. Paul Yoon K. & Hwang, C. L. (1981), Multiple attribute decision making methods and applications: A state-of-the-art survey, Lecture Notes in Economics and Mathematical Systems, Berlin: Springer-Verlag 29. Popović, Ž., Stanković, J., Veselinović, I. (2013) Multi-criteria analysis application in the investment projects assessment, FACTA UNIVERSITATIS, Series: Economics and Organization Volume 10 (4): 401 - 418 30. Serbian Public Procurement Law, http://www.parlament.gov.rs/upload/archive/files/cir/pdf/zakoni/ 2012/4113-12.pdf (21.07.2014) 31. Serbian Public Procurement Office, Strategy of Public Procurement Development in the Republic of Serbia (in Serbian), http://www.ujn.gov.rs/sr.html (12.08.2014) 32. Srdjevic, B., Medeiros, Y. D. P., Faria A. S., Schaer, M. (2003). Objective evaluation of performance criteria for the system of accumulations (in Serbian), Water management (in Serbian), 35: 163-179 33. Soreide, T. (2002) Corruption in public procurement. Causes, consequences and cures, Chr. Michelsen Institute, http://hdl.handle.net/10202/185 (10.08.2014) 34. Triantaphillou, E. (2000), Multi-criteria decision making methods: A comparative study. Dordrecht, itd: Kluwer 35. Torra, V., Yasuo N. (2007), Modeling Decisions. Berlin: Springer-Verlag 36. Tzeng, G.-H., Huang, J.-J. (2011), Multiple Attribute Decision Making: Methods and Applications, Chapman and Hall/CRC 37. Tzeng, G.-H, Yanga, J. L., Chiub, H. N., Yeh, R. H. (2008) Vendor selection by integrated fuzzy MCDM techniques with independent and interdependent relationships, Information Sciences, Volume 178 (21): 4166-4183 38. Varinac S. (2012) Corruptional map of public procurements in Republic of Serbia (in Serbian), Belgrade, http://www.ujn.gov.rs/sr.html (15.07.2014) 39. Weber, C. A., Ellram, L. M., (1993) Supplier Selection Using Multi-objective Programming: A Decision Support System Approach, International Journal of Physical Distribution & Logistics Management, Volume 23 (2): 3-14 40. Zeleny M. (1982) Multiple Criteria Decision Making, New York: McGraw-Hill 41. Zopounidis, C., Doumpos, M. (2002) Multi-criteria Decision Aid in Financial Decision-Making: Methodologies and Literature Review, Journal of Multi-Criteria Decision Analysis, Special Issue: MCDA Methodologies in Finance, Volume 11 (4-5): 167-186. VIŠEKRITERIJUMSKI METODI I MODELI ODLUČIVANJA U JAVNIM NABAVKAMA Izbor najboljeg ponuđača u procesu javne nabavke predstavlja tipičan primer višekriterijumskog problema odlučivanja. Osnovna svrha ovog rada jeste prezentovanje mogućih pristupa za izračunavanje pondera u cilju olakšavanja postupka donošenja odluka u procesu javne nabavke. Imajući u vidu činjenicu da težinski koeficijenti kriterijuma mogu da utiču na konačni redosled alternative, veoma je važno ozbiljno i odgovorno pristupiti procesu određivanja pondera. Adekvatno određeni težinski koeficijenti smanjuju mogućnost pojave zloupotreba i prevara u sistemu javnih nabavki. Određivanje pondera u radu je izvršeno na bazi subjektivnog pristupa (Analitički hijerarhijski proces). Ključne reči: Izbor ponuđača, analitički hijerarhijski proces, višekriterijumsko odlučivanje, javne nabavke, težinski koeficijenti. http://www.parlament.gov.rs/upload/archive/files/cir/pdf/zakoni/2012/4113-12.pdf http://www.parlament.gov.rs/upload/archive/files/cir/pdf/zakoni/2012/4113-12.pdf http://www.ujn.gov.rs/sr.html http://hdl.handle.net/10202/185 http://www.ujn.gov.rs/sr.html