Operational Research in Engineering Sciences: Theory and Applications Vol. 4, Issue 2, 2021, pp. 102-123 ISSN: 2620-1607 eISSN: 2620-1747 DOI: https://doi.org/10.31181/oresta20402102y * Corresponding author. yorulmaz@istanbul.edu.tr (Ö. Yorulmaz), sultan.kuzu@istanbul.edu.tr (S. Kuzu Yıldırım), bahadirf.yildirim@istanbul.edu.tr (B. F. Yıldırım) ROBUST MAHALANOBIS DISTANCE BASED TOPSIS TO EVALUATE THE ECONOMIC DEVELOPMENT OF PROVINCES Özlem Yorulmaz 1, Sultan Kuzu Yıldırım 2, Bahadır Fatih Yıldırım 3* 1 Department of Econometrics, Faculty of Economics, Istanbul University, Turkey 2 Department of Quantitative Methods, School of Business, Istanbul University, Turkey 3 Department of Logistics, Faculty of Transportation and Logistics, Istanbul University, Turkey Received: 07 April 2021 Accepted: 07 June 2021 First online: 01 July 2021 Research paper Abstract: In this paper, 81 Turkish provinces with different development levels were ranked using the TOPSIS method. To evaluate the ranking with TOPSIS, we presented an improvement to Mahalanobis distances, by considering a robust MM estimator of the covariance matrix to deal with the presence of outliers in the dataset. Additionally, the homogenous subsets, which were obtained from the robust Mahalanobis distance- based TOPSIS were compared with robust cluster analysis. According to our findings, robust TOPSIS-M scores reflect the inter-class differences in economic developments of provinces spanning from the extremely low to the extremely high level of economic developments. Considering indicators of economic development, which are often used in the literature, İstanbul ranked first, Ankara second, and İzmir third according to the Robust TOPSIS-M method. Moreover, with the Robust Cluster analysis, these provinces were diagnosed as outliers and it was seen that obtained clusters were compatible with the ranking of Robust TOPSIS-M. Keywords: Economic Development, Mahalanobis Distance, Robust Clustering, Robust TOPSIS-M, Outliers. 1. Introduction In today's world where globalization and competition are rapidly increasing, countries are trying to gain an advantage with both their economic activities and social policies. To increase the international competitiveness of the countries, it is aimed to keep the economic indicators in the national context. Because it has been observed that regional and local economies also affect the global economy and increase competition (Kılıç et al., 2011). Economic development has generally been Robust Mahalanobis Distance Based TOPSIS to Evaluate the Economic Development of Provinces 103 conceptualized as a balance increase in per capita income (Ascani et al., 2012). However, studies draw attention to the importance of determining the factors affecting per capita income. For regional development, the necessity of both increasing exports and following import substitution strategies have been put forward (Shaffer, 1989; Blair and Carroll, 2008; Cooke and Watson, 2011). Exports are generally considered in two dimensions as the export of goods and services. Advanced technology and advanced industrial facilities used in developed countries increase the sales potential for the foreign market by enabling these countries to produce fast and high quality (Contractor and Mudambi, 2008). On the other hand, developing countries, follow a policy that will increase exports by utilizing their raw materials and underground resources. The service sector has been identified as a new growth engine for both developed and developing countries (Noland et al., 2012, Akın and Özsağır, 2012). Regions and provinces in the country carry out export activities according to the characteristics of their geographical location, production, and service types. According to these characteristics, there are important differences between the export capacities of the provinces and the development levels accordingly. Economic development, in another definition, focuses on increasing wealth (Mathur, 1999). According to this view, domestic savings are one of the most important sources of development. The positive relationship between saving and growth has been noted in studies of many countries (Room, 2002; Carroll and Weil, 1994). In recent years a decline was observed in domestic savings in Turkey. This decline causes a negative impact on the economy through a deficit and it has led to the emergence of domestic savings again. (Peace and Space, 2015). Another factor that is thought to have an impact on economic development is population. However, the direction and strength of the relationship between economic development and population are still under debate. While some argue that rapid population growth has a negative effect on economic development (Srinivasan, 1988; Kentor, 2001), there are also studies showing that the relationship between them is not significant (Easterlin, 1967). The population-oriented economic growth hypothesis, which states that population growth supports economic development, also maintains its validity. It is seen that population growth has positive effects on economic development, especially in developing countries (Furuoka, 2009). Increasing population brings some needs with it. The most important of these is the need for housing. With the sale of housing, not only the construction sector but also many sub-sectors such as cement, ready-mixed concrete, iron, and steel are affected. Specifically, when the economic contraction begins in developing countries, a way out of this bottleneck is sought by increasing investment expenditures in the construction sector. Thus, economic recovery is provided. TOPSIS (Technique for Order Preference by Similarity to Solution) method makes it possible to assess the objects concerning multidimensional economic phenomena based on the group of economic variables (Yoon and Hwang, 1995; Balcerzak and Pietrzak, 2016). Most economists think that international comparisons of the level of sustainable development must be done with an application of quantitative methods (Balcerzak and Pietrzak, 2016). TOPSIS is referred to be a very useful and informative technique for ranking and selecting variables (Shih et al., 2007; Bhutia and Phipon, 2012, Kizielewicz et al. 2021). For this reason, TOPSIS is widely used in studies that Yorulmaz et al./Oper. Res. Eng. Sci. Theor. Appl. 4 (2) (2021) 102-123 104 are based on the comparisons of economic and financial performances and real- world problems. Eyüboğlu (2016) compared the developing countries considering macro performances as economic growth, inflation rate, unemployment rate, and the current account balance/GDP using Analytic Hierarchy Process (AHP) and TOPSIS methods. Using similar variables, Dinçer (2011) ranked both European Union members and candidate countries using TOPSIS and similarly, Kuncova (2012) made the comparisons of European countries in terms of e-commerce. TOPSIS method was also preferred to evaluate economic performances of countries during the financial crisis period (Mangır and Erdoğan, 2011) and used to examine the development achievement by European countries in the field of implementing the concept of sustainable development (Balcerzak and Pietrzak, 2016). TOPSIS method was employed to evaluate the good governance development in the European Union countries for the years of 2007-2017 (Ardielli, 2019). To assess the e-Government in the countries TOPSIS was used (Ardielli and Halaskova, 2015). Besides the comparisons of countries, municipalities were evaluated considering environmental sustainability using DEMATEL based TOPSIS (Kiliç and Yalçın, 2020). Slovak municipalities were assessed according to management criteria using TOPSIS (Vavrek, et al, 2015). Different from the listed studies here, TOPSIS was also used to identify suitable health indicators to evaluate the efficiency of Slovak municipalities (Vavrek et al., 2021). In this study, it was aimed to evaluate the level of economic competition of 81 Turkish provinces considering the economic indicators using TOPSIS-M (Mahalanobis distance-based TOPSIS) which is based on the robust covariance matrix. The TOPSIS method is used to construct the ranking of items considering many variables and it is based on Euclidean distance that assumes the criteria of monotonically increasing or decreasing and this approach disregards the dependence among variables. Conversely, TOPSIS-M uses dependencies between variables considering the correlation matrix. However, in the presence of outliers, the use of methods based on covariance matrix should be approached with attention. Because the covariance matrix can be manipulated by outliers and give misleading results. TOPSIS method is based on the distances from the model values (“positive ideal solution” and “negative ideal solution”) and in case of the existence of outliers in a dataset, the maximum and minimum values of the variables affect the model values inevitably and this leads to excessive remoteness from typical values of the considered variables that narrow the range of variability of the constructed synthetic measure (Luczak and Just, 2020). Several studies in the literature suggested limiting the effect of outliers on the TOPSIS method. Khalif, et al. (2017) proposed the Spearman correlation matrix to handle outlier effects in the TOPSIS method. Luczak and Just (2020) used robust standardization and spatial median to make the TOPSIS method resistant against outliers. De Andrede, et al. (2020) used Singular Value Decomposition (SVD) TOPSIS approach to decrease the impacts of outliers while evaluating the performance of TV programs. In this study, different from the previous approaches we presented an improvement to TOPSIS-M by using robust Mahalanobis distances which are resistant to outliers. To make Mahalanobis distances resistant to outliers, a robust covariance matrix was used. The covariance matrix employed in this study is based on the MM estimator. However, MCD, OGK, and S estimators were also evaluated, but Robust Mahalanobis Distance Based TOPSIS to Evaluate the Economic Development of Provinces 105 since the results were very similar, only the results based on the MM estimator are included here. To evaluate the level of economic competition of provinces in this study, per capita GDP, the trade deficit (import-export), the population, the total housing sales numbers, and the total bank deposit accounts were determined as variables. Since this dataset includes socioeconomic variables belonging to the provinces, due to the provinces with different development levels, the existence of outliers and dependency between variables are expected. Therefore, in the first stage of the application, descriptive statistics and correlation matrices were used to evaluate the dataset and outliers were diagnosed. In the next stage, the findings obtained from TOPSIS, TOPSIS-M, and robust MM covariance matrix based TOPSIS-M were evaluated. In addition to rank the provinces by taking into account the economic indicators, it was also included to classify provinces with robust cluster analysis. At the final stage, findings of robust cluster analysis were compared homogenous subsets obtained from robust Mahalanobis distance-based TOPSIS. 2. Methodology TOPSIS method, originally developed by Hwang and Yoon (1981), is a simple and efficient Multi-Criteria Decision-Making (MCDM) method to identify solutions from a finite set of alternatives. The main idea is based on determining the best alternative which should have the closest geometric distance from the ideal solution. However, there are some main disadvantages in the traditional TOPSIS model: (i) correlations between criteria, (ii) uncertainty in obtaining the weights only by objective and subjective methods, finally, (iii) possibility of alternative closed to positive and negative ideal points concurrently (Li et al., 2011). Additionally, when the data set does not only include regular observations, outliers may have effects on the definition of ideal solutions and the calculation of distances it is important to consider robust estimators to deal with outliers. Because of the listed disadvantages, traditional TOPSIS can lead to biased estimation of relative significances of alternatives and can cause inaccurate ranking results. To overcome the deficiency of correlation between criteria in the TOPSIS model, Mahalanobis distance-based TOPSIS was preferred. Mahalanobis distance is a measure that takes into consideration the correlation in the data by using the covariance matrix. However, outliers have a major influence on the covariance matrix. Because covariance matrix is known as a low breakdown estimator. Outliers attract mean and inflate variance towards its direction (Becker and Gather, 1999). To make Mahalanobis distances resistant against outliers, robust estimates of the covariance matrix are preferred to use (Rocke and Woodruff, 1996). Robust estimators are used to reducing and limiting the effect of outliers and strong asymmetry when calculating Mahalanobis distance. The robustness of an estimator can be evaluated by considering breakdown points and influence function properties (Huber, 1981; Maronna et. al., 2006). Minimum Covariance Determinant (MCD) estimator, S-estimators, Orthogonalized Gnanadesikan-Kettenring (OGK) estimator, and MM-estimators are well-known high-breakdown robust estimator of mean and covariance matrix. The covariance matrix employed in this study is based on the MM estimator. Yorulmaz et al./Oper. Res. Eng. Sci. Theor. Appl. 4 (2) (2021) 102-123 106 2.1. Mahalanobis Distance-Based TOPSIS (TOPSIS-M) The Euclidean distance approach used by the TOPSIS method is insufficient in terms of investigating the relationship between the criteria in the MCDM problem and including it in the decision process. Therefore, it is more appropriate to use Mahalanobis distance in calculating the deviations from the ideal solutions. TOPSIS-M method is a type of analysis in which deviations are computed using Mahalanobis distance in traditional TOPSIS algorithm. Mahalanobis distance measurement also takes into account the correlation between variables in measuring the distance between two points. This measurement was proposed by Mahalanobis in 1936 and is used under his name. Mahalanobis distance between 1 x and 2 x points is calculated with the help of the following equation:      11 2 1 2 1 2, T d x x x x C x x     (1) C in Eq. (1) shows the variance-covariance matrix of the X set consisting of x values. (Xiang et al., 2008). Analysis of the decision problem with the TOPSIS-M method consists of the following steps. Step 1. As in all MCDM problems, the analysis process in the TOPSIS method starts with generating a decision matrix in which is the performance score of the alternative according to the criterion is expressed together. The A matrix created by the decision-maker is shown as below: 11 12 1 21 22 2 1 ij n n m mn a a a a a a A a a              (2) Step 2. Since the performance values created in the decision matrix represent different units or sizes according to different criteria, the evaluation process is continued by standardizing the decision matrix. Standardized performance scores to standardize the decision matrix, represented by ij r , are obtained as follows: 2 1 1, 2, , 1, 2, , ij ij m kj k a r i m j n a      (3) R standardized decision matrix is obtained by making use of Eq. (3). Step 3. As mentioned in the definition of the TOPSIS-M method, it is based on the principle of proximity calculation to ideal solutions. In this step of the TOPSIS-M method, in which the ideal solution is handled in two directions, the ideal positive solution and the ideal negative solution sets are created, and the process continues. While creating the ideal solution clusters, the attributes of the criteria included in the decision problem are taken into account, considering the benefits and cost conditions. Robust Mahalanobis Distance Based TOPSIS to Evaluate the Economic Development of Provinces 107 In the TOPSIS-M method, the positive ideal solution set is calculated with Eq. (4), and the negative ideal solution set is calculated with the help of Eq. (5).  * '(max ), (minij ijiiA v j J v j J   (4)  '(min ), (maxij ij ii A v j J v j J     (5) In the equations, J refers to Benefit Index and J’ refers to Cost Index. Step 4. In the TOPSIS-M method, the Mahalanobis distance approach is used to calculate deviations from ideal solution sets. As a result of the process, ideal separation values are calculated for each solution set. The Positive ideal discrimination measure * i S is calculated using Eq. (6) and the negative ideal discrimination measure i S  is calculated using Eq. (7).      * * * 1 *, T T i i i i S d x A A x C A x        (6)      1, T T i i i i S d x A x A C x A            (7) The C value in the equations represents the variance-covariance matrix of the X decision matrix of mxn, and  represents the square root of the elements of the weight vector on the diagonal matrix. The diagonal matrix  is obtained using Eq. (8).  1 2, , , ndiag w w w  (8) Step 5. In the calculation of the * i C value, which expresses the relative proximity of each alternative to the ideal solution, the ideal separation measures obtained in Step 5 are used. * * * , 0 1i i i i i S C C S S       (9) As the * i C values that take values between 0 and 1 grow, it expresses the absolute proximity to the positive ideal solution. The * i C value obtained as a result of the analysis steps is ranked in descending order and a ranking based on the closeness of the alternatives to the ideal is obtained (Wang and Elhag, 2006). 2.2. Robust MM Estimator The MM-estimator is a high breakdown value estimator, and it is an extension of the S-estimator (Maronna et. al., 2006). S-estimator was proposed by Rousseeuw and Leroy (1987). S-estimators of location μ and covariance S are defined such that the Yorulmaz et al./Oper. Res. Eng. Sci. Theor. Appl. 4 (2) (2021) 102-123 108 determinant of the matrix S is minimized under the constraint (Maronna et. al., 2016):     ' 1 1 1 n i i i X S X b n         (10) where b is a constant and ( )X is the loss function. A popular choice loss is Tukey’s bi-weight function (Hubert and Rousseuw, 2013): 3 22 2 1 1 , 6 ( ) , 6 k x x k k x k x k                   (11) For the estimation of the MM estimator the following steps should be considered (Maronna et. al., 2006.): a) Define a loss function ρ to compute the S-estimators of location and covariance, (  and  ). b) Calculate 1/ 2 ˆ p    c) Find the MM-estimator of the location and the shape parameter, ˆˆ( , )  , that minimize: ' 1 1/ 2 1 1 1 ˆ(( ) ( )) / ) n i i i X X n          (12) d) Compute the MM-estimator of the covariance matrix ˆ ˆ̂   2.3. Robust Cluster Cluster analysis is based on identifying homogeneous clusters with large heterogeneity among them. Many studies emphasize outliers may impair clustering ability and clustering methods need to be robust if they are to be useful in applications (García-Escudero et al. 2010, Ruwet et al. 2012). For handling outliers, robustness in cluster analysis is needed because outliers appear many times joined together (Garcia-Escudero et.al. 2011). To refrain from the outlier effects García- Escudero et al. (2008) introduced the TCLUST approach. The TCLUST approach performs robust clustering to find clusters with different distribution structures and weights (Ruwet et al. 2012). The TCLUST algorithm allows for Eigenvalue Rate restriction and trimming of a specific observation rate determined by the researchers to eliminate the effect of outliers. The T-CLUST method is known as the trimmed k- means technique. In this study, TCLUST was used to identify clusters with trimming a rate of 5%. The flowchart in Figure 1 summarizes the steps followed throughout the methodology. As can be seen from the flow chart in the first stage, Mahalanobis distances based on the solid MM covariance matrix were calculated using the first Robust Mahalanobis Distance Based TOPSIS to Evaluate the Economic Development of Provinces 109 decision matrix and these distances were used for ranking in the TOPSIS process. Similarly, based on this decision matrix, TOPSIS scores, and TOPSIS-M scores based on the classical covariance matrix were obtained. In the last step, provinces were classified using robust cluster analysis and the findings were evaluated considering the MM covariance-based TOPSIS-M, TOPSIS-M, and TOPSIS rankings. Normalized decision matrix Robust Mahalanobis Distances Initial decision matrix Robust Clustering Determine the positive and negative ideal solutions sets Calculation of the similarity distances Obtain C* values Rank provinces Calculation of the similarity distances Obtain C* values Rank provinces Calculation of the similarity distances Obtain C* values Rank provinces TOPSIS Robust TOP SIS-M TOPSIS-M C o m p a re R e su lt s Robust MM Covariance Matrix Figure 1. Flowchart of the evaluation methodology used. 3. Dataset and Results In this study, the variables of GDP per capita, the trade deficit of the provinces (import-export), the population of the provinces, the total housing sales figures in the provinces, and the total bank deposit accounts of the provinces are used for the years 2019 and 2020. Datasets have been created through the official web page of the Turkish Statistical Institute and the Banking Supervision and Regulatory Agencies. The reason why the TOPSIS method based on Mahalanobis distance was preferred in this study is the strong correlation coefficients between the variables. When the correlation values in Table 1 are examined, it is seen that there is a strong relationship. However, it was observed that the relationships were slightly weaker in the MM correlation matrix. Table 1. Pearson Correlation Matrix Population GDP per capita Housing Sales Trade deficit Bank deposit Population 1.00 GDP per capita 0.52 1.00 Housing Sales 0.97 0.61 1.00 Trade deficit 0.85 0.39 0.77 1.00 Bank deposit 0.96 0.52 0.93 0.94 1.00 Yorulmaz et al./Oper. Res. Eng. Sci. Theor. Appl. 4 (2) (2021) 102-123 110 Descriptive statistics were presented in Table 2. As can be seen from Table 2, the difference between mean and median values of variables (except GDP per capita) seem significantly different. This raises the suspicion of the existence of outliers. As a matter of fact, in a way to confirm this situation, outlying observations can be seen in Figure 2. Figure 2 corresponds to the distance-distance plot defined by Rousseeuw and van Zomeren (1991). This plot is based on classical Mahalanobis distances versus robust Mahalanobis distances (based on MM covariance estimator), it enables the classification of regular observations and outliers. The dashed line depicts the points where both distances are equal. The vertical and horizontal lines were drawn at the points ( 2  df=5, 0.975). Observations beyond these lines (Istanbul, Ankara, and Izmir) are defined as outliers. Table 2. Descriptive statistics of development indicators Variables Mean Std. Dev. Median MAD Bank deposit 45353637,4 171658598 9715929 8497261 Housing Sales 18510,07 35694,54 7625 7168,37 Population 1032276,07 1872575,82 537762 419343 Trade deficit -402315,83 4976466,74 35118 142794 GDP per capita 39506,76 13648,03 36820,7 10774,7 Figure 2. Distance-Distance plot (detection of outlying provinces). Robust TOPSIS-M analysis steps and final scores of 81 provinces which obtained based on robust MM covariance matrix, are included in the Appendix. However, in Figure 3, provinces are divided into homogeneous groups based on these robust TOPSIS-M scores. As can be seen from this map, the provinces with the highest scores are respectively Istanbul, Ankara, Izmir, and Antalya. The scores with the lowest provinces are Ardahan, Bayburt, and Tunceli. These rankings are consistent with the Robust Mahalanobis Distance Based TOPSIS to Evaluate the Economic Development of Provinces 111 actual values, considering the development levels of the provinces. Robust TOPSIS-M scores reflect the inter-class differences in the economic developments of provinces. Figure 3 presents ten classes of provinces, spanning from the extremely low to the extremely high levels of economic development. Figure 3. Classification of provinces according to Robust TOPSIS-M scores. Figure 4. Classification of provinces according to robust clustering. In Figure 4, robust clustering results were given. According to the TCLUST algorithm, four clusters and an outlier group were obtained. Cluster 0 consists of the outlying provinces. The map in Figure 4 also includes rank values of provinces according to robust TOPSIS-M scores. As can be seen, provinces were divided into four groups according to the robust clustering. Following the "distance-distance plot" in Figure1, Istanbul, Ankara, and Izmir have been determined as outliers here as well, and these provinces are in the top three with the robust TOPSIS-M ranking. It is seen that the homogeneous groups defined based on robust TOPSIS-M scores in Figure 3 are compatible with the clusters in Figure 4. Although there are fewer clusters in Figure 4, only four clusters, these clusters can show the inter-class differences in terms of development indicators. Yorulmaz et al./Oper. Res. Eng. Sci. Theor. Appl. 4 (2) (2021) 102-123 112 Table 3 presents the ranking of provinces according to TOPSIS, TOPSIS_M, and TOPSIS-MM approaches. This table also contains information about the cluster to which each province belongs. Rankings of provinces in the same cluster in Table 3 are expected to be close to each other. Although the order of provinces falling into clusters with 0 and 4 codes is close to each other in all three approaches, the order of provinces in clusters with codes 1-2 and 3 seems compatible only in TOPSIS-MM. Denizli, Kocaeli, Şırnak, Hatay, and Çorum are not compatible in the clusters in which they are ranked according to TOPSIS and TOPSIS-M approaches. Table 3. Ranking of Provinces based on TOPSIS, TOPSIS-M, and TOPSIS-MM approaches Province T O P S IS T O P S IS -M R o b u st T O P S IS -M R o b u st C lu st e r Province T O P S IS T O P S IS -M R o b u st T O P S IS -M R o b u st C lu st e r İstanbul 1 1 1 0 Adıyaman 47 50 42 1 Ankara 2 2 2 0 Kırklareli 35 39 43 1 İzmir 3 3 3 0 Kastamonu 42 38 44 1 Antalya 5 5 4 4 Giresun 40 42 45 1 Bursa 4 4 5 4 Uşak 45 36 46 1 Gaziantep 6 9 6 3 Isparta 37 35 47 1 Kocaeli 12 6 7 3 Düzce 41 52 48 1 Konya 7 8 8 3 Aksaray 44 37 49 1 Adana 10 7 9 3 Yalova 38 40 50 1 Denizli 14 15 10 3 Yozgat 57 46 51 1 Mersin 8 10 11 2 Siirt 64 75 52 1 Hatay 24 13 12 2 Batman 54 54 53 1 Muğla 17 11 13 2 Bolu 46 51 54 1 Kayseri 9 12 14 2 Amasya 55 60 55 1 Manisa 19 16 15 2 Niğde 53 59 56 1 Balıkesir 16 14 16 2 Bilecik 49 65 57 1 Tekirdağ 13 19 17 2 Karabük 68 49 58 1 Aydın 15 17 18 2 Nevşehir 59 44 59 1 Samsun 21 20 19 2 Kırşehir 63 57 60 1 Kahramanmaraş 25 25 20 2 Karaman 52 55 61 1 Diyarbakır 20 23 21 2 Burdur 51 56 62 1 Sakarya 11 22 22 2 Şırnak 39 73 63 1 Eskişehir 22 18 23 2 Ağrı 67 70 64 1 Şanlıurfa 18 27 24 2 Kırıkkale 56 64 65 1 Trabzon 23 21 25 2 Çankırı 62 67 66 1 Erzurum 36 43 26 1 Bitlis 74 76 67 1 Elazığ 32 34 27 1 Kars 72 69 68 1 Ordu 30 30 28 1 Muş 65 72 69 1 Afyonkarahisar 27 28 29 1 Erzincan 58 61 70 1 Malatya 28 29 30 1 Sinop 66 63 71 1 Van 31 45 31 1 Bartın 70 62 72 1 Mardin 26 58 32 1 Artvin 60 66 73 1 Çanakkale 29 26 33 1 Hakkari 77 78 74 1 Sivas 33 31 34 1 Bingöl 73 68 75 1 Çorum 81 32 35 1 Iğdır 71 74 76 1 Kütahya 34 41 36 1 Gümüşhane 78 77 77 1 Zonguldak 75 24 37 1 Kilis 76 79 78 1 Rize 50 47 38 1 Ardahan 79 80 79 1 Edirne 43 33 39 1 Tunceli 69 71 80 1 Osmaniye 61 53 40 1 Bayburt 80 81 81 1 Robust Mahalanobis Distance Based TOPSIS to Evaluate the Economic Development of Provinces 113 Figure 5. Comparision of TOPSIS, TOPSIS-M, and Robust TOPSIS-M results The provinces that exists in cluster 0 and cluster 4 are also consistent in terms of rankings. While Denizli and Kocaeli should be in the third cluster, they are in the second cluster according to TOPSIS and TOPSIS-M rankings. The province of Zonguldak, which should be in the first cluster, falls in the second cluster according to the TOPSIS and TOPSIS-M rankings, and Şanlıurfa, which should be in the second Yorulmaz et al./Oper. Res. Eng. Sci. Theor. Appl. 4 (2) (2021) 102-123 114 cluster, falls into the first cluster. However, as can be seen in Figure 5, there is no inconsistency between Robust TOPSIS-M and clusters. 4. Conclusion The TOPSIS method is an MCDM method that is frequently used to sort the observations and divide them into homogeneous groups, considering various variables. However, the TOPSIS method is calculated based on the Euclidean distance and ignores the relationship between variables. The TOPSIS-M method calculated based on the Mahalanobis distance takes into account the dependency structure between variables. However, since Mahalanobis distances are calculated based on the covariance matrix, these distances calculated when there are outliers in the data set give misleading results. In this study, it was proposed to make the TOPSIS-M method resistant with the use of the MM covariance matrix, which is resistant to outliers. Robust Mahalanobis distances are used frequently in the literature by using robust covariance matrix. However, to the best of our knowledge, this approach has not been applied to the TOPSIS-M method in studies conducted so far. In this study, it was aimed to rank 81 Turkish provinces by taking into account the variables of per capita GDP, foreign trade deficit (import-export), population, total housing sales, and total bank deposit accounts. The limitation of this study is that the most up-to-date values of statistics collected by provinces are 2019. The fact that the provinces have quite different levels of economic development inevitably made it necessary to consider the effect of outlying observations in the data. For this reason, since the TOPSIS-M method is based on the classical covariance estimator and this estimator is a low breakdown estimator, the covariance matrix was made resistant to outliers using the robust MM estimator. In addition, provinces were classified using the robust clustering method. According to the Robust Cluster Analysis, Istanbul, Ankara, and Izmir, which are obtained as outliers were found to be the top 3 provinces with the Robust TOPSIS-M method. Antalya and Bursa, which are in the first cluster, are ranked as the fourth and fifth provinces in the ranking. Gaziantep, Kocaeli, Konya, Adana, and Denizli, which are in the second cluster, were ranked from 6 to 10 in the Robust TOPSIS-M ranking, again producing consistent results. The last 3 provinces in the ranking for economic development are Ardahan, Tunceli, and Bayburt. The top provinces in the robust TOPSIS-M ranking and observations in clusters number three and four (including outliers) correspond to important industrial and trade centers. Likewise, it is seen that the population density is concentrated in these provinces. For this reason, housing sales are also high in these provinces. When the provinces that are the last in the ranking are examined, it is known that these provinces have some disadvantages such as natural disasters and terrorism due to their geographical location, and therefore economic development is lower. This situation both accelerates migration and prevents investment in these regions. According to our findings, obtained robust clusters and homogenous groups that are based on MM estimator based TOPSIS-M and the actual situation seem compatible. This research presents that robust MM estimator based TOPSIS-M performs correct rankings and partitions homogeneous groups in case of variables with outliers. The ranking of the provinces taking into account the socio-economic Robust Mahalanobis Distance Based TOPSIS to Evaluate the Economic Development of Provinces 115 indicators are included in various studies. However, while ranking in these studies, the dependency between indicators and the potential effects of outliers were not taken into account. The TOPSIS approach based on robust Mahalanobis distance, which is resistant to outliers, was used because the data used in this study consisted of provinces with different development levels and correlated variables. To make Mahalanobis distances resistant to outliers, a robust covariance matrix was used. The covariance matrix employed in this study is based on the MM estimator. The Findings obtained in this study are consistent with the real situation. For this reason, we recommend using robust MM estimator based TOPSIS-M for the evaluation of the economic development of provinces described by variables with outliers. In this study, the importance of the criteria was accepted as equal and the ranking was made accordingly. The importance of criteria can also be determined by subjective methods such as AHP, ANP, DEMATEL or objective weighting methods such as CRITIC and Entropy-based on expert opinion. In addition, the results can be compared by considering the VIKOR, ARAS, COPRAS methods. Another suggestion is that Robust estimators can be used when analyzing data sets containing outliers. Acknowledgement: The authors would like to thank the editor and two anonymous referees who kindly reviewed the earlier version of this manuscript and provided valuable suggestions and comments. Appendix Appendix 1. Initial Decision Matrix opt. direction max max max min max Provinces C1 C2 C3 C4 C5 Adana 55967796.32 28014.82016 1877332.698 -311514.1 2250.26969 Adıyaman 1863197.32 2070.82016 251073.698 -31357.12 -12116.15031 Afyonkarahisar 6888921.32 4691.82016 355526.698 234945.9 1539.61969 Ağrı -2645137.68 -2840.17984 154049.698 -87890.12 -17843.02031 Aksaray 713663.32 1693.82016 41625.698 19646.88 2482.87969 Amasya -295349.68 150.82016 -45891.302 427.8769 1172.38969 Ankara 492294499.3 151783.8202 5281936.698 -3483295 36456.87969 Antalya 132877245.3 58586.82016 2166922.698 780921.9 26061.19969 Ardahan -5284044.68 -5119.17984 -285224.302 -49309.12 74.52969 Artvin -3411236.68 -3236.17984 -211884.302 -15058.12 16262.21969 Aydın 23029583.32 28466.82016 737698.698 511716.9 3318.19969 Balıkesir 25442739.32 26952.82016 858899.698 149926.9 9731.58969 Bartın -3302268.68 -2577.17984 -182406.302 -31205.12 -2380.33031 Batman -675291.68 -32.17984 238892.698 -58419.12 -11171.71031 Bayburt -5952091.68 -4568.17984 -299475.302 -52799.12 -588.14031 Bilecik 24778.32 -1558.17984 -162668.302 2948.877 22498.06969 Bingöl -4503701.68 -2761.17984 -99617.302 -50850.12 -7248.06031 Bitlis -3181196.68 -3062.17984 -30391.302 -49285.12 -12390.60031 Bolu 203256.32 1250.82016 -66583.302 -106321.1 19585.19969 Burdur -950198.68 -1557.17984 -114293.302 144657.9 7718.20969 Bursa 103889557.3 49910.82016 2720447.698 1892625 24386.28969 Çanakkale 5925653.32 7541.82016 160162.698 7834.877 19109.07969 Çankırı -2747841.68 -2633.17984 -188957.302 46700.88 3018.99969 Çorum 6674460.32 3493.82016 148740.698 -1790497 -2984.09031 Yorulmaz et al./Oper. Res. Eng. Sci. Theor. Appl. 4 (2) (2021) 102-123 116 opt. direction max max max min max Provinces C1 C2 C3 C4 C5 Denizli 53935630.32 12770.82016 659529.698 1355305 11958.85969 Diyarbakır 15438764.32 14021.82016 1402045.698 97005.88 -10925.07031 Düzce 1390404.32 2540.82016 14293.698 37837.88 9178.98969 Edirne 3728462.32 2354.82016 26377.698 -93268.12 9517.71969 Elazığ 7909597.32 5613.82016 206574.698 69351.88 -2341.99031 Erzincan -3181739.68 -2104.17984 -146954.302 -33907.12 12717.06969 Erzurum 8618759.32 4625.82016 376893.698 -73632.12 -4335.33031 Eskişehir 18079022.32 16869.82016 507442.698 96372.88 21037.71969 Gaziantep 85062990.32 30046.82016 1719771.698 2837014 3062.58969 Giresun 1361999.32 2313.82016 67335.698 183646.9 -3347.60031 Gümüşhane -4913512.68 -3789.17984 -239683.302 -17616.12 -5486.51031 Hakkari -4193366.68 -5115.17984 -100871.302 -51723.12 -4378.33031 Hatay 35258735.32 19863.82016 1277934.698 -1128064 -3548.01031 Iğdır -4458034.68 -3356.17984 -180071.302 20415.88 -3832.76031 Isparta 1270343.32 1548.82016 58918.698 115079.9 6658.64969 İstanbul 1461674123 259786.8202 15081066.7 -44100660 52227.99969 İzmir 184423821.3 88145.82016 4013308.698 3085818 25983.10969 Kahramanmaraş 22580684.32 10205.82016 786777.698 -173798.1 -464.72031 Karabük 206223.32 -727.17984 -137771.302 -253143.1 4144.99969 Karaman -772106.68 -2061.17984 -126466.302 102636.9 12431.04969 Kars -3063130.68 -2282.17984 -96462.302 -52533.12 -8298.62031 Kastamonu 1818376.32 1564.82016 -5008.302 98800.88 4187.53969 Kayseri 30238897.32 24721.82016 1040069.698 1343356 9640.35969 Kırıkkale -1904612.68 1071.82016 -102682.302 -52676.12 4675.43969 Kırklareli 2668193.32 3197.82016 -19648.302 -25132.12 22464.03969 Kırşehir -311130.68 -763.17984 -138343.302 -100793.1 -798.78031 Kilis -5406827.68 -2276.17984 -238593.302 -35270.12 -5464.94031 Kocaeli 86408382.32 31458.82016 1615872.698 -1863656 46657.65969 Konya 60562975.32 31884.82016 1868634.698 1237525 6322.18969 Kütahya 3786265.32 2929.82016 195302.698 50603.88 7249.71969 Malatya 6182146.32 7375.82016 424770.698 120705.9 -4147.51031 Manisa 31011819.32 20323.82016 1069230.698 -208845.1 14896.21969 Mardin 4259936.32 3300.82016 473330.698 582686.9 -5707.20031 Mersin 37674319.32 38184.82016 1487371.698 317713.9 2502.11969 Muğla 33571393.32 16931.82016 619387.698 276033.9 21892.43969 Muş -3682386.68 -2947.17984 29731.698 -29532.12 -11243.22031 Nevşehir -435112.68 -1773.17984 -76423.302 -24949.12 2160.34969 Niğde -370740.68 1959.82016 -19314.302 -50039.12 1344.10969 Ordu 6967854.32 6100.82016 380014.698 159544.9 -4303.13031 Osmaniye 3450177.32 1594.82016 167170.698 -342117.1 -4603.69031 Rize 4066309.32 -1308.17984 -37026.302 78696.88 6147.14969 Sakarya 16619602.32 17106.82016 661263.698 1655493 15186.54969 Samsun 22123112.32 20644.82016 974693.698 -115775.1 229.80969 Siirt 531142.32 -2521.17984 -50315.302 8529.877 -7978.65031 Sinop -3238326.68 -1697.17984 -164925.302 -30873.12 -3016.23031 Sivas 4712271.32 4699.82016 254503.698 2109.877 418.87969 Şanlıurfa 13184870.32 20959.82016 1733870.698 -130692.1 -17105.90031 Şırnak -2774068.68 -4059.17984 156376.698 528624.9 -7290.49031 Tekirdağ 23358321.32 29306.82016 699679.698 256729.9 36217.16969 Tokat 2076276.32 1249.82016 216475.698 -31300.12 -7668.86031 Trabzon 15582031.32 6753.82016 430515.698 909257.9 2743.69969 Tunceli -5448571.68 -4318.17984 -297942.302 -52940.12 13259.17969 Uşak 1847229.32 680.82016 -11952.302 8509.877 9212.63969 Robust Mahalanobis Distance Based TOPSIS to Evaluate the Economic Development of Provinces 117 opt. direction max max max min max Provinces C1 C2 C3 C4 C5 Van 4106565.32 2366.82016 767956.698 -45450.12 -15861.99031 Yalova 1351893.32 6788.82016 -105335.302 -173924.1 20458.87969 Yozgat 409397.32 136.82016 37709.698 -58432.12 -5858.70031 Zonguldak 4575397.32 1537.82016 209818.698 -818261.1 2122.10969 Appendix 2. Normalize Decision Matrix Provinces C1 C2 C3 C4 C5 Adana 0.036 0.082 0.106 -0.007 0.017 Adıyaman 0.001 0.006 0.014 -0.001 -0.093 Afyonkarahisar 0.004 0.014 0.020 0.005 0.012 Ağrı -0.002 -0.008 0.009 -0.002 -0.137 Aksaray 0.000 0.005 0.002 0.000 0.019 Amasya 0.000 0.000 -0.003 0.000 0.009 Ankara 0.313 0.446 0.298 -0.078 0.281 Antalya 0.084 0.172 0.122 0.017 0.201 Ardahan -0.003 -0.015 -0.016 -0.001 0.001 Artvin -0.002 -0.010 -0.012 0.000 0.125 Aydın 0.015 0.084 0.042 0.011 0.026 Balıkesir 0.016 0.079 0.048 0.003 0.075 Bartın -0.002 -0.008 -0.010 -0.001 -0.018 Batman 0.000 0.000 0.013 -0.001 -0.086 Bayburt -0.004 -0.013 -0.017 -0.001 -0.005 Bilecik 0.000 -0.005 -0.009 0.000 0.173 Bingöl -0.003 -0.008 -0.006 -0.001 -0.056 Bitlis -0.002 -0.009 -0.002 -0.001 -0.095 Bolu 0.000 0.004 -0.004 -0.002 0.151 Burdur -0.001 -0.005 -0.006 0.003 0.059 Bursa 0.066 0.147 0.153 0.042 0.188 Çanakkale 0.004 0.022 0.009 0.000 0.147 Çankırı -0.002 -0.008 -0.011 0.001 0.023 Çorum 0.004 0.010 0.008 -0.040 -0.023 Denizli 0.034 0.037 0.037 0.030 0.092 Diyarbakır 0.010 0.041 0.079 0.002 -0.084 Düzce 0.001 0.007 0.001 0.001 0.071 Edirne 0.002 0.007 0.001 -0.002 0.073 Elazığ 0.005 0.016 0.012 0.002 -0.018 Erzincan -0.002 -0.006 -0.008 -0.001 0.098 Erzurum 0.005 0.014 0.021 -0.002 -0.033 Eskişehir 0.011 0.050 0.029 0.002 0.162 Gaziantep 0.054 0.088 0.097 0.063 0.024 Giresun 0.001 0.007 0.004 0.004 -0.026 Gümüşhane -0.003 -0.011 -0.014 0.000 -0.042 Hakkari -0.003 -0.015 -0.006 -0.001 -0.034 Hatay 0.022 0.058 0.072 -0.025 -0.027 Iğdır -0.003 -0.010 -0.010 0.000 -0.030 Isparta 0.001 0.005 0.003 0.003 0.051 İstanbul 0.929 0.763 0.850 -0.987 0.402 İzmir 0.117 0.259 0.226 0.069 0.200 Kahramanmaraş 0.014 0.030 0.044 -0.004 -0.004 Karabük 0.000 -0.002 -0.008 -0.006 0.032 Karaman 0.000 -0.006 -0.007 0.002 0.096 Yorulmaz et al./Oper. Res. Eng. Sci. Theor. Appl. 4 (2) (2021) 102-123 118 Provinces C1 C2 C3 C4 C5 Kars -0.002 -0.007 -0.005 -0.001 -0.064 Kastamonu 0.001 0.005 0.000 0.002 0.032 Kayseri 0.019 0.073 0.059 0.030 0.074 Kırıkkale -0.001 0.003 -0.006 -0.001 0.036 Kırklareli 0.002 0.009 -0.001 -0.001 0.173 Kırşehir 0.000 -0.002 -0.008 -0.002 -0.006 Kilis -0.003 -0.007 -0.013 -0.001 -0.042 Kocaeli 0.055 0.092 0.091 -0.042 0.359 Konya 0.038 0.094 0.105 0.028 0.049 Kütahya 0.002 0.009 0.011 0.001 0.056 Malatya 0.004 0.022 0.024 0.003 -0.032 Manisa 0.020 0.060 0.060 -0.005 0.115 Mardin 0.003 0.010 0.027 0.013 -0.044 Mersin 0.024 0.112 0.084 0.007 0.019 Muğla 0.021 0.050 0.035 0.006 0.169 Muş -0.002 -0.009 0.002 -0.001 -0.087 Nevşehir 0.000 -0.005 -0.004 -0.001 0.017 Niğde 0.000 0.006 -0.001 -0.001 0.010 Ordu 0.004 0.018 0.021 0.004 -0.033 Osmaniye 0.002 0.005 0.009 -0.008 -0.035 Rize 0.003 -0.004 -0.002 0.002 0.047 Sakarya 0.011 0.050 0.037 0.037 0.117 Samsun 0.014 0.061 0.055 -0.003 0.002 Siirt 0.000 -0.007 -0.003 0.000 -0.061 Sinop -0.002 -0.005 -0.009 -0.001 -0.023 Sivas 0.003 0.014 0.014 0.000 0.003 Şanlıurfa 0.008 0.062 0.098 -0.003 -0.132 Şırnak -0.002 -0.012 0.009 0.012 -0.056 Tekirdağ 0.015 0.086 0.039 0.006 0.279 Tokat 0.001 0.004 0.012 -0.001 -0.059 Trabzon 0.010 0.020 0.024 0.020 0.021 Tunceli -0.003 -0.013 -0.017 -0.001 0.102 Uşak 0.001 0.002 -0.001 0.000 0.071 Van 0.003 0.007 0.043 -0.001 -0.122 Yalova 0.001 0.020 -0.006 -0.004 0.157 Yozgat 0.000 0.000 0.002 -0.001 -0.045 Zonguldak 0.003 0.005 0.012 -0.018 0.016 Appendix 3. Covariance Matrix C1 C2 C3 C4 C5 C1 7.76E+13 71414896520 2.90942E+12 3.62346E+11 30383494618 C2 7.14E+10 71756333 2661769030 374774730 34537249 C3 2.91E+12 2661769030 1.33093E+11 18567394308 -212823424 C4 3.62E+11 374774730 18567394308 32006766617 107912881 C5 3.04E+10 34537249 -212823424 107912881 165872410 Robust Mahalanobis Distance Based TOPSIS to Evaluate the Economic Development of Provinces 119 Appendix 4. Robust TOPSIS-M results and Robust Clusters Provinces S- S* C* Rank Robust Cluster İstanbul 3806970.55 0.00 1.0000 1 0 Ankara 1315344.95 2494550.68 0.3452 2 0 İzmir 540509.74 3274118.34 0.1417 3 0 Antalya 396058.35 3419752.48 0.1038 4 4 Bursa 330753.25 3487381.04 0.0866 5 4 Gaziantep 277507.93 3542972.56 0.0726 6 3 Kocaeli 276162.66 3541735.68 0.0723 7 3 Konya 219392.06 3603602.94 0.0574 8 3 Adana 207115.82 3615707.45 0.0542 9 3 Denizli 194374.17 3630396.97 0.0508 10 3 Mersin 163408.66 3664411.81 0.0427 11 2 Hatay 154532.08 3673023.35 0.0404 12 2 Muğla 147826.70 3681960.66 0.0386 13 2 Kayseri 145282.08 3686437.22 0.0379 14 2 Manisa 145128.76 3684833.95 0.0379 15 2 Balıkesir 132516.62 3700386.28 0.0346 16 2 Tekirdağ 127596.18 3706567.06 0.0333 17 2 Aydın 127282.54 3707360.65 0.0332 18 2 Samsun 126212.93 3708004.87 0.0329 19 2 Kahramanmaraş 125600.94 3708571.90 0.0328 20 2 Diyarbakır 116569.03 3721282.25 0.0304 21 2 Sakarya 116280.49 3723596.74 0.0303 22 2 Eskişehir 115656.03 3721757.37 0.0301 23 2 Şanlıurfa 114373.51 3724234.47 0.0298 24 2 Trabzon 111612.58 3728659.48 0.0291 25 2 Erzurum 98085.95 3747016.37 0.0255 26 1 Elazığ 96291.94 3750128.47 0.0250 27 1 Ordu 96003.68 3751018.17 0.0250 28 1 Afyonkarahisar 95901.64 3751339.97 0.0249 29 1 Malatya 95048.94 3752610.14 0.0247 30 1 Van 93597.31 3755186.68 0.0243 31 1 Mardin 93352.18 3756901.33 0.0242 32 1 Çanakkale 93207.95 3755240.61 0.0242 33 1 Sivas 91889.38 3757735.11 0.0239 34 1 Çorum 91063.00 3754443.44 0.0237 35 1 Kütahya 90466.14 3760470.56 0.0235 36 1 Zonguldak 90079.02 3758778.28 0.0234 37 1 Rize 89697.81 3761760.02 0.0233 38 1 Edirne 89321.95 3762063.33 0.0232 39 1 Osmaniye 89151.99 3761846.64 0.0232 40 1 Tokat 88192.13 3764793.87 0.0229 41 1 Adıyaman 88081.94 3765080.11 0.0229 42 1 Kırklareli 87977.97 3764919.61 0.0228 43 1 Kastamonu 87144.59 3767103.59 0.0226 44 1 Giresun 87058.94 3767686.80 0.0226 45 1 Uşak 87010.94 3767075.77 0.0226 46 1 Isparta 86840.86 3767906.40 0.0225 47 1 Düzce 86662.16 3767969.21 0.0225 48 1 Aksaray 85953.20 3769527.17 0.0223 49 1 Yalova 85801.86 3768973.01 0.0223 50 1 Yorulmaz et al./Oper. Res. Eng. Sci. Theor. Appl. 4 (2) (2021) 102-123 120 Provinces S- S* C* Rank Robust Cluster Yozgat 85421.17 3770463.34 0.0222 51 1 Siirt 85271.04 3770921.94 0.0221 52 1 Batman 85096.64 3771558.44 0.0221 53 1 Bolu 84797.12 3771567.26 0.0220 54 1 Amasya 84458.37 3772823.86 0.0219 55 1 Niğde 84410.23 3772788.28 0.0219 56 1 Bilecik 84407.81 3772772.88 0.0219 57 1 Karabük 84164.54 3772408.90 0.0218 58 1 Nevşehir 84142.10 3773447.97 0.0218 59 1 Kırşehir 83876.31 3773707.63 0.0217 60 1 Karaman 83874.45 3774547.24 0.0217 61 1 Burdur 83803.32 3774923.73 0.0217 62 1 Şırnak 83736.19 3777277.36 0.0217 63 1 Ağrı 82723.75 3777316.70 0.0214 64 1 Kırıkkale 82594.90 3777297.79 0.0214 65 1 Çankırı 81699.87 3780150.88 0.0212 66 1 Bitlis 81664.43 3780126.62 0.0211 67 1 Kars 81542.53 3780329.88 0.0211 68 1 Muş 81497.20 3780856.44 0.0211 69 1 Erzincan 81377.80 3780799.86 0.0211 70 1 Sinop 81221.97 3781266.28 0.0210 71 1 Bartın 81109.97 3781574.80 0.0210 72 1 Artvin 81017.96 3781881.92 0.0210 73 1 Hakkari 80631.68 3783181.03 0.0209 74 1 Bingöl 80405.97 3783954.88 0.0208 75 1 Iğdır 80323.38 3784453.71 0.0208 76 1 Gümüşhane 79753.45 3786130.63 0.0206 77 1 Kilis 79407.02 3787346.82 0.0205 78 1 Ardahan 79335.75 3787401.81 0.0205 79 1 Tunceli 79231.01 3787762.23 0.0205 80 1 Bayburt 78884.23 3789196.40 0.0204 81 1 References Ardielli, E. (2019). Use of TOPSIS method for assessing of good governance in european union countries. Review of Economic Perspectives, 19(3), 211-231. doi:10.2478/revecp-2019-0012. Ardielli, E.,& Halaskova, M. (2015). Evaluation of Good Governance in EU countries. Acta academica karviniensia. 5(3), 5-17. DOI: 10.25142/aak.2015.027 Ascani, A., Crescenzi, R., & Iammarino, S. (2012). Regional economic development. A Review, SEARCH WP01/03, 2-26. Balcerzak, A. P. & Pietrzak, M. P. (2016). Application of TOPSIS Method for Analysis of Sustainable Development in European Union Countries. In: T. Loster & T. Pavelka (Eds.). The 10th International Days of Statistics and Economics. Conference Proceedings. September 8-10, 2016. Prague, 82-92. Becker, C., & Gather, U. (1999). The masking breakdown point of multivariate outlier identification rules. Journal of the American Statistical Association. 94(447), 947– 955. Robust Mahalanobis Distance Based TOPSIS to Evaluate the Economic Development of Provinces 121 Bhutia, P. W., & Phipon, R. (2012). Appication of ahp and topsis method for supplier selection problem. IOSR Journal of Engineering. 2, 43-50. Blair, John P., and Michael C. Carroll. 2008. Local economic development: analysis, practices, and globalization. 2nd ed. Los Angeles: Sage Publications. Carroll, C. D., & Weil, D. N. (1994). Saving and growth: a reinterpretation. In Carnegie- Rochester conference series on public policy, North-Holland. Vol. 40, 133-192. Contractor, F. J., & Mudambi, S. M. (2008). The influence of human capital investment on the exports of services and goods: An analysis of the top 25 services outsourcing countries. Management International Review, 48(4), 433-445. Cooke, S., & Watson, P. (2011). A comparison of regional export enhancement and import substitution economic development strategies. Journal of Regional Analysis & Policy, 41(1), 1-15. De Andrade LH, Antunes JJM, & Wanke P. (2020). Performance of TV programs: a robust MCDM approach. Benchmarking: an International Journal 27(3):1188-1209. Dinçer, H., & Görener, A., (2011). Performans Değerlendirmesinde AHP-Vıkor ve AHP- TOPSIS Yaklaşımları: Hizmet Sektöründe Bir Uygulama. Sigma: Mühendislik ve Fen Bilimleri Dergisi, 29, 244-260. Easterlin, R. A. (1967). Effects of population growth on the economic development of developing countries. The Annals of the American Academy of Political and Social Science, 369(1), 98-108. Furuoka, F. (2009). Population growth and economic development: New empirical evidence from Thailand. Economics Bulletin, 29(1), 1-14. García-Escudero, L.A., Gordaliza,A., Matrán, C. & Mayo-Iscar, A. (2008). A general trimming approach to robust cluster Analysis. Ann. Statist. 36 (3) 1324 -1345. https://doi.org/10.1214/07-AOS515 García-Escudero, L.A., Gordaliza, A., Matrán, C. et al. (2011). Exploring the number of groups in robust model-based clustering. Stat Comput 21, 585–599. https://doi.org/10.1007/s11222-010-9194-z Huber, P. (1981). Robust Statistics. New York: Wiley. Hwang, C.L., Yoon, K., (1981). Multiple Attributes Decision Making Methods and Applications, Springer, Berlin. Kaya, V., Yalçınkaya, Ö., & Hüseyni, İ. (2013). Ekonomik büyümede inşaat sektörünün rolü: Türkiye örneği (1987-2010). Atatürk Üniversitesi İktisadi ve İdari Bilimler Dergisi, 27(4), 148-167. Kentor, J. (2001). The long term effects of globalization on income inequality, population growth, and economic development. Social Problems, 48(4), 435-455. Kizielewicz, B., Więckowski, J., Shekhovtsov, A., Wątróbski, J., Depczyński, R., & Sałabun, W. (2021). Study towards the time-based mcda ranking analysis–a supplier selection case study. Facta Universitatis, Series: Mechanical Engineering. Khalif, K.M.K, Gegov, A., & Abu Bakar A. (2017). Z-TOPSIS approach for performance assessment using fuzzy similarity. 2017 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Naples, 1-6, doi: 10.1109/FUZZ-IEEE.2017.8015458. https://scholar.google.com/scholar_lookup?title=Performance%20of%20TV%20programs:%20a%20robust%20MCDM%20approach&author=De%C2%A0Andrade&publication_year=2020 https://scholar.google.com/scholar_lookup?title=Performance%20of%20TV%20programs:%20a%20robust%20MCDM%20approach&author=De%C2%A0Andrade&publication_year=2020 Yorulmaz et al./Oper. Res. Eng. Sci. Theor. Appl. 4 (2) (2021) 102-123 122 Kilic, H.S, & Yalcin A. (2020). Comparison of municipalities considering environmental sustainability via neutrosophic DEMATEL based TOPSIS. Socio- Economic Planning Sciences,2020, Kuncova, M., (2012). Elektronické obchodování - srovnání zemí EU v letech 2008- 2009 s využitím metod vícekriteriálního hodnocení variant. In: IRCINGOVÁ, J. and J. TLUČHOŘ, Trendy v podnikání 2012. Plzeň: ZČU, 1–9. ISBN 978-80-261-0100-0. Luczak, A., & Malgorzata J. (2020). The positional MEF-TOPSIS method for the assessment of complex economic phenomena in territorial units. Statistics in Transition New Series, Polish Statistical Association, 21(2), 57-172. Mahalanobis P C, 1936. On the generalized distance in statistics. Proceedings of the National Institute of Sciences (Calcutta), 2: 49–55. Mangır, F., & Erdogan, S. (2011).Comparison of Economic Performance Among Six Countries in Global Financial Crisis: The Application of Fuzzy TOPSIS Method. Economics, Management and Financial Markets, 6 (2), 122-136. Maronna R., Martin, R. D.,& Yohai, V. (2006). Robust Statistics: Theory, Computation and Methods. New York: Wiley. Mathur, V. K. (1999). Human capital-based strategy for regional economic development. Economic Development Quarterly, 13(3), 203-216. Noland, M., Park, D., & Estrada, G. B. (2012). Developing the service sector as engine of growth for Asia: an overview. Asian Development Bank Economics Working Paper Series, (320). Özsağır, A., & Akın, A. (2012). Hizmetler Sektörü İçinde Hizmet Ticaretinin Yeri Ve Karşilaştirmali Bir Analizi. Elektronik Sosyal Bilimler Dergisi, 11(41), 311-331. Rocke, D.M., & Woodruff, D.L.(1996). Identification of outliers in multivariate data. Journal of the American Statistical Association. 91(435), 1047–1061. Romm, A. T. (2002). The relationship between savings and growth in South Africa: An empirical study (Doctoral dissertation, University of the Witwatersrand). Rousseeuw P., & Hubert M. (2013). High-Breakdown Estimators of Multivariate Location and Scatter. In: Becker C., Fried R., Kuhnt S. (eds) Robustness and Complex Data Structures. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642- 35494-6_4. Rousseeuw, P.J, & van Zomeren,B. (1991). Robust distances: simulation and cutoff Values. In: Directions in Robust Statistics and Diagnostics, Part II. (W. Stahel, S. Weisberg, eds.), Springer-Verlag, New York. Ruwet, C., García-Escudero, L.A., & Gordaliza, A. (2012). The influence function of the TCLUST robust clustering procedure. Adv Data Anal Classif, 6, 107–130. Shaffer, Ron. 1989. Community economics: economic structure and change in smaller communities. 1st ed. Ames: Iowa State University Press Shih,H.S., Shyur,H.J.,& Lee, E.S. (2007). An extension of TOPSIS for group decision making,Mathematical and Computer Modelling,45, (7–8), 801-813. Srinivasan, T. N. (1988). Population growth and economic development. Journal of Policy Modeling, 10(1), 7-28. Vavrek, R., Becica, J., Papcunova, V., Gundova, P.,& Mitríková, J.(2021). Number of Financial Indicators as a Factor of Multi-Criteria Analysis via the TOPSIS Technique: A Municipal Case Study. Algorithms, 14, 64. https://doi.org/10.3390/a14020064 Robust Mahalanobis Distance Based TOPSIS to Evaluate the Economic Development of Provinces 123 Vavrek,R., Kotulic,R,& Adamisin, P. (2015) Evaluation of municipalities management with the topsis technique emphasising on the impact of weights of established criteria, Lex Localis, 13 (2), 249 Wang, Y.-M., & Elhag, T. M. S. (2006). Fuzzy TOPSIS method based on alpha level sets with an application to bridge risk assessment. Expert Systems with Applications, 31(2), 309–319. doi:10.1016/j.eswa.2005.09.040 Wang, Z.-X., & Wang, Y.-Y. (2014). Evaluation of the provincial competitiveness of the Chinese high-tech industry using an improved TOPSIS method. Expert Systems with Applications, 41(6), 2824–2831. doi:10.1016/j.eswa.2013.10.015 Xiang, S., Nie, F., & Zhang, C. (2008). Learning a Mahalanobis distance metric for data clustering and classification. Pattern Recognition, 41(12), 3600–3612. doi:10.1016/j.patcog.2008.05.018 Yoon, K. P., & Hwang, C.-L. (1995). Quantitative applications in the social sciences, No. 07-104.Multiple attribute decision making: An introduction. Sage university papers series. Sage Publications, Inc. © 2021 by the authors. Submitted for possible open access publication under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/). Robust Mahalanobis Distance based TOPSIS to Evaluate the Economic Development of Provinces Özlem Yorulmaz 1, Sultan Kuzu Yıldırım 2, Bahadır Fatih Yıldırım 3* 1. Introduction 2. Methodology 2.1. Mahalanobis Distance-Based TOPSIS (TOPSIS-M) 2.2. Robust MM Estimator 2.3. Robust Cluster 3. Dataset and Results 4. Conclusion Appendix References