FACTA UNIVERSITATIS Series: Mechanical Engineering Vol. 19, No 3, Special Issue, 2021, pp. 381 - 399 https://doi.org/10.22190/FUME210130048K © 2021 by University of Niš, Serbia | Creative Commons License: CC BY-NC-ND Original scientific paper STUDY TOWARDS THE TIME-BASED MCDA RANKING ANALYSIS – A SUPPLIER SELECTION CASE STUDY Bartłomiej Kizielewicz1, Jakub Więckowski 1, Andrii Shekhovtsov1, Jarosław Wątróbski2, Radosław Depczyński2, Wojciech Sałabun1 1Research Team on Intelligent Decision Support Systems, Department of Artificial Intelligence Methods and Applied Mathematics, Faculty of Computer Science and Information Technology, West Pomeranian University of Technology, Szczecin, Poland 2Institute of Management, University of Szczecin, Szczecin, Poland Abstract. Decision-making processes increasingly use models based on various methods to ensure professional analysis and evaluation of the considered alternatives. However, the abundance of these methods makes it difficult to choose the proper method to solve a given problem. Also, it is worth noting whether different results can be obtained using different methods within a single decision problem. In this paper, we used three selected Multi- Criteria Decision Analysis (MCDA) methods called COMET, TOPSIS, and SPOTIS in order to examine how the obtained rankings vary. The selection of material suppliers was taken into consideration. The equal weights, entropy and standard deviation methods were used to determine the weights for criteria. Final preferences values were then compared with the WS similarity coefficient and weighted Spearman correlation coefficient to check the similarity of the received rankings. It was noticed that in the given problem, all of the methods provide highly correlated results, and the obtained positional rankings are not significantly different. However, practical conclusions indicate the need to look for improved solutions in the correct and accurate assessment of suppliers in a given period. Key Words: MCDA, Supplier Selection, COMET, TOPSIS, SPOTIS 1. INTRODUCTION Various methods support the decision-making process with the indicators that favor one solution over another [1, 2]. It allows the experts to choose the optimal alternatives with a greater precision than relying only on their feelings. However, many different methods are used to create such decision-support models, making it a challenge to identify the method which is the right choice for a given decision problem [3, 4]. Received January 30, 2021 / Accepted May 05, 2021 Corresponding author: Wojciech Sałabun West Pomeranian University of Technology, Szczecin; Żołnierska Street 49, 71-210 Szczecin, Poland E-mail: wojciech.salabun@zut.edu.pl 382 B. KIZIELEWICZ, J. WIĘCKOWSKI, A. SHEKHOVTSOV, ET AL. One of the main approaches to creating support systems is the Multi-Criteria Decision Analysis (MCDA) methods [5]. They allow evaluating a defined set of alternatives based on the selected criteria describing the quality of alternatives in the considered aspects numerically [6]. However, the number of methods belonging to this group is constantly growing, and the results obtained often give different results within one problem [7, 8, 9]. Hence the question arises, how do MCDA rankings vary? The Multi-Criteria Decision Analysis is of great interest to experts in problems where the choice of an optimal solution is influenced by numerous factors that determine the quality of alternatives. These methods are applied in multiple issues, such as evaluating means of transport [10, 11], the selection of industrial suppliers [12, 13], the assessment of medical patients’ health [14, 15], or the management of resources [16]. The possibilities of a broad application of the MCDA methods make them a significant support and are effectively used by a larger group of experts. The calculation of the rankings, in some cases, requires the expert to define the values of the weights for the criteria, thus describing their importance in the evaluation process [17]. It is also possible to determine weights by the methods that do not require expert knowledge, and this is possible by using the methods such as entropy or standard deviation [18, 19]. What is more, in the case of testing the MCDA methods’ performance, it is worth determining the extent to which the obtained rankings are similar to each other [20]. For this purpose, similarity coefficients may be used, which use the preferences of alternatives obtained as a result of the MCDA methods in order to calculate the correlation value. The selection and evaluation of suppliers play a very important role in an enterprise’s functioning [21]. These activities occupy a key role in implementing the organization’s strategic objectives while being an important determinant of competitive advantage [22]. Due to its nature, the problem of correct supplier selection occupies a key role in managing the entire supply chain (Supply Chain Management - SCM) of the enterprise [23]. In the literature, it is emphasized that the correct selection and assessment of the supplier determines the possibility of focusing the entity on key competencies [24] and the appropriate response of the enterprise to the challenges posed by the market [25]. Pro-environmental initiatives and pro-social activities occurring in recent years have been reflected in the current principles of management of economic entities [23, 26]. Taking into account environmental imperatives in business activities caused the problem of selection while evaluation of suppliers is considered not only from the technological and economic perspective but also from pro-environmental factors [27]. This is confirmed by numerous literature studies on green supplier selection [22]. Additionally, including pro-environmental [26] and pro-social [23] imperatives in the business sphere and an attempt to find solutions satisfying various perspectives [22] resulted in a model issue of sustainable supplier selection [27]. It should be pointed out that, by definition, the problem of selection and evaluation of suppliers requires the consideration of different factors as well as different perspectives in one process [21]. Additionally, some of them have a conflicting character, e.g. the quality of services and price. It should be pointed out that these factors may be of both quantitative and qualitative nature, which implies the necessity of applying an appropriate methodological approach [21]. In scientific research, this has shifted the problem of selection and evaluation to the construction of multi-criteria models. The Multiple- Criteria Decision Analysis (MCDA) methodology is widely used here. The literature on the subject shows a wide spectrum of multi-criteria supplier selection and evaluation Study towards the Time-Based MCDA Ranking Analysis – a Supplier Selection Case Study 383 models built using both the methods of the so-called "American school" of multi-criteria decision support [22] and its European counterpart [27]. In the area of the American school, for example, AHP and TOPSIS [28], VIKOR [29], ANP [23] or DEMATEL [24] methods proved their effectiveness in the problems of supplier selection and evaluation. Also, the methods based on the superiority relation (European school) are widely used in this issue and, for example, include methods ELECTRE I [25], ELECTRE II [30], ELECTRE TRI [31] or Promethee II [32]. Several works have also used rule-based and mixed-mode expansions of MCDA methods, and the current state of research in this area is available in Ref. [21]. Also, fuzzy expansions of the well-known MCDA methods have proven their effectiveness in the problem of supplier evaluation and selection [33]. Both simple fuzzy developments of the MCDA methods based on the triangular or trapezoidal form of membership function (fuzzy AHP [34], BWM and TOPSIS [26, 35]), as well as based on subsequent generalizations, e.g. Intuitionist fuzzy sets MCDA methods (TOPSIS [36, 37]) proved to be powerful tools for dealing with the uncertainty of measurements and preferences in supplier selection models. The current state of the art in the use of multi-criteria methods in supplier selection and evaluation is contained, for example, in Refs. [21, 22, 27]. On the other hand, despite the intensive development of research of the MCDA methods development, it should be pointed out that none of them is universal. Moreover - despite the same input data, the results (supplier rankings) obtained by different MCDA methods may differ [38, 39, 40]. Confirmation of this fact can be found in literature, where the problem of objectification in the MCDA methodology [38, 39, 40] and benchmarking of the MCDA methods was analyzed [8, 41, 42, 43, 44]. What is important is that the authors’ conclusions do not contain generalized conclusions. They are fragmentary (limited to a narrow subset of the assessed methods and the specifics of the domain). The authors unambiguously confirm the validity of the recommendations in the paper [3] on the need for a broader analysis and benchmarking of the MCDA methods in particular areas (domains) of application. Aggregation Equal weights Expert knowledge based supplier selection problem stucturing Set of criteria Decision variants Set of reference MCDA methods TOPSIS method The similarity coefficients COMET methodSPOTIS method WS similarity coefficient Set of time based supplier evaluation rankings Rankings similarity checking Weights Determining Methods Analysis and interpretation Entropy method Standard deviation method Performance tables Fig. 1 Research framework As an answer to the shortcomings of the MCDA methods indicated above, in this paper, we used the Characteristic Objects Method (COMET), the Stable Preference Ordering Towards Ideal Solution (SPOTIS) and the Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS) methods to solve the problem of the suppliers’ selection. To provide the criteria weights, the methods of determining the weights were used to omit defining this vector by an expert. Obtained preferences were then used to 384 B. KIZIELEWICZ, J. WIĘCKOWSKI, A. SHEKHOVTSOV, ET AL. compare rankings similarity with two selected coefficients, namely the weighted Spearman correlation coefficient and WS similarity coefficient. The aim was to check the impact of the used method to received rankings and answer how MCDA rankings vary. A detailed framework presenting the study procedure is shown in Fig. 1. The rest of the paper is organized as follows. In Section 2, preliminaries of three MCDA methods are presented, namely COMET, SPOTIS and TOPSIS methods. Section 3 presents the assumptions of determining the weights for criteria in multi-criteria problems. Section 4 presents the correlations coefficients. Section 5 includes the study case, in which the theoretical problem of supplier selection was solved. In Section 6 the conclusions from the research are presented. 2. PRELIMINARIES Multi-Criteria Decision Analysis (MCDA) methods have been developed to evaluate a considered set of alternatives in order to obtain numerical values indicating the quality of individual options [45, 46]. They are often used as support models in the decision-making process made by the decision-maker [47, 48, 49]. The main assumptions of the selected MCDA methods should be presented in order to introduce the principle of their operation. 2.1. The COMET method The Characteristic Objects Method (COMET) performance is based on the definition of the Characteristic Objects, which are used in the process of the pairwise comparison made by expert [50, 51]. The preferences for the set of alternatives are then being calculated based on the obtained rule base. The main advantage is that it is completely free of the rank reversal phenomenon [52], meaning that the change in the number of alternatives will not affect the received ranking. The formal notation of the COMET method should be shortly recalled [53, 54, 55]. Step 1 Define the Space of the Problem – the expert determines the dimensionality of the problem by selecting number r of criteria, C1,C2, ...,Cr. Then, the set of fuzzy numbers for each criterion Ci is selected: 1 2 ,..., }{ , r rn r r rn C C C C= (1) where nr is a number of the fuzzy numbers for criterion r. Step 2 Generate Characteristic Objects – The characteristic objects (CO) are obtained by using the Cartesian Product of fuzzy numbers cores for each criterion as follows: 1 2 ( ) ( ) ( ) r CO C C C C C C=   (2) Step 3 Rank the Characteristic Objects – the expert determines the Matrix of Expert Judgment (MEJ). It is a result of pairwise comparison of the COs by the problem expert. The MEJ matrix contains results of comparing characteristic objects by the expert, where αij is the result of comparing COi and COj by the expert. Function fexp denotes the mental function of the expert. It depends solely on the knowledge of the expert and can be presented as Eq. (3). Afterwards, the vertical vector of the Summed Judgments (SJ) is obtained as given in Eq. (4). Study towards the Time-Based MCDA Ranking Analysis – a Supplier Selection Case Study 385 exp exp exp exp exp exp 0.0, ( ) ( ) 0.5, ( ) ( ) 1.0, ( ) ( ) i j ij i j i j f CO f CO f CO f CO f CO f CO     = =   (3) 1 t i ij j SJ  = =  (4) Finally, the values of preference are approximated for each characteristic object. As a result, vertical vector P is obtained, where i - th row contains the approximate value of preference for COi. Step 4 The Rule Base – each characteristic object and value of preference is converted to a fuzzy rule as follows: ( ) ( )1 2 ii iIF C C AND C C AND THEN P (5) In this way, the complete fuzzy rule base is obtained. Step 5 Inference and Final Ranking – each alternative is presented as a set of crisp numbers (e.g., Ai = {a1i, a2i, ..., ari}). This set corresponds to criteria C1, C2, ..., Cr. Mamdani’s fuzzy inference method is used to compute preference of i - th alternative. The rule base guarantees that the obtained results are unequivocal. 2.2. The SPOTIS method The Stable Preference Ordering Towards Ideal Solution (SPOTIS) method is a recently developed method [56] and its main assumption is to define the data boundaries so as to determine the Ideal Solution Point (ISP). Further calculations to obtain the final preferences for alternatives are being made based on the received ISP. The method is declared to be fully resistant to the rank reversal phenomenon, similarly to the COMET method. For each criterion Cj, the data boundaries should be defined. It is required to select the maximum Sj max and minimum Sj min bound for every Cj. The definition of Ideal Solution Point Sj * depends on the type of criterion, where for profit type it should meet the condition of Sj *=Sj max, and for cost type it should be Sj *=Sj min. The following steps of SPOTIS performance are presented below. Step 1 Calculation of the normalized distances to Ideal Solution Point: ),( ij j ij i j max min j j S S d A S S S   − = − (6) Step 2 Calculation of weighted normalized distances d(Ai, S *) ∈ [0,1], according to: 1 ,( )( , ) N i j ij i j j d A S w d A S   = =  (7) Step 3 Final ranking should be determined based on d(Ai, S *) values. Smaller values d(Ai, S *) which are preferences of alternatives result in a better position in general ranking. 386 B. KIZIELEWICZ, J. WIĘCKOWSKI, A. SHEKHOVTSOV, ET AL. 2.3. The TOPSIS method The Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS) method was developed by Chen and Hwang in 1992 [57, 58]. Authors proposed to examine the set of alternatives based on the calculation of the distance to the ideal solution. During this process, the Positive Ideal Solution (PIS) and Negative Ideal Solution (NIS) are used to calculate alternatives’ final preferences. Moreover, TOPSIS requires defining of the weights vector describing the relevance of each criterion [59]. Proper application of this method should begin with normalizing the decision matrix. Next step is to calculate the weighted normalized decision matrix: , 1, , ; 1, , ij i ij v w r j J i n=  =  =  (8) Positive and negative ideal solutions for a defined decision-making problem should also be identified: 1 1 { , , } {(max | ), (min | )} { , , } {(min | ), (max | )} P C n j ij j ij P C n j ij j ij A v v v i I v i I A v v v i I v i I    − − − =  =   =  =   (9) where IC stands for cost type criteria and IP for profit type. Negative and positive distance from an ideal solution should be calculated using the n-dimensional Euclidean distance. To apply such calculations, formula presented below should be used: 2 1 2 1 ( ) , 1, , ( ) , 1, , n j ij i i n j ij i i D v v j J D v v j J   = − − = = − =  = − =    (10) The last step is to calculate the relative closeness to the ideal solution: , 1, , ( ) j j j j D C j J D D −   − = =  + (11) 3. WEIGHTS DETERMINING METHODS The methods for determining the criteria’ weights can be divided into two groups: subjective and objective. In the first one, the weights are chosen based on the expert’s knowledge and feelings, while in the second one, certain features of the data from the decision matrix are used in determining the weights. The calculation of weights in an objective way for the selected three methods is presented below. 3.1. Equal weights method This method assigns weights equally to all criteria. The number of criteria equal to n will be the denominator in the calculation of the values of weights, where the nominator is 1 by the necessity of meeting the condition of summing up the values of weights to this Study towards the Time-Based MCDA Ranking Analysis – a Supplier Selection Case Study 387 particular number. This method is used in the sensitivity analysis of solutions obtained by different MCDA methods. The formula for calculating equal weights is shown below: 1 j w n = (12) 3.2. Entropy method The entropy method involves measuring the average amount of information using an appropriate normalization given by Eq. (13). Based on this, the entropy value for each criterion is calculated according to Eq. (14). The weights’ final value is calculated using Eq. (15). 1 , 1, , ; 1, , ij ij m ij i x p i m j n x = = =  =   (13) 1 0 ln , 0 , ) 0 ( m i i ij j ij ij j p p p E p =   = =      (14) 1 1 (1 ) j j n i i E w E = − = − (15) 3.3. Standard deviation method The standard deviation method in calculating the criteria weights’ values is based on the determination of the standard deviation, where larger values obtained result in assigning a greater weight to the more diverse criteria (Eq. (16)). The final values of the criteria weights are determined by means of Eq. (17), where normalization is performed using sums of standard deviation values for individual criteria. 2 1 ( ) , 1, , m ij j i j x x j n m  = − = =   (16) 1 / , 1, , n j j j j w j n  = = =  (17) 4. SIMILARITY COEFFICIENTS As the rankings obtained using MCDA methods were compared using the weighted Spearman correlation coefficient and WS similarity coefficient, their explanations are given below. 388 B. KIZIELEWICZ, J. WIĘCKOWSKI, A. SHEKHOVTSOV, ET AL. 4.1. Weighted Spearman’s Rank Correlation Coefficient This coefficient differs from the classic Spearman measure because it considers where the alternatives under study were in the rankings under comparison. However, it is a symmetrical measure and does not consider which of the rankings the reference is. This coefficient can be represented by Eq. (18). 2 1 3 2 1(6 ( ) ( 1) ( ) 1 ( 1) ) n i i i i i w x y N x N y r n n n n =  − − + + − + = −  + − −  (18) 4.2. Rank Similarity Coefficient The WS ranking similarity coefficient is a new coefficient distinguished by its high sensitivity to significant changes in the ranking [60]. This index is asymmetric and strongly dependent on the difference between the two considered rankings at specific positions. It can be represented by Eq. (19). 1 | | 1 2 max{| 1 |,| |} i n x i i i i i x y WS x x N − =  − = −   − −   (19) 5. CASE STUDY The conducted research concerned the evaluation of the quality of suppliers of materials for the manufacturing of metallurgical products. The company produces steel structures using purchased materials, so the price, quality, and security of supply aspects are important elements. Using three selected MCDA methods, it was decided to solve material suppliers’ problem and evaluate the alternatives considered, the criteria presented in Table 1 were taken into account. Their relevance determined the selection of criteria in evaluating suppliers and the extent to which they affect their attractiveness. The selection of such a set of criteria resulted from the analysis of reference literature - bibliographic studies [61, 62, 63, 64, 65]. On its basis, a set of 53 potential supplier evaluation criteria was identified. Subsequently, the company's panel of experts identified criteria relevant from the point of view of the business entity under study. Table 1 Criteria Ci taken into consideration in solving multi-criteria problem Ci Name C1 Price C2 Materials quality C3 Deliveries timeliness C4 Discounts C5 Payment condition C6 Cooperation assessment C7 Supplier communication C8 Complaint handling C9 Delivery terms Study towards the Time-Based MCDA Ranking Analysis – a Supplier Selection Case Study 389 The COMET, SPOTIS and TOPSIS methods were used to evaluate the suppliers' quality and materials. This analysis was carried out using data collected over the last four years, taking monthly deliveries. Also, in evaluating a set of alternatives, it was decided to use the methods of determining the weights for the criteria to check whether the method used would affect the results obtained in the given problem. The obtained preference results were then subjected to a comparative analysis, which consisted of checking the extent to which the obtained rankings are correlated. For this purpose, the weighted Spearman correlation coefficient and WS similarity coefficient were used. The rankings' correlation values are shown in Fig. 2 and A1 for the first and second selected coefficients. In individual Fig. 2 and A1, the horizontal axis represents the consecutive month number, and the vertical axis represents the value of the obtained correlation coefficient. It should also be noted that the correlation studied concerns rankings obtained by the selected MCDA method based on data for two consecutive months. In practice, the resulting broken line represents an analysis of the variability of the rankings of the assessed suppliers. Besides, each of Figs. (2, 3, 4, and 5, 6, 7) represents results obtained based on different techniques for determining the objectified weights in the model, i.e., entropy, equal and standard. Fig. 2 Month to month MCDA based supplier rankings comparison with weighted Spearman correlation coefficient for entropy type weights Fig. 3 Month to month MCDA based supplier rankings comparison with weighted Spearman correlation coefficient for equal type weights 390 B. KIZIELEWICZ, J. WIĘCKOWSKI, A. SHEKHOVTSOV, ET AL. Fig. 4 Month to month MCDA based supplier rankings comparison with weighted Spearman correlation coefficient for standard deviation type weights Fig. 5 Month to month MCDA based supplier rankings comparison with weighted Spearman correlation coefficient for standard deviation type weights Fig. 6 Month to month MCDA based supplier rankings comparison with WS similarity coefficient for equal type weights Study towards the Time-Based MCDA Ranking Analysis – a Supplier Selection Case Study 391 Fig. 7 Month to month MCDA based supplier rankings comparison with WS similarity coefficient for standard deviation type weights A relatively high correlation of rankings is shown in both cases, both for all used methods of determining the relative importance of attributes (Entropy, Equal and Standard deviation methods) and MCDA methods (COMET, SPOTIS and TOPSIS). As a result of the analysis of Figs. 2, 3 and 4, it is easy to show that the highest variability of rankings was observed for the period from month 11 to 14 and for the pairs of months 1 and 2, 23 and 24, 37 and 38 as well as 45 and 46. As a result of the analysis of Figs. 5, 6, and 7, it is easy to show that higher sensitivity of WS coefficient concerning Weighted Spearman's Rank Correlation Coefficient results in the fact that in addition to the previously indicated significant variability of rankings, there was an additional period of significant variability of rankings covering months 18 to 20. In investigating and analyzing the results, periods were sought where the greatest differences in the rankings obtained were recorded, and consequently, where the correlation of the rankings was lowest. Tables 2, 3 and 4 present, in turn, the results obtained for the COMET, SPOTIS and TOPSIS methods taking into account the lowest similarity suppliers final rankings. For each of the weighing criteria methods, two top lowest correlated rankings are included, along with data describing the year and month in which the suppliers were evaluated. As can be seen, these Tables do not differ since, as rw, the WS value was the lowest for these rankings. Table 2 Lowest correlated suppliers final rankings (COMET method compared with WS similarity coefficient) Alternative Entropy Equal Std A1 7.0 3.0 2.0 3.0 7.0 3.0 A2 3.0 1.0 3.0 3.0 3.0 1.0 A3 7.0 5.5 5.5 3.0 7.0 5.5 A4 4.0 2.0 5.5 5.0 4.0 2.0 A5 7.0 5.5 1.0 10.0 7.0 5.5 A6 1.0 8.0 8.0 6.0 1.0 8.0 A7 7.0 5.5 11.0 7.5 7.0 5.5 A8 7.0 5.5 7.0 7.5 7.0 5.5 A9 2.0 9.0 4.0 1.0 2.0 9.0 A10 10.0 11.0 10.0 9.0 10.0 11.0 A11 11.0 10.0 9.0 11.0 11.0 10.0 Date 2017-11 2017-12 2018-12 2019-01 2017-11 2017-12 392 B. KIZIELEWICZ, J. WIĘCKOWSKI, A. SHEKHOVTSOV, ET AL. Table 3 Lowest correlated suppliers final rankings (SPOTIS method compared with WS similarity coefficient) Alternative Entropy Equal Std A1 8.0 5.0 8.0 9.0 9.0 4.0 A2 4.0 3.0 8.0 7.0 7.0 5.0 A3 10.0 6.0 2.5 7.0 10.0 8.0 A4 9.0 9.0 8.0 7.0 8.0 9.0 A5 1.0 10.0 10.0 4.0 1.0 10.0 A6 5.0 8.0 5.0 1.5 3.0 7.0 A7 6.5 7.0 5.0 4.0 5.5 6.0 A8 6.5 4.0 5.0 4.0 5.5 3.0 A9 11.0 11.0 11.0 11.0 11.0 11.0 A10 3.0 2.0 2.5 1.5 2.0 2.0 A11 2.0 1.0 1.0 10.0 4.0 1.0 Date 2018-02 2018-03 2018-01 2018-02 2018-02 2018-03 Table 4 Lowest correlated suppliers final rankings (TOPSIS method compared with WS similarity coefficient) Alternative Entropy Equal Std A1 7.0 3.0 2.0 3.0 9.0 3.0 A2 2.0 1.0 4.0 3.0 6.0 1.0 A3 7.0 5.5 8.0 3.0 9.0 5.5 A4 4.0 2.0 6.0 5.5 3.0 2.0 A5 7.0 5.5 1.0 5.5 9.0 5.5 A6 1.0 8.0 10.0 11.0 2.0 8.0 A7 7.0 5.5 10.0 8.5 9.0 5.5 A8 7.0 5.5 10.0 8.5 9.0 5.5 A9 3.0 9.0 3.0 1.0 1.0 9.0 A10 10.0 11.0 6.0 10.0 5.0 11.0 A11 11.0 10.0 6.0 7.0 4.0 10.0 Date 2017-11 2017-12 2018-12 2019-01 2017-11 2017-12 Fig. 8 gives a graphical representation of the correlation between the suppliers month to month rankings obtained for the full 48 month period. The Figures below also show detailed correlations for all three MCDA methods (TOPSIS, SPOTIS, COMET) using WS and rw coefficients. Fig. 8 shows the case where equal attribute weights were used in the MCDA models. These Figures illustrate the density distribution obtained using a month-to-month comparison of the individual similarity coefficients between the different MCDA and weighting methods. The most similar results were obtained with the COMET and TOPSIS methods. The Appendix sets of Figs. 9 and 10 were developed for the MCDA models using standard deviation and entropy-based weighting methods. Study towards the Time-Based MCDA Ranking Analysis – a Supplier Selection Case Study 393 Fig. 8 Month to month MCDA based supplier rankings correlation analysis – equal weights case, 48 month period 394 B. KIZIELEWICZ, J. WIĘCKOWSKI, A. SHEKHOVTSOV, ET AL. Fig. 9 Month to month MCDA based supplier rankings correlation analysis – standard deviation based weights case, 48 month period Study towards the Time-Based MCDA Ranking Analysis – a Supplier Selection Case Study 395 Fig. 10 Month to month MCDA based supplier rankings correlation analysis – entropy based weights case, 48 month period The conducted research clearly shows significant variability of the month-to-month- suppliers of rankings. Objectification of further research leads to the analysis of differences in rankings resulting from different MCDA methods (TOPSIS, SPOTIS, COMET). One form of such analysis and presentation of detailed quantitative differences between MCDA methods in suppliers' rankings is a set of Fig. 11. In a nutshell, it can be stated that these are so-called "error charts" showing each time the lack of convergence of the place in the ranking for 2 analyzed MCDA methods. A dot plot is expected in perfect correlation, where all dots are located on the main diagonal. It turns out again that the best results were obtained for the COMET and TOPSIS methods, whose results are characterized by the most significant similarity. 396 B. KIZIELEWICZ, J. WIĘCKOWSKI, A. SHEKHOVTSOV, ET AL. Fig. 11 Month to month MCDA based supplier rankings misfits analysis – entropy based weights case, 48 month period 6. CONCLUSIONS The problem of selecting the best alternative in the considered set, in which many criteria influence the quality assessment, is complex. Many support systems are developed to improve the decision-making process, based on different approaches, using MCDA methods. However, their number and use of different approaches in solving the problem by these methods often lead to different results. Nevertheless, it is worth identifying how obtaining the rankings of MCDA methods vary from one another. Three methods were selected to solve the problem of evaluating material suppliers in the metallurgical industry: COMET, SPOTIS and TOPSIS [2, 5, 66, 67]. It was also decided to use criterion weighting methods to check their performance on the results obtained. The rankings were compared using the WS similarity coefficient and the weighted Spearman correlation coefficient to determine the resulting correlations. Despite the use of different methods to determine the relevance of the weights and to determine the alternatives’ final quality preferences, similar results were observed, which shows that in the given problem of evaluating material suppliers, the method used did not significantly affect the rankings obtained. This study shows that the significant variability in time-based supplier rankings warrants further research in seeking detailed, reliable mapping models proper reflecting collected data over time. Forgetting or recalling functions adopted MCDA methodology seems to be promising research fields. Acknowledgements: The work was supported by the National Science Centre, Decision number UMO-2018/29/B/HS4/02725 (K.B., S.A. and S.W.) and by the project financed within the framework of the program of the Minister of Science and Higher Education under the name “Regional Excellence Initiative” in the years 2019–2022, Project Number 001/RID/2018/19; the amount of financing: PLN 10.684.000,00 (D.R. and W. J.). Study towards the Time-Based MCDA Ranking Analysis – a Supplier Selection Case Study 397 REFERENCES 1. De Montis, A., De Toro, P., Droste-Franke, B., Omann, I., Stagl, S., 2000, Criteria for quality assessment of mcda methods, In 3rd Biennial Conference of the European Society for Ecological Economics, Vienna, pp. 3-6. 2. Kizielewicz, B., Sałabun, W., 2020, A new approach to identifying a multi-criteria decision model based on stochastic optimization techniques, Symmetry, 12(9), 1551. 3. Guitouni, A., Martel, J.-M., 1998, Tentative guidelines to help choosing an appropriate mcda method, European journal of operational research, 109(2), pp. 501-521. 4. Jacquet-Lagreze, E., Siskos, Y., 2001, Preference disaggregation: 20 years of mcda experience, European Journal of Operational Research, 130(2), pp. 233-245. 5. Badi I., Pamucar D., 2020, Supplier selection for steelmaking company by using combined Grey- MARCOS methods, Decision Making: Applications in Management and Engineering, 3(2), pp. 37-48. 6. Nutt, D. J., Phillips, L. D., Balfour, D., Curran, H. V., Dockrell, M., Foulds, J., Fagerstrom, K., Letlape, K., Milton, A., Polosa, R., Ramsey J., Sweanor D., 2014, Estimating the harms of nicotine-containing products using the mcda approach, European addiction research, 20(5), pp. 218-225. 7. Kizielewicz, B., Wątróbski, J., Sałabun, W., 2020, Identification of relevant criteria set in the mcda process—wind farm location case study, Energies, 13(24), 6548. 8. Sałabun, W., Wątróbski, J., Shekhovtsov, A., 2020a, Are mcda methods benchmarkable? a comparative study of topsis, vikor, copras, and promethee ii methods, Symmetry, 12(9), 1549. 9. Shekhovtsov, A., Kołodziejczyk, J, 2020, Do distance-based multi-criteria decision analysis methods create similar rankings?, Procedia Computer Science, 176, pp. 3718-3729. 10. Ambrasaite, I., Barfod, M. B., Salling, K. B., 2011, Mcda and risk analysis in transport infrastructure appraisals: The rail baltica case, Procedia-Social and Behavioral Sciences, 20, pp. 944-953. 11. Nalmpantis, D., Roukouni, A., Genitsaris, E., Stamelou, A., Naniopoulos, A., 2019, Evaluation of innovative ideas for public transport proposed by citizens using multi-criteria decision analysis (mcda), European Transport Research Review, 11(1), pp. 1-16. 12. Karczmarczyk, A., Wątróbski, J., Ladorucki, G., Jankowski, J., 2018, Mcda-based approach to sustainable supplier selection, In 2018 Federated Conference on Computer Science and Information Systems (FedCSIS), pp. 769-778. 13. Stević, Ž., Pamučar, D., Puška, A., Chatterjee, P., 2020, Sustainable supplier selection in healthcare industries using a new MCDM method: Measurement of alternatives and ranking according to COmpromise solution (MARCOS), Computers & Industrial Engineering, 140, 106231. 14. Angelis, A., Kanavos, P., 2017, Multiple criteria decision analysis (mcda) for evaluating new medicines in health technology assessment and beyond: the advance value framework, Social Science & Medicine, 188, pp. 137-156. 15. Goetghebeur, M. M., Wagner, M., Khoury, H., Levitt, R. J., Erickson, L. J., Rindress, D., 2012, Bridging health technology assessment (hta) and efficient health care decision making with multicriteria decision analysis (mcda) applying the evidem framework to medicines appraisal, Medical decision making, 32(2), pp. 376-388. 16. Ensslin, L., Dutra, A., Ensslin, S. R., 2000, Mcda: a constructivist approach to the management of human resources at a governmental agency, International transactions in operational Research, 7(1), pp. 79-100. 17. Triantaphyllou, E., Baig, K., 2005, The impact of aggregating benefit and cost criteria in four mcda methods, IEEE Transactions on Engineering Management, 52(2), pp. 213-226. 18. Huang, J, 2008, Combining entropy weight and topsis method for information system selection, In 2008 ieee conference on cybernetics and intelligent systems, pp.1281-1284. 19. Wang, Y.-M., Luo, Y., 2010, Integration of correlations with standard deviations for determining attribute weights in multiple attribute decision making, Mathematical and Computer Modelling, 51(1-2), pp. 1-12. 20. Asuero, A. G., Sayago, A., Gonzalez, A., 2006, The correlation coefficient: An overview. Critical reviews in analytical chemistry, 36(1), pp. 41-59. 21. Chai, J., Liu, J. N., Ngai, E. W., 2013, Application of decision-making techniques in supplier selection: A systematic review of literature, Expert systems with applications, 40(10), pp. 3872-3885. 22. Biswas, T. K., Das, M. C., 2020, Selection of the barriers of supply chain management in Indian manufacturing sectors due to COVID-19 impacts, Operational Research in Engineering Sciences: Theory and Applications, 3(3), pp. 1-12. 398 B. KIZIELEWICZ, J. WIĘCKOWSKI, A. SHEKHOVTSOV, ET AL. 23. Tavana, M., Yazdani, M., Di Caprio, D., 2017, An application of an integrated anp--qfd framework for sustainable supplier selection, International Journal of Logistics Research and Applications, 20(3), pp. 254-275. 24. Liu, T., Deng, Y., Chan, F., 2018, Evidential supplier selection based on dematel and game theory, International Journal of Fuzzy Systems, 20(4), pp. 1321-1333. 25. Zhong, L., Yao, L., 2017, An electre i-based multi-criteria group decision making method with interval type-2 fuzzy numbers and its application to supplier selection, Applied Soft Computing, 57, pp. 556-576. 26. Javad, M. O. M., Darvishi, M., Javad, A. O. M., 2020, Green supplier selection for the steel industry using bwm and fuzzy topsis: a case study of khouzestan steel company, Sustainable Futures, 2, 100012. 27. Schramm, V. B., Cabral, L. P. B., Schramm, F., 2020, Approaches for supporting sustainable supplier selection-a literature review, Journal of Cleaner Production, 123089. 28. Jain, V., Sangaiah, A. K., Sakhuja, S., Thoduka, N., Aggarwal, R., 2018, Supplier selection using fuzzy ahp and topsis: a case study in the indian automotive industry, Neural Computing and Applications, 29(7), pp. 555-564. 29. Awasthi, A., Govindan, K., Gold, S., 2018, Multi-tier sustainable global supplier selection using a fuzzy ahp-vikor based approach, International Journal of Production Economics, 195, pp.106-117. 30. Wan, S.-p., Xu, G.-l., Dong, J.-y., 2017, Supplier selection using anp and electre ii in interval 2-tuple linguistic environment, Information Sciences, 385, pp. 19-38. 31. Costa, A. S., Govindan, K., Figueira, J. R., 2018, Supplier classification in emerging economies using the electre tri-nc method: A case study considering sustainability aspects, Journal of Cleaner Production, 201, pp. 925-947. 32. Abdullah, L., Chan, W., Afshari, A., 2019, Application of promethee method for green supplier selection: a comparative result based on preference functions, Journal of Industrial Engineering International, 15(2), pp. 271-285. 33. Chai, J., Ngai, E. W., 2020, Decision-making techniques in supplier selection: Recent accomplishments and what lies ahead, Expert Systems with Applications, 140, 112903. 34. Ecer, F., 2020, Multi-criteria decision making for green supplier selection using interval type-2 fuzzy ahp: a case study of a home appliance manufacturer, Operational Research, pp. 1-35. 35. Pamucar, D., Yazdani, M., Montero-Simo, M. J., Araque-Padilla, R. A., Mohammed, A., 2021, Multi- criteria decision analysis towards robust service quality measurement, Expert Systems with Applications, 170, 114508. 36. Pamučar, D., Behzad, M., Božanić, D., Behzad, M., 2020, Decision making to support sustainable energy policies corresponding to agriculture sector: Case study in iran’s caspian sea coastline, Journal of Cleaner Production, 125302. 37. Rouyendegh, B. D., Yildizbasi, A., Üstünyer, P., 2020, Intuitionistic fuzzy topsis method for green supplier selection problem, Soft Computing, 24(3), pp. 2215-2228. 38. Cinelli, M., Kadziński, M., Gonzalez, M., Slowiński, R., 2020, How to support the application of multiple criteria decision analysis? let us start with a comprehensive taxonomy, Omega, 102261. 39. Wątróbski, J., Jankowski, J., Ziemba, P., Karczmarczyk, A., Ziolo, M., 2019a, Generalised framework for multi-criteria method selection, Omega, 86, pp. 107-124. 40. Wątróbski, J., Jankowski, J., Ziemba, P., Karczmarczyk, A., Ziolo, M. 2019b, Generalised framework for multi-criteria method selection: Rule set database and exemplary decision support system implementation blueprints, Data in brief, 22, 639. 41. Ceballos, B., Lamata, M. T., Pelta, D. A., 2016, A comparative analysis of multi-criteria decision-making methods, Progress in Artificial Intelligence, 5(4), pp. 315-322. 42. Chang, Y.-H., Yeh, C.-H., Chang, Y.-W., 2013, A new method selection approach for fuzzy group multicriteria decision making, Applied Soft Computing, 13(4), pp. 2179-2187. 43. Kolios, A., Mytilinou, V., Lozano-Minguez, E., Salonitis, K., 2016, A comparative study of multiple- criteria decision-making methods under stochastic inputs, Energies, 9(7), 566. 44. Özcan, T., Çelebi, N., Esnaf, S., 2011, Comparative analysis of multi-criteria decision making methodologies and implementation of a warehouse location selection problem, Expert Systems with Applications, 38(8), pp. 9773-9779. 45. Matsatsinis, N. F., Samaras, A. P., 2001, Mcda and preference disaggregation in group decision support systems, European Journal of Operational Research, 130(2), pp. 414-429. 46. Oliveira, G. D., Dias, L. C., 2020, The potential learning effect of a mcda approach on consumer preferences for alternative fuel vehicles, Annals of Operations Research, 293(2), pp. 767-787. 47. Chakraborty S, Chattopadhyay R, Chakraborty S., 2020, An integrated D-MARCOS method for supplier selection in an iron and steel industry, Decision Making: Applications in Management and Engineering, 3(2), pp. 49-69. Study towards the Time-Based MCDA Ranking Analysis – a Supplier Selection Case Study 399 48. Mousseau, V., Figueira, J., Dias, L., da Silva, C. G., Clímaco, J., 2003, Resolving inconsistencies among constraints on the parameters of an mcda model, European Journal of Operational Research, 147(1), pp. 72-93. 49. Pietersen, K., 2006, Multiple criteria decision analysis (mcda): A tool to support sustainable management of groundwater resources in south Africa, Water SA, 32(2), pp. 119-128. 50. Sałabun, W., Karczmarczyk, A., 2018, Using the comet method in the sustainable city transport problem: an empirical study of the electric powered cars, Procedia computer science, 126, pp. 2248-2260. 51. Wątróbski, J., Sałabun, W., Karczmarczyk, A., Wolski, W., 2017, Sustainable decision-making using the comet method: An empirical study of the ammonium nitrate transport management, In 2017 Federated Conference on Computer Science and Information Systems (FedCSIS), pp. 949-958. 52. Sałabun, W., Ziemba, P., Wątróbski, J., 2016, The rank reversals paradox in management decisions: The comparison of the ahp and comet methods, In International Conference on Intelligent Decision Technologies, pp. 181-191. 53. Sałabun, W., 2014, The Characteristic Objects Method: a new approach to identify a multi-criteria group decision-making model, Intl J Comput Tech Appl, 5(5), pp. 1597-1602 54. Sałabun, W., 2015, The Characteristic Objects Method: A New Distance‐based Approach to Multicriteria Decision‐making Problems, Journal of Multi‐Criteria Decision Analysis, 22(1-2), pp. 37-50. 55. Watróbski, J., Sałabun, W., 2016, The characteristic objects method: A new intelligent decision support tool for sustainable manufacturing, In International Conference on Sustainable Design and Manufacturing, pp. 349-359. 56. Dezert, J., Tchamova, A., Han, D., Tacnet, J.-M., 2020, The spotis rank reversal free method for multi- criteria decision-making support, In 2020 IEEE 23rd International Conference on Information Fusion (FUSION), pp. 1-8. 57. Behzadian, M., Otaghsara, S. K., Yazdani, M., Ignatius, J., 2012, A state-of the-art survey of topsis applications, Expert Systems with applications, 39(17), pp. 13051-13069. 58. Lai, Y.-J., Liu, T.-Y., Hwang, C.-L., 1994, Topsis for modm, European journal of operational research, 76(3), pp. 486-500. 59. Opricovic, S., Tzeng, G.-H., 2004, Compromise solution by mcdm methods: A comparative analysis of vikor and topsis, European journal of operational research, 156(2), pp. 445-455. 60. Sałabun, W., Urbaniak, K., 2020, A new coefficient of rankings similarity in decision-making problems, In International Conference on Computational Science, pp. 632-645. 61. da Silva, E. M., Ramos, M. O., Alexander, A., Jabbour, C. J. C., 2020, A systematic review of empirical and normative decision analysis of sustainability-related supplier risk management, Journal of Cleaner Production, 244, 118808. 62. Jabbour, C. J. C., de Sousa Jabbour, A. B. L., Sarkis, J., 2019, Unlocking effective multi-tier supply chain management for sustainability through quantitative modeling: Lessons learned and discoveries to be made, International Journal of Production Economics, 217, pp. 11-30. 63. Wetzstein, A., Feisel, E., Hartmann, E., Benton Jr, W. C., 2019, Uncovering the supplier selection knowledge structure: a systematic citation network analysis from 1991 to 2017. Journal of Purchasing and Supply Management, 25(4), 100519. 64. Hoque, I., Rana, M. B., 2020, Buyer–supplier relationships from the perspective of working environment and organisational performance: review and research agenda. Management Review Quarterly, 70(1), pp. 1-50. 65. Chatterjee, P., Stević, Ž, 2019, A two-phase fuzzy AHP-fuzzy TOPSIS model for supplier evaluation in manufacturing environment, Operational Research in Engineering Sciences: Theory and Applications, 2(1), pp. 72-90. 66. Kizielewicz, B., Kołodziejczyk, J., 2020, Effects of the selection of characteristic values on the accuracy of results in the comet method, Procedia Computer Science, 176, pp. 3581-3590. 67. Paradowski, B., Więckowski, J., Dobryakova, L., 2020, Why topsis does not always give correct results?, Procedia Computer Science, 176, pp. 3591-3600.