Plane Thermoelastic Waves in Infinite Half-Space Caused


Operational Research in Engineering Sciences: Theory and Applications 
Vol. 2, Issue 3, 2019, pp. 77-91 
ISSN: 2620-1607 
eISSN: 2620-1747 

 DOI: https:// 10.31181/oresta1903077l 

* Corresponding author. 
ilazarevic@grf.bg.ac.rs 

USING THE ELECTRE MLO MULTI-CRITERIA 
DECISION-MAKING METHOD IN STEPWISE 

BENCHMARKING – APPLICATION IN HIGHER EDUCATION  

Ivan Lazarević  

Faculty of Civil Engineering, University of Belgrade, Serbia  
 

Received: 08 October 2019  
Accepted: 01 December 2019  
First online: 16 December 2019 

 
Research paper 

Abstract. The purpose of this paper reflects in a study of an optimal development path 
in the ELECTRE-based stepwise benchmarking context. In the paper, multi-criteria 
decision-making is first described as a tool for stepwise benchmarking, where the 
ELECTRE MLO ranking method is used. In order to make the problem of finding the 
optimal path easier and significantly reduce the number of the paths that have to be 
considered, we are proving the theorem showing that it is better to make gradual 
progress than “skip steps”. As an illustration of these considerations, the ELECTRE MLO 
method is applied to the benchmark teaching assistants of one faculty of Belgrade 
University, according to the marks given by their students. We are looking for an 
optimal development path by using our theorem that substantially reduces the number 
of cases. We are also checking that the paths with no steps skipped are superior to the 
paths in which steps are skipped, in accordance with the theoretical result we have 
obtained. Thus, we are demonstrating that one should first look up to the colleague 
who is a little better than him/her, and then gradually improve until he/she has 
reached the level of the individual given the best mark. 

Key words: multi-criteria decision-making, ELECTRE, benchmarking, evolution path, 
higher education 

1. Introduction 

Benchmarking is a management tool representing a systematic process of 
measuring the quality of products or services against the best representative ones in 
the field of interest. This process includes comparison with the direct competitor and 
comparison against the given benchmark, or standard one strives to achieve. In this 
paper, an example of the teaching assistants of one faculty of Belgrade University, in 
which the teaching assistants are compared with one another according to the marks 
they have received from students, is used as an illustration. The marks are based on a 
total of ten criteria. 



Lazarević/Oper. Res. Eng. Sci. Theor. Appl. 2 (3) (2019) 77-91 

 

78 
 

Benchmarking is mostly used for the purpose of comparing the state policies at 
the international level. Benchmarks are always provided by the most developed 
countries. There are a lot of studies on this topic; see (Arrowsmith et al.,2004; 
Petrović et al., 2012; P. Hong et al., 2012; Petrović et al., 2014; Brehmer et al., 2019; 
M. Petrović et al., Omega, 2018; Petrović et al., Journal of Sustainable Business and 
Management Solutions in Emerging Econo, 2018). The socioeconomic, geostrategic 
and cultural influences of one country are often neglected during a mutual 
comparison, so the question is whether the measures transferred from other 
countries are always applicable; see (Dolowitz and Marsh, 1996; Bauer, 2010; 
Lundvall and Tomlison, 2012). In spite of the differences, it is clear that country 
leaders, especially the leaders of those in the same region, in the European Union, or 
those tending to enter the European Union, can follow one another (Rose 1991). 
International benchmarking is broadly applied even in information and 
communication technologies.  

The benchmarking process includes making different decisions, ranging from the 
manner of choosing the most relevant statistical data, all the way to the role model 
which is considered as the best to improve certain characteristics. The main 
question is as follows: Who or what should we look up to in order to become better? 
To learn from the best in a certain field is not always the best of options. One should 
also be realistic when assessing abilities. The main purpose of this paper is to more 
closely examine this topic and particularly answer the question of whether it is 
better to make gradual progress or “skip steps”. The answer is provided as the 
central theorem of this paper. There are many studies on striving towards slightly 
better, gradual progress; see (Moore, 1999; Hambelton and Gross, 2008; Lim et al., 
2011). We look for someone or something who/which is a bit better, i.e. for an 
appropriate benchmark in each step of such progress, thus coming to the so-called 
evolution of progress. At this point, the most important thing is to choose the best 
evolution path. In Chapter 3 herein, an example of the teaching assistant who 
obtained the worst marks is presented. He should first look up to the colleague who 
is a little better than him, after which he should gradually improve until he has 
reached the level of the teaching assistant who has received the best marks. If 
uniform progress is made, then the ideal evolution path is obtained, which is difficult 
to achieve in practice because a non-uniform benchmark distribution is typical for 
situations in which we deal with realistic data. The DEA (Data Envelope Analysis) 
method is one of the popular operational research methods often used in 
benchmarking; see (Ramon et al., 2018; Ji et al., 2019; De Blas et al., 2018; Gidion et 
al., 2019). It is based on linear programming and was created in the paper (Charnes 
et al., 1981). In this paper, a modification of the ELECTRE I method developed in 
order to serve as a benchmarking tool is used. This is the ELECTRE MLO method that 
first appeared in the study (Petrović et al., 2012). The ELECTRE I method was 
introduced by Roy B., in the paper (Roy 1968). The method is now only of a historical 
interest as the method representing the base on which other, more useful methods 
have been created. The most popular and the most frequently used modifications of 
ELECTRE I are ELECTRE IV (Figueira et al., 2005) and ELECTRE Is (Roy and Skalka, 
1987). The family of the ELECTRE methods solve the following three very important 
problems, namely: making a choice (Hassan. et al., 2018; Wang Y and Xeo., 2018; 
Tavassoli et al., 2018), ranking (Dias et al., 2018; Harsoyo and Jati, 2018) and sorting 
(Pereira et al, 2019; Pereira and Ishizaka, 2019; Ishizaka et al., 2019; Singh, 2019). 
The methods which solve the alternatives ranking problem are especially important 



Using the ELECTRE MLO Multi-Criteria Decision-Making Method in Stepwise Benchmarking 
– Application in Higher Education 

 

79 
 

for benchmarking. The ELECTRE III method deals with these issues; see (Bouysson 
and Roy, 1986; Papadopulos and Karagiannidis, 2008; Ishizaka and Giannoulis, 
2010; Hashemi et al., 2016; La Fata et al., 2019). Over time, modifications of 
ELECTRE III have developed; see (Galo et al., 2018; Doumpos and Figueira, 2019). 
Before the ELECTRE MLO method appeared, the alternatives forming a cycle had 
been thought to be indifferent and had been ranked at the same hierarchical level. 
This approach can lead to obtaining imprecise levels (i.e. levels containing many 
more alternatives than other levels). In the paper (Petrović et al., 2012), the problem 
of cycles for the ELECTRE MLO method is solved based on an important result 
obtained in the study (Anic and Larichev, 1996) which solved the problem of cycles 
for the original ELECTRE method. The problem of cycles is solved by introducing a 
modified concordance index and the AST (Absolute Significance Threshold), which 
represent its limit, above which no cycle will appear in a graph. The ELECTRE MLO 
method will help us find the best evolution path. By this method, alternatives are 
ranked into levels, so that we can clearly see a hierarchy between them. By applying 
this method, a tree (a graph without a cycle) is obtained. The best alternative, i.e. the 
one being a benchmark to all other alternatives, is on top of the tree. The worst 
candidate needs to make progress gradually towards the top, choosing the best 
benchmark every step of the way. He looks for the optimal path, the path which is 
closest to the ideal one.  

Although benchmarking is mostly used in foreign policies, its specific application 
in higher education is demonstrated in Chapter 3. Benchmarking is applied in higher 
education; see (Ganushchak-Yefimenko et al., 2017; Padro and Sankey, 2012; Placek 
et al., 2017; Paliulis and Labanaskis, 2015). Various studies on the quality of lectures, 
the lecturer’s capability and the students’ evaluation of their lecturers in higher 
education have been carried out; see (Millis and Cottell, 1997; Ramsden, 2003; Wei, 
2007; Spehl et al., 2019). They have been aimed at improving the quality of 
higher-education facilities. The paper (Wachtel, 1998) provides the arguments “for” 
and “against” students’ evaluation of their lectures. The authors of the paper 
(Sullivan and Skanes, 1974) pay special attention to the characteristics of the 
lecturers with succesful academic carriers who were given excellent marks by their 
students. 

In the Methodology chapter of this paper, our main result is proven. In Chapter 3 
of this paper, the theorem is applied to a concrete example of benchmarking the 
teaching assistants of one faculty of Belgrade University, and how to choose an 
optimal development path and make gradual progress towards the top is illustrated. 

2. Methodology 

As stated in the Introduction, ELECTRE MLO is a good benchmarking tool.  

ELECTRE MLO (Multi-Level Outranking) first appeared in the study (Petrović et 
al, 2012) as a tool in stepwise benchmarking; it is a modification of ELECTRE I.  The 
result of the application of ELECTRE MLO to realistic data is a hierarchical structure 
of alternatives (e.g. in Figure 1 of Chapter 3). 

The sets of the criteria Gij+, Gij-, Gij=  are now defined for two alternatives, Ai  and Aj, 
in the following manner: 



Lazarević/Oper. Res. Eng. Sci. Theor. Appl. 2 (3) (2019) 77-91 

 

80 
 

 Gij+={gk |gk(Ai)>gk(Aj)}, 

 Gij-={gk |gk(Ai)<gk(Aj)},  (1) 

Gij=={gk |gk(Ai)=gk(Aj)}; 

where gk(Ai) are marks for the alternative Ai and the criterion k, and the ωk  is a 
weight factor for the criterion k. Let the I1,…,Im be a set of marks for any criteria and 
|Ik|=maxIk-minIk be a scaled score range of the criterion k. Allow us to define  the 
normalized value of the marks and the normalized value of the weight factor, as 
follows: 

    (2) 

For the ELECTRE I method, the concordance and discordance indices are defined 
in the following manner: 

 (3) 

 (4) 

The given concordance index is modified by applying the following 

equation:   (5) 

Let us then define the parameter  (i, j), applying the following equation: 

 (6) 

Theorem 1 (Anić, Larichev): The parameter λ is chosen, so that, for each 
arranged pair of alternatives: 

{( ) S S  } holds inequality: 

 (7) 

At this point, S is a binary relation, where S  implies that  is at least as good 

as the alternative  

This theorem provides a sufficient condition for the construction of a relationship 
when cycles do not appear. 

The parameter λ is the limit value of the modified concordance index, an there is 
no cycle. Alternatives are not indifferent, either.  

The following equation defines the AST:  



Using the ELECTRE MLO Multi-Criteria Decision-Making Method in Stepwise Benchmarking 
– Application in Higher Education 

 

81 
 

AST=  .  (8) 

 =  

For each criterion k, the difference between the scores of the alternatives from 
adjacent levels of performance is as follows: 

  (9) 

The ideal step is as follows: 

 (10) 

For each criterion k on the path π, the variation measuring the mean-squared 
difference from the increment step is depicted, thus obtaining the distance from the 
ideal path: 

 (11) 

The overall value of the variation for all criteria is as follows: 

 (12) 

The value is the worst path, where the total difference between the scores 

of the alternatives  and the target  for each criterion k is obtained when only 

moving by one level.  The number  is the relative measure of the evolution path π. 

   (13) 

Now, the difference between the scores of the alternatives from the two concrete 
levels are subjected to examination: 

 =  (14) 

 =  (15) 

 =c (16) 

 (17) 

The following inequalities read as follows:   Ultimately, the 
main result of our study is formulated and proven. 

Theorem 2: If alternatives are compared according to one single criterion, it is 
always better to follow the order of events and move one level at a time than move 
on to the next level only to immediately have a “break” after that. 



Lazarević/Oper. Res. Eng. Sci. Theor. Appl. 2 (3) (2019) 77-91 

 

82 
 

Proof: 

Invoking the previously said, the next inequality must be proven:  

 (18) 

From the right-hand side of the inequality, the addendum   

represents the skipped step and the pause of the value . 

The rest of the left-hand and the right-hand sides is the same; so, after canceling 
out, the following is obtained:  

 

  (19) 

Now, the theorem is proven. 

Remark 1: That the step skipped at the j-th moment is equivalent to the step 
skipped at any other moment has been proven. 

3. Application 

Human resources allow us to understand the sum of all the knowledge and skills 
of a certain group of people. These skills and knowledge should be a product of one’s, 
especially higher, education in today’s developed world. It is very important that, 
during their education process, students should be taught by high-quality lecturers, 
who will prepare them for making further progress.  

The students of one faculty of Belgrade University evaluated their teaching 
assistants according to a total of 10 criteria (Regularity of practice, Regularity of 
consultation, Comprehensibility and manner of presentation, Encouraging students 
to be more active, Importance of practice, Providing useful information, Assistant 
responds to students’ questions, Being professional and ethical in communication, 
Being objective and impartial in evaluation, General impression) on a scale from 1 to 
5. The evaluation was anonymous. We accessed and used the data, the average 
marks given for the teaching assistants according to each of the 10 criteria. The 
teaching assistants of this faculty were generally given good marks, so that the 
normalization of those marks was performed by giving the value 1 to the marks 
below 3 and creating a uniform integer scale from 2 to 9 for the values ranging from 
3 to 5.  



Using the ELECTRE MLO Multi-Criteria Decision-Making Method in Stepwise Benchmarking 
– Application in Higher Education 

 

83 
 

The example only included the teaching assistants who were evaluated by more 
than fifty students, for the reason of which fact the marks can be considered as 
realistic. Table 1 shows the evaluation criterion, the weight factor for the criterion 
and the obtained normalized marks for the teaching assistants. Each criterion is 
given an appropriate weight factor according to its importance (Table 1). Since the 
faculty at which our research study was carried out is a technical science faculty and 
the teaching assistants hold practical classes, the order of the most important 
criteria is as follows: Encouraging students to be more active, Importance of practice, 
Comprehensibility and manner of presentation. Regularity of practice and Regularity of 
consultation is a duty on the part of all those employed at the faculty, but it does not 
affect much the quality of the lecturers. 

Table1. The marks obtained (after the normalization) for teaching 

assistant of the faculty of Belgrade University which is the subject matter 

of the research. 

  C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 
ω 1 1 4 5 4 3 3 2 2 3 
A1 7 7 7 6 7 7 7 6 7 6 
A2 7 7 5 6 6 6 6 6 6 6 
A3 8 7 7 6 7 6 6 6 7 7 
A4 7 7 3 4 4 4 5 5 5 4 
A5 6 7 4 4 4 5 5 5 6 5 
A6 6 4 3 2 2 2 3 3 3 2 
A7 8 7 7 7 7 7 7 7 7 7 
A8 9 8 8 7 8 8 8 8 8 8 
A9 8 7 6 3 5 4 5 5 5 4 
A10 8 7 7 6 7 7 7 7 7 6 
A11 9 9 8 8 8 8 8 8 8 8 
A12 9 9 9 9 9 9 9 9 9 9 
A13 9 8 4 6 5 6 6 7 7 6 
A14 6 6 5 4 5 4 5 4 5 5 
A15 9 9 9 8 8 8 9 9 9 9 
A16 9 9 8 8 9 8 8 8 8 8 
A17 9 8 8 7 8 8 8 7 8 8 
A18 6 5 3 3 3 3 3 3 3 3 
A19 6 6 5 5 5 5 6 5 5 5 
A20 8 8 6 5 4 7 5 6 6 6 

Specialized software applied the ELECTRE MLO ranking method to these data. The 
value of the modified concordance index of 0.85 was chosen. The AST was 0.75. The 
tree in Figure 1 is a result of the application of the ELECTRA MLO method. The 
hierarchy amongst the alternatives and a possible development path for each 
alternative can clearly be seen. Weaker candidates have the aim to make the most 
possible uniform progress. Since the data are realistic, the ideal evolution path is 
non-existent. The candidates, however, need to choose the optimal evolution path in 
order to achieve the level of the teaching assistant who has received the highest of 
marks. 



Lazarević/Oper. Res. Eng. Sci. Theor. Appl. 2 (3) (2019) 77-91 

 

84 
 

In every educational institution, cooperation among colleagues is advisable. 
Evolution paths were considered for Alternative 5. For each of those paths (Figure 2), 

 was calculated and the example of the calculation is given at the end of this 
chapter. The effect of skipping steps, either one step or two at a time, was also 
considered. That skipping does not improve the evolution paths (in the cases of the 

the two paths in Figure 2, they are clearly worse, because  is greater   
of the first four paths in Figure 2) is clearly seen (the bottom of Figure 2), which is in 
accordance with Theorem 2. That the same value ρ was obtained for the two paths 
should not be a surprise, either, given the fact that they only differ in one alternative, 
A15 replacing A16 (these two alternatives have the marks that only differ from each 
other in a few criteria, and these two alternatives are at same tree level). So, the 
teaching assistant number 5 should choose one of these two best paths (the first and 
the second paths in Figure 2). The suggestion for the teaching assistant number 5 
implies that he should first look up to the colleague who is a little better than him, 
and that is the teaching assistant number 2 or the teaching assistant number 20, 
after which he should start gradually improving until he has reached the level of the 
teaching assistant given the best of marks. 

In our opinion, it is very important for the faculty and with respect to the quality of 
knowledge that the said teaching assistant should  be making gradual progress. We 
suggest that the teaching assistants should be attending each other’s practical 
classes so as to be able to take the advantage of learning from a better-evaluated 
colleague. Workshops without students could be organized for the teaching 
assistants in order to enable them to improve their personal teaching methods.  

 

Figure1. The result of the application of the ELECTRE MLO method to the 

realistic data. 



Using the ELECTRE MLO Multi-Criteria Decision-Making Method in Stepwise Benchmarking 
– Application in Higher Education 

 

85 
 

 

Figure 2. The evolution paths for the alternative A5, without “skipped 

steps” (paths 1, 2, 3, and 4), and with “skipped steps” (paths 5 and 6). 

For example, we calculated   according to the equations given in the 
Methodology chapter for the first path in Figure 2 
(A5-A2-A1-A10-A7-A17-A8-A11-A15-A12) in the following manner: 

For the Regularity of practice criterion (k=1), the marks of the alternatives of 
these paths are as follows: 

6-7-7-8-8-9-9-9-9-9 and  for this criterion is calculated as follows: 

=  

 

+ +

 

It is clear that, if the marks of the adjacent alternatives for the criterion k are 

equal, then we obtain   

For the Regularity of consultation criterion (k=2), the marks for the teaching 

assistant of these paths are as follows: 7-7-7-7-7-8-8-9-9-9; now,  is calculated 

as for the previous criterion,  ;   

 



Lazarević/Oper. Res. Eng. Sci. Theor. Appl. 2 (3) (2019) 77-91 

 

86 
 

For the Comprehensibility and manner of presentation criterion (k=3), the 
normalized marks are as follows: 

4-5-7-7-7-8-8-8-9-9;     and     

For the Encouraging students to be more active criterion (k=4), the following 
marks were obtained: 

4-6-6-6-7-7-7-8-8-9;   and    

For the Importance of practice criterion (k=5), the obtained normalized marks are 
as follows: 

4-6-7-7-7-7-8-8-8-9;      and  

5-6-7-7-7-8-8-8-8-9 ;     and   

For the Assistant responds to students’ questions criterion (k=7), the following 
marks were obtained and the DPV was calculated in a manner similar to the previous 
one: 

5-6-7-7-7-8-8-8-9-9;  ;  after the calculation, we obtained  

For the Teaching assistants are professional and ethical in communication 
criterion (k=8), the obtained marks are as follows: 

5-6-6-7-7-7-8-8-9-9    ; after the calculation, we obtained  

For the Objective and impartial in evaluation criterion (k=9), the obtained 
normalized marks are as follows: 

6-6-7-7-7-8-8-8-9-9   ; after the calculation, we obtained  

For the General impression criterion (k=10), the obtained normalized marks are 
as follows: 

5-6-6-6-7-8-8-8-9-9  ; after the calculation, we obtained  

 

The sum of the weight factors for all the criteria concerned is 28. 

According to this equation:  

 

Now, the worst DPV is calculated for this concrete path: 

For the Regularity of practice criterion (k=1): 

=     



Using the ELECTRE MLO Multi-Criteria Decision-Making Method in Stepwise Benchmarking 
– Application in Higher Education 

 

87 
 

For the Regularity of consultation criterion (k=2): 

  and     

For the Comprehensibility and manner of presentation criterion (k=3) : 

    and  

In following example, the DPV is calculated in the same manner as the previous 
one was.  

For the Encouraging students to be more active criterion (k=4), the following was 
obtained: 

  and    

For the Importance of practice criterion (k=5), the following was obtained: 

   and  ; 

For the Providing useful information criterion (k=6), the following was obtained: 

    and   ;  

For the Assistant responds to students’ questions criterion (k=7), the following was 
obtained: 

   and   ; 

For the Teaching assistants are professional and ethical in communication 
criterion (k=8), the following was obtained: 

     and  ; 

For the Objective and impartial in evaluation criterion (k=9), the following was 
obtained: 

    and ; 

For the General impression criterion (k=10), the following was obtained: 

     and     

According to the equation in the Methodology chapter,   

The same calculation was performed in the other five paths shown in Figure 2. It 
was concluded that one of the first two paths is the best choice for the teaching 
assistant A5.  



Lazarević/Oper. Res. Eng. Sci. Theor. Appl. 2 (3) (2019) 77-91 

 

88 
 

4. Conclusion and Discussion  

There is a saying, according to which if you do not become better, you will 
become worse. There is an existent constant tendency of people to make progress. 
This research study has shown that the ELECTRE MLO ranking method is a good tool 
in stepwise benchmarking.  

In the Methodology chapter, a detailed description of the ELECTRE methods of 
multi-criteria decision-making, especially the ELECTRE MLO ranking method, which 
is a good stepwise benchmarking tool is given. It prescribes how to pick the best 
evolution path of all possible paths in the graph by calculating ρ, which is very 
important for everyday practice. People very often want to make progress as fast as 
they can so as to become the best in a certain field. The main contribution of this 
paper rests on Theorem 2, which shows how one should behave in order to make 
general progress. According to this theorem, “skipping steps” is a worse choice than 
making progress gradually. The best thing to do is always to learn from a colleague 
who is slightly better than us. The ELECTRE MLO ranking method helps us to plan 
our progress, i.e. “who or what we should look up to in order to become better”.  

The decision-making based on the application of this method has been illustrated 
on an example pertaining to higher education, based on the data obtained from one 
faculty of Belgrade University. In this example, it is concluded that there is a 
suggested development path for every teaching assistant. All possible evolution 
paths for the teaching assistant number 5 (A5) were considered as well. The best 

evolution path was chosen after the calculation of  . The effect of skipping steps on 
this concrete example was also considered, and a conclusion is drawn that it is in 
accordance with our Theorem 2. Since we are dealing with the realistic data, our 
research has resulted in two equally good evolution paths because they only differ in 
one alternative (i.e. A15 or A16), with very similar marks for all the criteria. The 
same calculation (described in detail in Chapter 2) can also be applied to any other 
teaching assistant. This model and approach to the problem could also be useful in 
other business progress planning fields.  

5. Acknowledgement  

The author would hereby like to thank professors Vladimir Božin and Ivan Anić 
for their encouragement and support during his work on this paper.  

References 

Anić, I., Larichev. O. "The ELECTRE method and the problem of acyclic relation 
between alternatives." Automation and Remote Control, 1996. 

Arrowsmith, J., Sisson K., & Margison. P. "What can banchmarking offer the open 
method of co-ordination?" (Journal of Europen Public policy) 2004. 

Bauer, J. "Learning from each other: promises and pitfalls of benchmarking in 
comunications policy." Info, 2010. 

Bouysson, D. Roy. B."Comaprasion of two decision-aid models applied to a nuclear 
power plant siting example." Europen Jurnal of Operational Research, 1986. 



Using the ELECTRE MLO Multi-Criteria Decision-Making Method in Stepwise Benchmarking 
– Application in Higher Education 

 

89 
 

Brehmer, J., Dawson, S., Homiller, S., Kling, F., & Plehn, T. " Benchmarking simplified 
template cross sections in WH production". Journal of High Energy Physics, 34.2019. 

Charnes, A., W. Cooper & E. Rhodes. "Evaluating program and managerical efficiency: 
an application of data envelopment analysis to program folow through." 
Management science, 1981. 

De Blas, C. S., Martin, J. S., & Gonzalez, D. G. " Combined social networks and data 
envelopment analysis for ranking. ." European Journal of Operational Research, 
266(3), 990-999. 2018. 

Dias, L. C., Antunes, C. H., Dantas, G., de Castro, N., & Zamboni, L. " A multi -criteria 
approach to sort and rank policies based on Delphi qualitative assessments and 
ELECTRE TRI: The case of smart grids in Brazil." Omega, 76, 100-111. 2018. 

Dolowitz, D. Marsh. D."Who learns what from whom: a review of the policy transfer 
literature." Political studies, 1996. 

Doumpos, M., & Figueira, J. R. " A multicriteria outranking approach for modeling 
corporate credit ratings: An application of the Electre Tri-nC method." Omega, 82, 
166-180. 2019. 

Figueira, J., Moussean V., & Roy. B.,"ELECTRE methodes. In Multiple criteria decision 
analysis: State of the art surveys." Springer, 2005. 

Galo, N. R., Calache, L. D. D. R., & Carpinetti, L. C. R. "A group decision approach for 
supplier categorization based on hesitant fuzzy and ELECTRE TRI. ." International 
Journal of Production Economics, 202, 182-196. 2018. 

Ganushchak-Yefimenko, L., Shcherbak, V., & Nifatova, O. " Managing a project of 
competitive-integrative benchmarking of higher educational institutions." 
Eastern-European journal of enterprise technologies. 2017. 

Gidion, D. K., Hong, J., Adams, M. Z., & Khoveyni, M. " Network DEA models for 
assessing urban water utility efficiency." Utilities Policy, 57, 48-58.  

Harsoyo, A., & Jati, H.Harsoyo, A., & Jati, H. " performance analysis of ministry’s 
website in Indonesia using entropy weightage method and electre ranking method. 
."E-JPTE (Jurnal Elektronik Pendidikan Teknik Elektronika), 7(1), 1-8. 2018. 

Hashemi, S. S., Hajiagha, S. H. R., Zavadskas, E. K., & Mahdiraji, H. A. (2016). 
"Multicriteria group decision making with ELECTRE III method based on 
interval-valued intuitionistic fuzzy information." Applied mathematical modelling, 
40(2), 1554-1564., (2016). 

Hassan, M. F. C., Rosli, M. U. M., & Redzuan, M. A. M. " Material selection in a 
sustainable manufacturing practice of a badminton racket frame using Elimination 
and Choice Expressing Reality (ELECTRE) Method. " Journal of Physics: Conference. 
2018. 

Hong, P., Hong, S., Roh, J. James, J., Park. K., "Evolving benchmarking practices: a 
review for research percpetives." (Benchmarking: An Internacional Journal), no. 
19(4/5): 444-462 (2012). 

Ishizaka, A., & Pereira, V. " Utilisation of ANPSort for sorting alternative with 
interdependent criteria illustrated through a researcher’s classification problem in 
an academic context." Soft Computing, 1-12. 2019. 



Lazarević/Oper. Res. Eng. Sci. Theor. Appl. 2 (3) (2019) 77-91 

 

90 
 

Ishizaka, A., Giannoulis. C. "A web-based decision support system with ELECTRE III 
for a personalised ranking of british universitetes." Decision Support Systems, 2010. 

Ishizaka, A., Tasiou, M., & Martínez, L. " Analytic hierarchy process-fuzzy sorting: An 
analytic hierarchy process–based method for fuzzy classification in sorting 
problems. ." Journal of the Operational Research Society, 1-20. 2019. 

Ji, X., Wu, J., Zhu, Q., & Sun, J. " Using a hybrid heterogeneous DEA method to 
benchmark China’s sustainable urbanization: An empirical study." Annals of 
Operations Research, 278(1-2), 281-335. 2019. 

La Fata, C. M., Lupo, T., & Piazza, T. " Service quality benchmarking via a novel 
approach based on fuzzy ELECTRE III and IPA: an empirical case involving the 
Italian public healthcare context. Health care management science, 22(1), 106-120." 
2019. 

Lundvall, B. Tomlison. M. "International benchmarking as a policy learning tool. The 
New Knowledge Economy in Europe." A strategy for Internacional Competitiveness 
and social Cohesion, 2012. 

Millis, B. J., & Cottell Jr, P. G. (1997). Cooperative Learning for Higher Education 
Faculty. Series on Higher Education. Oryx Press, PO Box 33889, Phoenix, AZ 
85067-3889. 

Moore, R. "Making common sense common practice: Models for manufacturing 
excellence." Gulf Publishing Company, 1999. 

Padró, F. F., & Sankey, M. " Benchmarking as an instrument for continuous 
improvement in a regulated higher education quality assurance environment. ." In 
Cases on Quality Initiatives for Organizational Longevity (pp. 35-73). IGI Global. 2018. 

Paliulis, N. K., & Labanauskis, R. " Benchmarking as an Instrument for improvement 
of quality management in higher education." Business, management and education, 
13(1), 140. 2015. 

Papadopulos, A. Karagiannidis. A., "Application of the multicriteria analysis method 
ELECTRE III for the optimisation of decentralised energy systems." Omega, 2008. 

Pereira, J., de Oliveira, E. C., Gomes, L. F., & Araújo, R. M. "Sorting retail locations in a 
large urban city by using ELECTRE TRI-C and trapezoidal fuzzy numbers. ." Soft 
Computing, 23(12), 4193-4206. 2019. 

Petrović, M., Bojković, N., Anić, I., Petrović. D, "Benchmarking the digital divide using 
a multi-level outraking framework: Evidence from EBRD countries of operation." 
(Goverment Information Quaterly) 2012. 

Petrović, M., Bojković, N., Anić, I., Stamenković, M., Pejčić-Tarle. S., "An 
ELECTRE-based decision aid tool for stepwise benchmarking: An aplication EU 
Digital Agenda targets." (Decision Support Systems) 2014. 

Petrovic, M., Bojkovic, N., Stamenkovic, M., & Anic, I. "A sensitivity analysis of 
ELECTRE based stepwise benchmarking for policy: The case of EU digital agenda." 
Management: Journal of Sustainable Business and Management Solutions in Emerging 
Econo, 2018. 

Petrović, M., Bojković, N., Stamenković, M., & Anić, I. "Supporting performance 
appraisal in ELECTRE based stepwise benchmarking model". Omega, 78, 237-251. 
2018 



Using the ELECTRE MLO Multi-Criteria Decision-Making Method in Stepwise Benchmarking 
– Application in Higher Education 

 

91 
 

Plaček, M., Ochrana, F., & Půček, M. " Benchmarking in Czech higher education. ." 
NISPAcee Journal of Public Administration and Policy, 8(2), 101-124. 2015. 

R. Hambelton, J. Gross. "Local management in a global era." Public Management, 
2008. 

Ramón, N., Ruiz, J. L., & Sirvent, I. " Two-step benchmarking: Setting more 
realistically achievable targets in DEA. ." Expert Systems with Applications, 92, 
124-131.  

Ramsden, P. (2003). Learning to teach in higher education. Routledge. 

Rose, R. "What is lesson-drawing?" Cambrige University Press, 1991. 

Roy, B. "Classement et choix en presence de points de vue multiples." RAIRO- 
Operations Research-Recharche Operationnelle, 1968. 

Roy, B., & Skalka. J. M. "ELECTRE Is: Aspects methodologiques etiquide d'utilisation 
LAMASADE." Unite associee an CNRS no 825, 1987. 

S. Lim, H.Bae& L. H. Lee. "A study on the selection of benchmarking paths in DEA." 
Expert System with Applications, 2011. 

Singh, A. " An integrated approach towards ranking hospitals using Fuzzy AHP and 
ELECTRE-I technique. ." International Journal of Healthcare Management, 1-10. 2019. 

Spehl, M. S., Straub, C., Heinzmann, A., & Bode, S. F. " Student-perceived exam 
difficulty may trump the effects of different quality improvement measures 
regarding the students’ evaluation of a pediatric lecture series. ." BMC medical 
education, 19(1). 2019. 

Sullivan, A. M., & Skanes, G. R. (1974). Validity of student evaluation of teaching and 
the characteristics of successful instructors. Journal of Educational Psychology, 
66(4), 584. 

Tavassoli, M. A., Darestani, S. A., & Tavassoli, S. A. " Supplier selection and evaluation 
using QFD and ELECTRE in quality management system environment (case study: 
Faravari & Sakht Company)." International Journal of Productivity and Quality 
Management. 2018. 

Wachtel, H. K. (1998). Student evaluation of college teaching effectiveness: A brief 
review. Assessment & Evaluation in Higher Education, 23(2), 191-212. 

Wang, Y., & Yeo, G. T. " Intermodal route selection for cargo transportation from 
Korea to Central Asia by adopting Fuzzy Delphi and Fuzzy ELECTRE I methods." 
Maritime Policy & Management, 45(1), 3-18. 2018. 

Wei, W. A. N. G. (2007). Dilemma of the Criteria and Anxiety for the Reality: 
Reflections on the Current System for Evaluation of College Teachers' Classroom 
Teaching [J]. Journal of Zhejiang Normal University (Social Sciences), 3, 025. 

© 2019 by the authors. Submitted for possible open access publication under the 
terms and conditions of the Creative Commons Attribution (CC BY) 
license (http://creativecommons.org/licenses/by/4.0/).