IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 1 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 SCHOOL AND ACADEMIC PERFORMANCE FOR RANKING HIGH SCHOOLS: SOME EVIDENCE FROM ITALY Gabriella Marcarelli University of Sannio gabriella.marcarelli@gmail.com Paola Mancini University of Sannio paola.mancini@unisannio.it ABSTRACT This paper aims to investigate and compare the performance of Italian public high schools (HS) to provide a ranking among different typologies of HS. In this paper, seven criteria that refer to students’ school and academic performance were considered. The sample includes 263 high schools in all Italian regions, grouped into 6 different types of schools and 3 geographic areas. Assuming that all criteria have the same weights, the Analytic Hierarchy Process (AHP) was applied to derive the ranking among the typologies of schools both at a national level and within each geographic area. The main results show that there are significant differences between HS according to criteria related to school and academic performance both within and between geographic areas. The ranking does not vary, but the intensity of preferences may be different according to the area and/or the criterion considered. The application of PROMETHEE to the same problem confirms the results obtained by the AHP. Keywords: school ranking; school performance; academic performance; AHP; PROMETHEE 1. Introduction The literature on school ranking is vast. Some scholars have mainly focused on school quality and student achievements (Eide & Showalter, 1998; Camanho et al., 2021), others have analyzed the school’s contribution to student academic performance discussing both the so-called “school value-added” (Jamelske, 2009; Kelly & Downey, 2010; Schiltz et al., 2018) and the question of “school accountability” affecting the school choice (Burgess et al., 2013; Hart & Figlio, 2015; Nunes et al., 2015). The analysis of performance assessments of the upper secondary education system requires the measurement of student achievements at both the school and academic levels. Students’ performance may be influenced by many factors including students’ socio-economic status, family background, geographic area of residence, the type of school attended (Agasisti & Murtinu, 2012; Lauer, 2003), school and class size, students’ features, and school management and resources (Giambona & Porcu, 2018; Masci et al., 2018). More recently, Aina et al. (2011,2021) demonstrate that differences in university students’ achievements across high schools cannot be limited to the first year at university and have to consider the geographic differences. mailto:gabriella.marcarelli@gmail.com IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 2 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 To evaluate high schools, an adequate data set and appropriate methods are required. Eduscopio (Giovanni Agnelli Foundation - https://eduscopio.it/) and ScuolaInChiaro (Ministry of Education - www.miur.gov.it/-/scuola-in-chiaro) data are considered in this paper. The Eduscopio project started in 2014 to provide students and their families with a ranking of high schools in the area of residence based on the FGA (Giovanni Agnelli Foundation) index 1 . The ScuolaInChiaro project started in 2011 in response to the need to make all information related to Italian schools of all levels available to the community in an organic and structured form. These two portals provide data annually on the students' school career and their academic achievements for each Italian school. To evaluate the performance of schools by considering the school and academic careers of students, a multi-criteria approach, which is a useful tool was used. Multiple Criteria Decision Analysis (MCDA) helps decision-makers face problems characterized by multiple conflicting criteria. A large number of methods have been developed to solve multi-criteria problems. Roy (1991) listed different kinds of decision problems such as choice problems, sorting problems, ranking problems and description problems. A comprehensive literature review of common multi-criteria decision making (MCDM) methods is provided by Figueira et al. (2005), Ishizaka and Nemery (2013), and Velasquez and Hester (2013). The use of combined MCDM methods has led to a new approach to decision analysis (Velasquez & Hester, 2013). The software available, such as web and smartphone applications, R routines, or ad hoc packages, has made MCDM methods more accessible, increasing their use amongst researchers and the user community (Ishizaka & Nemery 2013). Each MCDM method has both advantages and disadvantages. The selection of the appropriate method depends on the features of the problem, the nature of the criteria, and data availability. When facing a decision problem, it is also important to define the type of output required and the input data. This paper aims to provide the ranking of different types of schools, taking into account the geographic area. The schools are evaluated considering quantitative criteria represented by performance indicators expressed in different units of measurement. Many studies have dealt with the application of multi-criteria methods in the field of education (Stamenkovic et al., 2016; Giannoulis & Ishizaka 2010; Mancini & Marcarelli 2019; Goztepe, 2020). By taking into account some performance indicators used by Eduscopio and ScuolaInChiaro and according to the approach proposed by Mancini and Marcarelli (2019), this study analyzes some criteria related to students’ school and academic achievements. Due to the characteristics of the problem (e.g., independence among the elements) and the output required, this work focuses on the Analytic Hierarchy Process (AHP) from among multi-criteria methods proposed in the literature. The available data allows us to avoid some disadvantages of the AHP. There is no inconsistency in the judgment matrices because entries of matrices are ratios between performance indices. After 1 The FGA index is an indicator that accounts for the percentage of academic credits at the end of the first year and the average exam score in order to measure the students’ “speed” and “profit” in academic education (https://eduscopio.it/dati-e-metodologia). For further details, see Bordignon et al. (2017). IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 3 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 obtaining a complete ranking with scores, a sensitivity analysis is conducted on the weights of criteria. Furthermore, in order to compare the results obtained by a full aggregation approach with those derived by an outranking approach, the Preference Ranking Organization Method for Enrichment of Evaluations (PROMETHEE) was applied to verify the robustness of the solution. The main contributions of this paper are as follows: - application of the AHP to derive the ranking among different types of Italian high schools at a national level; - highlighting any significant differences between HS within each geographic area; - application of the PROMETHEE method to verify the robustness of the solution. The remainder of this article is structured as follows. Section 2 defines the elements of the problem and makes a descriptive analysis of the data; Section 3 introduces the methods applied in our study; Section 4 provides the ranking of high schools at a national level and by geographical area and includes the sensitivity analysis; in Section 5, a comparative analysis is carried out. In the last section, some conclusions are provided. 2. Data Data are from the Eduscopio and ScuolaInChiaro portals and refer to the 2019/2020 school year. In addition to the FGA index, Eduscopio provides information on students' school and academic performance annually. Based on the above data, the following performance indicators were considered: - HSLS = high school leaving score (calculated as the weighted average between the high school leaving score of academic enrolled and not enrolled students), - RegGrad = proportion of regular graduates, - AES = proportion of academic enrolled students, - AESpFY = percentage of academic enrolled students passing the first year (calculated as the proportion of academic enrolled students), - ACFY = percentage of academic credits at the end of the first year, - EAS = exam average score. Moreover, the study considers the INVALSI test score (computed as the average between individual students' math, reading, and foreign language test scores) provided by ScuolaInChiaro. The reference universe is made up of 761 public high schools located in the regional capitals. A sample of 263 public HS, stratified by region and type of school, was selected. The schools, ranked by the Eduscopio FGA index in each capital region, were selected ensuring the representativeness of the various positions in the ranking 2 . Following Aina et al. (2021), six types of schools were taken into account as follows: 60 scientific lyceums (SL), 40 classic lyceums (CL), 41 linguistic lyceums (LL), 27 human sciences lyceums (HSL), 44 commercial technical high schools (CTHS), and 51 technological technical high schools (TTHS). 2 For example, in the case of Rome, 13 SL are selected. We consider the 4 best schools, the 5 schools in the middle of the ranking and the 4 worst schools, according to the FGA index. IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 4 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 Table 1 shows the performance matrix showing the average 3 scores computed for each type of school for the varying criteria. The descriptive analysis shows that, as concerns students’ achievements, CL and SL have the best performance, and technical HS have the worst performance, whereas the performance of LL and HSL are not clearly defined and may vary depending on the criterion. CL are better than SL with regard to most of the criteria related to school career (e.g., HSLS, RegGrad, and INVALSI) and AES; the opposite holds for ACFY; they scored similarly as concerns AES and AESpFY. Data related to 263 HS were processed by hierarchical cluster analysis 4 and organized into groups based on how closely associated they are. This method aims to capture similarities and/or differences among Italian high schools and create homogeneous groups according to the criteria taken into account. The main results highlight 6 clusters almost matching the six types of schools (Table 2). The first group contains 50.0% of SL and 38.8% of CL; the second group includes about 40% of HSL and 35% of LL; CTHS and TTHS schools are located mainly in the fourth, fifth and sixth clusters; the third cluster consists of a residual group of schools. Table 1 Average scores related to 6 school types and 7 criteria in Italy Average scores School performance Academic performance School type HSLS INVALSI RegGrad AES AESpFY ACFY EAS SL 78.32 4.49 62.62 90.63 90.84 73.24 26.62 CL 80.80 4.67 72.08 91.42 91.30 69.76 27.07 LL 78.41 4.21 59.93 74.20 87.38 58.28 25.32 HSL 76.11 3.98 58.80 69.09 84.09 49.40 24.17 CTHS 74.37 3.78 45.79 47.88 80.39 46.64 23.55 TTHS 74.11 4.12 40.98 45.54 76.51 50.17 24.61 3 The average values have been obtained as weighted arithmetic averages according to the number of students in each school. 4 The objective of this method is to find similar groups of subjects, where “similarity” between each pair of subjects means some global measure over the whole set of characteristics. The Ward's method was the criterion applied in this hierarchical procedure. IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 5 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 Table 2 High schools grouped using cluster analysis School type Cluster 1 2 3 4 5 6 SL 50.6 3.1 35.7 0.0 0.0 0.0 CL 39.2 18.8 2.4 0.0 0.0 0.0 LL 10.1 34.4 31.0 12.0 4.5 0.0 HSL 0.0 40.6 14.3 6.0 18.2 0.0 CTHS 0.0 3.1 4.8 38.0 50.0 38.5 TTHS 0.0 0.0 11.9 44.0 27.3 61.5 TOT (a.v.) 79 32 42 50 22 26 The best performing high schools are in the first cluster (Table 3). Students’ high school leaving score is almost 80 and the INVALSI test score is 4.85, 90% of their graduates enroll in university and, of these, more than 9 out of 10 pass the first year exceeding at least 71% of the academic credits and having an average exam score of 26.9. On the contrary, less performing schools are in the fourth and sixth clusters. For these, the high school leaving score is lower than 74, only 45% of graduates enroll in university; particularly in the fourth group, only 70% of enrolled students pass the first year having obtained about 38% of the academic credits and an average exam score below 24. Table 3 Average values of 6 clusters according to 7 criteria Criteria Cluster 1 2 3 4 5 6 HSLS 79.75 77.16 75.82 73.95 75.34 73.37 INVALSI 4.85 4.16 3.67 3.96 3.91 3.69 RegGrad 67.79 66.12 46.46 42.53 56.49 31.37 AES 90.24 76.69 77.12 44.76 52.63 44.15 AESpFY 91.54 83.59 85.79 70.45 84.93 81.10 ACFY 71.73 49.51 56.42 37.78 55.34 52.49 EAS 26.88 24.57 24.84 23.61 24.26 24.20 IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 6 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 3. Methodology 3.1. AHP The AHP is a multi-criteria method developed by Saaty in the 1970s in order to prioritize different and potentially conflicting objectives and give a ranking of alternatives through pairwise comparisons (Saaty & Vargas, 1982; Aczel & Saaty, 1989). The multi-criteria approach has found many fields of application in both individual and collective decision contexts (Triantaphyllou & Mann, 1995). The AHP makes it possible to analyze a complex problem by combining both qualitative and quantitative aspects into a single framework and generating a set of priorities for alternatives. A detailed explanation of the AHP is found in the study by Vargas (1990). This method involves four phases described in Table 4. Table 4 Steps of the AHP Step Description 1 Decomposition of the decision problem into a multi-level hierarchy 2 Data collection by means of pairwise comparisons 3 Determination of the relative weights, reflecting the relative importance of the elements belonging to each hierarchical level considered with respect to the elements of the immediately higher level 4 Aggregation of relative weights to obtain the overall priorities, expressing the importance of alternatives with respect to the overall objective of the evaluation (by applying the principle of hierarchical composition). This priority vector provides the ranking of alternatives Saaty uses a pairwise comparison matrix (PCM) to derive the relative weights, which reflect the relative importance of the elements belonging to each hierarchical level with respect to the elements of the level immediately higher. To each pair of elements (xi, xj) of a fixed level, a positive number (aij) is assigned expressing how much xi is preferred to xj as regards to a given criterion; by comparing all n elements of a level, a positive square matrix of the order n is obtained. The value aij > 1 implies that xi is strictly preferred to xj, whereas aij < 1 expresses the opposite preference and aij = 1 means that xi and xj are indifferent. The matrix is obviously reciprocal (aji=1/aij for i, j =1, 2, …, n). Before deriving priority vectors, a check of logical consistency in the allocation of judgments is required. The inconsistency may be caused by errors, inaccuracies, or simply by the violation of transitivity or proportionality. To verify the judgments expressed by the DM, several measures of consistency have been proposed in the literature. Saaty suggested the Consistency Index (CI): and the Consistency Ratio (CR): IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 7 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 where RI (Random index) is the average of CI values associated with several PCM (of size n) randomly generated (Saaty & Vargas, 1982). If CR is less than 0.1, then the matrix may be considered to have acceptable consistency; otherwise, the judgments must to be revised. When a PCM is perfectly consistent, its entries are given by the ratios between pairs of the components of the priority vector (aij = wi/wj): the matrix has unit rank and all its eigenvalues except one are zero; since the sum of the eigenvalues of a matrix is equal to its trace, then n is the only non-zero eigenvalue of A (as a consequence, CI=0 and CR=0). In order to calculate the priority vector associated to each PCM, Saaty proposed adopting the eigenvector method (EM), that is, the eigenvector associated to the maximum eigenvalue of A. This vector represents the relative weight of the elements of a level with respect to an element of the level immediately higher. Finally, by applying the principle of hierarchical composition, relative weights are aggregated to obtain the overall priorities expressing the importance of alternatives with respect to the overall objective of the evaluation. This priority vector provides the ranking of alternatives. Saaty (1994) developed a software package called Expert Choice for use with the AHP that allows priority vectors to be calculated, inconsistencies of matrices to be identified and a sensitivity analysis to be performed. 3.2. PROMETHEE PROMETHEE is an outranking method introduced by Vincke (1992). It uses preference degrees to derive the ranking of alternatives. For each criterion, the decision-maker assigns a preference degree to pairs of alternatives. The preference degree expresses how an alternative is preferred to another one. If the value is 1, then there is a strong preference for an alternative on a given criterion; 0 means that there is no preference; and values between 0 and 1 express that there is some preference. The pairwise preference degrees related to a given criterion are synthesized in the positive, negative, and net flows related to that criterion. The positive flow indicates how an alternative “a” is preferred to all other alternatives, with regard to a criterion, and the negative flow indicates how the other alternatives are preferred to “a”. The positive and negative flows are scores between 0 and 1; net flows, a combination of the above two flows, are valued between -1 and 1. The final ranking of the alternatives is given by the global net flow, obtained as a difference between the global positive and negative flows when considering all criteria and their weights. PROMETHEE makes it possible to represent the decision problem in the Gaia plane as a two-dimensional plot containing the alternatives, criteria, and information on the preferences of the decision-maker (thresholds and weights). IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 8 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 4. Ranking high schools by typology A decision problem consists of a finite set of alternatives (A), a set of criteria with respect to the alternatives are compared, and a weak order relation on A. A multi-criteria problem can be represented by a decision matrix, where each entry represents the value of the i th alternative regarding the j th criterion and each criterion has a weight (Triantaphyllou & Mann, 1995). Given this matrix, several methods exist for aggregating judgments related to each alternative into a synthetic value in order to select the best one. In the first step of our analysis, the AHP is applied to obtain a ranking among the different types of schools taking into account the geographical area. In the second step, the PROMETHEE procedure is performed on the same problem in order to compare the results. Figure 1 shows the elements of our problem arranged in a hierarchical structure consisting of 3 levels as follows: the bottom level contains the alternatives, that is, the school typologies; the second level contains the criteria, that is, the performance indicators with respect to which the alternatives are evaluated; and the top level is represented by the goal, the ranking among the 6 types of high schools. Figure 1 Elements of the problem Data have been processed using Expert Choice and R routines. PROMETHEE was used from among the different packages containing functions for solving multiple-criteria decision-making problems. Using the data in Table 1, seven pairwise comparison matrices (PCM) were constructed to evaluate pairs of school typologies with respect to each criterion. All of the PCM are perfectly consistent because their entries are given by the ratios between pairs of performance values (as a consequence, CI and CR are null). Therefore, the consistency issue may be disregarded in our study. The weights of the criteria are not derived by pairwise comparison of the criteria with regard to the goal, but they are a priori assigned; in particular, the criteria all have the same weight. Next, in order to determine if any geographical differences emerge, the six types of schools were analyzed by area. Table 5 contains the performance matrices which show IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 9 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 the average scores computed with regard to the six types of schools for each geographic area (North, Center, and South of Italy) on the varying criteria. Table 5 Average scores related to 6 school types and 7 criteria by geographical area Area School type School performance Academic performance HSLS INVALSI RegGrad AES AESpFY ACFY EAS N O R T H SL 77.44 4.83 60.82 91.87 90.84 73.24 26.62 CL 78.57 5.02 68.05 91.87 91.30 69.76 27.07 LL 78.26 3.97 63.05 75.46 87.38 58.28 25.32 HSL 74.34 3.50 52.01 71.01 84.09 49.40 24.17 CTHS 73.95 3.64 46.34 47.44 80.39 46.64 23.55 TTHS 73.78 4.40 41.07 47.10 76.51 50.17 24.61 C E N T E R SL 77.99 4.53 62.44 90.51 91.85 76.99 27.11 CL 81.44 4.31 69.31 91.32 91.98 71.81 27.61 LL 78.14 4.39 56.97 72.76 88.91 60.76 25.39 HSL 76.16 4.23 59.91 71.55 84.55 51.06 24.56 CTHS 74.32 3.84 47.87 48.58 80.23 45.93 23.57 TTHS 73.29 3.96 43.05 48.34 79.83 53.68 24.98 S O U T H SL 79.41 4.07 64.45 89.61 90.56 72.81 26.77 CL 81.97 4.80 77.38 91.15 90.87 69.41 27.31 LL 78.81 4.16 60.59 74.74 86.07 56.37 25.35 HSL 76.85 4.06 61.01 66.34 85.33 50.14 24.15 CTHS 74.79 3.91 44.33 47.98 79.70 48.46 24.16 TTHS 75.10 3.89 39.58 41.49 73.10 45.79 24.45 The results obtained by applying the AHP (Figure 2) show that 3 groups of schools may be identified nationwide. CL and SL have the best performance, LL and HSL occupy the intermediate level, and technical HS have the worst performance. It is interesting to note that the ranking does not vary moving from North to South, but the intensity of preferences may be different according to the area and/or the criterion considered (relative priority vectors are listed in the Appendix). For example, CL exceeds SL for IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 10 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 most of criteria related to school performance, particularly in Southern Italy, whereas SL overcomes CL regarding ACFY, especially in the North. Figures 3 and 4 show the relative priorities of the alternatives with respect to each criterion. In each geographic area, CL is the best alternative for all criteria except for ACFY. The sensitivity analyses showed that the global ranking is a stable solution since varying the weights of the criteria did not produce any significant changes. As Figure 5 highlights, minor variations in the ranking of the alternatives occur in particular when the weight of the criterion ACFY is greatly increased (for which SL is the best alternative). Figure 2 Ranking of school typologies for Italy by geographical area Figure 3 Performance sensitivity plot - Italy IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 11 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 Figure 4a Performance sensitivity plot - North Figure 4b Performance sensitivity plot – Center Figure 4c Performance sensitivity plot - South IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 12 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 Figure 5 Gradient sensitivity for ACFY - Italy 5. Comparative analysis In the second step of the study, the PROMETHEE procedure was applied to the performance matrices in Tables 1 and 5 in order to compare the results with the solution provided by AHP. The preference and indifference thresholds (Table 6) are identified based on a One Way Analysis of Variance test 5 on the differences among 6 means for each criterion. Table 6 Preference and indifference thresholds Scores School performance Academic performance HSLS INVALSI RegGrad AES AESpFY ACFY EAS Max-min max Max max max Max max max thresholds Indif. (q) 1.74 0.70 1.13 5.11 3.87 3.54 0.71 Pref. (p) 2.21 0.89 9.47 16.43 6.75 8.89 1.06 weights Wj 0.10 0.15 0.10 0.10 0.15 0.15 0.15 The PROMETHEE method provides the final ranking of the alternatives given by the global net flow, which is a result of the difference between the global positive flow and 5 ANOVA is used to test for differences among two or more independent groups, providing a statistical test to verify if two or more means are equal or not, and therefore generalizes the t-test beyond two means. IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 13 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 the negative flow. Figure 6 shows that positive and negative flows provide the same ranking at the national level. CL has the best rank because its global positive and negative flows are simultaneously better than the others; on the contrary, CTHS has the worst rank. There are no incomparable school types because the global positive flow and the negative flow provide the same ranking. Table 7 and Figure 7 show the rankings of the six types of schools both in Italy and in 3 geographic areas: North, Center, and South. A comparison of the rankings among areas shows the following: - each area confirms the ranking results at the national level: CL ranks at the top, SL has the second rank, LL composes the third group, HSL the fourth, and technical schools (CTHS and TTHS) are the last group; - with regard to the last positions (TTHS has the worst rank), there is a rank reversal in the Center and the South. The rankings obtained by the AHP and PROMETHEE models are similar, so the stability of the solution is verified. Table 7 Net flows and ranking of school types at a national level and by geographic area Alternatives Net flows Italy rI North rN Center rC South rS SL 0.07521521 2 0.06766490 2 0.08951429 2 0.07315623 2 CL 0.10569414 1 0.08274755 1 0.09345714 1 0.11788571 1 LL 0.02295296 3 0.02202566 3 0.01081549 3 0.03469699 3 HSL -0.03566422 4 -0.03282828 4 -0.01501180 4 -0.03198341 4 CTHS -0.08756714 6 -0.07544853 6 -0.08723914 5 -0.08919477 5 TTHS -0.08063096 5 -0.06416129 5 -0.09153598 6 -0.10456076 6 Figure 6 Positive and negative flows for the types of schools in Italy IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 14 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 Figure 7 Positive and negative flows for the types of schools by geographic area The GAIA plane summarizes the results of the PCA applied to the uni-criterion net flows matrix. The criteria are represented by vectors (Figure 8a) and alternatives by points (Figure 8b). The length of each vector is a measure of its strength in the alternatives’ differentiation. The horizontal axis distinguishes the alternatives by typology of school. The best schools (lyceums) are on the right, and the worst school types (technical schools) are on the left. In other words, moving from left to right, in the plane, school and academic performance improves. IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 15 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 Figure 8a PCA graph of variables Figure 8b PCA graph of individuals 6. Conclusions This paper investigates and compares the performance of Italian high schools in order to derive a ranking considering the typology and the geographic area. The results show that the ranking among the types of schools does not vary moving from North to South (classic lyceums are the best, technical schools are the worst). The model may assist students and their parents in selecting the type of school to attend; the information makes it possible to make an appropriate choice according to their academic perspectives. High school types are evaluated using criteria that provide a measure of performance under specific (school and academic) features. When applied to small territorial districts, this model may be a useful tool to help public administrations distribute additional financial resources to public schools based on their performance rank. Our future works will address the classification of high schools into different categories such as ‘over-performing schools’, ‘average-performing schools’ and ‘weak-performing schools’ by using ELECTRE TRI (Corrente et al., 2016). Furthermore, it could be IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 16 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 interesting to verify the ranking of HS when varying the key parameters (e.g., preference and indifference thresholds). IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 17 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 APPENDIX Relative priority vectors of school typologies with respect to each criterion Alternatives HSLS INVALSI RegGrad AES AESpFY ACFY EAS Italy SL 0,169 0,178 0,185 0,216 0,178 0,211 0,176 CL 0,175 0,185 0,212 0,218 0,180 0,201 0,179 LL 0,170 0,167 0,176 0,177 0,171 0,168 0,168 HSL 0,165 0,157 0,173 0,165 0,165 0,142 0,160 CTHS 0,161 0,150 0,135 0,114 0,157 0,134 0,155 TTHS 0,160 0,163 0,120 0,109 0,150 0,144 0,163 North SL 0,169 0,191 0,183 0,216 0,178 0,214 0,177 CL 0,172 0,198 0,206 0,216 0,178 0,199 0,180 LL 0,172 0,156 0,190 0,177 0,172 0,169 0,166 HSL 0,163 0,138 0,157 0,168 0,163 0,142 0,160 CTHS 0,162 0,143 0,140 0,112 0,155 0,128 0,154 TTHS 0,162 0,174 0,124 0,111 0,154 0,149 0,163 Center SL 0,169 0,191 0,183 0,216 0,214 0,211 0,177 CL 0,172 0,198 0,206 0,216 0,199 0,201 0,180 LL 0,172 0,157 0,190 0,177 0,169 0,168 0,166 HSL 0,163 0,138 0,157 0,168 0,142 0,142 0,160 CTHS 0,162 0,143 0,140 0,112 0,128 0,134 0,154 TTHS 0,162 0,174 0,124 0,111 0,149 0,144 0,163 South SL 0,170 0,164 0,186 0,218 0,179 0,207 0,175 CL 0,176 0,193 0,223 0,222 0,180 0,203 0,178 LL 0,169 0,167 0,175 0,182 0,173 0,172 0,169 HSL 0,164 0,163 0,175 0,161 0,164 0,142 0,161 CTHS 0,160 0,157 0,128 0,117 0,160 0,137 0,156 TTHS 0,161 0,156 0,114 0,101 0,145 0,140 0,162 IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 18 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 REFERENCES Aczel, J., & Saaty, T.L. (1989). Procedures for synthesizing ratio judgements. Journal of Mathematical Psychology, 27(1), 93–102. Doi: https://doi.org/10.1016/0022- 2496(83)90028-7 Aina, C., Baici, E., & Casalone, G. (2011). Time to degree: Students’ abilities, university characteristics or something else? Evidence from Italy. Education Economics, 19(3), 311–325. Doi: https://doi.org/10.1080/09645292.2011.585016 Aina, C., Bratti, M., & Lippo, E. (2021). Ranking high schools using university student performance in Italy. Economia Politica, 38, 293–321. Doi: https://doi.org/10.1007/s40888-020-00208-2 Agasisti, T., Murtinu, S. (2012). Perceived competition and performance in Italian secondary school: new evidence from OECD-PISA 2006. British Educational Research Journal, 38(5), 841–858. Doi: https://doi.org/10.1080/01411926.2011.588314 Bordignon M., Carapella P., & Turati G. (2017). Eduscopio: le determinanti dell'accesso all'informazione, Programma Education FGA Working Paper, n.56, Fondazione Giovanni Agnelli. Burgess, S., Wilson, D., & Worth, J. (2013). A natural experiment in school accountability: the impact of school performance information on pupil progress, Journal of Public Economics, 106(C), 57-67. Doi: https://doi.org/10.1016/j.jpubeco.2013.06.005 Camanho, A.S., Varriale, L., Barbosa, F., & Sobral, T. (2021). Performance assessment of upper secondary schools in Italian regions using a circular pseudo-Malmquist index, European Journal of Operational Research, 289(3), 1188-1208. Doi: https://doi.org/10.1016/j.ejor.2020.07.050 Corrente, S., Greco, S. & Slowinski, R. (2016). Multiple Criteria Hierarchy Process for ELECTRE Tri methods. European Journal of Operational Research, 252, 191-203. Doi: https://doi.org/10.1016/j.ejor.2015.12.053 Eide, E., and Showalter, M.H. (1998). The effect of school quality on student performance: A quantile regression approach. Economics Letters, 58, 345–350. Doi: https://doi.org/10.1016/s0165-1765(97)00286-3 Giambona, F., & Porcu, M. (2018). School size and students ’ achievement: Empirical evidences from PISA survey data. Socio-Economic Planning Sciences, 64(C), 66-77. Doi: https://doi.org/10.1016/j.seps.2017.12.007 Giannoulis, C., & Ishizaka, A. (2010). A web-based decision support system with ELECTRE III for a personalised ranking of British universities. Decision Support Systems, 48(3), 488–497. Doi: https://doi.org/10.1016/j.dss.2009.06.008 IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 19 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 Goztepe, K. (2020). Applying Choquet Integral Approach for ranking high school innovative education. International Journal of Electrical Communication Engineering, 6(1), 27-40. Hart, C.M.D., & Figlio, D.N. (2015). School accountability and school choice: effect on student selection across schools. National Tax Journal, 68(3S), 875–900. Doi: https://doi.org/10.17310/ntj.2015.3s.07 Ishizaka, A, & Nemery, P. (2013). Multi-criteria decision analysis methods and software. New York: Wiley. Jamelske, E. (2009). Measuring the impact of a university first-year experience program on student GPA and retention. Higher Education, 57(3), 373–391. Doi: https://doi.org/10.1007/s10734-008-9161-1 Kelly, A., & Downey, C. (2010). Value-added measures for schools in England: Looking inside the ‘black box’ of complex metrics. Educational Assessment, Evaluation and Accountability, 22(3), 181–198. Doi: https://doi.org/10.1007/s11092-010-9100-4 Lauer, C. (2003). Family background, cohort and education: A French-German comparison based on a multivariate ordered probit model of educational attainment. Labour Economics, 10, 231–251. Doi: https://doi.org/10.1016/s0927-5371(03)00007-1 Mancini, P., & Marcarelli, G. (2019). High school choice: how do parents make a choice. International Journal of the Analytic Hierarchy Process, 11(1) 91-109. Doi: https://doi.org/10.13033/ijahp.v11i1.633 Masci, C., Witte, K. D., & Agasisti, T. (2018). The influence of school size, principal characteristics and school management practices on educational performance: An efficiency analysis of Italian students attending middle schools. Socio-Economic Planning Sciences, 61, 52–69. Doi: https://doi.org/10.1016/j.seps.2016.09.009 Nunes, L., Reis, A., & Seabra, C. (2015). The publication of school rankings: A step toward increased accountability? Economics of Education Review, 49(C), 15-23. Doi: https://doi.org/10.1016/j.econedurev.2015.07.008 Roy, B. (1991). The outranking approach and the foundations of ELECTRE methods. Theory and Decision, 31, 49-73. Doi: https://doi.org/10.1007/bf00134132 Saaty, T.L. (1994). How to make a decision: the analytic hierarchy process, Interfaces, 24, 41-42. Doi: https://doi.org/10.1287/inte.24.6.19 Saaty, T.L. & Vargas, L.G. (1982). The logic of priorities, Applications in business, energy, health, and transportation. Boston: Kluwer-Nijhoff. Reprinted in Paperback (1991), Pittsburgh: RWS Publications. Schiltz, F, Sestito, P, Agasisit, T., & De Vitte, K. (2018). The added value of more accurate predictions for school rankings. Economics of Education Review, 67, 207-215. Doi: https://doi.org/10.1016/j.econedurev.2018.10.011 IJAHP Article: Marcarelli, Mancini/School and academic performance for ranking high schools: Some evidence from Italy International Journal of the Analytic Hierarchy Process 20 Vol. 14 Issue 2 2022 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v14i2.948 Stamenkovic, M., Anic, I., Petrovic, M., & Bojkovic, N. (2016). An ELECTRE approach for evaluating secondary education profiles: evidence from PISA survey in Serbia. Annals of Operational Research, 245, 337–358. Doi: https://doi.org/10.1007/s10479- 015-1823-7 Triantaphyllou, E. & Mann, S.H. (1995). Using the analytic hierarchy process for decision making in engineering applications: some challenges. International Journal of Industrial Engineering: Applications and Practice, 2(1), 35–44. Vargas, L.G. (1990). An overview of the analytic hierarchy process and its applications. European Journal of Operational Research, 48(1), 2–8. Doi: https://doi.org/10.1016/0377-2217(90)90056-h Velasquez, M. and Hester, P. T. (2013). An analysis of Multi-Criteria Decision Making Methods. International Journal of Operations Research, 10(2), 56-66. Vincke, P. (1992). Multicriteria Decision-Aid. Chichester: John Willey & Sons. https://doi.org/10.1016/0377-2217(90)90056-h