BCP Education & Psychology EED 2022 Volume 5 (2022) 25 Design of evaluation system for higher vocational institutions based on AHP hierarchical analysis Weitao Pang1,a, Zhipeng Wang*1,b, Yuan Yuan2, and Mengqiu Zhang3 aGuangzhou Panyu Polytechnic, Guangzhou 510000, China; bXi’an Jiaotong University, Xi’an 710049, China Abstract. The development of higher vocational education is inseparable from a country's economic and social development and industrial structure development. Strengthening scientific evaluation of the development status of higher vocational education is an important prerequisite and foundation for scientifically promoting the development of higher vocational education. Educational evaluation is an effective scientific method for discovering the shortcomings of institutions' development and promoting their improvement. Based on this, it is important to construct a systematic, rich, measurable, forward-looking and robust as an evaluation index system. This paper uses a combination of subjective expert scoring and objective AHP hierarchical analysis to construct an evaluation system for higher vocational institutions, and comprehensively analyses the relative strength indicators and relative weakness indicators of each vocational institution, with a view to helping relevant institutions understand their overall development situation more comprehensively and objectively, and promoting the sustainable development of higher vocational education. Keywords: Institutional evaluation; AHP; Vocational education. 1. Introduction As human socio-economic development enters a new stage, the refinement of social division of labour and the promotion of industrial collaboration models are increasingly appearing in different fields. Vocational education is a bridge between the training of talents and the demand for social skills, and assumes a more important task in promoting industrial upgrading and economic restructuring. β€œHow to develop vocational education well” is inevitably dependent on educational evaluation as an important tool for conveying quality information and assessing development. Research on education evaluation has been increasing by scholars and experts in related fields. The focus of research is mainly on universities and the evaluation of their professional, research, teaching and management aspects. In the area of institutional evaluation, OCTAVIO proposes a visual data extraction method to analysis the developmental changes of institutions[1], and Samuel Gratzl proposes a method of comparing multi-attribute rankings to compare the rankings of universities in different years[2]. This paper uses AHP hierarchical analysis to discover and study the comprehensive strength of the collected higher vocational institutions, and further improve the evaluation system through significance and consistency tests, with a view to providing a scientific basis for the planning and construction of higher vocational institutions in the future. 2. The theory of evaluation system of higher education institutions The evaluation system of higher education institutions established in this paper is analyzed by AHP (Analytic Hierarchy Process), which is a method combining subjective judgment and scientific calculation proposed by Professor Thomas L. Saaty of the University of Pittsburgh in the early 1970's[3]. AHP embodies the idea of decomposing and synthesizing, Representation of different relationships by stratifying and categorizing the problems analyzed and step-by-step[4]. The application of hierarchical analysis is divided into three main steps: the first step is to construct the index system; the second step is to calculate the index weights; and the third step is to calculate the data of each institution according to the weights, resulting in an evaluation score table[5]. BCP Education & Psychology EED 2022 Volume 5 (2022) 26 2.1 Constructing a hierarchical evaluation model According to the development characteristics of higher vocational institutions, reference to the research results of relevant aspects. According to the principles of scientificity, systematicity, comparability, and operability[6], the evaluation factors were selected. The scale of school operation B1, major and curriculum construction B2, faculty strength B3, talent cultivation B4, social service B5 and scientific research strength B6 are initially determined as the guideline layers of the evaluation system. To ensure the independence and objectivity of the evaluation factors, relevant experts were invited to score the 30 evaluation indicators in Table 1. The scoring is based on the importance scale 1-9 recommended by Saaty, as shown in table 2. Table 1 Importance Scale Score Description 1 Both indicators are equally important 3 x indicator 3 times more important than y indicator (marginally important) 5 7 9 2,4,6,8 decimal x indicator 5 times more important than y indicator (clearly important) x indicator 7 times more important than y for (strongly important) X indicator 9 times more important than y-indicator (extremely important) Importance indices between the above descriptions x indicator is less important than y indicator The experts judge the importance of all the indicators collected and give them a corresponding score. Of these, the very important ones are left in the indicator system with a greater weight and the less important ones are given a smaller weight. The final results are calculated according to the different scores. Since the sample size is less than 30, the test is carried out using the t-distribution commonly used for small samples. First, because the overall variance is unknown, the entire data obeys a distribution with degrees of freedom of n-1 and the test statistic is 𝑑𝑑 = π‘šπ‘šπ‘šπ‘šπ‘šπ‘šπ‘šπ‘š(π‘₯π‘₯)βˆ’πœ‡πœ‡0 𝑠𝑠 βˆšπ‘šπ‘šοΏ½ (1) In equation (1), the overall mean was replaced using the sample variance s^2. Results showing significance levels much less than 0.05, it proves that parameters are statistically insignificant[7].all indicators in Table 2 are retained for the next step of the calculation. 2.2 Determination of weights and consistency tests for first-level indicators In this paper, a hierarchical single ranking method is used to calculate the importance of all the first level indicators in the criterion level for the target. There are two ways to achieve this, and this paper uses the square root method for calculation. In the judgment matrix A constructed in the previous step, the Mth power of the product of each row is calculated to obtain an M-dimensional vector π‘Šπ‘Šπ‘–π‘– π‘Šπ‘Šπ‘–π‘– = �∏ π‘Žπ‘Žπ‘–π‘–π‘–π‘–π‘šπ‘š 𝑖𝑖=1 π‘šπ‘š (2) In Equation (2), π‘Žπ‘Žπ‘–π‘–π‘–π‘–denotes the vectors in the judgement matrix A and m is the total number BCP Education & Psychology EED 2022 Volume 5 (2022) 27 Table 2 Hierarchical evaluation model Objective level Criteria level Sub-criteria level Program level Evaluation system For higher vocational institutions college Scale New Student Arrival Rate Data associated with 52 higher vocational institutions Number of full-time students (including undergraduates) The Average value of financial allocation per student Programs and Courses The Average value of teaching and research equipment per student (RMB) National-level Excellence Courses National-level Practical Training Bases Number of specialties National-level Professional Teaching Resources Library for Vocational Education Teacher Capability Number of teaching staff Number of full-time teachers Number of senior staff Number of associate professors Associate or above Percentage of senior title Percentage of graduate degree teachers Number of doctoral teachers The Proportion of doctoral teachers Number of National Master Teachers/Talent Program of the Ministry of Organization Number of national-level teaching achievement awards Number of national-level teaching teams Talent Cultivation Initial Employment Matching Rate The Average value of monthly income from initial employment Employer satisfaction Social Services Social training services (person-days) Training volume of foreign personnel (abroad) Technology service receipts (RMB million) Amount of technology transactions (RMB million) Research Strength Number of scientific research projects at national and ministerial levels Number of scientific research platforms at or above prefectural and municipal levels First-class awards at national level of row vectors. The ratio of each vector to the sum of all vectors is then calculated using equation (3), which is the weight of each indicator π‘Šπ‘Šπ‘–π‘– π‘Šπ‘Šπ‘–π‘– = π‘Šπ‘Šπ‘–π‘– βˆ‘ π‘Šπ‘Šπ‘–π‘– π‘šπ‘š 𝑖𝑖=1 (3) To ensure that the data is reasonable, a consistency test needs to be performed on each matrix. Use equation (4) to find the maximum characteristic roots of the judgment matrix and the corresponding eigenvectors[8]. πœ†πœ†π‘šπ‘šπ‘šπ‘šπ‘₯π‘₯ = 1 π‘šπ‘š βˆ‘ (π΄π΄π‘Šπ‘Š)𝑖𝑖 π‘Šπ‘Šπ‘–π‘– π‘šπ‘š 𝑖𝑖=1 (4) In Equation (4), πœ†πœ†π‘šπ‘šπ‘šπ‘šπ‘₯π‘₯is the largest characteristic root in the judgment matrix, and w is the weight value of the index after standardization[9]. Then the Consistency index is calculated as follows: 𝐢𝐢𝐢𝐢 = πœ†πœ†π‘šπ‘šπ‘šπ‘šπ‘šπ‘šβˆ’π‘šπ‘š π‘šπ‘šβˆ’1 (5) In Equation (5), n denotes the total number of indicators. The consistency ratio CR is then derived to determine whether there is a logical problem with the constructed matrix. 𝐢𝐢𝐢𝐢 = 𝐢𝐢𝐢𝐢 𝑅𝑅𝐢𝐢 (6) BCP Education & Psychology EED 2022 Volume 5 (2022) 28 Equation (6), the random consistency index (Random Index, RI) by looking up the table 3, we use Saaty simulated 50,000 times to obtain a table of random consistency values[10]. Table 3 Mean random consistency index Order 1 2 3 4 5 6 7 8 RI 0 0 0.52 0.89 1.11 1.25 1.35 1.4 The judgment matrix of the first-order indicators in this paper is six, and the CR value calculated by bringing in the RI value is 0.093, which is less than 0.1. The results show that the degree of consistency of the judgment matrix A is considered to be within the tolerable range, and the weight vector calculation can be carried out using the eigenvectors of A. The final calculation results are shown in Table (2.3). 2.3 Weight determination and consistency test of secondary indicators The weight determination method of the secondary indicators is the same as that of the primary indicators, using the hierarchical single ranking method to obtain the weights of each secondary indicator for the corresponding primary indicators respectively. The two-factor judgment matrix of the second-level indicators under each first-level indicator is constructed, and the CR value of the matrix is calculated to see if it meets the consistency requirements. In this paper, six CR values were obtained for the second-level indicators under the six first-level indicators, which were 0.0083, 0. 0042, 0. 0358, 0. 0858, 0. 0452 and 0.0653 respectively, all of which were less than 0.1 and met the consistency requirement. Finally, according to the weight relationship between the primary indicators and the total target, the weights of the secondary indicators were calculated relative to the total target, as shown in Table (2.4) 3. Analysis of Evaluation Results The data in this paper comes from the reports provided by institutions in the Ministry of Education's statistical annual report, and the evaluation analysis of 52 double-high institutions. The top 10 institutions with high overall evaluation scores are shown in Table 3.1. From Table 4 we can see that the development of vocational institutions on different evaluation indicators is uneven, and each institution has its strengths and weaknesses in different aspects. Taking the second-ranked institution as an example, we can find out by comparing the relevant information on the official website of this institution. This institution is a national key vocational institution and has been selected to participate in the national " High level development Program". Under the guidance of the program, the university has achieved remarkable results in terms of professional and curriculum development and talent training, indicating that it has achieved good results in teaching and learning, and the quality of employment of its students is high. However, when compared vertically with other institutions, it can be found that its score in social services is low, indicating that this school has a low contribution to social and technical training and occupational Science Popularization, and it has not effectively driven the development of regional vocational education- related economic industries.it also proves that the integration of industry and education and school- enterprise cooperation needs to be strengthened. Other vocational institutions can also adopt this similar analysis method for interpretation and analysis. BCP Education & Psychology EED 2022 Volume 5 (2022) 29 Table 4 Factor weights of criterion layer and indicator layer Criteria level/ Weights Sub-criteria level Weights Total target weighting college Scale/ 0.1309 New Student Arrival Rate 0.1436 0.01879724 Number of full-time students (including undergraduates) 0.4589 0.06007001 The Average value of financial allocation per student 0.3975 0.05203275 Programs and Course/ 0.2683 The Average value of teaching and research equipment per student (RMB) 0.3524 0.09454892 National-level Excellence Courses 0.3521 0.09446843 National-level Practical Training Bases 0.2135 0.05728205 Number of specialties 0.2342 0.06283586 National-level Professional Teaching Resources Library for Vocational Education 0.082 0.0220006 Teacher Capability/0.2577 Number of teaching staff 0.1125 0.02899125 Number of full-time teachers 0.1257 0.03239289 Number of senior staff 0.0622 0.01602894 Number of associate professors 0.0511 0.01316847 Associate or above 0.0602 0.01551354 Percentage of senior title 0.0536 0.01381272 Percentage of graduate degree teachers 0.0623 0.01605471 Number of doctoral teachers 0.0651 0.01677627 The Proportion of doctoral teachers 0.0686 0.01767822 Number of National Master Teachers/Talent Program of the Ministry of Organization 0.1132 0.02917164 Number of national-level teaching achievement awards 0.1022 0.02633694 Number of national-level teaching teams 0.1233 0.03177441 Talent Cultivation/0.1308 Initial Employment Matching Rate 0.3233 0.04228764 The Average value of monthly income from initial employment 0.4253 0.05562924 Employer satisfaction 0.2514 0.03288312 Social Services/0.1086 Social training services (person-days) 0.2311 0.02509746 Training volume of foreign personnel (abroad) 0.1051 0.01141386 Technology service receipts (RMB million) 0.3564 0.03870504 Amount of technology transactions (RMB million) 0.3074 0.03338364 Research Strength/0.1037 Number of scientific research projects at national and ministerial levels 0.5388 0.05587356 Number of scientific research platforms at or above prefectural and municipal levels 0.358 0.0371246 First-class awards at national level 0.1032 0.01070184 Table 5 Score and aRanking of each Vocational college Rank B1 B2 B3 B4 B5 B6 Score 1 0.829467 0.678584 0.608623 0.373939 0.453848 0.285442 0.5752829 2 0.377415 0.659737 0.584312 0.952648 0.008602 0.265686 0.5025287 3 0.316276 1.324412 0.013789 0.333413 0.095411 0.426404 0.4542657 4 0.630166 0.141483 0.597899 0.451599 0.846699 0.006333 0.4255478 5 0.276415 0.532531 0.634525 0.384583 0.195882 0.607607 0.4141541 6 0.187405 0.343579 0.718036 0.569949 0.166789 0.288191 0.3944140 7 0.182427 0.423676 0.748725 0.443132 0.008915 0.444984 0.3894282 8 0.275417 0.344244 0.483898 0.790677 0.218102 0.542368 0.3802196 9 0.37695 0.525249 0.180176 0.4349 0.056014 0.588216 0.2996664 10 0.480168 0.175933 0.164723 0.307537 0.800271 0.777778 0.2796412 BCP Education & Psychology EED 2022 Volume 5 (2022) 30 4. Summary and outlook In this paper, the method of AHP was used to evaluate the comprehensive development of vocational institutions. The indicators were selected using the expert scoring method to count the data involved in the relevant indicators. In order to ensure that the questionnaire has objectivity, a mean t- test was conducted on different questionnaires, and the significance level showed that the probability of falling into β€œI error” did not exceed the given level, and the indicators could be retained for calculation. In terms of weight determination, the square root method of the hierarchical single ranking approach was chosen to calculate the weights of the indicators in the criterion level relative to the target level separately and Weighting of indicators at sub-criterion level relative to indicators at guideline level. Then this paper calculates the weights of the second-level indicators relative to the total target according to the weights of the first-level indicators, and finally calculates the evaluation results of vocational institutions. It should be noted that the indicators in this paper are selected with the help of experts and related scholars’ scoring tables, and some more objective methods can be selected for analysis in the subsequent research. The weights of the secondary indicators in the sub-criteria level can also be calculated using the hierarchical total ranking method, which is not repeated here. Acknowledgements 1.Project support by the Central Universities Basic Scientific Research Business Fund "Research on Strategies and Paths of Online Civic Work under the Perspective of National Defense Education",(xxj012022002) 2.Project support by the Fundation of Guangzhou Panyu Polytechnic Humanities and Social Sciences Projects"Constructing the periodical evaluation system of Vocational Colleges and universities in the new era",(2021060903) 3.Project support by the Fundation of the 14th Five-Year Plan for the Development of Philosophy and Social Science in Guangzhou,"Research on the mechanism and path of reforming the comprehensive evaluation of academic journals", (2021GZGJ261) References [1] O Loyola-GonzΓ‘lez, et al. A Contrast Pattern-Based Scientometric Study of the QS World University Ranking. (IEEE Access 8,2020),p.1-1. [2] Gratzl, S. , et al. LineUp: Visual Analysis of Multi-Attribute Rankings. (IEEE Transactions on Visualization & Computer Graphics 19.12,2013),p.2277-2286. [3] Cheng, Ewl , and H. Li . Analytic hierarchy process. (Measuring Business Excellence 5.3,1997),p.30 - 37. [4] Saaty, T. L. . Decision making with the analytic hierarchy process.(International Journal of Services Sciences 1.1,2008),p.83-84. [5] Vaidya, O. S. , and S. Kumar . Analytic hierarchy process: An overview of applications. (European Journal of Operational Research 169.1,2006),p.1-29. [6] Liu, J. J. , and Q. Liu . Design on the Evaluation Index System for Education Quality of the Self-Study Examination of Higher Education. (Journal of Tongren University 57a.1,2011),p.22–33. [7] Lury, D. A. , and R. A. Fisher . Statistical Methods for Research Workers. by R. A. Fisher. (Botanical Gazette 87.3,1929),p.229. [8] Lin Lin. An Evaluation System and Its Model for Educational Competitiveness of Universities, (International Journal of Emerging Technologies in Learning, 2020),p.1 [9] Maoxin Su, Congcong Li, Yiguo Xue, Peng Wang, Kai Cheng, Yimin Liu. Engineering application of fuzzy evaluation based on comprehensive weight in the selection of geophysical prospecting methods, (Earth Science Informatics, 2021)No.3,p21-22. BCP Education & Psychology EED 2022 Volume 5 (2022) 31 [10] Saaty T L.Relative measurement and its generalization in decision making: Why pairwise comparisons are central in mathematics for the measurement of intangible factorsβ€”The Analytic Hierarchy/Network Process. (Review of the Royal Spanish Academy of Sciences A: Mathematics,2008), p.102 1. Introduction 2. The theory of evaluation system of higher education institutions 2.1 Constructing a hierarchical evaluation model 2.2 Determination of weights and consistency tests for first-level indicators 2.3 Weight determination and consistency test of secondary indicators 3. Analysis of Evaluation Results 4. Summary and outlook Acknowledgements References