IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 64 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 THE USE OF THE ANALYTIC HIERARCHY PROCESS FOR ADMISSION TO AN AIR TRAFFIC CONTROL DEPARTMENT Mustafa Özdemir* 1 Department of Air Traffic Control, Anadolu University Eskişehir, Turkey mozdemir26@anadolu.edu.tr Müjgan Sağır Özdemir Department of Industrial Engineering, Eskisehir Osmangazi University Eskişehir, Turkey mujgan.sagir@gmail.com ABSTRACT Meeting the need for qualified staff in the field of air traffic control depends primarily on selection of appropriate candidates among many applicants. The selection of competent air traffic controllers, who play a vital role in the air traffic system, is of critical importance and is usually conducted through multi-stage examinations. It is also important to use correct methods in the selection process in order to identify the most suitable candidates. In particular, the application of subjective examinations, such as interviews in a standard way, and determination and standardization of criteria can assist in selecting the right candidates. Within this context, the student selection process in an air traffic control department is investigated, with the aim of supporting the interview examination through the Analytic Hierarchy Process. The criteria weighting is determined by expert opinion. Thirty-nine candidates are ranked according to the AHP, with the current and proposed rankings being compared. As a result, the ranking has changed significantly with the proposed approach. Keywords: Analytic Hierarchy Process; air traffic control; interview 1. Introduction Expertise and ability play an important role in the aviation industry. This role increases the responsibility of institutions providing education on the related subject. In particular, qualified staff in areas such as pilotage or air traffic control (ATC) have a direct impact on the efficiency of education, as well as work in this area, and the successful fulfillment of responsibilities that are undertaken. The selection of staff for the intended purpose is the first and one of the most important processes in the training of qualified staff. Therefore, it is necessary to comply with new conditions and to develop the selection Acknowledgements: The authors would like to express their gratitude to Prof. Aydan Cavcar and Prof. Oznur Usanmaz for their help granting the permission for and providing the expertise to conduct the surveys at Air Traffic Control Department, Faculty of Aeronautics and Astronautics. *Corresponding author IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 65 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 process. In this study, the air traffic controller (ATCO) candidate selection process that was applied in 2014 is investigated and a new methodology for the interview examination is proposed, based on the Analytic Hierarchy Process (AHP). The aim of the study is to select the most appropriate candidates for the ATC department by minimizing any shortcomings of the interview process and to compare the proposed solution with the current one. There have been a number of studies concerning personnel and student selection using the AHP. Altunok at al. (2010) suggest multi-criteria decision-making (MCDM) methods for postgraduate student selection. They compare results obtained by the AHP, Weighted Product (WP) and the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS). In order to determine the most appropriate method, they compute the total deviation between student’s cumulative grade point average (CGPA) and scores calculated by each method. As a result, they state that the AHP is the most appropriate method among the investigated techniques, as it has the least total deviation from the CGPA (Altunoluk et al, 2010). Gibney and Shang (2007) propose the AHP method for a dean selection process, which consists of qualitative criteria, such as candor and honesty, trustworthiness, inclusive style of leadership and ability to develop vision. They state that the AHP can be applied for personnel selection in academia since it is a powerful tool for evaluating intangible criteria, and it can reflect judgments about feelings and emotions (Gibney& Shang, 2007). Fuzzy AHP is also applied for similar problems. Taşkın et al. (2013) use the fuzzy AHP to score and rank students during oral examinations for Erasmus Student Mobility, with the results obtained from the fuzzy AHP being compared with the results of the Rubric, which is a scoring tool that lists criteria for a piece of work. The authors state that the rank obtained by the fuzzy AHP is more satisfactory for decision-makers because it is more flexible, i.e., the criteria weightings can easily be changed (Taşkın, Üstün & Deliktaş, 2013). In another study, Kasim et al. (2012) use the AHP method to determine the importance and prioritization of each criterion for the teacher candidate selection process. They identify three main criteria; namely, content of knowledge, communication skill and personality, based on the literature and expert opinion. The weights of the criteria were determined by a group of experts using pairwise comparisons (Kasim et al, 2012). Taylor et al. (1998) conducted a study to research a college dean at Texas A&M University using the AHP method. A committee, comprising faculty, administrators, and community representatives, identified four attributes including experience with AACSB (the Association to Advance Collegiate Schools of Business) accreditation processes, experience in an administrative position, a good publication record and proven ability at fundraising. In their study, there were thirty-three candidates, and they divided the candidates into three groups with respect to each attribute in order to reduce the number of pairwise comparisons. They assumed that within these groups the candidates were approximately equal with respect to that attribute. This significantly reduces the evaluation time of decision-makers, especially when there are a large number of alternatives (Taylor, Ketcham & Hoffman, 1998). Norddin et al. (2012) investigated a new lecturer’s selection model to overcome shortcomings of the current selection process, such as difficulties in making comparisons among candidates and correctly evaluating their performance at all stages of the process before a final decision. They state that they chose the AHP method, since it possesses advantages over other MCDMs, and that the criteria were identified as a mock teaching performance, a face-to-face interview, and academic qualifications (Norddin, Ibrahim & Aziz, 2012). IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 66 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 Most decision problems involving qualitative criteria have difficulty combining with quantitative criteria. When quantitative techniques are used, intangibles may have to be ignored. However, the AHP is a decision-making theory of measurement through pairwise comparisons which helps decision-makers to select the best alternatives or to rank the alternatives by taking into account qualitative and quantitative aspects of decisions (Bogdanovic & Miletic, 2014; Saaty, 2008). It is especially useful for evaluating complex multi-attribute alternatives involving qualitative criteria (Kwong &Bai, 2002; Saaty & Sagir, 2009). In addition, when there are many alternatives, rating is a more effective and practical method to evaluate the alternatives. In this study, an AHP model based on a rating scale is proposed to be used in the interview stage where candidates are evaluated based on their qualitative characteristics. The ratings mode is explained in greater detail later. The use of the AHP is not a new approach in student selection problems. However, it has not yet been implemented in the area of ATC to standardize and minimize the shortcomings of interviews. It is also believed that this study will improve the student selection process in the related department, and that the criteria and prioritization identified in the study will contribute to personnel selection problems with similar criteria. 2. Problem statement and a decision support model for the student selection process In civil and military aviation, a great majority of accidents are caused by human error (Wiegmann & Shappell, 2001). For this reason, the selection of qualified ATCOs is especially important in order to improve safety in flight operations. One measure that can be taken to achieve this is to develop and implement an appropriate selection process or to overcome deficiencies in the existing system. Interviews are frequently used as a method for assessing the suitability of candidates for a particular profession by questionnaire and giving points to evaluate their discernment, comprehension ability, overall appearance, appropriateness, competence, and suitability of behavior and reactions. Selection of candidates by interview only can prevent selection of the most appropriate candidate. This can lead to one being skeptical of the results because interviews are considerably open to subjective evaluation and are often far from objective. Furthermore, appealing interview results are common, and it becomes a burden for the interviewers and institutions conducting the interview. Therefore, conducting the interview examination in a more systematic way can avoid possible inconveniences and lead to a more effective selection procedure. 2.1 Current system The department of ATC admits its students according to scores achieved in a special talent exam which is held every year. It consists of four stages (four examinations) including mathematics, aptitude, interview and simulations. The admissions process is organized by the department management. The candidates are evaluated based on a method of elimination. On the other hand, due to the elimination method, candidates who are going to be evaluated by interviewers are those who have already maintained their chances of being selected; i.e. candidates passing the first two examinations can IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 67 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 participate in the interview procedure. The interview panel consists of the decision- makers at this phase, but their task is limited to this stage. In addition, we can say that each candidate is evaluated with respect to different criterion at each step. In the study, the special talent examination applied in 2014 is investigated. The candidates need to meet certain conditions in order to participate in this examination. They are required to be under the age of 21 years and have received at least 250 points (increased to 280 in 2016) in the university entrance examination, conducted by the Student Selection and Placement Center (OSYM), in order to apply for the department. In addition, the candidates should have no physical disabilities, speech disorders, hearing loss, or color blindness. 194 candidates fulfilling these requirements were entitled to participate in the examinations. Figure 1 illustrates the entire selection and elimination process. The number of candidates eligible to take the mathematics examination was 194. Following the mathematics examination, 94 were eliminated and 100 candidates took the aptitude examination. Following this, 60 of the 100 candidates were eliminated. The remaining candidates went on to interview. Finally, 20 remaining candidates were ranked according to their scores. At the end, the first 15 of the 20 candidates were determined as principals with the next 5 candidates being determined as alternates. Figure 1 Current selection process A commission consisting of five members conduct the interview stage. Every member evaluates candidates based on various criteria and gives a score in order to measure a candidate's ability for the profession. Each candidate achieves five scores from every member of the commission. Under the current system, the interview grade is calculated by arithmetic mean, with 40 candidates being graded by this method. All candidates Math exam (194 candidates) Aptitude exam (100 candidates) Interview exam (40 candidates) Simulation exam (30 candidates) Principal: 15 Alternate: 5 94 candidates eliminated 60 candidates eliminated 10 candidates eliminated 10 candidates eliminated IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 68 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 2.2 Proposed approach The proposed approach is applied to the interview stage, since it has qualitative factors and subjective assessment. The other stages could be considered as being straightforward, and are based on quantitative and objective assessment. The study is conducted in two stages; firstly, criteria and intensity levels are determined, and their weights are calculated by pairwise comparison of the AHP, then the alternatives are evaluated by the rating method using Expert Choice software. 2.3 The Analytic Hierarchy Process The AHP is a tool developed by Thomas Saaty to help decision-makers in complex decision problems involving many attributes of varying degrees of subjectivity (Taylor, Ketcham & Hoffman, 1998). The AHP is applied in different decision problems, such as planning, resource allocation, resolving conflict, optimization, and so on (Vaidya & Kumar, 2006). When applying the AHP in a decision problem, the factors affecting the decision are transformed into a hierarchy consisting of a goal, criteria and alternatives. The goal of the decision problem is stated at the top of the hierarchy, and criteria are placed at the bottom of the goal. Alternatives are at the lowest level of the hierarchy (An, Kim & Kang, 2007). After the hierarchy has been created, decision-makers determine the relative importance of the main and sub-criteria by way of a pairwise comparison process. In this process, the criteria at the same level of the hierarchy are compared and scored using Saaty's 1-9 scale of pairwise comparisons shown in Table 1. For example, suppose that two of the criteria in a decision problem are called A and B. If criterion A has strong importance over criterion B with respect to the goal, then it will be shown by the number 5 in the pairwise comparison matrix. If criterion B has strong importance over criterion A, it will be shown by 1/5. In this way, pairwise comparison matrices are obtained. Next, each element of the pairwise comparison matrix is divided by the sum of its corresponding column elements. Finally, the relative weights of the criteria are calculated by taking the average of each row (Al-Harbi, 2001). IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 69 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 Table 1 The fundamental scale of absolute numbers (Saaty, 2008). Intensity of Importance Definition Explanation 1 Equal Importance Two activities contribute equally to the objective 2 Weak or slight 3 Moderate importance Experience and judgment slightly favor one activity over another 4 Moderate plus 5 Strong importance Experience and judgment strongly favor one activity over another 6 Strong plus 7 Very strong or demonstrated importance An activity is favored very strongly over another, its dominance demonstrated in practice 8 Very, very strong 9 Extreme importance The evidence favoring one activity over another is of the highest possible order of affirmation For the final step, on evaluating the alternatives, each alternative is compared with the other alternatives with respect to each criterion and sub‐criterion. The pairwise comparison process allows decision-makers to measure the inconsistency ratio at each level of the hierarchy. If this exceeds 0.10, judgments should be re-examined. If the inconsistency is at an acceptable level, the alternatives are ranked from the most preferred to the least preferred (Saaty, 1987; Swiercz & Ezzedeen, 2001). The use of pairwise comparison in this final step becomes difficult as the number of alternatives increases, because the number of pairwise comparisons reaches an unreasonable level at large numbers of alternatives. If each candidate is compared to every other candidate, the total number of comparisons that must be performed is: 𝑇 = 𝑛(𝑛−1) 2 (1) where T is the total number of comparisons, and n is the number of alternatives (Taylor, Ketcham & Hoffman, 1998). In our case, n is 39. In the case of using relative mode, the total number of comparisons required would be 741, and this process must be applied for nine criteria. In such cases, the use of the rating method would be more appropriate. In the rating method, alternatives are not compared pairwise, but are evaluated independently using intensity levels for each criterion, and this significantly reduces the number of pair-wise comparisons. The two methods, relative and rating modes, do not deliver the same priorities exactly. The relative method where alternatives are compared with each other under the various criteria is more accurate. The ratings method has the advantage that one can rate large numbers of alternatives rather quickly, and the results are adequately close (Saaty, 2008). In light of the above explanations, we prefer using the rating mode to evaluate the alternatives for this study. IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 70 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 2.4 Determination of the criteria ATCOs are responsible for ensuring the safe, regular and rapid flow of aircraft traffic in airspace and at airports (ICAO, 2007). Their tasks are based mainly on cognitive skills, such as knowledge, attention, memory, judgment and evaluation, reasoning and computation, problem-solving, fast and accurate decision-making under stress, quick verbal communication skills and so on. (EUROCONTROL, 1998). They consistently carry out their duties in coordination and cooperation with pilots, technical personnel, management and other supervisors. In such interactions, accurate and rapid communication becomes crucial. In particular, an error that may occur during communication between a pilot and an air traffic controller may jeopardize flight safety. Therefore, the communication skills and diction of the candidates are vital for a proper understanding of instructions. Non-verbal communication skills make verbal communication more effective and are considered to be important during an interview in accordance with expert opinion. An air traffic controller’s job is a profession which comes with a high level of stress. Air traffic controllers need to make and apply a lot of decisions within a limited time in an intensive and isolated working environment. Such factors cause physical and mental stress on controllers. Therefore, fatigue and pressure from high workplace stress negatively impact professional skills, responsiveness and prudence of ATCOs. As a result, one of the most important characteristics of those wishing to enter this profession is the ability to cope with stress and fulfill their duty under heavy stress (Jou, Kuo & Tang, 2013). The air traffic system is a dynamic environment with only short periods of time available for control; any lack of attention may cause serious consequences. It is also difficult to maintain attention while traffic is being monitored over a long period. A controller should not confuse tasks when doing two different things simultaneously, such as monitoring traffic and following a written procedure. Consequently, candidates are expected to have a high level of attention (EUROCONTROL, 2010). The job interest criterion aims to test whether candidates really want the job and have a basic knowledge of ATC. Self-reliance refers to the confidence that a candidate has in his/her ability to manage the work and the impression gained by interviewers in this regard. Previously, five items of criteria have been used to rank candidates. These can be defined as communication skills, stress control, attention level, job interest and self-reliance. In the proposed approach, the criteria have been revised and expanded by noting the opinions of interview commission members. We suggest that the communication skill criterion should be divided into sub-criteria, because it contains many different qualifications. Therefore, five main criteria and eight sub-criteria have been determined. After the new criteria set was identified, thirteen experts were surveyed in order to determine the criteria’s relative importance. An example of the survey is given in Appendix 1. The professions and years of experience of the experts are given in Table 2. The importance level of the criteria according to each other are obtained by geometric mean of the scores given by the experts. IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 71 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 Table 2 Professions and years of experience of the experts Experts Area of expertise Years of experience 1 Licensed ATCO, ATC instructor, and member of interview commission 25 2 Licensed ATCO, ATC instructor, and member of interview commission 19 3 Licensed ATCO, and ATC instructor 13 4 Licensed ATCO, and ATC instructor 9 5 Licensed ATCO, and ATC instructor 7 6 Licensed ATCO, actively works as an ATCO 10 7 Licensed ATCO, actively works as an ATCO 5 8 Academician in ATC, and member of the interview commission 29 9 Academician in ATC, and member of the interview commission 27 10 Academician in ATC, and member of the interview commission 17 11 Academician in ATC 5 12 Academician in ATC 4 13 Academician in ATC 3 2.5 The proposed AHP model In order to implement the rating method, intensity levels should be created for each criterion. The criterion with higher weightings are divided into further levels in order to achieve more precise assessments. After the identification of the criteria and the intensities for each criterion, an AHP model is developed, as shown in Figure 2. The model consists of the goal, the criteria and the weightings obtained from paired comparisons, and the intensities for each criterion. The rating categories are prioritized by pair-wise comparison. For example, the rating categories for the manner of speaking criterion in Figure 2, is decided as very good, good, fair and poor; we compare these for preference using a pair-wise comparison matrix in the usual way, as given in Table 3. The rating categories for all the covering criteria and their priorities are established in a similar way. Next, the idealized values are obtained by dividing by the largest value in the vector of priorities, derived from their paired comparisons matrix. The idealized priorities are always used for ratings, and this allows the outstanding to receive the full criterion weighting. Alternatives are then evaluated by selecting the appropriate rating category on each criterion. IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 72 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 Table 3 Deriving priorities for ratings on manner of speaking Manner of Speaking Very good Good Fair Poor Priorities Idealized Priorities Very good 1 2 3 4 0.467 1.000 Good 1/2 1 2 3 0.277 0.593 Fair 1/3 1/2 1 2 0.160 0.343 Poor 1/4 1/3 1/2 1 0.095 0.204 In the model, there are four different intensity levels, with the highest inconsistency ratio among them being obtained as 0.02. A final matrix inconsistency ratio for the criteria is obtained equal to 0.006, which indicates that the model is well-structured. Of the five main criteria selected for the AHP model, the most important is stress control, and the least important is job interest. A candidate should simultaneously satisfy a number of requirements to receive a high score in communication skills, since it consists of several sub-criteria. IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 73 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 Goal: Student selection for air traffic control department Attention level (0.254) Self-reliance (0.173) Job interest (0.100) Communication skills (0.217) Stress control (0.256) Poor Excellent Good Above average Average Below average 1.000 0.655 0.417 0.263 0.168 0.112 1.000 0.655 0.417 0.263 0.168 0.112 1.000 0.593 0.343 0.204 1.000 0.593 0.343 0.204 1.000 0.627 0.382 0.232 0.148 1.000 0.627 0.382 0.232 0.148 1.000 0.550 0.303 1.000 0.550 0.303 1.000 0 0.593 0.343 0.204 Average Excellent Good Above average Below average Poor Very good Good Fair Poor Very good Good Fair Poor Verbal communication (0.798) Non-verbal communication (0.202) Manner of speaking (0.424) Appearance (0.330) Body language (0.670) Tone of voice (0.364) Pronunciation (0.636) Diction (0.576) Very good Good Fair Poor Very good Good Acceptable Poor Very poor Very good Good Acceptabl e Poor Very poor Good Fair Bad Good Fair Bad Figure 2 AHP model for student selection problem for air traffic control department 2.6 Application of rating method During the interview phase, a number of procedures are carried out to assess a candidate according to each criterion. For example, candidates are asked basic questions about the ATC in order to determine their job interest, or they are asked to read a specified text that can reveal their level of communication skills, such as pronunciation and tone of voice. Following the same procedure described above, based on the rating method, the candidates are re-evaluated using the criteria weights and the intensity levels associated with each criterion. In applying the rating method, there are two parts in our individual score calculation for each candidate, and we divide the criteria in Figure 2 into two sets as follows: IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 74 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503  The criteria set-1: It consists of the criteria that have already been used in the current interview process, namely stress control, attention level, job interest and self-reliance.  The criteria set-2: It consists of the newly added criteria, i.e. sub-criteria of communication skills. We already have the score (1-100) of each candidate for each item of the criteria set-1. We also define the intensity levels for these criteria. In the first part, in order to transform the numerical score of a candidate for the criteria set-1 into the intensity levels, we use the following methodology. If we assume that a candidate’s score is 60 for job interest in 2014, we determine the intervals for this criterion depending on the number of the intensity levels. Job interest includes four levels, so the intervals are defined as 0-25 (poor), 26-50 (fair), 51-75 (good) and 76-100 (very good). The intervals for the other criteria in the criteria set-1 are established in a similar way. Therefore, since this candidate has a score of 60 in the job interest criterion, then, this candidate's score category is determined as ‘good’ according to the uniform category above. The calculation is then conducted by using the numerical equivalence of ‘good’, which is 0.593 for the related criterion (see Figure 2). This way we are able to transform the previous scores into our categories. Since we obtain the priority for each intensity level by paired comparisons (i.e. 0.593 for ‘good’ here), these adjusted values are more appropriate to use. We use this simple transformation to protect the original judgements of the experts for the criteria set-1, instead of asking them their opinions again. The second part is related to the rating of the alternatives in terms of each item of the criteria set-2. In this part, the evaluations are carried out by the experts through watching the videos recorded during the interview. These new criteria were not used before so the evaluation scores of the candidates for these criteria are not available. Finally, the score of each candidate for each criterion is obtained by multiplying the related criterion weighting by the corresponding intensity level. A candidate's final score is calculated by totaling these scores. For example, if the evaluations of candidate 1 (C1), with respect to manner of speaking, pronunciation, tone of voice, body language, appearance, stress control, attention level, job interest and self-reliance are good, good, acceptable, bad, good, poor, poor, poor and fair, respectively, the final score for this candidate would be obtained as follows: (0.073×0.593) + (0.064×0.627) + (0.036×0.382) + (0.029×0.303) + (0.014×1.00) + (0.256×0.112) + (0.254×0.112) + (0.1×0.204) + (0.173×0.343) = 0.257 Therefore, a new ranking based on the AHP is obtained, and the evaluation of other candidates is given in Appendix 2. 2.7 Results and sensitivity analysis The rankings of candidates according to interview and AHP grade are listed in Table 4. It can be seen that the rankings of nine candidates have not changed while the rankings of five candidates have changed the most (C3, C16, C26, C35 and C36). It is believed that the main reason for the differences between the interview and the AHP ranking is the differences in the criteria weightings used by both methods. In the proposed method, for instance, the weighting of stress control is 25.6% while the weighting of job interest is IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 75 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 10%. However, these criteria were equally important in the current process. In addition, we propose to use more criteria by constructing an AHP model for the problem under consideration. As previously stated, the first 30 candidates according to the ranking obtained in the interview phase proceed to the next stage, and the remaining candidates are eliminated. The candidates who are eliminated in the interview process are C4, C18, C20, C23 C24, C26, C34, C38 and C39. With the AHP, not only is the ranking changed, but also two of the candidates to be eliminated are changed. If the AHP method was applied, C10 and C36 would be eliminated instead of C18 and C26. Although only two eliminated candidates are changed according to these two methods, the grades obtained from the interview and AHP are quite different from each other. For example, in the interview process, C36 is the latest candidate with a right to pass to the next stage ranking 30 th with 26.40 points while, in the AHP method, this candidate is C32 ranking 30 th with 23.96 points. This situation has created a number of differences in the rankings. In addition, these grades are also important, since they are taken into account in a final assessment to determine successful candidates. The criteria with a high level of importance have more influence on the ranking of candidates than those with low levels of importance. For instance, a candidate can be placed among the top fifteen if he/she gets a full score in one item of criteria including stress control, attention level and communication skills; however, self-reliance and job interest are insufficient by themselves. IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 76 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 Table 4 Comparison of the candidate’s interview and AHP grades Candidate Number Interview grade Ranking based on interview AHP grade Ranking based on AHP Difference C1 33.12 22 25.74 20 2 C2 77.80 3 73.97 3 0 C3 53.40 8 33.11 13 -5 C4 26.12 32 21.87 36 -4 C5 32.80 23 26.67 19 4 C6 35.68 19 27.43 18 1 C7 67.80 5 47.49 5 0 C8 49.04 10 36.07 10 0 C9 34.32 21 25.39 22 -1 C10 27.24 29 23.54 33 -4 C11 28.88 26 24.30 27 -1 C12 80.00 2 77.02 2 0 C13 30.40 24 24.96 25 -1 C14 53.20 9 37.70 8 1 C15 85.00 1 87.59 1 0 C16 42.60 12 31.43 17 -5 C17 41.44 14 32.90 14 0 C18 26.28 31 24.18 29 2 C19 53.92 6 43.29 7 -1 C20 24.88 34 23.84 31 3 C21 70.80 4 55.68 4 0 C22 53.84 7 44.80 6 1 C23 22.80 37 22.81 34 3 C24 24.56 36 23.66 32 4 C25 37.08 17 32.67 15 2 C26 25.44 33 24.29 28 5 C27 30.00 25 25.63 21 4 C28 34.60 20 25.07 24 -4 C29 41.64 13 35.28 11 2 C30 46.60 11 37.03 9 2 C31 40.80 15 33.37 12 3 C32 28.80 27 23.96 30 -3 C33 39.80 16 32.24 16 0 C34 24.80 35 20.02 39 -4 C35 36.40 18 25.31 23 -5 C36 26.40 30 22.47 35 -5 C37 28.12 28 24.31 26 2 C38 16.00 38 20.91 38 0 C39 14.00 39 21.75 37 2 IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 77 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 A sensitivity analysis allows decision-makers to test the robustness of the rankings for the alternatives. To perform a sensitivity analysis, it is necessary to change the weighting of a criterion and see how they affect the overall priorities of the alternatives. Increasing or decreasing a criterion weighting changes the weightings of the other criteria, proportionally. The sensitivity analysis is conducted in three scenarios: (a) when all the main criteria have the same weighting, i.e. 0.20 each; (b) increasing the weighting of stress control by 10%; and (c) increasing the weighting of job interest by 10%. When the weightings of the main criteria are taken as equal, the weightings of stress control, attention level and communication skills are decreased, while the others are increased. In Figures 3 and 4, bars above and below the axis represent the increase and decrease in the ranking, respectively. For example, C20 is ranked as 31st in normal ranking while his raking is raised to 28th when we assume that all the main criteria have an equal weighting, and the difference is 3. Candidates such as C11, C13 and C20 are raised in the ranking as can be seen in Figure 3. On the other hand, the rankings of certain candidates, such as C25, C26 and C28, who are categorized as ‘poor’ in job interest and self-reliance, are reduced. As a result, the rankings of twelve candidates are changed. Figure 3 Sensitivity analysis when all the main criteria have equal weighting Increasing the weighting of stress control by 10% changes the rankings of 10 candidates as shown in Figure 4. However, changing the weighting of job interest by 10% does not make any changes in the rankings, since it is a criterion with low weighting, and thus its reflection on the scores is limited. -4 -3 -2 -1 0 1 2 3 4 C 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 C 9 C 1 0 C 1 1 C 1 2 C 1 3 C 1 4 C 1 5 C 1 6 C 1 7 C 1 8 C 1 9 C 2 0 C 2 1 C 2 2 C 2 3 C 2 4 C 2 5 C 2 6 C 2 7 C 2 8 C 2 9 C 3 0 C 3 1 C 3 2 C 3 3 C 3 4 C 3 5 C 3 6 C 3 7 C 3 8 C 3 9 D if fe re n c e IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 78 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 Figure 4 Sensitivity analysis of alternatives when stress control is increased by %10 3. Discussion and conclusions The main concern about the interview process that consists of many qualitative factors is the objectivity of the process. It is not easy to evaluate qualitative factors objectively because an interviewer's overall impression of a candidate may influence the interviewer's feelings and thoughts about that candidate's character. To prevent such situations, it is necessary to determine the criteria and criteria weightings by appropriate methods, and to systematize the interview process. In the study, a method based on the AHP is proposed for use in the interview examination of an air traffic control department. As a result, thirty-nine candidates are re-ranked according to this method and compared with the current ranking. In addition, a sensitivity analysis is conducted for different scenarios. In the current interview process, each criterion is considered to be equally weighted. However, in this study, the weightings of the criteria are determined by expert opinions, and experts have noted that, for ATCOs, certain criteria, such as stress control and attention level, are far more important than others. Therefore, the criteria with higher levels of importance have greater influence on the ranking of candidates than those with lower levels of importance. As a result, the use of the AHP at the interview phase may ensure that more appropriate candidates are selected. -4 -3 -2 -1 0 1 2 3 4 C 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 C 9 C 1 0 C 1 1 C 1 2 C 1 3 C 1 4 C 1 5 C 1 6 C 1 7 C 1 8 C 1 9 C 2 0 C 2 1 C 2 2 C 2 3 C 2 4 C 2 5 C 2 6 C 2 7 C 2 8 C 2 9 C 3 0 C 3 1 C 3 2 C 3 3 C 3 4 C 3 5 C 3 6 C 3 7 C 3 8 C 3 9 D if fe re n c e IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 79 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 APPENDIX I The comparison of the criteria according to the importance scale from 1 to 9 is shown in the following form. Example A 9 8 7 6  4 3 2 1 2 3 4 5 6 7 8 9 B The criterion A is five times more important than the criterion B A 9 8 7 6 5 4 3 2 1 2  4 5 6 7 8 9 B The criterion B is three times more important than the criterion A A 9 8 7 6 5 4 3 2  2 3 4 5 6 7 8 9 B The criteria A and B have equal importance Paired Comparisons Communication 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Stress Control Skills Communication 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Attention Level Skills Communication 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Job Interest Skills Communication 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Self-reliance Skills Stress Control 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Attention Level Stress Control 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Job Interest Stress Control 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Self-reliance Attention Level 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Job Interest Attention Level 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Self-reliance Job Interest 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Self-reliance Verbal 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Nonverbal Communication Communication Manner of 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Diction Speaking Pronunciation 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Tone of Voice Body Language 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9 Appearance IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 80 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 APPENDIX II The values in the column titled “total” indicate the total score that the candidates have gained from the all criteria. L and G indicate, respectively, local and global weights of the criteria. IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 81 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 REFERENCES Al-Harbi, K.M.A.S. (2001). Application of the AHP in project management. International Journal of Project Management, 19(1), 19-27. Doi: https://doi.org/10.1016/S0263-7863(99)00038-1 Altunoluk, T., Özpeynirci, Ö., Kazançoğlu, Y., & Yılmaz, R. (2010). Comparative analysis of multicriteria decision making methods for postgraduate student selection. Eurasian Journal of Educational Research, 40, 1-15. An, S., Kim, G., & Kang, K. (2007). A case-based reasoning cost estimating model using experience by Analytic Hierarchy Process. Building and Environment, 42(2007), 2573- 2579. Doi: https://doi.org/10.1016/j.buildenv.2006.06.007 Bogdanovic, D., & Miletic, S. (2014). Personnel evaluation and selection by multicriteria decision making method, Economic Computation and Economic Cybernetics Studies and Research, 48(3), 179-196. European Organization for the Safety of Air Navigation (EUROCONTROL). (1998). Integrated task and job analysis of air traffic controllers, Belgium, Brussels: EUROCONTROL. European Organization for the Safety of Air Navigation (EUROCONTROL). (2010). Human performance in air traffic management safety. Belgium, Brussels: EUROCONTROL. Gibney, R., & Shang, J. (2007). Decision making in academia: A case of the dean selection process. Mathematical and Computer Modelling, 46, 1030-1040. Doi: https://doi.org/10.1016/j.mcm.2007.03.024 International Civil Aviation Organization (ICAO). (2007). Procedures for air navigation services air traffic management (Doc 4444). Canada, Montreal: ICAO. Jou, R.C., Kuo, C.W., & Tang, M.L. (2013). A study of job stress and turnover tendency among air traffic controllers: The mediating effects of job satisfaction, Transportation Research Part E, 57 (2013), 95-10. Doi: https://doi.org/10.1016/j.tre.2013.01.009 Kasim, M., Ramli, R., Ibrahim, H., Ghazali, M., Kamal, F., & Vikneswari S. (2012). Prioritization of criteria in teacher-candidate selection process by a pairwise comparison method, Knowledge Management International Conference. Johor Bahru, Malaysia. Kwong, C.K., & Bai, H. (2002). A fuzzy AHP approach to the determination of importance weights of customer requirements in quality function deployment. Journal of Intelligent Manufacturing, 13(5), 367-377. Norddin, N.I., Ibrahim, K., & Aziz, A.H. (2012). Selecting new lecturers using the Analytical Hierarchy Process (AHP). 2012 International Conference on Statistics in Science, Business and Engineering (ICSSBE). Langkawi, Malaysia. Doi: 10.1109/ICSSBE.2012.6396585 IJAHP Article: Özdemir M., Özdemir MS/ The use of the Analytic Hierarchy Process for admission to an air traffic control department International Journal of the Analytic Hierarchy Process 82 Vol. 10 Issue 1 2018 ISSN 1936-6744 https://doi.org/10.13033/ijahp.v10i1.503 Saaty, T.L. (2008). Decision making with the Analytic Hierarchy Process. International Journal of Services Sciences, 1(1), 83-98. Doi: 10.1504/IJSSCI.2008.017590 Saaty, T.L. (1987). The Analytic Hierarchy Process-what it is and how it is used. Mathematical Modelling, 9(3-5), 161-176. Doi: https://doi.org/10.1016/0270- 0255(87)90473-8 Saaty, T.L., & Sagir, M. (2009). Extending the measurement of tangibles to intangibles. International Journal of Information Technology & Decision Making, 8(01), 7-27. Doi: https://doi.org/10.1142/S0219622009003247 Swiercz, P., & Ezzedeen, S. (2001). From sorcery to science: AHP, a powerful new tool for executive selection. HR. Human Resource Planning, 24(3), 15-26. Taşkın, H., Üstün, Ü., & Deliktaş, D. (2013). Fuzzy MCDM approach for oral examination in erasmus student selection process. DPÜ Fen Bilimleri Enstitüsü Dergisi, 32, 21-40. Taylor, F., Ketcham, A., & Hoffman, D. (1998). Personnel evaluation with AHP. Management Decision, 36, 679-685. Doi: https://doi.org/10.1108/00251749810245336 Vaidya, O.S., & Kumar, S. (2006). Analytic Hierarchy Process: An overview of applications. European Journal of Operational Research, 169(1), 1-29. Doi: https://doi.org/10.1016/j.ejor.2004.04.028 Wiegmann, D.A., & Shappell, S.A. (2001). Human error analysis of commercial aviation accidents: Application of the Human Factors Analysis and Classification System (HFACS). Aviation, Space, and Environmental Medicine, 72(11), 1006-1016.