Microsoft Word - 109-758-1 WORD N.docx New articles in this journal are licensed under a Creative Commons Attribution 4.0 United States License. This journal is published by the University Library System of the University of Pittsburgh as part of its D-Scribe Digital Publishing Program and is cosponsored by the University of Pittsburgh Press. This journal is supported by the Carnegie Project on the Education Doctorate: A Knowledge Forum on the EdD (CPED) cpedinitiative.org impactinged.pitt.edu ISSN 2472-5889 (online) Vol.5 (2020) DOI 10.5195/ie.2020.109 Predicting EdD Applicant Potential Using Structured Behavioral Interviews During the Application Process Laurie A. Kimbrel, EdD Assistant Professor University of West Georgia lkimbrel@westga.edu Mary Alice Varga, PhD Associate Professor University of West Georgia maryv@westga.edu ABSTRACT This essay describes the work of faculty at a public university in the southeast to align their application process with CPED principles through the addition of structured behavioral interviews. Their work was grounded in the premise that redefinition of the Ed. D. program to focus on the successful preparation of scholarly practitioners also requires a redefinition of the type of student who is initially identified and admitted to the program. A structured interview was added to the application process as a tool to gather data to accurately predict the likelihood of student success in the program and as a leader of change in schools. Selection science literature guided the development of the interview with questions aligned to essential student attributes. Although the interview questions and process are still evolving, data from the initial administration suggests that the interview has promise as a useful element of the candidate selection process. Keywords: Structured behavioral interviews, Ed.D. application process, predictive interviews, Ed.D. program improvement INTRODUCTION As colleges of education work to refine their doctorate in educational leadership (Ed.D) programs to align with the Carnegie Project on the Educational Doctorate (CPED) principles, attention typically focuses on updating coursework and the dissertation process to ensure rigor and relevance. This work, while essential, may be insufficient if we truly desire to achieve the shared vision of preparing educational leaders who are scholarly practitioners, ready to meet the complex challenges of the 21st century. Refining the Ed.D. to focus specifically on the effective preparation of scholarly practitioners also requires a redefinition of the type of student who is most likely to benefit from such a program and, consequently, how the applicants most likely to achieve success are identified and admitted. If we embrace the assumption that Ed.D. programs are unique, rather than less rigorous versions of Ph.D. programs, then relying on our traditional Ph.D. program admission processes may no longer be sufficient. The Ed.D. program in school improvement at University of West Georgia has a school improvement focus, yet when faculty examined the admission process the program, they expressed concern that decisions based solely on the review of written materials may not provide sufficient evidence that applicants have the potential to identify problems of practice and to use research to solve those problems. Faculty concerns stemmed from perceptions that students experience difficulty connecting their dissertation proposals to school improvement initiatives, despite coaching and instruction. ADMISSION SELECTION The program receives approximately 100 applicants annually, and yet only 16-18 students are typically admitted. Given this acceptance rate, making the best admission decisions possible is of the utmost importance. In order to ensure that the most appropriate applicants are selected for admission to the School Improvement Ed.D program at the (name of University), the faculty posed the following question: How can we use selection science research to create an Ed.D. application process that will accurately predict the potential of each student for success in a program designed for practitioner- researchers focused on school improvement? EXAMINATION OF APPLICATION PROCESS To answer this question, the School Improvement Program faculty began with an examination of the existing application process. Potential students are required to submit an on-line application as well as GRE scores, three letters of recommendation, vitae, official transcripts, and a short essay detailing personal goals related to school improvement. Completed applications are initially screened for minimum GPAs and GRE scores, and then qualifying applications are sent for additional review by research and education leadership faculty members. Given the school improvement focus of the program, faculty review applications to determine if potential students are in a position to identify and implement change 27 Kimbrel & Varga Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 5 (2020) DOI 10.5195/ie.2020.109 initiatives; understand the basic concepts of school improvement; and have the academic capacity and persistence to complete rigorous coursework and a dissertation. Faculty feedback is used to assemble a list of finalists, who then complete an unstructured, conversational phone interview with the program director. This brief interview addresses questions that arise in the application review but do not include a formal or consistent set of questions. Final admission decisions are based on the totality of the information gathered and reviewed. Faculty review of the admission process resulted in the agreement that existing components of the process should remain; however, elements to more accurately assess student writing ability and potential to engage in school improvement activities would increase our ability to identify the best applicants. To assess student writing ability, the decision was made to add a requirement that applicants submit a professional writing sample and scores for the writing portion of the GRE. The faculty were aware of the disagreement regarding the degree of correlation between GRE scores and academic success. The Educational Testing Service (ETS) has found that GRE scores have only a moderate correlation with first year Grade Point Averages and that critical skills associated with academic and professional competence are not measured by the assessment (ETS, 1998). Given these limitations, the professional writing sample was also added to assess a previous work product and to determine what candidates identified as their best-written work. To assess student potential to identify and solve problems of practice related to school improvement, faculty committed to the creation of a structured behavioral interview process designed to measure the attributes necessary for success in the Ed.D. program and as a leader of school improvement. THE STRUCTURED BEHAVIORAL INTERVIEW AS A PREDICTIVE MEASURE OF SUCCESS The conversational interview has been a widely used tool to make employment decisions for over a century, and as such, there has been a great deal of study of its effectiveness for predicting future job-related success (Levashina et al., 2014). The interview remains popular despite the findings from decades of selection science literature, which has reported that traditional conversational interviews are often predictive of little more than an applicant's appearance, manners, and likability (Buckley, Norris, & Wiese, 2000). The lack of predictive ability of conversational interviews has been reported to be a result of a variety of factors including, interviewers who do not agree among themselves about what types of questions should be asked and how interviewee responses should be evaluated (Judge, Higgins, & Cable, 2000). Other research has found that interviews are susceptible to interviewer bias and the impact of factors such as race, gender, age, and appearance, which are the primary determinants of the successful applicant (de Kock & Hauptfleisch, 2018; Nadler, Lowery, Grebinoski, & Jones, 2014; Segrest Purkiss, Perrewe, Gillespie, Mayes & Ferris, 2006). Furthermore, traditional interviews can be problematic. Interviewers can control the outcome by talking for the majority of the interview, asking questions that are not meaningful, and by conveying their opinions of the applicant’s responses through their own verbal and non-verbal responses (Delli & Vera, 2003). Interviewers can also focus on unrelated issues, including the appearance of the interviewee (Friedman, 2014). The addition of elements of structure in the employment interview process can reduce bias and inconsistency (Moore, 2017). Twelve meta-analyses conducted from 1988 through 1995 found strong evidence that structured interviews predict success more consistently than unstructured interviews (Levashina et al., 2014). Conway, Jako, and Goodman (1995) used meta-analysis of 111 studies to demonstrate the reliability of interviews, in terms of the extent to which the questions asked actually measure necessary job skills. Results indicated that reliability varies widely depending on the format of the interview. They found that the average reliability of unstructured individual interviews was 0.37, but reliability increases to .59 for individual structured interviews. If we use Cohen’s (1988) conventions, where bivariate correlations of .10. .30, and .50 represent small, medium, and large effect sizes, respectively, the addition of structure moves interviews to the highly reliable category. A meta-analysis by Lavashina et al. (2014) identified six essential elements of the structured interview: 1. Job analysis used to create questions. 2. Identical questions asked of each applicant. 3. Behavioral question format focused on situational questions based on past behavior. 4. Individual answers rated with a predetermined scale. 5. Presence of anchor answers. 6. Trained interviewers. STRUCTURED BEHAVIORAL INTERVIEW DEVELOPMENT Faculty at the University of West Georgia were purposeful in their decision to use selection science research regarding structured employment interviews to create a customized interview protocol and questions aligned with the qualities necessary for student success in the School Improvement Ed.D. program and as a school improvement leader. The development process was guided by the six elements of structured interviews, as identified by Levashina et al. (2014). Attributes Related to Ed.D Success The structured interview development process begins with an analysis to determine the essential skills required for success in a given position. The faculty addressed this step by conducting a brief literature review to identify the attributes essential for success in an Ed.D. program. Unfortunately, no published studies outline specific pre-requisite attributes necessary for success in practitioner-based education programs. The faculty then searched for attributes of successful students identified by other EdD programs on their websites or in their program literature. Faculty focused on program materials from universities and colleges who are members of the Carnegie Project on the Education Doctorate (CPED). An extensive list of attributes was assembled from approximately ten CPED member programs. Faculty members then discussed the list and narrowed it to pre-requisite attributes they believed were related explicitly to success in the (name of university) online Ed.D. focused on school improvement. Attributes or behaviors that could be improved through participation in the program were not considered at this early stage of interview development, although this might become a consideration in the future. Individual attributes were then categorized into broad classifications, as indicated in table 1. 28 Predicting Ed.D. Applicant Potential Using Structured Behavioral Interviews During the Application Process Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 5 (2020) DOI 10.5195/ie.2020.109 Table 1. Attributes of an Effective School Improvement Ed.D. Student Doctoral Student Attributes School Improvement Leader Attributes Practitioner-Researcher Attributes Online Learner Attributes Effective time- management Skilled writer Flexible Optimistic Persistent Motivated Organized Curious Self-advocate Self-disciplined Coachable Ethical Analytical Creative Purpose Commitment/passion Critical thinker/ problem solver Curious Effective communicator Collaborative Optimistic Persistent Researcher self-efficacy Ethical Analytical Creative Curious Effective communicator Effective communicator Social presence Engagement Self-disciplined Motivated Organized Once the list of attributes was finalized and prioritized, the faculty collaborated to identify essential attributes that would be measured using a structured interview for each applicant. Faculty members worked together to write operational definitions for each of the essential attributes. The definitions were written to focus on descriptions of optimal behaviors necessary for either school improvement leadership or success in a fully on-line doctorate program. Table 2 lists the operational definitions for essential attributes. Table 2. Operational Definitions of Essential Attributes Attribute Operational Definition Purpose This student knows that schools are in a constant state of development and never loses focus on the need to continually strive for significant improvement within their system. This person has a willingness to take action on approaches that improve the effectiveness of schools in terms of equitable outcomes for students. Commitment/passion This student has a mission of service to others and is committed to the development of teachers and staff. They keep a focus on the students and works as a partner with parents and community members. Collaborative This student actively seeks to understand the perspectives of others, seeks to align the work of team members to talents and strengths, requests input from those whom decisions will impact. Coachable This student not only accepts but actively seeks feedback that is then utilized to continue to improve their performance and as well as that of their team. Time management/ Organization This student can multi-task, break long term projects down into manageable pieces, and make deadlines through the application of specific organizational strategies. Persistent This student views obstacles as a part of any process and as surmountable. He or she has effective strategies to work through difficult circumstances. 29 Kimbrel & Varga Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 5 (2020) DOI 10.5195/ie.2020.109 Optimistic This student is positive about their work and has ways of encouraging others to do their best. Also, he or she has high expectations for self and others. Flexible This student remains open to change, admits mistakes and rectifies them, adapts to new circumstances, accepts alternate perspectives. Problem solver This student identifies problems through the examination of data or evidence, uses research, best practice, and literature to arrive at logical, sound, and objective decisions. He or she ultimately wants schools to be accountable for student development and growth. Ethical This student has personal values that are clearly defined and is willing to behave in a manner that demonstrates their sense of purpose. They are willing to tactfully speak out when necessary and confront inappropriate actions or behaviors of others. He or she can remain firm on personal values, even in the face of criticism. Motivated This student continues to grow as a professional and has demonstrated that their ongoing learning benefits their school, colleagues, and students. Interview Protocol Development The work of Levashina, et al. (2014) guided interview question development. One interview question was developed to align with each of the 11 essential attributes, and an additional question was added to assess the capability of applicants to learn in an online setting. Questions were developed according to research-based best practices for structured interviews. All questions were behavioral in nature and designed to uncover the presence of the attribute by asking applicants to describe how they have responded in the past or may respond in the future to specific situations. This focus on behavior allowed the interviewers to determine if there was a connection between stated applicant values and the ability to operationalize those values in their everyday professional practice. A rating scale for the interview was developed after all questions were written. Each interviewee's response could be scored two, one, or zero. Target answers based on the operational definitions were developed for each question and assigned a score of two. Answers that approached but did not meet the target were assigned a score of one, and a negative or contrary answer was scored as a zero. A perfect score for the 12-question interview was 24, and the lowest possible score was zero. See table 3 for an example of an interview question and rating scale. Table 3. Sample Interview Question for the Attribute of Persistence The program director and a faculty member with a background in selection science research and practical field experience designing structured interviews developed interview procedures. Procedures reflected recommendations from selection science literature so that every interviewee would have a similar experience. For example, procedures were implemented to reduce bias from factors such as the appearance of the applicant. All interviews were conducted on the GoToMeeting platform using audio-only. The faculty developed an interview script to ensure all interviewees would have the same experience: This interview is designed so that you can express your beliefs and behaviors as an education professional and graduate student. We are interested in hearing about the essence of your beliefs and behaviors without much detail. If there is not enough information in your response, we will ask you to tell us more. When we think that we are ready to move on, we will interject a new question. Although this may feel somewhat awkward, please do not be worried if this occurs! In addition, we will not be able to interpret a question for you, since it is your interpretation of the question that is most important to us. You may ask to have a question repeated if necessary. There are a total of 12 questions, and the interview should take about 15 minutes. The program director and the faculty member with expertise in structured interviews conducted the interviews together. The program director had reviewed written applications, and the other faculty member had not. The interviewers scored each interview independently and then compared and discussed results to establish inter-rater reliability. STRUCTURED INTERVIEW IMPLEMENTATION Students currently enrolled in the (name of university) School Improvement Ed.D. program were contacted via e-mail to seek volunteers to participate in a pilot of the structured interview process. The program director and faculty member with expertise in structured interviews conducted four pilot interviews with students using the Briefly describe a project that you led that did not go as planned. How did you resolve the situation? (2) accurately determined barriers or issues, created a plan to address, the project was completed satisfactorily (1) created a plan, but it was not specifically aligned with the barriers, the project completed. (2) blame others, project abandoned 30 Predicting Ed.D. Applicant Potential Using Structured Behavioral Interviews During the Application Process Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 5 (2020) DOI 10.5195/ie.2020.109 established process and questions.. Students who participated in the pilot interviews also had the opportunity to share feedback regarding how the process felt from the perspective of an interviewee. Feedback from students was generally positive, but in response to their ideas, small changes were made to the wording of three questions to increase clarity. One line was also added in the directions to reinforce that being interrupted by the interviewer was a normal part of the process and should not be interpreted as a negative sign. Structured interviews with 20 School Improvement Ed.D. program applicants were conducted during February 2019. Each interview was limited to 30–40 minutes, and the interview protocol was strictly followed to ensure that each applicant had a similar experience. Interviewers gently interrupted to ask the next question in cases where interviewees gave extremely long answers. This strategy ensured that all interviews remained in the 30–40-minute timeframe. As planned, all interviews were audio only with no video and recorded so that interviewers could review the data as they established inter-rater reliability. Data from one interview was omitted due to the applicant’s decision to defer admission to the following year. Final admission decisions were ultimately made by the program director who considered feedback from the faculty review of written applications and to a lesser extent, the results from the structured interview. Given that the interview is still considered to be in a pilot phase, no candidate was denied admission based on interview performance alone. In the near future, program faculty will work together to determine the weight and importance of each application component. As changes to the application process were discussed, consideration was given to the CPED principle of social justice. Faculty members were mindful that changes must enhance and not impede the admission of a diverse group of Ed.D. students who work in settings that serve the most vulnerable K-12 students. Preliminary Findings When examining all applicant response scores, the average score was 16.89 out of a possible 24 points. The 14 applicants who were offered admission to the program obtained a higher average score of 18.06. Cohen’s kappa was run to determine if there was agreement between the two interviewers on whether the applicant effectively communicated a connection between their stated values and the ability to operationalize those values in their everyday professional practice. Altman’s (1999) guidelines were used to determine the strength of agreement as poor (<0.20), fair (0.21- 0.40), moderate (0.41-0.60), good (0.61-0.80) or very good (0.81- 1.00). There was moderate agreement between the two interviewers’ ratings, kappa = .542, p < .0001. When examining the relationship between applicant behavioral interview scores and programmatic success, Grade Point Averages (GPAs) of the 14 accepted students achieved during their first semester in the program were examined. The average GPA after the first semester was 3.93. GPAs for each student and their corresponding interview score are listed in Table 4. Table 4. Student Grade Point Averages and Interview Scores Student Grade Point Average Interview Score A 4 21 B 4 20 C 4 20 D 3.5 20 E 4 19 F 4 19 G 4 18 H 4 18 I 3.5 18 J 4 17 K 4 17 L 4 16 M 4 15 N 4 14 Attempts at examining a relationship between interview scores and GPA were made; however, the data violated the assumption of a monotonic relationship between the two variables. Since this is required to run a Spearman Correlation, the relationship could not be statistically analyzed. Faculty recognize that GPA is only one limited measure of student success. Work is in progress to determine what other measures of program success will be used as we continue to refine and use the structured interview as a component of the application process. Ultimately, the percentage of students who complete the program in three, four, and five years will become important measures of student success. Dissertations in this program are required to focus on identifying and solving a problem of practice through action research or program improvement, and so successful defense will be a measure of effectiveness in the area of school improvement. NEXT STEPS School Improvement faculty at (name of university) recognize that the development and the first round of implementation of the structured behavioral interview process was only the first step in a long-term project. Several important actions are being taken to strengthen both the interview questions and the protocol. A presentation was made at the June 2019 CPED convening to solicit feedback of faculty members from colleges and universities who have also committed to the CPED principles. The feedback from this group focused on ideas to strengthen questions to align more closely with the attributes they seek to measure. Interview questions will be revised to consider their feedback and the performance of applicants during the first round of implementation. Any question that resulted in uniformly high or low scores will be examined by faculty to consider if the question presents an opportunity for applicants to differentiate themselves from one another. The revised interview questions will 31 Kimbrel & Varga Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 5 (2020) DOI 10.5195/ie.2020.109 remain aligned to the same essential attributes and will be used in the spring 2020 admission process. Concerns relating to diversity were not found with the initial use of the structured interview. The Ed.D. program is in a fortunate position to receive a diverse pool of applicants each year. Of the 14 students who were interviewed and offered admission, approximately 30% (n = 4) identified as African American/Black or Hispanic. Admitted applicants who identified as males represented 20% (n = 3) of the group. These numbers are consistent with the demographics of applicants who are admitted to the program over the past four years. Faculty at (name of university) will compare the structured interview results with measures of student success in the program including GPA, successful completion of coursework, and successful completion of the program including the dissertation, in order to establish if the structured interview is a predictive tool and enhances the ability of faculty to admit the students to the Ed.D. program who are most likely to experience success. Ultimately, the value of the structured interview is dependent on the alignment of the essential attributes as listed in table 1, the operational definitions as listed in table 2 and the structured interview questions. As more interview data is gathered and analyzed, adjustments will be made to improve this alignment and to ensure that the interview measures the attributes as they are defined. (name of university) faculty are also committed to a collaborative relationship with their colleagues in other CPED aligned Ed.D. programs and as such, are willing to share the interview protocol and questions upon request. REFERENCES Altman, D. G. (1991). Practical statistics for medical research. London: Chapman and Hall. Buckley, R. M., Norris, A.C., & Wiese, D. S. (2000). A brief history of the selection interview: May the next 100 years be more fruitful. Journal of Management History, 6(3), 113- 126. https://doi.org/10.1108/eum0000000005329 Chambers, B.A. (2015). Acing the Interview. Collector, 81(1), 32-33. Retrieved from https://search.ebscohost.com/login.aspx?direct=true&AuthType=ip,shib& db=edb&AN=108892454&site=eds-live&scope=site Cohen J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum. Conway, J.M, Jako, R.A., & Goodman, D.F. (1995). A meta-analysis of inter- rater and internal consistency reliability of selection interviews. Journal of Applied Psychology, 80, 565-579. https://doi.org/10.1037/e518362013-291 deKock, F.S., & Hauptfleisch, D.B. (2018). Reducing racial similarity bias in interviews by increasing structure: A quasi-experimental using multilevel analysis. International Perspectives in Psychology: Research, Practice, Consultation, 7(3), 137-15. https://doi.org/10.1037/ipp0000091 Deli, D.A., & Vera, E.M. (2003). Psychological and contextual influences on the teacher selection interview: A model for future research. Journal of Personnel Evaluation in Education, 17(2), 137-218. https://doi.org/10.1023/b:peev.0000032425.42878.f3 Educational Testing Service. (1998). GRE Guide to the Use of Scores, 1998- 1999. Princeton, NJ. Friedman, R. (2014). The Best Place to Work. New York, NY: Penguin- Random House Publishing. Judge, T.A., Higgins, C.A., & Cable, D.M. (2000). The employment interview: A review of recent research and recommendations for future research. Human Management Review, (10)4, 383-406. https://doi.org/10.1016/s1053-4822(00)00033-4 Levashina, J., Hartwell, C. J., Morgeson, F. P., & Campion, M.A. (2014). The structured employment interview: Narrative and quantitative review of the research literature. Personnel Psychology, (67)1, 241-293. https://doi.org/10.1111/peps.12052 Moore, D. A. (2017). How to improve the accuracy and reduce the cost of personnel selection. California Management Review, (60)1, 8-17. https://doi.org/10.1177/0008125617725288 Nadler, J.T., Lowery, M.R., Grebinoski, J., & Jones, R.G. (2014). Adverse discrimination in employment interviews: Reducing effects of sexual orientation bias with accountability. Psychology of Sexual Orientation and Gender Diversity, 1(4), 480- 488. https://doi.org/10.1037/sgd0000079 Schmidt, F.L. & Hunter, J.E. (1998). The validity and unity of selection and methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124 (2), 262- 274. https://doi.org/10.1037//0033-2909.124.2.262 Seagrest Purkiss, S.L., Perrewe, P.L., Gillespie, T.L., Mayes, B.T., & Ferris, G.R. (2006). Implicit sources of bias in employment interview judgments and decisions. Organizational Behavior and Human Decision Processes, 101(2), 152-167. https://doi.org/10.1016/j.obhdp.2006.06.005 32