Impacting Education: Journal on Transforming Professional Practice New articles in this journal are licensed under a Creative Commons Attribution 4.0 United States License. This journal is published by Pitt Open Library Publishing. 36 This journal is supported by the Carnegie Project on the Education Doctorate: A Knowledge Forum on the EdD (CPED) cpedinitiative.org impactinged.pitt.edu ISSN 2472-5889 (online) Vol. 8, No. 2 (2023) DOI 10.5195/ie.2023.361 The Tension Between Rigor and Relevance: Redesigning EdD Applied Research Methods Coursework within an R1 Institution Christine M. Neumerski University of Maryland cneumers@umd.edu ABSTRACT This essay examines the tension between rigor and relevance in a newly designed EdD program in School System Leadership in a Research 1 institution. After a three-year redesign of this program that prepares students to become superintendents, our faculty continue to wrestle with questions around how to make methodology coursework meaningful, impactful, timely, and useful. This article examines the perspective of one of the EdD faculty leading the redesign of the program and concludes with lingering questions and tensions around best practices. KEYWORDS EdD, improvement science, research methods, education leadership, leadership preparation Shulman’s (2006) call for two separate, distinct education doctorate programs provided the field with a framework for how to improve both the EdD and the PhD in education leadership. Yet, universities are still grappling with how best to develop EdD programs that strike a balance between relevance and rigor (Perry, Zambo, & Crow, 2020). On the one hand, EdD programs that focus on relevance for practitioners often suffer from the critique that they are PhD Lite, lacking the appropriate methodological rigor to remain credible within higher education institutions (Perry, Zambo, & Crow, 2020). On the other hand, EdD programs that focus more exclusively on rigor often suffer from the critique that they have become too similar to the PhD, teaching EdD students the research methodologies that were developed for aspiring researchers rather than for practitioners (Hochbein & Perry, 2013). Where is the right balance that Shulman and colleagues (2006) urge us to consider? In this essay, I draw on a three-year redesign of an EdD in School System Leadership in a Northeast Research 1 university that has tried to strike this balance, maintaining both relevance and rigor (Hochbein & Perry, 2013; Shulman et al., 2006) for an EdD for aspiring superintendents. Although the tension around relevance and rigor is evident throughout the entire EdD, I center this article on the program’s approach to applied research methodology. I focus on research methods because: 1) they are a programmatic area of intense debate in the field (Berliner, 2002), 2) they are an area of debate among EdD faculty, and 3) they form the anchor courses that lead to our program’s capstone. In what follows, I briefly describe the overall EdD redesign, examine the ways in which we approach research methods courses, and conclude with questions around rigor and relevance that have emerged during the ongoing redesign. A NOTE ABOUT METHODS AND POSITIONALITY Note that this is not a formal research article, but instead an essay in which I draw on my experience over the past three years as part of a collaborative team of faculty members engaged in EdD program redesign. I utilize my reflective practice (Malen, 2016; Metz, 2001) in crafting this essay, and the points I make throughout represent my understanding of and sense-making behind the redesign process (Weick, 1995). I use the first-person plural of “we” in order to reflect the collaborative nature of our redesign, noting that although I was one of four faculty members leading the programmatic overhaul, this was embedded in a larger team approach. While I draw on the collective work and thought-processes of the faculty that comprise our EdD team, the assertions in this article are mine. I also acknowledge that each member of that team likely understands and makes sense of our collaborative redesign efforts somewhat differently (Weick, 1995) and that includes me. My position (Holmes, 2020) is as a faculty researcher who has been trained in traditional quantitative and qualitative methodology at a Research 1 institution and has subsequently worked as a researcher in multiple Research 1 institutions. I began my career as a practitioner, a teacher and teacher educator in urban Northeast and Midwest schools. Currently, I work as a scholar of policy and leadership, not as a practitioner. My prior and current positions and experiences will, of course, color my perspectives. That said, I offer my reflective practice on this work as part of the ongoing conversation in the field about how to best approach research methods within EdD programs in education. https://library.pitt.edu/e-journals http://creativecommons.org/licenses/by-nc-nd/3.0/us/ http://cpedinitiative.org/ https://orcid.org/0000-0002-9520-1141 The Tension Between Rigor and Relevance Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8, No. 2 (2023) DOI 10.5195/ie.2023.361 37 OVERVIEW OF THE EDD REDESIGN Over the past three years, eight faculty members have engaged in the redesign of a 36-month EdD in School System Leadership. Led by 4 faculty – 2 PhD research scholars and 2 EdD former district practitioners – our team has debated how to respond to the university’s shift away from a more traditional dissertation to a capstone for all advanced practitioner degrees. The freedom to redesign the EdD opened the door for innovation; however, like many programs wrestling with what the ideal doctorate for practitioners should encompass (Perry, Zambo, & Abruzzo, 2020), we were plagued with several stubborn questions. What should leaders know and be able to do by the time they receive their doctorate? How should students demonstrate their knowledge and skills in a culminating capstone that is rigorous and credible within the university, while also relevant for practitioners? Furthermore, what research methodology courses would appropriately prepare students for this type of capstone work? The diversity of the EdD faculty - including former policy makers, teachers and school board members, improvement science experts and traditional research experts, former school and district leaders, and clinical and tenure-track faculty - enhanced our conversations and debates and likely led to a more well-thought out program. However, that diversity also led to some unresolved and ongoing tensions, particularly around research methodology. As we explored these tensions, different faculty camps emerged. Some lobbied for improvement science (Bryk et al., 2015) as the sole methodology for the new EdD, while others pushed for the inclusion of some traditional research methodology coursework. Yet, all of us advocated for the development of students who would become critical thinkers, consumers of research, and leaders who could actively solve local, pressing problems of practice. That led to a decision akin to the ways in which Hochbein and Perry (2013) describe the use of research methods in an EdD: “…Explicit instruction of high quality research skills should be a hallmark of professional doctorate programs. The definition of high quality need not, and possibly should not, be identical to that of a PhD research sequence… [It should include] tools that not only connect the knowing-doing gap, but also empowers the ‘foot soldiers’ to empirically attack problems of practice” (p. 192). To this end, we created a newly designed version of the EdD, one that encompasses National Educational Leadership Preparation (NELP) standards, a portfolio assessment based on the practice of leadership in students’ day-to-day work, mentorship from local district leaders, and a capstone grounded in a problem of practice. In sum, our program would build the capacity of our students as foot soldiers doing the on-the-ground work of leadership every day. In the newly designed program, students can work individually or in small groups of 2 or 3 to build their capstones, which includes two stand-alone products: 1) a scholarly-practitioner paper based on a study of a problem of practice (PoP) and 2) an impact product that targets specific practitioners and enables the use of the study findings. The former includes many of the traditional aspects of a research paper (e.g. introduction, literature review, methods, findings), while drawing on an improvement science framework (Bryk et al., 2015) to push students to develop a practical aim (e.g. reduce district-wide disproportionate suspension rates 5% by 2025) and to investigate a PoP that is meaningful to their leadership contexts. The latter includes actionable recommendations to a district or other organization based on study findings (e.g. recommendations on what to do to reduce disproportionate suspension rates) and can take any number of formats. For example, students might create a presentation to the board of education, a white paper for a think tank, recommendations for a district, a policy brief for an office on Capitol Hill, an EdWeek article, or another medium of their choosing. The proposal defense must include the EdD students’ articulation of how the entire capstone is meaningful, relevant, timely, and useful for a defined practitioner audience. In what follows, I describe the research methodology coursework that our program developed to prepare students to engage in this type of capstone. I couch the investigation of research methodology in the tensions that have emerged throughout our program redesign, and the lingering questions that we continue to ask ourselves as EdD faculty. We see this questioning not as problematic, but as essential to the iterative nature of redesigning and refining the new program. YEAR 1: THE PROBLEM OF PRACTICE: DEVELOPING CRITICAL THINKERS One challenge we noticed with our EdD students is that many of them came from compliance and accountability driven districts, in which they must often implement directives. We suspected that this might be related to their desire for us to tell them what to do for their capstone. To put it differently, our hunch is that many of our students are highly-skilled in making quick decisions, following directives in school districts, and leading complex organizations. We want to put them in a space in which they can slow down, wrestle with, rethink, and reexamine their ideas over time. We want our students to develop critical thinking skills as potential future superintendents and allow them to experience the sometimes uncomfortable role that comes with having full autonomy over designing their capstones. To do so, we grounded their applied research methodology around an exploration of problems of practice throughout the three years of the program. We introduce students in their first course to the concept of a PoP and the tools of improvement science (Bryk et al., 2015). Although some EdD programs have students identify a PoP immediately so that they can begin building their dissertations in year 1, we decided to instead scaffold students’ exploration of different problems that were relevant to them and to their contexts. Our theory was that providing space for students to contend with different PoPs and to rewrite and refine them will build their critical thinking skills and become comfortable with the ambiguity of this type of undefined space in year 1. We built support into their year 1 coursework for various touchpoints to offer check-ins around their thinking. YEAR 2: CREDIBLE EVIDENCE: DEVELOPING CONSUMERS OF RESEARCH After a first year of exploration, students begin year 2 with a public presentation to their EdD cohort and faculty on the PoP they have chosen to pursue for their capstone. During this “Evening at the PoPs,” students briefly describe the importance of their problems to their work as practitioners. This is followed by an applied research methods course, in which they examine the importance of their problems to the larger field, focusing on understanding what counts as credible evidence. They move from exploring the importance of Neumerski Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8, No. 2 (2023) DOI 10.5195/ie.2023.361 38 their PoP within their own contexts to exploring what is known about their PoPs in broader contexts; they do so by beginning their literature reviews. Part of their investigation into credible evidence is an introduction to quantitative and qualitative methods. The scaffolding in this course is designed not to teach these students how to become formal researchers, but to become consumers of research as they dig into the literature, identifying the tenants that make particular studies more or less salient to their work, and to discriminate high quality studies from those that are poorly done. Students are supported to begin exploring the research questions they might want to pursue for their capstones, working first to examine the literature, brainstorming questions, engaging again in the literature, and further refining their questions. We emphasize the iterative nature of this process to our students, namely that both the PoPs and research questions are refined – or sometimes altogether changed – as the student builds her expertise around not only her practitioner knowledge but knowledge in the field. This course is followed by an advanced improvement science course, in which students use the framework of continuous improvement to develop a causal systems analysis, a driver diagram, an aim statement, and a theory of improvement (Bryk et al., 2015; Lewis, 2015). This shifts the students’ focus from their perspectives as practitioners – and the perspectives of the field writ large – to considering the “micro details as to how any proposed set of changes is actually supposed to improve outcomes” (Bryk et al., 2015). This type of on-the-ground systems thinking provides a set of tools for students to begin to think more closely about the potential impact they can make through their capstone work (Bryk et al., 2015). It is the shift in research from being consumers to being foot soldiers charged with actionable change. Table 1. The Arc of Research Methodology in the Redesigned EdD Year 1 Introduction to Problems of Practice and Improvement Science PoP “Check-ins” occur in each course in Year 1 Year 2 Evening at the PoPs Applied Research Methods 1 Advanced Seminar in Improvement Science and Theory of Improvement Night Year 3 Applied Research Methods 2 & Advisor Assignment Defend Capstone Proposal Implement Capstone Study YEAR 3: GROUNDING IN IMPACT: MEANINGFUL, RELEVANT, TIMELY, AND USEFUL In Year 3, students take their second applied research methods course, which has the explicit goal of helping them build their capstone proposals. This course is designed as a workshop in which students receive individualized support from their instructor and advisor/mentor). Throughout the course, students refine their research questions, align those questions with the study purpose, and explore methodological options. During this process, they review improvement science tools, as well as qualitative and quantitative methods. The emphasis in this course is on thinking about a study design that will ultimately lead to an impact on a specific practitioner group. Students are asked to constantly evaluate and reevaluate how their proposed work is meaningful, relevant, timely, and useful for practitioners. They build a capstone study and accompanying methods, while simultaneously considering an impact product. This product is their choice, but it must convey or use findings from the capstone to impact a specific practitioner audience. The two-product capstone, therefore, requires students to demonstrate proficiency in the rigor of conducting studies and relevance to practitioners. The course instructor and advisor ensure that the student’s capstone committee consists of experts on the student’s PoP content, methodology (e. g. an expert in Plan Do Study Act cycles, focus groups, surveys, interviews, etc.), and includes a practitioner perspective. The goal is for students’ coursework to culminate in the second applied research methods course in order to ensure they have the tools needed to move forward in their individualized work of implementing a capstone study and drawing on the practitioner expertise in their cohorts as they design impact products and mechanisms for sharing them. RETURNING TO RIGOR AND RELEVANCE: UNANSWERED QUESTIONS Our first cohort of the new EdD in School System Leadership is in the midst of their third year of the program. As such, we have had an opportunity to reflect and further refine our program, but this has not yet given us the answers to our initial questions. We do not yet know if we have struck the right balance between rigor and relevance. We do not yet know if we have succeeded in creating a program that is appropriately tailored for school and school system leaders. We do not yet have evidence as to whether this redesign will work as intended. Time will tell. And in the meantime, the redesign process has left us with new questions. One question that remains is if in our effort to maintain rigor within a Research 1 institution, we have designed our capstone scholarly-practitioner paper to look too much like a traditional dissertation. Is it necessary for students to undertake their own research studies in order to demonstrate proficiency for the EdD in School System Leadership? On the one hand, they are unlikely to conduct original research in the future; instead, we are preparing them to be consumers of research, disentangling high from low quality studies, and as leaders, to know how and when to hire professional researchers for their district. On the other hand, the process of engaging with a set of research questions grounded in a real PoP and determining how to study it in a way that leads to impact appears to be fruitful in developing students who are critical thinkers, critical consumers of research, and ultimately, informed doers. A second question lies at the opposite end of the rigor and relevance tension. Our EdD program is completely separate from our PhD program. These two groups of students do not take coursework together; their research methodology coursework is not taught by the same faculty members. In fact, our EdD and PhD faculty have no formal interaction with one another, nor do our EdD and PhD students. We have achieved what Shulman and colleagues (2006) suggested is important – two separate and distinct programs with two separate and distinct aims. And yet, I continue to wonder if it is possible to err too far, if separate and distinct might have an unintentional consequence: that of perpetuating the research- practice divide. In a field in which we struggle to find ways for researchers to engage in studies that are truly meaningful for practitioners, to translate findings in ways that are digestible for The Tension Between Rigor and Relevance Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8, No. 2 (2023) DOI 10.5195/ie.2023.361 39 practitioners, and to focus their work on that which is useable and timely for practitioners, should we be doing more in graduate programs to build those bridges? Would providing opportunities for our current practitioners (EdD students) to learn and study and socialize alongside our aspiring researchers (PhD students) help bridge the research-practice divide? Could we maintain separate and distinct preparation programs while also teaching a shared language for researchers and practitioners? And if they learned this shared way of communicating and learning together around problems of practice from the field, would they be more likely to carry that with them into the real world? I question whether we truly want to prepare our EdD and PhD students to be so separate in doctoral programs that they maintain that separation as practitioners and researchers – or whether our EdD programs should prepare scholar-practitioners to critically engage with researchers in the service of problems of practice. Put differently, is our drive for relevance perpetuating a divide that plagues our field of education? Clues to the answers to these questions are unfolding as I write, as students provide their ongoing feedback around the ways in which the program is supporting them and in the ways they need more or different support. We plan to formally assess the quality and content of our redesigned EdD through an external review of the program that will include surveys and interviews with faculty, current students, and alumni. We will continue to seek input as we untangle the rigor and relevance questions pervading our redesign. REFERENCES Berliner, D. C. (2002). Comment: Educational research: The hardest science of all. Educational researcher, 31(8), 18–20. Bryk, A. S., Gomez, L. M., Grunow, A., & LeMahieu, P. G. (2015). Learning to improve: How America’s schools can get better at getting better. Harvard Education Press. Hochbein, C., & Perry, J. A. (2013). The role of research in the professional doctorate. Planning and Changing, 44(3/4), 181–194. Holmes, A. G. D. (2020). Researcher Positionality--A Consideration of Its Influence and Place in Qualitative Research--A New Researcher Guide. Shanlax International Journal of Education, 8(4), 1–10. Lewis, C. (2015). What is improvement science? Do we need it in education?. Educational Researcher, 44(1), 54–61. Malen, B. (2017). Enriching the preparation of education researchers and practitioner-scholars: Linking school district research priorities and university-based policy evaluation research courses. Journal of Research on Leadership Education, 12(1), 97–115. Metz, M. H. (2001). Intellectual Border Crossing in Graduate Education: A Report from the Field. Educational Researcher, 30(5), 1–7. https://doi.org/10.3102/0013189X030005012 Perry, J. A., Zambo, D., & Abruzzo, E. (2020). Faculty Leaders Challenges and Strategies in Redesigning EdD Programs. Impacting Education: Journal on Transforming Professional Practice, 5(1), 1–6. https://doi.org/10.5195/ie.2020.143 Perry, J. A., Zambo, D., & Crow, R. (2020). The improvement science dissertation in practice: A guide for faculty, committee members, and their students. Myers Education Press. Shulman, L., Golde, C., Bueschel, A., & Garabedian, K. (2006). Reclaiming education’s doctorates: A critique and a proposal. Educational Researcher, 35(3), 25–32. https://doi.org/10.3102/0013189X035003025 Weick, K. E. (1995). Sensemaking in organizations (Vol. 3). Sage. https://doi.org/10.3102/0013189X030005012 https://doi.org/10.5195/ie.2020.143