New articles in this journal are licensed under a Creative Commons Attribution 4.0 United States License. This journal is published by the University Library System of the University of Pittsburgh as part of its D-Scribe Digital Publishing Program and is cosponsored by the University of Pittsburgh Press. 8 This journal is supported by the Carnegie Project on the Education Doctorate: A Knowledge Forum on the EdD (CPED) cpedinitiative.org impactinged.pitt.edu ISSN 2472-5889 (online) Vol. 6 No. 4 (2021) DOI 10.5195/ie.2021.154 Strategies for Promoting Evidence Use Through the Education Doctorate William A. Firestone Rutgers University wilfires@gmail.com Andrew S. Leland Southern Illinois University aleland519@gmail.com ABSTRACT One goal of the education doctorate is to prepare educational leaders who can use research-based evidence to solve complex problems related to education and improve lives. We recently completed a mixed methods study of four EdD programs that showed the kinds of experiences that encourage their graduates to use evidence. This paper uses qualitative data from the study to describe in more detail the strategies these programs used to promote evidence use. These strategies helped students develop skills in finding, assessing, and doing research; applying research; and working with others to use research. They ranged in size from the kinds of in- class activities professors used to help students collectively process what they were learning to the coordinated set of assignments spread across three years to help students turn a work-related issue into a research problem while designing and conducting their capstone doctoral project. KEYWORDS doctoral programs, leadership curriculum, leadership knowledge and skills, leadership program design, university programs, qualitative research methods Educational leaders are the key brokers of evidence to their institutions (Daft & Becker, 1978; Daly, et al., 2014; Neal, et al., 2015). Yet, they often find research confusing and jargon filled (Penuel et al., 2018). When they do not understand data analysis or lack the knowledge and networks to access research, they may adopt practices with questionable evidentiary support (Coburn & Turner, 2011; Farley-Ripple, 2012). Several strategies have been tried to facilitate educational leaders’ use of evidence, including early federally funded dissemination programs (Lagemann, 2000) and modern interest in improvement science (Bryk, et al., 2015). Academic preparation is another tool. The EdD should be ideal for helping leaders learn to use evidence because most students are currently educational leaders who, by job responsibility or inclination, are well placed to help colleagues learn about and, when appropriate, act on evidence. The EdD curriculum is supposed to help students understand and assess different kinds of evidence as well as think about how to apply evidence to practical education problems. Moreover, principles of the CPED Framework include that the professional doctorate in education:  Prepares leaders who can construct and apply knowledge to make a positive difference in the lives of individuals, families, organizations, and communities.  Is grounded in and develops a professional knowledge base that integrates both practical and research knowledge, that links theory with systemic and systematic inquiry. (Carnegie Project on the Education Doctorate, n.d., para. 5). Yet, the education doctorate has been criticized for neither adequately developing leaders’ research skills or giving them necessary scholarly habits of mind (Prestine & Malen, 2005; Shulman, et al., 2006) nor preparing leaders for the hurly-burly of practice (Murphy, 2007). Improving EdD programs often requires attention to redesign, experiment, and evaluation, in order to “bring the education doctorates for practice and scholarship in better alignment with their professional and disciplinary analogs (Shulman et al., 2006, p. 30). Findings from recent studies provide some insight for such alignment, particularly in developing practically oriented research skills, such as connecting coursework to practice (Cosner et al., 2015; Honig & Donaldson Walsh, 2019), encouraging and often requiring students to address a problem in their workplace as part of their capstone projects (Buss & Zambo, 2016; Honig & Donaldson Walsh, 2019), and contextualizing workplace problems within a broader body of research (Belzer & Ryan, 2013). In addition to these authentic learning experiences, research has also captured mechanisms that help students learn about research including methods courses that are inquiry based (Bengston et al., 2016); coursework that develops specific skills in finding, assessing, and understanding research (Firestone, Perry, Leland, & McKeon, 2021); about:blank about:blank about:blank about:blank about:blank about:blank about:blank https://orcid.org/0000-0001-9448-3274 Strategies for Promoting Evidence Use Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 6 No. 4 (2021) DOI 10.5195/ie.2021.154 9 as well as hands-on research experiences embedded throughout coursework (Osterman et al., 2014). Despite the insight that these studies have provided, few of these studies have examined the extent to which EdD programs influence graduates’ capacities to understand and use research in their work. A recent exception to this trend is a mixed methods study we conducted that included a survey of EdD graduates. This research provides evidence that the EdD can influence its graduates when it provides methodological training experience grounded in students’ everyday work experience as well as strong social bonding (Firestone, Louis, Perry, Leland, & McKeon, 2020)1. However, the previous study lacked guidance about how EdD programs aspiring to better prepare leaders might use this research to modify their own programs. To address that issue, this paper turns to the study’s qualitative data which suggest that EdD programs helped their students develop abilities in three areas: research skills, application skills, and social skills. These categories emerged from our analysis of each program. The research skills are central to what is taught at the doctoral level and would seem to be fundamental to using research. They are part of what is meant by scholarly habits of mind (Prestine & Malen, 2005; Shulman et al., 2006). Application skills reflect in part what is different about the work of researchers and leaders who use evidence to solve place-specific problems. It is why various kinds of research users are often so concerned to ensure that findings are generated in settings like the user’s own (Weiss & Bucuvalas, 1980). Social skills are required if leaders are to be effective research brokers (Daly, et al., 2014; Honig & Donaldson, 2019) and by the nature of educational decision making where so many decisions require reaching consensus after considerable teaching and learning, politicking and persuasion (Nutley, et al., 2007; Penuel et al, 2018). The remainder of the paper describes our study methods before describing how the programs addressed each of these areas in turn. METHODS For the qualitative portion of this mixed methods study, we adopted a multiple-case design (Yin, 2018). The programs studied were broadly distributed across the country. Three ran programs that were largely face-to-face but that included some online instruction. With the one program that had separate face-to-face and online cohorts, we only report on the face-to-face component. Two required students to complete relatively conventional, individual dissertations; two required group dissertations. All had redesigned their EdD programs in keeping with CPED principles—some in the late 2000s and some more recently. We visited each site to learn about its operation and how it was perceived by participants. Actual visits were made by the lead author and another member of the research team. Before and after the visit, the team collected information from its website and available 1 A copy of this paper can be obtained from the first author at wilfires@gmail.com. documents, including program handbooks, course syllabi, and sample dissertations. Visits lasted two to four days to allow for both interviews and observation. Table 1 provides details about program characteristics and data collection. The faculty interviewed in each program were the bulk of its core instructors. Because students came to the programs from widely distributed work places, they were recruited by program directors to represent the first-, second-, and third-year cohorts. The interviews included a number of criticisms as well as descriptive and complimentary observations. Table 1. Case Study Program Characteristics and Data Arizona State Portland State Michigan State Boston College Dissertation Type Individual Individual Group Group Year Adopted Current Program 2007 2010 2010 2013 Program Enrollment 190 91 70 68 Faculty Interviews 7 6 9 7 Administrator Interviews 1 1 2 1 Current Student Interviews 7 6 8 6 Class Observations 2 4 2 2 Interviews ranged from 45 minutes to an hour, were guided by semi-structured interview guides (Kvale, 2008; Patton, 2002) and were audio recorded. Faculty and administrator interviews addressed the program’s history, the individual’s vision for the program and of evidence use, how the person taught students to understand and use evidence as well as other program goals during coursework, how the person worked with dissertation students, and interactions with peers and students. Questions for students focused on their background, experience with and reaction to their course work and their dissertations, and interactions with faculty and fellow students. Interview guides triangulated the views of students with faculty and administrators. Our two- to three-day site visits were scheduled to maximize opportunities to observe courses and other student-faculty interactions. Researchers attended two to four, face-to-face evening and weekend class sessions that included group data analysis activities, debates, and jigsaws as well as lectures. Access to work with dissertation students was more limited, but at one visit, we observed morning meetings where dissertation groups—with input from faculty—organized and analyzed recently collected field data. We documented all observations with extensive field notes. We entered field notes, program documents, and interviews in Dedoose, a platform for analyzing qualitative data. We began our analysis by focusing on select data from two institutions, to which we applied a set of broadly defined codes that we developed out of the conceptual framework that guided the larger study—namely drawn from studies on research evidence use (e.g., Weiss & Bucuvalas, 1980), situated learning (Lave & Wenger, 1991), communities of practice (Wenger, 1998), and prior research on EdD programs (e.g., Honig & Donaldson Walsh, 2019). Throughout this initial coding process, we met bi-weekly to engage in procedures for inter-rater reliability, allowing us to define codes and refine them into parent and child codes. Once in agreement, we applied the coding scheme to all data sources, reduced codes to patterns and themes, and created thematic memos that were shared with each other for continued refinement and understanding. Firestone & Leland Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 6 No. 4 (2021) DOI 10.5195/ie.2021.154 10 We then developed within-case reports of each site to a common outline (Yin, 2018) organized around program history, mission, program features, and participation patterns. These reports were shared with the respective program, which provided for member-checking and allowed us to ask specific clarifying questions. During this process, we received permission to use institution names in subsequent presentations and publications. Upon completion of all case reports, we analyzed for cross-cutting themes and several emerged around strategies that programs used to promote evidence use. FINDINGS We present our findings by focusing on three skills areas that all four programs emphasized in developing students’ capacities to use research evidence. The first, research skills, includes how all programs taught students to find and understand research, as well as the ways in which these learning experiences provided the foundations for students to be able to do research. The second, application skills, addresses skills developed in doing research in more depth, particularly in how programs prepared students to apply research skills in identifying a problem in their workplace, gathering and analyzing data, and determining the practical implications to find possible solutions. The third, social skills, shows how program structures and classroom interactions presented several opportunities for students to use research in communication, collaboration, and persuasion. Research Skills Through their programs, EdD students developed research skills or how to find, understand, and do research. A number of courses provided students with learning experiences on different methodological approaches to conducting research. These learning experiences occurred primarily in research methods classes much like those found in many social science doctoral programs and began with activities for students to collect and gather studies. Learning to find research was relatively simple, but foreign to people whose work demands do not regularly challenge them to locate empirical studies. Thus, early courses in several programs provided students with formal opportunities to understand how and where to find research. In some programs, students met with a librarian or watched a university-recorded video lesson to learn about different electronic search tools. In others, class meetings in early courses focused on introducing students to these skills. In addition to learning how to find studies, research methods courses allowed students to learn about the principles of qualitative and quantitative approaches to research. Lectures, course meetings, and online response posts contributed to an understanding of the fundamentals of these approaches, while conventional exercises focused on the utility of different techniques. One such exercise included the design of instruments and collecting, analyzing, and interpreting data. A qualitative methods class in one program, for example, assigned students to perform several steps of qualitative analysis from transcribing interviews to conducting an initial data analysis and developing coding scheme. Then, according to the syllabus, students created a data gathering report, allowing them to document their experiences and reflect on how the gathering and analysis process “unfolded over the [term].” In another program, students constructed and piloted survey instruments to deepen their understanding of descriptive and inferential statistics, as well as to use this information to revise their instruments. Literature reviews were another exercise that developed students’ understanding of research. Students conducted literature reviews throughout their coursework, or as one professor stated, “they get a lot of experience in terms of how to read research articles, how to summarize research articles, how to synthesize and analyze themes from research articles.” These assignments served multiple purposes in developing students’ research skills. For instance, they helped students construct conceptual frameworks for what would become the focus of their capstone projects and ultimately understand how this process contributes to gaining knowledge about a given topic. Moreover, the fact that these assignments existed in multiple courses allowed students to build upon previous iterations of literature reviews and contributed to students’ depth of knowledge. Second, literature reviews helped students develop the necessary skills needed in organizing studies. For example, one professor taught her students to approach literature reviews similar to binning excerpts in analyzing qualitative research (Miles & Huberman, 1994), in which students would organize research according to a set of criteria such as similar findings or methods used to examine a topic. In other programs, instructors gave assignments that asked students to search for their own documents and start an account with an online reference manager like Refworks, Endnote or Mendeley and fill it with references to the documents they found. The use of such software not only helped students with organizing studies, but also with learning how to logistically look across multiple studies. One student noted: I think comfort wise I started utilizing some software, so I could store things in a more logistical way. I was making, I assumed, more sense of the research because I was able to organize it in a better way to go back to it and learning how to utilize research so if I read something in an article that I felt contradicted what I thought it was gonna say, I knew how to kind of label that so I knew I could go back to that. If I read something that was spot on, I could pick that out to the point where you also start to ... A couple of the instructors have called it, ‘You're in conversation with the research.’ A student in another program similarly emphasized their training on examining the literature through the use of software—in this case as it developed their capacity to connect prior research to problems that they observed in their working contexts, saying: Even though I've always been a thinker, it's thinking in a different way, and more expansively and I think very quickly trains you to search the literature, that it's very literature based and rooted and how that connects to your daily work. Finally, the use of literature reviews allowed faculty to help students engage in critical discussion on the credibility of sources. At the simplest level, as one professor noted, “they usually learn that journal articles are more favorable.” Beyond that, the use of literature reviews served as useful modes of instruction for students to identify the strengths and limitations of different methodological approaches, how to critique a research article, and how to frame research through prior studies. An instructor at one program highlighted this process: Strategies for Promoting Evidence Use Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 6 No. 4 (2021) DOI 10.5195/ie.2021.154 11 And so, I ask them to go into the literature and find -- First I have them to find five sources, and then they realize that they don't know what different sources are. And so, we talk about what different sources are and how do you evaluate the credibility of the source, and where do you go in the articles themselves to evaluate the methods? To look at whatever potential bias there might be of the person who's writing it. The development of students’ research skills was not siloed to methods courses. That is, all four programs provided numerous opportunities for students to learn how to critique research in substantive coursework. Often the mechanism for this is a mix of reading and class discussion. One professor described how she had students read Piaget’s theory of stages of development. Through class discussion, she helped students see that all the research subjects studied were members of the dominant culture and used a critical theory perspective to point out the sampling bias of that approach. Another professor used the idea of causal stories (Stone, 1989) to critique study design. As students read quantitative articles, he had them pick out methodological aspects designed to give the study credibility--whether it be a sampling strategy, analysis choice or measurement instrument--and have them assess why it might make the study believable and to whom. Doing research also had a place in substantive coursework. In some cases, programs contextualized the process of understanding methods through courses in which students needed to draw from their workplaces to gather and analyze research. Opportunities like these most often occurred in data-informed decision making courses. One program also embedded data analysis in a course on human resources where one of the primary objectives was to develop students’ capacities to make evidence-based decisions around staffing. The professor of that course noted: I turned that course into a research course about human resources [HR], and the students engage in many research projects focused on HR. Again, the focus there is to bridge, we try and learn what people think of when they talk about human resources, but a lot of what we're doing is looking at staffing data. And thinking about how we might make evidence-based decisions around staffing, by looking at that data. As this example also shows, the contextualized application of data analysis also served as a key component in helping students develop research skills, which existed across all four programs. The next section details how programs accomplished these tasks. Application Skills While research instruction is largely similar to what is found in many social science doctoral programs such as learning about methodological approaches, teaching students how to apply such approaches to improve their contexts is not. Programs often taught application skills by adjusting how they taught research skills to better help students employ those skills to use evidence. They did so by having students collect and analyze data (and sometimes share results of those analyses) in their own work settings or settings like theirs. Although some application skills were developed through discrete activities, what was notable was that the individual dissertation programs in particular developed an overarching strategy to help students link all their doctoral work to application issues through the problem of practice (PoP) while group dissertation programs embedded opportunities to apply research skills in course activities. Individual Dissertation Programs The PoP began before students entered. Students were expected to describe their PoP--typically from their own workplace-- in their application to the program and were only selected if their description of the PoP was judged to be adequate. However, faculty only took this initial description as a starting point. Course work was designed to help students develop their PoPs, formulate them into researchable problems, and acquire the skills to conduct a study to address the problem. A student’s dissertation would be that study. Presumably the study results could be used to address the studied problem although that would take place after graduation. The PoP became the laboratory of practice comparable to the residency clinical setting in medical school. We note two main themes in how the PoP was used to help students learn to apply research to practical problems while maintaining a high level of motivation among practice-oriented students. The first was finding a workable problem. Especially in one program, faculty did not assume that students had a clear understanding of what their problem really was. As one professor noted, “When you come in and you have a PoP, the only things that you’re sure of right now is that you have a problem with it. You don’t know if anybody else in your workplace thinks that's a problem.” What the student understood to be the problem might not be the most effective way to understand the problem to ameliorate the situation. One way to address this situation was to have students conduct reconnaissance beyond their additional assumptions to get a broader understanding of the situation in their own workplace including perceptions of others. This school required students to conduct and write up such reconnaissance. This reconnaissance was captured as an initial “cycle of action research” intended to help a student better understand the pragmatics of a problem. In this program, later cycles of action research would be used to pilot instruments and an intervention to address the problem. These activities combined learning about research methods with refining the ultimate PoP in a way that helped students understand how to use research to address practical problems more generally. For instance, assessing instrument validity became, in part, an exercise in assessing the fit with the local context. An instructor who taught instrument development as part of quantitative methods insisted that part of assessing instrument validity was sharing a new questionnaire with critical friends on site to get their perceptions of it. Related to the challenge of defining the PoP in a practical manner was avoiding solutionitis (Bryk et al., 2015). In both individual-dissertation programs, faculty warned students not to settle on a solution to their PoP in the form of a specific intervention before they had a clear idea of what the underlying problem was. In addition to conducting further reconnaissance to understand a PoP, students used existing research to refine and clarify it. Such use of research was the second theme. A substantial amount of this work took place through writing the dissertation proposal and course assignments that led up to it. These assignments were often centrally specified. That is, a program would agree on the writing tasks for each course so they would be assigned regardless of the instructor. How assignments were interpreted would depend somewhat on the professor, however. They varied somewhat in how Firestone & Leland Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 6 No. 4 (2021) DOI 10.5195/ie.2021.154 12 they suggested using past research to clarify their understanding of their problem. According to one: …[I] get a lot of questions about what's the right theory?... I talk to them a lot about... evaluating what... people have said about the theory in that sense. It's not maybe a discussion of evaluating what their data specifically is, but does it feel like it's concrete…. Does it feel like it helps you explain the situation? Does it feel like it helps you ask research questions that feel true to what you want to think about in that context? It should be noted that the courses provided opportunities to apply research to more than one’s own problem. Most provided opportunities for students to discuss their problems with their peers. In the process, they learned about other people’s problems and practiced applying the literature they were learning about to a variety of issues. In addition, some individual-dissertation programs designed the dissertation qualifying examination as proposal preparation activity that helped students learn to use research literature to clarify a practical problem. One program had a very elaborate examination at the end of the first year. It tested how well students could use what they had learned in early courses to clarify their PoPs. As a student explained: you have section A... it's identifying our problem of practice. Section B… everything is all about the problem...but looking at it from the learning theories. Section C, looking at it from the organizational and leadership theories. Section D, looking at it from the policy and politics theory. This examination was scaffolded in two ways. First, the first- year courses on the three areas mentioned in the quote above were all designed to help students prepare for it, as well as the dissertation proposal. Students had numerous opportunities in these courses to write short papers where they synthesized course readings and related literature in a way that clarified some aspect of their research problem. Second, the qualifying paper had a scoring rubric that was distributed in advance. It specified that papers would be scored on how well they analyzed a problem of practice using the literature from those three areas and specified what constituted a good analysis. Group Dissertation Where programs pointed students towards a group dissertation rather than a PoP dissertation, other strategies were used to help students learn to conduct applied research. One program required students to take a “problem of practice” course. According to the professor, the goal of the course was to give students experience “us[ing] the literature to actually inform how you attack that problem of practice?” In addition to finding and reading relevant literature, students learned to apply some implementation science tools for diagnosing issues and sometimes collected data to clarify the issue. By the end of the course, according to the professor, “the goal is that they will leave… with a whole intervention laid out that they can begin to put in place.” Other courses provided students with opportunities to conduct research to address real problems, often with real users of the results. In the human resources course mentioned above, students spent most of the summer collectively analyzing personnel data from one district. At the end of the course, they would present both an oral and a written report to the superintendent who provided the information. Both group dissertation programs had students conduct equity audits (Skrla et al., 2004). In these audits a district examines data to assess both the equity of outcomes for various groups of students and of the services that they receive to identify modifications to increase equity of outcomes across groups. When conducting such audits, students would identify an equity issue in a specific school or district, collect and analyze relevant data and report back to decision makers. In one program, the equity audit was an assignment in a course on social justice. In the other it was a joint assignment for simultaneously taught courses on social justice and “Data and Decisions.” This course, the main opportunity for students to learn about quantitative methods, focused on a variety of applied problems. When students doing equity audits were decision makers, they might spend some time after the course was over working on the issue. One district administrator told how after finishing a course- based equity audit he worked with high school teachers to change prerequisites for AP courses so enrollments would better represent the minority population of the school. While group dissertations were almost never a PoP in one student’s workplace, they always addressed practical problems. This was in keeping with faculty understanding of their programs’ missions which focused on inculcating a “social justice mission” in their graduates, preparing school administrators, and, as one said “is to really prepare school leaders, both at the building level as well as the district level, and state level for that matter, to attack pressing problems of practice.” Students had to reach consensus on the problem the group would address, but because members of these groups were and aspired to be educational leaders, the problems they chose were always practical. These took different forms in the two programs. One required students to do a district case study. While the problem the case study addressed was always defined by the students, the process of requesting permission and data collection generated local interest so the dissertation group was required to debrief with relevant officials in written and oral form after the analysis was completed. The other program allowed teams greater freedom in study design. We identified dissertations where a team would  conduct and study parallel interventions in each member’s district,  conduct a state-level policy study and--in addition to the dissertation--present a report to a relevant state-level audience, and  conduct design research that culminated in the development of a product useful for a specific purpose. Thus, while approaches to learning to apply research were more diverse in the group dissertation programs, instructive models were clearly provided. Social Skills In addition to research and application skills, students developed skills in negotiating the social dynamics of interacting and collaborating with others—particularly in using research evidence to make informed decisions about changes to practices and policies in P-20 education. This emphasis was deeply embedded in all four programs, beginning with overall goals and objectives for students to strengthen partnerships and build relationships with various Strategies for Promoting Evidence Use Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 6 No. 4 (2021) DOI 10.5195/ie.2021.154 13 stakeholders. Program handbooks and promotional materials, for example, highlighted intended outcomes for students in their programs to lead organizational change and community advancement through collaborative action. In addition to these materials, faculty understood collaboration as a cornerstone for each program. For example, a faculty member at one program noted that “I think that our goal is to provide students a really intensive experience with collaboration. There’s the big focus on our program on distributed leadership and what that looks like in practice.” A professor at another program similarly said, “We want [students] to be leaders, we want them to be collaborators, we want them to be I think innovators in terms of practice.” Operationally, developing students’ capacities to effectively collaborate with others occurred in three ways: understanding forms of meaningful and constructive feedback and engaging in in-class group activities and group assignments. Student participation was a critical component in nearly all courses, and course syllabi outlined a number of parameters for what instructors believed would be most effective in interacting and collaborating with other students. One course, for example, described participation as: not just... offering your ideas in class. It also means that you have demonstrated the ability to listen carefully to others’ ideas, respond to others’ ideas (as in discussion), and to monitor your own talking in relation to the others in class. This means that you are aware of when you have talked more than others and give others an opportunity to present their ideas. Syllabi in other courses similarly identified that students were expected to be “actively listening and sharing in the discussion and activities,” as well as asking students to move beyond statements such as “I agree,” “I liked what you said,” and “yes that’s correct” when interacting with peers. One class observation captured an instructor unpacking student conversations—particularly around the use of “should”—when discussing implications. Field notes documented, “Instructor said, ‘listen to yourself and your classmates use the word should. What does that shut down or allow?’” In-class activities consisted of fishbowls, turn-pair-shares, think- aloud activities, and the use of critical friends that appeared to exist regularly in their respective classes. Fishbowl strategies, for example, asked students to engage in structured discussions on issues in view of their classmates by carefully listening and actively presenting ideas to each other. Although ways to structure fishbowls vary, one particular syllabus captured how the strategy would develop students’ skills in collaboration and communication, noting that it would help students “to share connections, topics you are engaged in/struggling with, and questions you have.” The use of critical friends also emphasized engagement with peers, where students were expected to, as another syllabus indicated, “submit the channel you used to connect (Facebook, LinkedIn, Google Alerts, discussion boards, e-mail, phone call, etc.), who you connected with (a group or individuals), and a summary of your interactions thus far.” Although slightly different in their formats, each approach used to facilitate engagement and participation encouraged students to make meaning of information in small group formats before returning to whole-group discussions with an entire class. Classroom observations captured the extent to which discussions and small-group work supported each program’s dedication to community-building and providing beginning spaces for collective learning opportunities. For example, one class structure allowed students to meet in small groups, workshop their assignment on communicating with families, use class-based readings to justify and evaluate their assignments, and talk with other small groups when reporting out. Field notes from a class at another institution also captured the extent to which one instructor’s constructivist approach to teaching and learning pivoted on in-class group work: Basically, we saw three [activities] this session: A quick ice-breaker where small groups introduced themselves to each other and then reported out on themselves to the whole class; one where groups had to construct mini- lectures for their peers on [learning] theorists; and one where different groups were assigned a learning tool. In addition to in-class activities, group projects offered students a number of opportunities to learn and develop some of the necessary skills for interacting and collaborating with others. These included group presentations and cycles of peer feedback on various assignments. They also included larger projects--including case studies, policy analysis assignments, and equity audits--that similarly challenged groups to analyze a particular task, topic, or problem collectively. Although all institutions included group assignments to some extent, they were even more present in institutions in which students completed group capstone projects. As an instructor from one institution noted, the use of group equity audits helped as a “kind of rehearsal for them working together as a research team.” This instructor described further the types of collective tasks in which students engage: As part of the semester-long project of doing equity audits, they have to go out and they have to collect data in the district so there's study. They have to interview people, they have to look at demographic data, they have to download [state test] data from the state websites and analyze where the gaps are and so on. Students similarly noted the benefits of working on major group assignments, especially in developing their skills to work with others. One student discussed: I think I worked with everybody in some arena and so it's nice to have worked on various projects with various people. They've done a lot to sort of say [...] they're pulling a group together, "okay, switch it up.” Now you have to work with this group or consider working with each other here." It's been nice to get to know people through that as well. Another student noted that, “Every course required us to do some work in groups. I think it improved my leadership skills, again in working with different groups of people and towards that goal.” As both students explain, faculty intentionally challenged students to work in different groups throughout the program, an approach that helped students develop the collaborative skills that are important to effective leadership. In addition to collaboration, some programs emphasized persuasive skills, particularly in using research evidence in decision- making processes. One way in which this occurred was conceptually, that is by critically unpacking the components of a persuasive argument. For example, in helping students learn how to critique research, the instructor who introduced causal stories, (Stone, 1989) simultaneously challenged them to connect pieces of evidence to support one causal story over another based on research and/or data. Through this process, students learned Firestone & Leland Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 6 No. 4 (2021) DOI 10.5195/ie.2021.154 14 valuable skills in persuasion, particularly in how to communicate a story to various stakeholders. As one student explained: The data can be taken and looked at in many different ways. That's been a big conversation as well. Not only how to delineate good data from bad data but then also using that data to tell the story that you want to tell. And making sure, again, that if you take this approach, your story might look a little bit different than if you take this approach, but it's all from the same data, it's how you frame it, how you explain it, those types of things. In some programs, opportunities to report data to stakeholders enriched their skills in persuasion and decision-making. The instructor of one course, for example, leveraged relationships with neighboring local districts for a project in which students analyzed data from an assigned district and presented their findings back to the district. This approach was also a primary component of one program’s group capstone project, in which students reported their findings to the schools and/or districts in which they conducted fieldwork by way of formal presentations and a written executive summary that became part of their written capstones. DISCUSSION AND CONCLUSION The focus of this paper on four EdD programs deepens and expands our understanding of mechanisms that promote practically oriented research skills. As our findings demonstrate, all four programs were committed to developing students’ capacity to use evidence in several areas. They were not only focused on the fundamentals of research (e.g., finding, assessing, and understanding), but also included a number of learning experiences around the application and communication of research. These programmatic and instructional components not only underscore CPED’s principles in constructing, applying, and integrating knowledge, they also serve as productive and meaningful approaches to connecting research and practice, developing students’ scholarly habits of mind, and influencing leaders to become more knowledgeable brokers of evidence in P-20 education. In the section that follows, we draw upon prior research to discuss some of these approaches in greater detail and conclude with implications for practice. Multiple Uses of Research Students learned about different qualitative, quantitative, and mixed-methods techniques; strengths and limitations to different techniques; as well as how to find research and assess credibility. What was noticeable about the four EdD programs, however, was that the development of those skills served as a foundation for students to move beyond a simple “research-for-problem-solving” paradigm and instead, learn about “the multiple functions that research can serve” (Weiss & Bucuvalas, 1980, p. 312). For instance, each program emphasized using research evidence to deepen one’s understanding of a particular issue in P-20 education (i.e., through shifts in attitudes or conceptually framing the issue), as well as to identify and suggest possible solutions to an issue through changes in policy and practice. Similar to what Belzer and Ryan (2013) found in students’ use of evidence to contextualize their dissertation topics in prior research, these four programs indicate a strong connection between coursework (e.g., research skills, theory) and practice that promoted students’ understanding and use of research. In addition to using research to deepen students’ understanding of a particular issue, some programs provided opportunities for students to develop skills in communicating evidence to stakeholders. Each program’s emphasis on collaboration and collective learning contributed to developing these skills through group projects, discussion-based classes, some noticeable constructivist approaches to teaching and learning, and one program’s use of causal stories (Stone, 1989). As students learned through these experiences and interactions, effective communication requires some depth of understanding of research, policy, and practice to help persuade others in decision-making processes. Although the development of students’ social skills was prominent in coursework, most programs did not formally integrate those skills into the capstone project beyond creating a chapter on implications and recommendations—a limitation of EdD capstone projects similarly identified by Honig and Donaldson (2019). An exception to this included one program’s requirement that students formally present their findings to the schools and districts in which they conducted their studies. Integrating Skills Development throughout Coursework and Milestones Cosner et al. (2015), Bengston et al. (2016), and Honig & Donaldson (2019) reminded us of the importance of connecting coursework to practice in developing scholarly-informed practitioners and P-20 brokers of evidence. Our findings expand these notions further, particularly in relation to how all four programs prepared their students to understand and use research. Similar to what Bengston et al. (2016) argued to be useful, the programs in the current study positioned research skills development throughout coursework. Students learned about the fundamentals of research in their methods classes, but they continued to deepen their understanding of content, credibility, and practical uses of research in substantive coursework on leadership, policy, and law, to name a few examples. In short, opportunities to develop the various skills in research (including applications of and communication) occurred throughout nearly all courses and continued as students completed their capstone projects. In addition, one hallmark of all four programs was that they engaged students in the process of designing and conducting research well before they began their capstone projects. For institutions with individual projects, these endeavors were centered on the PoP, which ultimately became the main focal point of helping students develop their research skills throughout coursework while leading to their qualifying exams and capstone projects. Institutions with group dissertation projects certainly encouraged students to identify practical research problems, but the development of their research skills occurred primarily through course assignments and other major projects throughout the program (e.g., equity audits, district case studies). Regardless of approach, the process of doing research occurred throughout coursework in a way that highlights how programs in previous research have used cycles of inquiry (Cosner et al., 2015) and action research (Buss & Zambo, 2016; Osterman et al., 2013). As these studies indicated and as our findings support, the process of doing research early on in programs provides students with learning experiences to understand some of the fundamentals of research that they could then apply and develop Strategies for Promoting Evidence Use Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 6 No. 4 (2021) DOI 10.5195/ie.2021.154 15 further as coursework progressed. Once students reach their capstone projects, and ultimately after they graduate, students will be better equipped to use these skills to identify problems and possible solutions in P-20 environments. From Research to Assessment for Improvement In this article, we used Yin’s (2018) multiple case study approach to explore how four EdD programs developed students’ skills in accessing, understanding, assessing, applying, and communicating evidence. This method helped us identify the approaches these programs used to promote multiple uses of evidence among their graduates. We conclude by reflecting on how our methods could help inform actionable recommendations for specific EdD programs and offer two suggestions to consider when evaluating and making changes to a specific program. A number of scholars have suggested the use of a continuous improvement model for leadership-preparation programs to collaboratively examine their practices and outcomes (e.g., Cosner, 2019; Honig & Donaldson, 2019). While we did not use this model, we echo Cosner’s (2019) sentiments that this model values the use of multiple data sources to understand program effectiveness and program impact. One such data source is interviews, which could complement the use of course evaluations that are typically used to gain student input. Through our interviews, we learned about the experiences and perspectives of both students and faculty members—an approach that could highlight specific instructional practices that are effective for student learning. In addition, a review of course syllabi and program documents (e.g., handbooks, marketing materials) provided information for how programs developed student knowledge and skills around research. Having access to these data sources allowed us to identify course objectives, pedagogical approaches, and major assignments and projects. Moreover, through this process, we were able to examine how individual courses connected to each other within a given program, as well as how individual course objectives aligned with each program’s overall goals. Such curriculum mapping would not only provide programs with additional data sources to analyze but could also be valuable in understanding a program’s scope and sequence of coursework and milestones. More generally, we urge EdD faculty to use their knowledge of research to assess their own processes to improve their students’ use of evidence. REFERENCES Belzer, A., & Ryan, S. (2013). Defining the problem of practice dissertation: Where's the practice, what's the problem? Planning and Changing, 44(3- 4), 195-207. Bengston, E., Lasater, K., Murphy-Lee, M., & Jones, S. J. (2016). The role of research courses. In J. A. Perry (Ed.), The EdD and the scholarly practitioner (pp. 79-104). Information Age Publishing, Inc. Bryk, A. S., Gomez, L. M., Grunow, A., & LeMahieu, P. G. (2015). Learning to improve: How America's schools can get better at getting better. Harvard Education Press. https://doi:10.1002/sce.21223 Buss, R., & Zambo, D. (2016). Using action research to develop scholarly practitioners. In J. A. Perry (Ed.), The EdD and the scholarly practitioner (pp. 137-153). Information Age Publishing, Inc. Carnegie Project on the Education Doctorate. (n.d.). The CPED framework. Retrieved from https://www.cpedinitiative.org/the-framework Coburn, C. E., & Turner, E. O. (2011). Research on data use: A framework and analysis. Measurement: Interdisciplinary Research and Perspectives, 9(4), 173-206. doi:10.1080/15366367.2011.626729 Cosner, S. (2019). What makes a leadership preparation program exemplary? Journal of Research on Leadership Education, 14(1), 98-115. Cosner, S., Tozer, S., Zavitkovsky, P., & Whalen, S. P. (2015). Cultivating exemplary school leadership preparation at a research intensive university. Journal of Research on Leadership Education, 10(1), 11-38. Daft, R. L., & Becker, S. W. (1978). Innovation in organizations: Innovation adoption in school organizations. Elsevier. Daly, A. J., Finnigan, K. S., Moolenar, N. M., & Che, J. (2014). The critical role of brokers in the access and use of evidence at the school and district level. In K. S. Finnigan, & A. J. Daly (Eds.), Using research evidence in education: From the schoolhouse door to Capitol Hill. (pp. 13-32). Springer. Farley-Ripple, E. N. (2012). Research use in school district central office decision making: A case study. Educational Management Administration & Leadership, 40(6), 786-806. doi:10.1177/1741143212456912 Firestone, W. A., Perry, J. A., Leland, A. S., & McKeon, R. T. (2021). Teaching research and data use in the education doctorate. Journal of Research on Leadership Education, 16(1), 81-102. doi:10.1177/1942775119872231 Firestone, W. A., Louis, K. S., Perry, J. A., Leland, A. & McKeon, R. (2020). Teaching evidence use to educational leaders: The Ed.D.'s contribution. AERA Annual Meeting, April, 2020 online. Honig, M. I., & Donaldson Walsh, E. (2019). Learning to lead the learning of leaders: The evolution of the University of Washington’s education doctorate. Journal of Research on Leadership Education, 14(1), 51-73. doi:10.1177/1942775118819673 Kvale, S. (2008). Doing interviews. Sage. Lagemann, E. C. (2000). An elusive science: The troubling history of educational research. University of Chicago Press. Lave, J. & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge University Press. Miles, M. B. & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Sage. Neal, J., Neal, Z., Kornbluh, M., Mills, K., & Lawlor, J. (2015). Brokering the research-practice gap: A typology. American Journal of Community Psychology, 56(3), 422-435. doi:10.1007/s10464-015-9745-8 Nutley, S. M., Walter, I., & Davies, H. T. O. (2007). Using evidence: How research can inform public services. Policy Press. Osterman, K., Furman, G., & Sernak, K. (2014). Action research in EdD programs in educational leadership. Journal of Research on Leadership Education, 9(1), 85-105. doi:10.1177/1942775113498378 Penuel, W. R., Farrell, C. C., Allen, A., Toyama, Y., & Coburn, C. E. (2018). What research district leaders find useful. Educational Policy, 32(4), 540- 568. doi:10.1177/0895904816673580 Prestine, N., & Malen, B. (2005). The case for revitalizing the dissertation. UCEA Review, 47(2), 7-9. Shulman, L. S., Golde, C. M., Bueschel, A. M., & Garabedian, K. J. (2006). Reclaiming education's doctorates: A critique and a proposal. Educational Researcher, 35(3), 25-32. Skrla, L., Scheurich, J. J., Garcia, J., & Nolly, G. (2004). Equity audits: A practical leadership tool for developing equitable and excellent schools. Educational Administration Quarterly, 40(1), 133-161. Stone, D. A. (1989). Causal stories and the formation of policy agendas. Political Science Quarterly, 104(2), 281-300. Weiss, C. H., & Bucuvalas, M. J. (1980). Social science research and decision- making. Columbia University Press. Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge University Press. Yin, R. K. (2018). Case study research and applications: Design and methods (6th ed.). Sage. about:blank about:blank Impacting Education: Journal on Transforming Professional Practice William A. Firestone Rutgers University Andrew S. Leland Southern Illinois University Methods Findings Research Skills Application Skills Individual Dissertation Programs Group Dissertation Social Skills Discussion and Conclusion Multiple Uses of Research Integrating Skills Development throughout Coursework and Milestones From Research to Assessment for Improvement References