Impacting Education: Journal on Transforming Professional Practice New articles in this journal are licensed under a Creative Commons Attribution 4.0 United States License. This journal is published by Pitt Open Library Publishing. 10 This journal is supported by the Carnegie Project on the Education Doctorate: A Knowledge Forum on the EdD (CPED) cpedinitiative.org impactinged.pitt.edu ISSN 2472-5889 (online) Vol.10 No.2 (2025) DOI 10.5195/ie.2025.451 Cultivating Change Agents through Effective Program Evaluation: Inaugural Year Insights from an Equity-Oriented EdD Program Tiffany Karalis Noel University at Buffalo–SUNY tbkarali@buffalo.edu Julie Gorlewski Hunter College–CUNY julie.gorlewski@hunter.cuny.edu ABSTRACT This article presents insights from X University’s inaugural EdD program, evaluating its curriculum, pedagogy, and innovations through an annual program evaluation. Designed for full-time professionals, this part-time, online program emphasizes interdisciplinary improvements to educational Problems of Practice (PoPs), aligned with the Carnegie Project on the Education Doctorate (CPED). The evaluation revealed students’ high satisfaction with faculty quality and program leadership. The cohort model's role in community building and the flipped defense model were highlighted as progressive educational approaches. Students appreciated the program's relevance to their career goals, with many aiming for academia or administration, while feedback suggested expanded course variety and increasingly flexible scheduling. A key finding is that traditional forms of program evaluation may not support the goals of innovative programs. This article, therefore, concludes with strategic recommendations for using the evaluation process as a core element for program enhancement, guided by its commitment to social justice and educational equity. KEYWORDS cohort model, curriculum and instruction, dissertation in practice, Doctor of Education, EdD, educational leadership, social justice education, scholar practitioners In the ever-evolving educational landscape, characterized by rapid advancements in pedagogical theory (Lam et al., 2021; Lucas & Kinsman, 2016) and the expansion of digital learning platforms (Akour & Alenezi, 2022; Park & Kim, 2022), the Doctor of Education (EdD) program stands as a paradigm of innovation. Established in a strategic alliance with the Carnegie Project on the Education Doctorate (CPED) — an international network of over 135 graduate schools of education committed to transforming the advanced preparation of educational professionals (2022) — the EdD program is meticulously designed with a twofold aim. It aspires to equip educational leaders and practitioners with the necessary skills to identify and address persistent and emergent Problems of Practice (PoPs), while also supporting the career progression of full-time educators and those transitioning from doctoral studies to roles in educational leadership and praxis. Launched in the Summer of 2022, the EdD program is on a mission to prepare educational change agents to lead and transform across various settings, from K-12 to higher education and beyond. This three-year, part-time doctoral program blends synchronous and asynchronous learning modalities, offering a flexible yet rigorous academic structure conducive to the busy schedules of working professionals. Hybrid programs, in general, are known for their ability to provide such flexibility, making it easier for professionals to balance their academic pursuits with work and personal commitments (Clark & Barbour, 2015; Glazer, 2012). The program’s innovative delivery is designed to leverage the strengths of an online format, fostering interactive and collaborative learning experiences that transcend geographical constraints (Castro & Tumibay, 2019). The EdD program is characterized by its cohort model, a deliberate structural choice that reinforces its pedagogical ethos. This approach facilitates a collective learning journey (Barnett et al., 2000; Butterwick et al., 2012; Seifert & Mandzuk, 2006), with students progressing together through a series of systematically sequenced academic milestones (Leland et al., 2020; Taylor, 2007). The program launched with a diverse inaugural cohort of 23 students in the Summer of 2022, expanded to 36 in the following year, and welcomed 39 PhD-to-EdD transfer students, nineteen of whom have graduated. Such diversity within and across cohorts, encompassing but not limited to race, ethnicity, gender, age, and educational and professional backgrounds, enriches the program's collaborative discourse and reflects the multiplicity of perspectives within the field of education (Bovill, 2020). With a curriculum grounded in real-world application, the program integrates design and research methodologies alongside a Dissertation in Practice (DiP), reinforcing its commitment to practical, outcome-driven scholarship. Supported by a comprehensive curriculum map and strategic course planning, the program guarantees a robust, scaffolded progression (Brauer & Ferguson, 2015) from theoretical foundations to applied research, tailored to address the unique PoP identified by each student. Additionally, elective courses provide further customization, allowing learners to https://library.pitt.edu/e-journals http://creativecommons.org/licenses/by-nc-nd/3.0/us/ http://cpedinitiative.org/ https://orcid.org/0000-0003-1989-1643 Karalis Noel & Gorlewski Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 10 No. 2 (2025) DOI 10.5195/ie.2025.451 11 explore disciplinary foci (Rawle et al., 2017) that align closely with their individual professional trajectories. The EdD program is anchored in a justice-focused framework aimed at cultivating educational leaders and practitioners proficient in addressing challenges across diverse educational landscapes. This framework transcends academic theories, serving as a call to action that encourages students to engage critically with issues of equity, diversity, justice, and inclusion (Furman, 2012; Jean-Marie et al., 2009). It implores them to apply these critical understandings to positively influence the communities and professional environments they seek to serve through their research (Shields, 2002). This article examines the EdD program's effectiveness in cultivating educational change agents during its first year, 2022- 2023. Utilizing a blend of curriculum and program evaluation analysis, the report elucidates how the program encourages interdisciplinary collaboration, supports progressive learning, and meets participants' career goals. It unfolds to offer a comprehensive view of the EdD’s pedagogical framework, innovative features, and its impact on the first cohort and students who transitioned into the program from a PhD program within the same department. In particular, this analysis emphasizes the potential for aligning program evaluation with program values in ways that are mutually reinforcing (Patton, 2017; Suàrez-Herrera et al., 2009). The insights drawn will inform ongoing improvements, ensuring the program's relevance and foresight in equipping future leaders for an evolving educational landscape. Purpose and Evaluative Approach This article presents a holistic analysis of the EdD program's inaugural year, focusing on its role in preparing educational leaders and practitioners to address complex PoPs in diverse settings. The purpose was to assess the program's effectiveness in aligning with its core values, fulfilling its curricular and pedagogical goals, and supporting its students’ professional development. Additionally, the evaluation sought to gauge the program's influence on students' professional trajectories. It is important to note that this evaluation aims to evolve as a mechanism for systemic improvement, incorporating student feedback as a central element of the process. Positionality Statement As the authors of this evaluation and program leaders, our positionality inherently influences our perspective and approach to this analysis. Our roles as program leaders involve a deep commitment to the principles of social justice and educational equity, which are core to the EdD program's mission. This commitment stems from our professional and personal experiences as educators, scholars, and advocates for marginalized communities within educational contexts. Being closely involved in the design, implementation, and continuous improvement of the EdD program, we bring both an insider’s understanding and a critical lens to the evaluation process. While this proximity allows for a nuanced and informed analysis, it also introduces potential biases. We are aware that our advocacy for the program’s goals and our interactions with students and faculty could influence the interpretation of the data and the conclusions drawn. To mitigate these biases, the evaluation process was designed to be as transparent and inclusive as possible, incorporating diverse student voices and feedback mechanisms. Efforts were made to ensure that the data collection and analysis processes were robust and reflective of the participants' genuine experiences. Despite these measures, it is important to acknowledge that our positionality as program leaders may shape the framing and emphasis of the evaluation findings. Evaluation Approach Drawing connections to scholarly literature, the analysis examined the program’s curriculum, pedagogy, and innovations via an annual program evaluation to capture a multidimensional view of its design, delivery, and the resultant outcomes as perceived by its stakeholders. The evaluation process is designed to be responsive, continually seeking to identify and address, from the student perspective, what might be irrelevant or missing. The evaluative review of the program's curriculum and structure was conducted through an analysis of program documentation, including the curriculum map, course syllabi, program handbook, and milestone descriptions. This review provided a foundation for understanding the program's projected path and the support mechanisms in place for student progression. The foundation was complemented by a systematic approach to include student voices in the evaluation process, aiming to draw lessons for systemic improvement. This was followed by an analysis of the annual evaluation results, which comprised both quantifiable satisfaction measures and qualitative feedback to appraise the program's efficacy. Table 1. Components of the EdD Program Evaluation Component Description Methodology Purpose Curriculum and Structure Review of curriculum map, course syllabi, program handbook, and milestone descriptions to outline the program's intended path. Document analysis To establish a foundational understanding of the program's design and the support mechanisms for students. Student Feedback Integration Systematic inclusion of student perspectives to ensure the evaluation reflects real-world experiences and needs. Course evaluations, focus groups, individual interviews To identify areas for systemic improvement and align the program more closely with student needs. To corroborate the Annual Evaluation Findings. Annual Evaluation Findings Analysis of quantitative data on student satisfaction and qualitative feedback from participants. Quantitative and qualitative analysis (descriptive statistics, thematic analysis) To evaluate the program's effectiveness, identify strengths, and pinpoint opportunities for enhancement. The subsequent section of the article will present the annual evaluation results, providing a detailed account of the program's strengths and opportunities for improvement. These insights inform the ensuing discussion and conclusions, offering actionable insights for the program's continuous enhancement and its responsiveness to the dynamic field of education. In addition, the results of this analysis can offer guidance for programs developing innovative opportunities for engaging in equity-oriented leadership preparation. Program Design and Pedagogy The EdD program is distinguished by its commitment to interdisciplinary methodologies, positioning itself as a model of Cultivating Change Agents Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 10 No. 2 (2025) DOI 10.5195/ie.2025.451 12 111 contemporary educational design that aims to develop scholarly practitioners who can make a significant difference in their fields. The following sections outline the program's design and the pedagogical principles that intersect to provide a transformative educational experience for educational leaders and practitioners. Each of these program features is addressed in the multilayered evaluation process. Curriculum Map and Milestone-Based Learning Central to the EdD program’s structure is a detailed curriculum map, a strategic tool that provides a transparent trajectory of learning from admission to program completion (Cuevas et al., 2010; Treadwell et al., 2019). The curriculum is meticulously designed to scaffold learning and guide students through a progression from foundational theoretical knowledge to the advanced application of research methodologies. This scaffolded approach ensures that students become well-versed in their selected disciplines, with the curriculum map acting as a guide for navigating their scholarly journeys (Brauer & Ferguson, 2015; Cuevas et al., 2010; Rawle et al., 2017; Treadwell et al., 2019). Also integral to the program’s structure are the academic milestones that students encounter. These include prequalifying papers, qualifying papers, DiP proposals, and DiP findings defenses. Each milestone is thoughtfully integrated into the curriculum, providing students with clear benchmarks for success and a tangible sense of progression through the program. These milestones represent critical moments that highlight the students’ increasing proficiency and capacity to conduct research and contribute original insights to the field of educational practice. Cohort Dynamics and Community Engagement The cohort model is integral to the EdD program, functioning as the fulcrum of community engagement and collaborative pedagogy. This paradigm fosters an enriched learning community, positioning students as collaborators within an interactive collective rather than as isolated learners, thus facilitating intellectual reciprocity throughout their scholarly pursuits. Within this microcosmic educational ecosystem, the individual’s academic growth is synergistically enhanced by the collective insights and mutual support of the cohort members (Barnett et al., 2000; Butterwick et al., 2012; Leland et al., 2020; Seifert & Mandzuk, 2006; Taylor, 2007). The program strategically utilizes synchronous and asynchronous modalities to promote learning community cohesion. Synchronous engagements, such as real-time seminars and interactive workshops, catalyze immediate intellectual exchange and relational dynamics (Racheva, 2018). Asynchronous activities, encompassing discussion forums and group-based inquiries, foster contemplative analysis and provide temporal flexibility. This blended integration of modalities is designed to accommodate the exigencies of working professionals, thereby ensuring an academically rigorous and interconnected learning environment, independent of physical locality (Jorgensen, 2003; Zinkhan, 2005). A Justice-Focused Educational Framework The EdD program is steeped in a justice-focused educational ethos which intends to permeate every facet of its curriculum. This framework is predicated on the belief that educational leaders and practitioners should be at the forefront of advocating for and actualizing social justice and equity within their spheres of influence. Accordingly, the program’s pedagogy is designed to cultivate leaders who are not only conversant in theories of justice but are also adept at applying these principles in practice to effect meaningful societal transformation (Al-Faham et al., 2019; Shields & Hesbol, 2020; Wang, 2018). This commitment to justice is woven into the curriculum via courses such as Curriculum Theory, Critical Interpretations of Research, and Intersectionality and Equity. Course content challenges students to grapple with complex social issues and consider the implications of their educational leadership and practice through a lens of equity and inclusivity. This approach ensures that graduates of the EdD program are well-equipped to address systemic inequities and lead with a deep understanding of the transformative potential of education. Program Features and Structural Innovations The EdD program distinguishes itself with a collection of features and innovations that contribute to a robust and dynamic learning environment attuned to the complexities of modern educational leadership and practice. This section explores these features and innovations in detail, examining how they serve the program's mission and enrich the educational experience. As with curricular components, the effectiveness of these features and innovations are assessed through program evaluation processes. Flipped Defense Model A distinctive feature of the EdD program is the flipped defense model implemented for the DiP milestones. This innovative approach transforms the traditional defense approach by requiring students to pre-record their presentations and distribute them to their committee and peers one to two weeks prior to the live defense session. As evidenced in a meta-analysis of 10 years of research (Lo & Hew, 2019), the flipped model significantly improves student achievement. In the context of doctoral education, the model deepens engagement and enables a more substantive and nuanced conversation during the actual defense; it underpins a more thorough critique while taking into account the busy schedules of working professionals. In addition, this model provides opportunities for non-traditional students to develop confidence through practice allowing time to polish a final version of their presentation and focus on discussion during the defense. Consequently, the flipped defense reflects and promotes the program commitment to equity and inclusion. While the flipped model is mandatory for all preliminary defenses leading up to the final DiP findings defense, students have the option to choose between a flipped model or a traditional live presentation for their final defense. This programmatic flexibility allows those who prefer a live presentation to do so, especially if they wish to have family, faculty, or peers present on location. Scaffolded Dissertation in Practice (DiP) Model The EdD program's DiP framework distinguishes itself from traditional dissertations by weaving its components into the curriculum at strategic points to maximize relevance and impact, concentrating on practical application of theoretical concepts. This process unfolds in a four-chapter DiP, mirroring a persistent, introspective academic journey. The prequalifying paper, forming the first milestone and a draft of Chapter 1 (Introduction), is defended at the end of the first year (Spring 1). The qualifying paper, as the second milestone and draft of Chapter 2 (Background Analysis), is Karalis Noel & Gorlewski Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 10 No. 2 (2025) DOI 10.5195/ie.2025.451 13 submitted for feedback in the second year (Fall 2). The third milestone, a draft of Chapter 3, the DiP Proposal, culminates in a defense by the end of the second year (Spring 2). Finally, the DiP Findings encompass a complete DiP from Chapters 1 to 4 and are defended by the end of the third year (Spring 3). Each stage is aligned with the corresponding academic phase, ensuring that the research is actionable and immediately applicable to the field of education. Moreover, this approach to the completion of the culminating project of the doctoral degree reinforces the effectiveness of the cohort model. Candidates collaborate in providing and receiving feedback as they progress through the program, strengthening their own skills as they contribute to the development of their peers' scholarly abilities. The scaffolded dissertation process also fosters an inclusive learning environment that supports diverse learners. Typically, doctoral programs involve a staged approach wherein coursework precedes dissertation work. While the former may include collaborative projects, the latter is generally individualized, even isolated. The scaffolded DiP represents a significant potential for improvement in student experience and ultimately program completion. Continuous Improvement through Evaluation The program's commitment to continuous improvement is evidenced by its systematic evaluation and check-in processes. Regular evaluations gather substantive data on student satisfaction and program effectiveness, while formative semester check-ins provide real-time feedback on student needs and experiences. These check-ins are integral to maintaining the program's adaptability and responsiveness, ensuring that it remains current and aligned with the professional landscape's evolving demands (Frye & Hemmer, 2012; Sallese et al., 2023). To further elucidate, these regular evaluations and check-ins are designed with specific content and methodologies to ensure comprehensive and meaningful data collection. The content of these evaluations includes targeted questions on curriculum relevance, instructional quality, learning outcomes, and program logistics. Additionally, evaluations seek insights on the program's impact on professional growth and personal development. Annual online surveys are typically used for broader questions and can reach a larger number of students efficiently, while each semester, focus groups and individual interviews allow for more in-depth exploration of specific issues or themes that emerge from the survey data. Findings are then discussed in department-wide meetings, where faculty and administrators deliberate on potential improvements and strategic changes. This iterative process ensures that the program remains agile and responsive to the evolving needs of its students and the educational sector. Moreover, findings from these evaluations are often shared with the students to maintain transparency and foster a collaborative atmosphere in the program’s continuous improvement journey. A key component of this process is a proposed future student task force, composed of current students, to provide critical evaluations of the program. This initiative, which emerged from an analysis of evaluation feedback, aligns with the program's design principles, cohort dynamics, and justice-focused framework. All evaluations are intended to reflect the program values of collaboration and transformation, aiming not just at high satisfaction scores but at fostering critical engagement and practical implementation of improvements identified by the students themselves. Program revisions are made on the basis of this feedback, as well as on insights and recommendations from participating faculty and staff. Candidates are frequently reminded about the importance of creating and participating in opportunities for feedback and on acting on information elicited from such opportunities. This process also includes reflecting on the growth in knowledge and theory that students experience, identifying which aspects of the program have been instrumental in this development. Systemic approaches to continuous improvement are intentionally modeled in program practices, allowing students to experience benefits that contribute to their own projects. Further enriching this evaluative process is the integration of multi-tiered faculty advising each semester. This advising, coordinated and implemented by the program director, is comprehensive, extending from program-wide to cohort-specific and from small group settings to individual sessions. The advising framework not only provides expansive programmatic guidance but also caters to the nuanced needs within each cohort, offers focused attention in small groups, and delivers personalized mentoring in one-on-one interactions. This layered advising structure ensures individualized support for each student's unique educational path, with meticulous monitoring of their progress, thus elevating the educational experience and ensuring it aligns with their professional goals (Brill et al., 2014; Holland, 1998; Schlosser et al., 2003). Utilization of Technology Technology plays a pivotal role in the EdD program, serving as a cornerstone for its innovative flipped defense model and facilitating both synchronous and asynchronous coursework. The strategic use of advanced digital platforms is central to this approach, enabling interactive and engaging learning experiences that transcend traditional classroom boundaries. Instructional platforms are carefully selected and optimized to foster a sense of community and collaboration, effectively bridging the gap between students and faculty regardless of physical distance. By prioritizing the integration of technology into its curriculum, the EdD program ensures that candidates are adept at utilizing digital resources and platforms, preparing them to be leaders and practitioners in an increasingly digital educational landscape. This holistic technological integration ensures that the virtual learning environment is not a barrier but a conduit to a more connected and accessible educational experience (Cañas et al., 2003; Haleem, 2022; Qureshi, 2021; Yordanova, 2007). Centering innovative, accessible technological utilization exemplifies leadership principles that the program intends to cultivate in candidates. Such technological integration, coupled with a critical orientation, is designed to prepare students to use technologies in ways that promote transformation toward equity and inclusion, while also developing a critical perspective about the broader context in which technological advances occur. Signature Pedagogies The EdD program is defined by its distinct signature pedagogies, reflective of its foundational values and educational goals. These pedagogies are characterized by their three- dimensional structure as delineated by Shulman (2005): the surface structure, which pertains to the visible, practical aspects of teaching and learning; the deep structure, which relates to the underlying beliefs about effective knowledge transmission; and the implicit structure, which carries the moral dimension of professional attitudes, values, and dispositions. In the first stage of the program, collaborative, inquiry-based learning (Karalis Noel et al., 2020) is at the forefront, fostering Cultivating Change Agents Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 10 No. 2 (2025) DOI 10.5195/ie.2025.451 14 111 curiosity and active engagement and blending theoretical knowledge with practical application. These aims are achieved through interdisciplinary courses focused on complex PoPs, enabling students to identify and address these issues with a critical, research-based lens. The second stage emphasizes equity-driven research (Karalis Noel et al., 2020), with a strong focus on fieldwork. Here, students are trained to be equity-minded practitioners, employing data and critical analysis to identify and address disparities in educational outcomes. This involves developing a deep understanding of the systemic nature of inequities across fields and contexts and a commitment to responsible, transformative practice. The final stage is dedicated to cultivating transformative leadership (Karalis Noel et al., 2020), rooted in questions of justice and democracy and aimed at creating inclusive and generative learning environments. During this phase, students work on bringing coherence to their DiPs, a scholarly work that encapsulates doctoral- level research with a strong relevance to practice. Through the consistent application of these signature pedagogies, the EdD program equips students with the skills and knowledge to confront and reform the pressing educational challenges of our time, shaping them into leaders who can effectuate inclusive and equitable change within diverse professional spheres. Annual Evaluation Results The 2022-2023 annual EdD program evaluation results reflect data from 26 students, offering a quantitative and qualitative account of the program's inaugural year. The program's demographics reveal that 73.08% of respondents were part of the first EdD cohort, while 26.92% represented PhD to EdD transfer students. The professional background of the respondents was diverse, with teachers representing 38.46%, administrators and faculty at teaching-oriented universities each at 15.38%, staff at research universities also at 15.38%, staff at teaching-oriented universities at 7.69%, faculty at research universities at 3.85%, and other professions at 3.85%. Full- time employment was reported by 84.62% of the respondents, part- time by 7.69%, and the remaining 7.69% were not working while studying. One student reported a smooth transition from a PhD to an EdD program and appreciated the advantages of its cohort structure. In terms of response rate, the data from 26 students needs to be contextualized within the total number of students enrolled in the program to understand the representativeness of these findings. If, for instance, the program enrolled 100 students, a 26% response rate would indicate a significant portion of the student body, but still less than a third, meaning the experiences and perspectives of the majority are not captured in this evaluation. The voices present in this evaluation predominantly come from full-time professionals, with a strong representation from teachers and university staff/administrators. However, there appears to be more limited representation from non-traditional students or those not in full-time employment, which might offer different perspectives on the program. The limited data from PhD to EdD transfer students, as well as the underrepresentation of certain professional backgrounds (such as research university faculty and other professions), suggests that the evaluation may not fully capture the diverse experiences and needs of the entire student body. This gap highlights an opportunity for future evaluations to engage more comprehensively with the entire cohort to ensure a more inclusive understanding of the program’s impact across different student demographics. Aspect Importance Ratings In evaluating the program, students rated the significance of various elements on a scale from 1 to 5, with 1 indicating the least importance and 5 indicating the highest (see Table 2). The highest- rated aspect was the program's ability to accommodate their schedules, averaging 4.71. Following closely was the importance attributed to the applicability of learned concepts in the workplace or community, with an average rating of 4.63. Participation in a learning community and the flexibility of obtaining a doctorate through an online format were both highly valued, each with an average score of 4.42. Additionally, the promotion of equity, diversity, justice, and inclusion (EDJI) within their field or community and the motivation for career advancement were considered important, scoring averages of 4.38 and 4.25, respectively. Feedback from the participants further emphasized these findings. One noted the significance of the program's flexibility, which facilitates a balance between work and academic commitments. Another participant highlighted the essential aspect of translating academic learning into practical application within their professional and community environments, stating that this practical application of theoretical principles was a primary reason for enrolling in the program. Table 2. Aspect Importance Aspect Mean Being part of a program that is designed to accommodate my schedule 4.71 Applying what I learn to my workplace/community 4.63 Being part of a learning community 4.42 Earning a doctorate in a flexible online program 4.42 Promoting EDJI in my field and/or community 4.38 Earning my EdD for career advancement 4.25 Developing as a leader in my workplace/community 4.13 Developing my ability to conduct research 3.96 Other 3.46 Earning my EdD for a career change 2.79 Aspect Rating Scores In the assessment of the program, students provided ratings for various elements on a scale from 1 to 5, with 1 denoting 'poor' and 5 denoting 'excellent' (see Table 3). The intellectual caliber of the faculty was highly praised, averaging a rating of 4.88. The effectiveness and support provided by the EdD program director were also recognized with a high average score of 4.79. Advising and guidance within the program were similarly commended, with an average score of 4.75. The quality of the coursework received positive feedback as well, with a rating of 4.50. Remarks from students included positive evaluations of the faculty's intellectual contribution and the significant role of their guidance in enhancing students' research abilities. Karalis Noel & Gorlewski Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 10 No. 2 (2025) DOI 10.5195/ie.2025.451 15 Table 3. Aspect Rating Aspect Mean The overall intellectual quality of the faculty 4.88 The helpfulness of the EdD program director 4.79 The helpfulness of staff in the department 4.63 The overall intellectual quality of peers in the program 4.63 The advising and guidance experience in the EdD program 4.75 The overall quality of the EdD program 4.71 The program's integration of EDJI principles 4.67 The helpfulness of the faculty in the department 4.67 The quality of the coursework in the EdD program 4.50 Program Experience The program experience was gauged using a 5-point scale, where participants rated aspects from 'Strongly Disagree' (1) to 'Strongly Agree' (5) (see Table 4). The approachability and helpfulness of the EdD program director received an exceptional average score of 4.96. The clarity of the program's milestones and the level of faculty support were also highly rated, with averages of 4.91 and 4.87, respectively. General satisfaction with the program was strongly positive, averaging 4.83, a figure that was echoed by ratings for faculty availability and beneficial interactions with the program director. Feedback indicated a particular appreciation for the program director's consistent availability and support, contributing to positive experiences within the program. Table 4. Program Experience Experience Mean The EdD program director is accessible. 4.96 The EdD program director is supportive of students 4.96 My interactions with the EdD program director are positive 4.96 The program milestones are clear 4.91 Faculty are supportive of students 4.87 My interactions with faculty are positive 4.87 Faculty are accessible 4.83 I am overall satisfied with my experience in the program 4.83 The program delivers on its mission to prepare education professionals to address problems of practice 4.78 My interactions with other students in the program are positive 4.78 My experience in the program meets or exceeds my expectations 4.78 The program design accommodates the needs of working professionals 4.78 I have a clear understanding of how to be successful in this program 4.78 Staff are accessible 4.74 Staff are supportive of students 4.74 My interactions with staff are positive 4.74 The program design is inclusive of local and non-local students 4.65 I have had opportunities to enhance my understanding of how to embed EDJI in research and practice 4.52 Feedback and Career Ambitions Students provided open-ended feedback on the overall program, indicating a desire for a wider selection of courses, specifically in higher education and social justice education. Regarding the prequalifying paper defenses, students felt adequately prepared, crediting the faculty's effective guidance. Career goals among the students varied, with 39.13% aspiring to become faculty at teaching-focused universities, 17.39% aiming for administrative positions, and 13.04% looking to pursue curriculum development. The responses revealed high satisfaction with the current course offerings and the supportiveness of the faculty, particularly highlighting the program's thoughtfully designed curriculum and the accessibility of faculty members like the program director. The feedback was predominantly positive, but students did suggest improvements such as timelier access to syllabi and an increase in course variety. These insights illustrate the program's strong points and provide clear guidance for future enhancements, reflecting the program's responsiveness to student needs and aspirations. This feedback also points to potential modifications to evaluation tools and techniques, an essential finding in relation to the implications of the alignment of evaluation design with program goals and values (Cochran-Smith & Mitescu Reagan, 2022). Discussion and Strategic Development The comprehensive evaluation of the EdD program for the academic year 2022-2023 underscores its strengths and identifies prospects for growth. A critical observation is the program's commitment to continuously transforming from a traditional university evaluation approach to one that is deeply reflective of its unique values and goals. Attracting a diverse group of professionals, the program enriches its interdisciplinary curriculum with and through a variety of perspectives, particularly from the significant number of full-time teachers and candidates transitioning from PhD programs. The evaluation process, with its emphasis on student voices and responsiveness, exemplifies the program's commitment to being a dynamic, student-centered learning environment. The robust initial enrollment figures underscore the program's strong reception and the effectiveness of its launch within the educational community. The program's structure and innovations, including the flipped defense model and scaffolded DiP approach, have been met with positive responses, reflecting their perceived effectiveness in enhancing students' academic experiences. High ratings for faculty support and accessibility, along with the flexibility in accommodating professional schedules, confirm the program's success in aligning with the academic, practical, and professional needs of its students. Feedback also points to the program’s well-executed alignment with student career aspirations, especially for those seeking academic and administrative roles, indicating that the pedagogical approaches and curriculum are aptly preparing students for their targeted career paths. The program's commitment to transformative leadership and equity-driven research through its signature pedagogies aligns with its justice-focused framework, resonating with students' ambitions to be change agents in education. However, the feedback calls for a deeper integration of equity, diversity, justice, and inclusion into the curriculum and into its evaluation processes, suggesting an area for the program to reinforce its mission. The program's responsiveness to student needs is evidenced by its proactive evaluations and adaptations, such as expanding course offerings and providing earlier access to Cultivating Change Agents Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 10 No. 2 (2025) DOI 10.5195/ie.2025.451 16 111 syllabi, responding to direct feedback from students. These insights are essential for the program's continuous improvement and relevance in the evolving landscape of educational leadership and praxis. It is also important to acknowledge the limitations of this evaluation. The use of a convenience sample may introduce bias, as students might feel obligated to provide positive feedback knowing that program leaders are aware of who took the survey. Additionally, the lack of longitudinal data, given that the program is only one year old, limits the ability to fully determine if the goals are being met as designed. These limitations highlight the need for ongoing, longitudinal evaluation to gain a more comprehensive understanding of the program's impact. Further, integrating findings from other educational research endeavors that have used the critical pedagogy model can provide additional insights and practical applications for those wishing to implement this approach at their institutions. Studies such as those by Furman (2012) and Jean-Marie et al. (2009) demonstrate the efficacy of critical pedagogy in fostering social justice-oriented educational leaders. Furman (2012) proposes a conceptual framework for social justice leadership as praxis, emphasizing the importance of developing capacities for both reflection and action across various dimensions, including personal, interpersonal, communal, systemic, and ecological. This approach aligns closely with the goals of the EdD program, which aims to cultivate leaders who are not only theoretically informed but also practically equipped to enact social justice within their professional contexts. Jean-Marie et al. (2009) extend this discussion by exploring how leadership preparation programs can effectively prepare school leaders to address social justice issues in a global context. Their findings suggest that comprehensive preparation programs that emphasize critical consciousness, reflective practice, and actionable skills are essential for developing leaders capable of fostering equitable educational environments. This insight is particularly relevant to the EdD program's mission to prepare practitioners who can serve as change agents and create synergy across communities. Patton (2017) provides further support for integrating critical pedagogy principles into program evaluation. By elucidating ten pedagogical principles derived from Paulo Freire's writings, Patton highlights how these principles can be applied to develop a critical pedagogy of evaluation. This approach not only enhances the evaluation process itself but also ensures that it remains aligned with the program's broader goals of equity and social justice. Cochran-Smith and Mitescu Reagan (2022) underscore the importance of centering equity in teacher education evaluation. Their work illustrates how equity-focused evaluation frameworks can drive transformative change within educational programs. By incorporating these frameworks, the EdD program can further ensure that its evaluative practices are not only rigorous but also deeply aligned with its commitment to social justice. These insights align with the CPED (2022) principles, reinforcing the program’s mission to prepare practitioners who can serve as change agents and create synergy across communities. By examining these and other related case studies, we can identify additional best practices and potential pitfalls, enabling us to refine our approach and better support our students in their transformative educational journeys (Cochran-Smith & Mitescu Reagan, 2022; Patton, 2017). Conclusion and Future Directions The inaugural year of the EdD program has been marked by a confluence of innovative pedagogy, robust curriculum design, and a focus on preparing educational leaders and practitioners to confront and address complex PoPs. The 2022-2023 program evaluation highlights both the significant accomplishments and the opportunities for enhancement. As the program evolves, an emphasis on a responsive and student-involved evaluation process will be key. The EdD program's commitment to systemic transformation will be further strengthened by the implementation of innovative, equity-oriented evaluation processes. Corresponding with a principles-based approach described by Patton (2017), evaluation procedures will seek to employ a “critical pedagogy of evaluation” (p. 49). As Patton explains, “principles-focused evaluation informs choices about which principles are appropriate for what purposes in which contexts,” thus allowing for the implementation of evaluation processes that reflect and strengthen progress toward program goals (p. ix). Driven by guiding principles of equity, inclusion, and collaborative leadership, these processes will actively incorporate student and alumni voices, ensuring that evaluations are not only collaborative and critically engaging but also contribute to the broader goal of fostering systemic change. Further, acknowledging the importance of emotional and academic support, we will enhance our focus on students' growth in both competency development and theory comprehension. This holistic approach to student development is a core aspect of our educational philosophy. To deepen our engagement with students and alumni, we will form and establish a task force by the end of the year. This task force, comprising students and alumni, will provide critical assessment and feedback. It aligns with our program design and values, aiming at proactive recommendations for program improvements. We aim to transcend traditional evaluation methods by engaging a broader spectrum of stakeholders in our evaluation process. This includes actively involving students, alumni, faculty, staff, leaders, and community partners, thereby capturing the full ethos and mission of the program. This strategic enhancement is designed to develop the next generation of equity-minded educational leaders. By broadening the scope of our evaluative framework to include these diverse perspectives, we intend to create a more holistic, inclusive, and representative assessment of our program's impact and efficacy. This evolution in our evaluation strategy is not just an improvement in methodology, but a reflection of our commitment to embodying the principles of equity, inclusivity, and collaborative leadership that we instill in our students. As the EdD program progresses, it is recommended to adopt several strategies to continue its path of excellence and adaptability. These include the expansion of the curriculum with new courses that tackle emerging educational challenges and cater to student interests, particularly in areas like social justice and educational policy. Enhanced flexibility is also essential, calling for revisions in synchronous course scheduling and the timely dissemination of syllabi to meet the varied needs of the student body. A deeper integration of EDJI principles is paramount, both within the program's pedagogical practices and curriculum content. The program should maintain its dedication to continuous evaluation, ensuring that student feedback is integral to its developmental strategy. Additionally, fostering a robust alumni network will be essential for ongoing program enhancement and for nurturing professional Karalis Noel & Gorlewski Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 10 No. 2 (2025) DOI 10.5195/ie.2025.451 17 collaborations that extend beyond the program's immediate community. In conclusion, the EdD program stands as a pivotal initiative in cultivating educational leaders and practitioners who are both academically proficient and ethically driven to enact positive change. By continuously evolving in response to the feedback from its students, alumni, and other key stakeholders, the program is not only teaching but also exemplifying the principles of collaborative, transformative leadership and change. The program's alignment with professional aspirations and its innovative approach to leadership development in education bode well for its continued success. Through ongoing assessments and related adaptations, the program is well-positioned to evolve and respond to the dynamic landscape of educational leadership and practice, ensuring that it remains at the forefront of cultivating the next generation of change agents in education. REFERENCES Akour, M., & Alenezi, M. (2022). Higher education future in the era of digital transformation. Education Sciences, 12(784). https://doi.org/10.3390/educsci12110784 Barnett, B. G., Basom, M. R., Yerkes, D. M., & Norris, C. J. (2000). Cohorts in educational leadership programs: Benefits, difficulties, and the potential for developing school leaders. Educational Administration Quarterly, 36(2), 255–282. Bovill, C. (2020). Co-creation in learning and teaching: The case for a whole- class approach in higher education. Higher Education, 79, 1023–1037. https://doi.org/10.1007/s10734-019-00453-w Brauer, D. G., & Ferguson, K. J. (2015). The integrated curriculum in medical education: AMEE Guide No. 96. Medical Teacher, 37, 312–322. Brill, J. L., Balcanoff, K. K., Land, D., Gogarty, M., & Turner, F. (2014). Best practices in doctoral retention: Mentoring. Higher Learning Research Communications, 4(2), 26–37. Butterwick, S., Cockell, J., McArthur-Blair, J., MacIver, S., & Rodrigues, J. (2012). Connectivity and collectivity in a doctoral cohort program: An academic memoir in five parts. Alberta Journal of Educational Research, 57(4), 446–459. Cañas, A. J., Coffey, J. W., Carnot, M. J., Feltovich, P., Hoffman, R. R., Feltovich, J., & Novak, J. D. (2003). A summary of literature pertaining to the use of concept mapping techniques and technologies for education and performance support. Report to the Chief of Naval Education and Training, 1–108. Carnegie Project on the Education Doctorate. (2022). https://www.cpedinitiative.org Castro, M. D. B., & Tumibay, G. M. (2021). A literature review: Efficacy of online learning courses for higher education institution using meta- analysis. Education and Information Technologies, 26, 1367–1385. https://doi.org/10.1007/s10639-019-10027-z Clark, T., & Barbour, M. K. (Eds.). (2015). Online, blended and distance education in schools: Building successful programs. Stylus Publishing. Cochran-Smith, M., & Reagan, E. (2022). Beyond “best practices”: Centering equity in teacher preparation evaluation. Education Policy Analysis Archives, 30(66), 1–39. https://doi.org/10.14507/epaa.30.7040 Cuevas, N. M., Matveev, A. G., & Miller, K. O. (2010). Mapping general education outcomes in the major: Intentionality and transparency. Peer Review, 12(1), 10–15. https://www.proquest.com/scholarly- journals/mapping-general-education-outcomes- major/docview/216598518/se-2 Frye, A. W., & Hemmer, P. A. (2012). Program evaluation models and related theories: AMEE Guide No. 67. Medical Teacher, 34(5), e288-e299. https://doi.org/10.3109/0142159X.2012.668637 Furman, G. (2012). Social justice leadership as praxis: Developing capacities through preparation programs. Educational Administration Quarterly, 48(2), 191–229. https://doi.org/10.1177/0013161X11427394 Glazer, F. S. (2012). Blended Learning: Across the disciplines, Across the academy. Stylus Publishing. Haleem, A., Javaid, M., Qadri, M. A., & Suman, R. (2022). Understanding the role of digital technologies in education: A review. Sustainable Operations and Computers, 3, 275–285. https://doi.org/10.1016/j.susoc.2022.05.004 Holland, J. W. (1998). Mentoring and the faculty development of African- American doctoral students. In H. T. Frierson, Jr. (Ed.), Diversity in Higher Education, Vol. 2 (pp. 17-40). Jean-Marie, G., Normore, A. H., & Brooks, J. S. (2009). Leadership for social justice: Preparing 21st-century school leaders for a new social order. Journal of Research on Leadership. Education, 4(Article 1). Jorgensen, D. (2003). The challenges and benefits of asynchronous learning networks. The Reference Librarian, 37(77), 3–16. https://doi.org/10.1300/J120v37n77_02 Karalis Noel, T., Gorlewski, J., & Kearney, E. (2020). SMACKtivism: A program redesign so good, you won’t know what hit you. Impacting Education: Journal on Transforming Professional Practice, 5(2), 3–10. https://doi.org/10.5195/ie.2020.112 Lam, P., Ng, H., & Tse, A.,Lu, M., & Wong, B. (2020). eLearning technology and the advancement of practical constructivist pedagogies: Illustrations from classroom observations. Education and Information Technologies, 26, 89–101. Leland, A. S., Firestone, W. A., Perry, J. A., & McKeon, R. T. (2020). Examining cohort models in the education doctorate. Studies in Graduate and Postdoctoral Education, 11(3), 249–262. https://doi.org/10.1108/SGPE-01-2020-0004 Lo, C. K., & Hew, K. F. (2019). The impact of flipped classrooms on student achievement in engineering education: A meta-analysis of 10 years of research. Journal of Engineering Education, 108, 523–546. Lucas, H., & Kinsman, J. (2016). Distance- and blended-learning in global health research: Potentials and challenges. Global Health Action, 9, 33429. Park, C. Y.; Kim, J. (2020). Education, skill training, and lifelong learning in the era of technological revolution. Asian Development Bank Economics Working Paper Series, 34, 3–9. Patton, M. Q. (2017). Pedagogical principles of evaluation: Interpreting Freire. In M. Q. Patton (Ed.), New Directions for Evaluation: No. 155. Pedagogy of evaluation (pp. 49-77). John Wiley & Sons. https://doi.org/10.1002/ev.20260 Qureshi, M. I., Khan, N., Raza, H., Imran, A., & Ismail, F. (2021). Digital technologies in education 4.0. Does it enhance the effectiveness of learning? A systematic literature review. International Journal of Interactive Mobile Technologies, 15(4). Rawle, F., Bowen, T., Murck, B., & Hong, R. (2017). Curriculum mapping across the disciplines: Differences, approaches, and strategies. Collected Essays on Learning and Teaching, 10, 75–88. Sallese, M. R., Fogarty, M.., Whiteside, E., & Montague, M. (2023). Multi- Tiered System of Supports for Teacher Preparation: A Framework to Attract, Retain, and Prepare Special Educators. Journal of Special Education Preparation, 3(1), 34–45. https://doi.org/10.33043/JOSEP.3.1.34-45 Schlosser, L. Z., Knox, S., Moskovitz, A. R., & Hill, C. E. (2003). A qualitative examination of graduate advising relationships: The advisee perspective. Journal of Counseling Psychology, 50, 178–188. Seifert, K., & Mandzuk, D. (2006). Student cohorts in teacher education: Support groups or intellectual communities? Teachers College Record, 108(7), 1296–1320. Shields, C. M. (2002). Thinking about community from a student perspective. In Furman, G. (Ed.), School as Community: From Promise to Practice (pp. 197-215). SUNY Press. Shields, C. M., & Hesbol, K. A. (2020). Transformative leadership approaches to inclusion, equity, and social justice. Journal of School Leadership, 30(1), 3–22. https://doi.org/10.1177/1052684619873343 Shulman, L. S. (2005). Signature pedagogies in the professions. Daedalus, 134(3), 52–59. http://www.jstor.org/stable/20027998 Suàrez-Herrera, J., Springett, J., & Kagan, C. (2009). Critical connections between participatory evaluation, organizational learning and intentional change in pluralistic organizations. Evaluation, 15(3), 321–342. Taylor, A. (2007). Learning to become researching professionals: The case of the doctorate of education. International Journal of Teaching and Learning in Higher Education, 19(2), 154–166. Treadwell, I., Ahlers, O., & Botha, G. C. (2019). Initiating curriculum mapping on the web-based, interactive learning opportunities, objectives, and https://doi.org/10.3390/educsci12110784 https://doi.org/10.3390/educsci12110784 https://doi.org/10.3390/educsci12110784 https://doi.org/10.1007/s10734-019-00453-w https://doi.org/10.1007/s10734-019-00453-w https://doi.org/10.1007/s10734-019-00453-w https://www.cpedinitiative.org/ https://www.cpedinitiative.org/ https://www.cpedinitiative.org/ https://doi.org/10.1007/s10639-019-10027-z https://doi.org/10.1007/s10639-019-10027-z https://doi.org/10.1007/s10639-019-10027-z https://doi.org/10.14507/epaa.30.7040 https://www.proquest.com/scholarly-journals/mapping-general-education-outcomes-major/docview/216598518/se-2 https://www.proquest.com/scholarly-journals/mapping-general-education-outcomes-major/docview/216598518/se-2 https://www.proquest.com/scholarly-journals/mapping-general-education-outcomes-major/docview/216598518/se-2 https://doi.org/10.3109/0142159X.2012.668637 https://doi.org/10.3109/0142159X.2012.668637 https://doi.org/10.3109/0142159X.2012.668637 https://doi.org/10.1177/0013161X11427394 https://doi.org/10.1177/0013161X11427394 https://doi.org/10.1016/j.susoc.2022.05.004 https://doi.org/10.1300/J120v37n77_02 https://doi.org/10.1300/J120v37n77_02 https://doi.org/10.1300/J120v37n77_02 https://doi.org/10.5195/ie.2020.112 https://doi.org/10.5195/ie.2020.112 https://doi.org/10.5195/ie.2020.112 https://doi.org/10.1108/SGPE-01-2020-0004 https://doi.org/10.1108/SGPE-01-2020-0004 https://doi.org/10.1108/SGPE-01-2020-0004 https://doi.org/10.1002/ev.20260 https://doi.org/10.33043/JOSEP.3.1.34-45 https://doi.org/10.1177/1052684619873343 https://doi.org/10.1177/1052684619873343 http://www.jstor.org/stable/20027998 Cultivating Change Agents Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 10 No. 2 (2025) DOI 10.5195/ie.2025.451 18 111 outcome platform (LOOOP). African Journal of Health Professions Education, 11(1), 27–31. https://doi.org/10.7196/AJHPE.2019.v11i1.1073 Wang, F. (2018). Social justice leadership—Theory and practice: A case of Ontario. Educational Administration Quarterly, 54(3), 470–498. Zinkhan, G. (2005). The marketplace, emerging technology and marketing theory. Marketing Theory, 5, 105–115. Yordanova, K. (2007). Mobile learning and integration of advanced technologies in education. In Proceedings of the 2007 International Conference on Computer Systems and Technologies (pp. 1-6). https://doi.org/10.7196/AJHPE.2019.v11i1.1073 https://doi.org/10.7196/AJHPE.2019.v11i1.1073 Tiffany Karalis Noel University at Buffalo–SUNY Julie Gorlewski Hunter College–CUNY Purpose and Evaluative Approach Positionality Statement Evaluation Approach Program Design and Pedagogy Curriculum Map and Milestone-Based Learning Cohort Dynamics and Community Engagement A Justice-Focused Educational Framework Program Features and Structural Innovations Flipped Defense Model Scaffolded Dissertation in Practice (DiP) Model Continuous Improvement through Evaluation Utilization of Technology Signature Pedagogies Annual Evaluation Results Aspect Importance Ratings Aspect Rating Scores Program Experience Feedback and Career Ambitions Discussion and Strategic Development Conclusion and Future Directions References