Impacting Education: Journal on Transforming Professional Practice New articles in this journal are licensed under a Creative Commons Attribution 4.0 United States License. This journal is published by the University Library System of the University of Pittsburgh as part of its D-Scribe Digital Publishing Program and is cosponsored by the University of Pittsburgh Press. 40 This journal is supported by the Carnegie Project on the Education Doctorate: A Knowledge Forum on the EdD (CPED) cpedinitiative.org impactinged.pitt.edu ISSN 2472-5889 (online) Vol. 6 No. 4 (2021) DOI 10.5195/ie.2021.171 Establishing Rigor and Quality in Doctoral Programs Through Program Assessment Maida Finch Salisbury University mafinch@salisbury.edu Jake D. Follmer University of West Virginia djakefollmer@gmail.com Heather Porter Salisbury University hdporter@salisbury.edu ABSTRACT This essay describes the development and implementation of a CPED-grounded program assessment system and the ways in which it contributes to quality assurance in Ed.D. programs broadly. We begin by articulating program quality and describing the contextual factors that guide our approach to program assessment. Next, we overview major components and processes of our program assessment system. Specific emphasis is placed on describing the development and evaluation of program effectiveness based on CPED-influenced student learning outcomes. We then briefly describe how we leverage an existing learning management system to implement program assessment efficiently, and outline continuous monitoring and improvement efforts that are based on our program assessment work. Finally, we describe our experiences with academic program review and discuss lessons learned and suggestions to promote program rigor and success. KEYWORDS program assessment, student learning, continuous improvement What do we mean by program quality in graduate and doctoral education? The answer is complicated because of the perspectives different stakeholders have on the matter. Accountability, program review, accreditation, quality assurance, and assessment are terms characterizing program quality and reflecting distinctive values of stakeholders. For example, faculty may focus on assessment as an indicator of learning, while prospective students value accreditation for assurance that the degree will help advance their careers, and universities emphasize program review as evidence for market competition. In light of these different values, scholars offer multidimensional definitions of quality (Bogue, 1998; Bonvillian & Dennis, 1995; Harvey & Knight, 1996), and Koslowski (2006) suggested quality in higher education as a theory rather than a measurable output. From this perspective, quality is a process, or way of thinking, that views the work as “valuable, measurable, and able to be improved” (p. 280). Assuring program quality requires a sustained effort at maintaining and improving academic standards (Bowker, 2017). Similarly, measuring program quality is complex and again reflects values of stakeholders, and increasingly, public sentiment. Indeed, the Commission on the Future of Higher Education recommended assessment and accountability as essential to quality assurance and emphasized the role of student outcomes in this process (National Commission on the Future of Higher Education, 2006). At the postsecondary level, accreditation and program reviews are common processes for examining program quality. With an emphasis on accountability, accreditation is associated with external stakeholders; however, critics claim this focus is too responsive to outside pressures, disregards theories of learning, and stifles innovation (Bogue, 1998; Koslowski, 2006; Openo et al., 2017). On the other hand, academic program review is undertaken by program faculty with the goal of assessing and improving and yet, it too is criticized as futile and meaningless (Bogue, 1998). Despite this, Brooks and Heiland (2007) argued that, in light of increased pressure for accountability in doctoral programs, faculty-driven assessment of program goals and outcomes is ideal. This raises questions about the feasibility of developing a quality assurance system that satisfies goals of both accountability and improvement. In an essay exploring best practices in quality assurance, Bogue (1998) recommended guidelines for developing a system to accomplish both purposes that include linking measurement instruments to teaching and learning. Graduate and doctoral faculty http://www.library.pitt.edu/ http://www.pitt.edu/ http://www.library.pitt.edu/articles/digpubtype/index.html http://www.library.pitt.edu/articles/digpubtype/index.html http://upress.pitt.edu/ http://creativecommons.org/licenses/by-nc-nd/3.0/us/ http://cpedinitiative.org/ Establishing Rigor and Quality in Doctoral Programs Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 6 No. 4 (2021) DOI 10.5195/ie.2021.171 41 are well-positioned to develop such a system because they know their program’s mission and goals and can develop meaningful student learning outcomes (Brooks & Heiland, 2007). From there, they can create assessments designed to measure student learning, and then use information from assessments to make program improvements. This emphasis on linking teaching and learning to improvement and assessment should not be overlooked. A recent survey conducted by the Council of Graduate Schools found that 65% of responding member institutions had developed student learning outcomes (Denecke et al., 2017). As Tagg (2010) asserted, “…assessment of student learning is the lever that, with the fulcrum of student learning outcomes, can move institutions to transformative rather than cosmetic change” (p. 57). Discussions about quality assurance are particularly germane to postgraduate education. Graduate and doctoral programs are facing increasing pressure to demonstrate their relevance and the applicability of the degrees they offer (Denecke et al., 2017). In particular, the emergence and viability of the professional doctorate demands a greater focus on transferable skills to meet workforce demands absent from terminal degrees that primarily prepare students for academia or other research-focused careers (Denecke et al., Shulman et al., 2016). Denecke et al. concluded their report on learning outcomes in doctoral education with recommendations for research about the process programs use to develop them, their specificity, and whether and how they are used by faculty and students. We take up these matters in this essay describing the program assessment process for our Ed.D. program at Salisbury University. We begin with an overview of our doctoral program as context for understanding our assessment practices. Following this, we detail the process of developing our system, including how we drew from the Carnegie Project for the Education Doctorate’s (CPED) framework and faculty collaboration throughout the process. We offer an in-depth description of the three components that comprise our assessment practices and highlight how we utilize our university’s learning management system to support this work. Finally, we share information about our first formal program review, reflecting on what we learned about our assessment practices and our plans for moving forward. We conclude with remarks about the affordances of this process for other graduate and doctoral programs. PROGRAM CONTEXT Salisbury University is a regional comprehensive university located on the Eastern Shore of Maryland. The Ed.D. program in Contemporary Curriculum Theory and Instruction: Literacy, the second of two doctoral programs on campus, admitted its first cohort of students in 2014. The program mission is to develop reflective and scholarly practitioners who are prepared to promote culturally- responsive, effective literacy learning across diverse contexts. Our students are pursuing careers in P-12 and higher education contexts as well as literacy-focused community organizations. To accommodate working professionals, the program is offered in a hybrid format; classes meet in-person one week and online the following week. Courses are offered year-round, and a new cohort is admitted each fall semester. Students have the option of enrolling in the program full-time, with degree completion expected in four years or part-time with a five-year projected graduation. From its inception, doctoral faculty and staff have utilized a shared governance approach that is guided by our program mission and core values of collaboration, community, diversity, innovation, professionalism, and respect. Monthly department meetings address procedural, policy, and instructional issues, and the department chair facilitates decision-making by consensus. The development of our program assessment practices (described below) followed departmental norms of review and revision based on faculty discussion and feedback. As a member of the Carnegie Project on the Education Doctorate, we engage with faculty at other institutions and draw from the CPED frameworks to develop a rigorous and relevant degree program. In particular, the drafting, refining, and implementing of our student learning outcomes are grounded in four of the seven CPED design concepts: signature pedagogy, inquiry as practice, laboratories of practice, and scholarly practitioner (The Carnegie Project on the Education Doctorate, 2019). According to these concepts, preparing professionals requires a pervasive set of practices (signature pedagogy) in order to understand problems of practice and use data to effectively address them (inquiry as practice). Laboratories of practice are contexts where practice and theory interact and inform each other while scholarly practitioner describes professionals who blend practical and professional knowledge to solve problems. We developed our assessment practices with these concepts in mind. COMPONENTS OF PROGRAM ASSESSMENT PRACTICES Assessment and evaluation of the Contemporary Curriculum Theory and Instruction: Literacy EdD program is based on three overarching and interrelated components: basic student information and program metric tracking; student learning assessment; and student exit surveying. The roles and functions of these components are described in turn. The first component of our program assessment approach is based on descriptive analysis of student information and tracking progression through the Ed.D. program. To support our analysis and reporting of this information, we record: contact and demographic information; student program status (i.e., part-time or full-time); employment information at enrollment in and completion of the program; student committee composition; performance on the comprehensive examination, dissertation proposal hearing, and dissertation defense; program exit information; and GPA at program completion. We use this information to gain insight into students’ successes and challenges as they progress through the doctoral program. Such analysis allows us to understand, for example, changes in employment that occur in part as a result of completion of the program; examination, proposal hearing, and defense pass rate information; and student withdrawal rate information as well as students’ reasons for exiting the program (i.e., for reasons other than completion). Taken together, this component allows us to examine broad trends over time in student enrollment, characteristics, completion, and successes and challenges across the program. A primary focus of program assessment efforts is to promote and ensure program quality and rigor (Brooks & Heiland, 2007; Ewell, 2010). These aims are typified by evaluation of student learning. Given the firm grounding of the literacy Ed.D. program in the CPED design concepts, we sought to develop a comprehensive Finch et al. Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 6 No. 4 (2021) DOI 10.5195/ie.2021.171 42 and integrated student learning assessment approach that was based directly on these principles. For this reason, we anchored our student learning assessment efforts to student learning outcomes (SLOs) that were informed by the language of the CPED design concepts. Program SLOs were drafted, evaluated, and formalized over time and through feedback and revision from all program faculty. In this way, refinement of the program SLOs was iterative. Table 1 summarizes the program SLOs guiding student learning assessment. To further support alignment between our assessment work and the CPED framework, we grouped the SLOs into areas in a manner consistent with the delineation of the CPED design concepts: signature pedagogy (SiPE), inquiry as practice (IP), laboratories of practice (LP) and scholarly practitioner (ScPr) (Table 1) (The Carnegie Project on the Education Doctorate, 2019). The broader lens of these SLOs is that our students are continually engaged in critical and transformative work that intends to address a relevant problem of practice – that is, a contextualized and persistent issue germane to individual students’ professional settings, the addressing of which aims to improve practice either through direct action or through contribution to the knowledge base. To ensure that our approach to developing, implementing, and refining these SLOs was reflective, we also provided explicit ratings of the degree to which each SLO is emphasized and addressed by the program at the current time (Table 1). These ratings, which were based on a Likert scale ranging from 1-Minor focus to 5-Major focus and were agreed-upon by faculty consensus, provided important information about current points of program focus and possible future curricular revisions. One intention in adopting these SLOs was to ensure that the CPED-driven outcomes are addressed fully in the program curriculum. To this end, we developed a curriculum map detailing broad alignment between program coursework, program milestones, and program-based SLOs (Table 2). Through such mapping, we extended evidence of overall alignment between the curriculum and SLOs and of each specific SLO being addressed by the major course objectives of the doctoral program. Each of the SLOs are addressed, at least broadly, by program core courses. As general trends, research strand courses address SLOs centered on the inquiry as practice CPED area. Next, theory and practice courses address SLOs centered on the signature pedagogy CPED area and, to some degree, the laboratories of practice area. Core courses emphasizing literacy assessment, policy and practice, and tools also address the laboratories of practice CPED area. Achievement of major program milestones, such as the comprehensive examination and the dissertation project, are intended to address the scholarly practitioner CPED area. Finally, as a whole, seminar and dissertation-based courses, and program milestones meaningfully address the integration of all the SLOs. We aimed to base our student learning assessment on a variety of meaningful representations of students’ work and understanding in the program, including:  a final project, completed in a quantitative strategies for inquiry course, designed to support students’ critical evaluative interpretation of data to understand the effect of developed innovations on literacy practice;  a research proposal paper, completed in a foundations of research course, designed to support students’ abilities to think critically and empirically about a relevant research question and develop a research proposal to support its investigation;  critical analysis and review of a literacy tool assignment, designed to promote students’ abilities to select and analytically evaluate an established literacy assessment tool from varied perspectives and using existing literacy theory and research;  results of the comprehensive examination, reflecting an assessment of students’ skill in synthesizing and conveying professional knowledge and linking theory with systemic and systematic inquiry;  and results of the dissertation hearing, reflecting a cumulative evaluation of students’ ability to evidence an in-depth understanding of existing literature and apply research skills to conduct an original, high-quality study intended to advance the field and profession. Table 1. Program-Based SLOS and Ratings Student Learning Outcome CPED Tag Rating SLO 1. Candidates will demonstrate the ability to ground professional practice in literacy theory and research. SiPe-1 5 SLO 2. Candidates will be able to write about literacy and education in relation to equity and social justice for multiple audiences. SiPe-2 5 SLO 3. Candidates will demonstrate the ability to identify, contextualize, and develop solutions that address problems of literacy practice in varied educational and community contexts. IP-1 5 SLO 4. Candidates will demonstrate the ability to use data to critically understand and evaluate the effects of developed innovations on problems of literacy practice. IP-2 5 SLO 5. Candidates will demonstrate the ability to apply critical inquiry to and through formal and informal contexts. LP-1 4 SLO 6. Candidates will demonstrate the ability to evaluate and make recommendations to improve the quality of services within literacy contexts through applied research and evaluation of local, state, and national policies and practices. LP-2 3 SLO 7. Candidates will demonstrate the ability to broadly disseminate research that has the potential to resolve critical problems of literacy practice. ScPr-1 4 SLO 8. Candidates will demonstrate an understanding of the importance of equity and social justice in shaping their approach(es) to solving problems of literacy practice. ScPr-2 4 Note: The Carnegie Project on the Education Doctorate (CPED) abbreviations in the CPED tag column are as follows: signature pedagogy (SiPe), inquiry as practice (IP), laboratories of practice (LP), and scholarly practitioner (ScPr). Ratings ranged from 1- Minor focus to 5-Major focus. Establishing Rigor and Quality in Doctoral Programs Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 6 No. 4 (2021) DOI 10.5195/ie.2021.171 43 These program-based activities allow for a holistic assessment of students’ learning and development across the trajectory of the doctoral program. To understand students’ progress on these assessments and further continuous improvement efforts, we regularly analyze and report the results of students’ performance on these key assessments. For course-based activities, such as the final project and the research proposal paper, we implement a simple, scalable three-point rubric that translates students’ work to one of three levels of performance: 1-Approaching Expectations, 2- Meeting Expectations, and 3-Exceeding Expectations. We also present basic descriptive statistics to understand, in aggregate, students’ performance on the program assessments. An example summary of these data, based on the final project described above, is presented in Table 3. Table 2. Alignment of Course- and Program-Based Assessments to SLOs Assignment Student Learning Outcomes Aligned with CPED Design Concepts SLO1 SiPE-1 SLO2 SiPe-2 SLO3 IP-1 SLO4 IP-2 SLO5 LP-1 SLO6 LP-2 SLO7 ScPr-1 SLO8 ScPr-2 1. Research Proposal Project X 2. Final Project X X 3. Equity Audit and Action Project X X 4. Critical Analysis and Review of Literacy Tool X X 5. Mini Ethnographic Inquiry Project X X 6. Literacy Policy Case Study X X 7.Comprehensive Exam X X 8. Dissertation Proposal Hearing X X 9. Graduate Exit Survey X X X X X X X X Note. SLOs 2 and 6 will be assessed directly in course-based assignments beginning in the 2020-2021 academic year. Table 3. Example Summary of Program-Based Activity SLO Data Assessment Metrics SLO3: IP-1 SLO4: IP-2 # Approaching Expectations (Score of 1 out of 3) 0 0 # Meeting Expectations (Score of 2 out of 3) 5 6 # Exceeding Expectations (Score of 3 out of 3) 4 3 M Rating on Outcome 2.44 2.33 SD for Rating on Outcome 0.53 0.53 Mdn Rating on Outcome 2.00 2.00 Note. n=9 students enrolled in the course. M=mean; SD=standard deviation; Mdn=median. Finch et al. Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 6 No. 4 (2021) DOI 10.5195/ie.2021.171 44 Table 4. Example Summary of Program Milestone Outcome Data Assessment Metrics SLO7: ScPr-1 SLO8: ScPr-2 # Passing: First attempt (P/F) 15 15 # Passing: First attempt conditional pass with oral or written revisions (P/F) 4 4 # Passing: Second attempt (P/F) 5 5 # Failing (P/F) 2 2 Final Pass Rate 92.3% 92.3% For program milestones, such as the comprehensive exam and the dissertation hearing, we examine and report students’ performance based on a modified pass rate system, describing students’ performance relative to the following categories: pass, first attempt; conditional pass, revisions; pass, second attempt; fail. An example summary of these data, based on the comprehensive examination, is presented in Table 4. One notable feature of our approach to program assessment is our anchoring of our assessment framework to a learning management system. In particular, to support implementation and reporting of student learning assessment, we embed our program- based SLOs in the outcomes function of Canvas. Through this function, each SLO is entered and tracked as a separate Canvas outcome. Next, the three-point rubric (described above) is attached to each outcome, and calculation method (e.g., most recent score) and criteria for mastery (e.g., 2 points) are specified. These rubric- aligned outcomes are then embedded in and attached to specific course-based activities. This approach allows specific faculty overseeing implementation and grading of the assessments to efficiently score each program-based activity. Using the reporting functions provided by the learning management system (e.g., outcome reports), we download, organize, and analyze pre-compiled student learning assessment data based on each outcome representing specific program-based SLOs. Finally, in our assessment approach, we devote specific effort to understanding students’ beliefs about, perceptions of, and suggestions for improving the Ed.D. program. To support broader evaluation of the program as well as understanding of students’ perceptions of their own progress on the program-based SLOs, we implement student exit surveying. In our use of student surveying, we implement an instrument to assess: graduates’ ratings of their development and competency in program CPED-aligned SLOs (e.g., As a result of completing the doctoral program, I am able to apply critical inquiry to my own educational contexts.); graduates’ ratings of satisfaction with and quality of the doctoral program; and graduates’ perceptions of strengths and improvements needed of the program. Taken together, this three-part assessment approach reflects an implementable and scalable framework for iteratively evaluating Ed.D. program implementation, successes, and challenges. At the same time, a hallmark of strong assessment work is the ability to leverage the results of such work to promote continuous monitoring and improvement. In the literacy Ed.D. program, for example, we support programmatic change based on the results of this assessment work through consistent review and discussion of major assessment findings (e.g., during allotted time in each faculty meeting); dissemination of program assessment results to both internal and external stakeholders; and the development, implementation, and monitoring of an assessment action plan. OUR FIRST ACADEMIC PROGRAM REVIEW As a public institution in the University System of Maryland, Salisbury University aligns with and is required to report program effectiveness at the system and state level, i.e. Maryland Higher Education Commission (MHEC), in addition to Middle States Commission on Higher Education, our regional accrediting agency. Institutionally, the APR is valued as both a process and a product through which assessment, reflection, and innovation are leveraged to facilitate discussion and action around program quality, continuous improvement, and strategic planning. As is common at other institutions, our process is implemented across a seven-year cycle and involves a self-study, student learning assessment reporting, an external reviewer visit, and meetings with school and university leadership. These steps culminate in the creation of a comprehensive report, including assessment results and narratives that document an academic program’s accomplishments and challenges as well as envisioned goals and initiatives for advancement. The final document is then shared with external stakeholders as evidence of academic rigor and program viability. Three years after the submission of the APR, a progress report meeting is held to discuss advancement toward the implementation of the recommended goals and review of the Student Learning Outcomes. As a way to support the development of a culture of assessment within academic departments, newly established programs are required to conduct an initial APR within the first five years of program inception. Our Ed.D. program’s review took place during the 2019-2020 academic year. What we learned about our assessment system from first APR One of the challenges we faced early on in the process was defining what it meant to evaluate quality within our program and in doctoral education, in general. Anchoring our SLOs in the CPED framework as a guiding hallmark of quality in doctoral education, we opted for a pragmatic assessment approach that would help us determine the extent to which our students were meeting designated learning outcomes. However, as we worked with our data, we realized the need to look beyond evaluative measures of assurance in order to obtain a more nuanced, qualitative understanding of needs for improvement (Hakkola & King, 2016). For example, the Establishing Rigor and Quality in Doctoral Programs Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 6 No. 4 (2021) DOI 10.5195/ie.2021.171 45 comprehensive exam is a demanding learning experience in which students are expected to articulate mastery of the conceptual and methodological knowledge needed to advance to doctoral candidacy. The exam consists of three questions, independently evaluated by a committee, and an overall holistic metric of pass/fail is used to assess student outcomes. As Table 4 shows, we have observed a trend in students needing additional attempts to successfully meet departmental expectations. As a measure of quality assurance, we grappled with how much credence this pass/fail rate should offer in demonstrating rigor to external stakeholders while also informing our understanding of specific teaching and learning issues. For instance, while faculty have voiced concerns with students’ abilities to synthesize literature and articulate rigorous research designs, the use of the pass rate as evidence of students’ SLO achievement did not offer specific insights into the nature and extent of students’ demonstrated competencies at the individual or cohort level. Our oversight in not developing a more descriptive evaluation method inhibited the kind of generative work that should stem from assessment efforts, such as leveraging the data to inform discussions about student preparation across the prerequisite coursework and ways to enhance outcomes. This is not to imply the department is not engaging in continuous improvement efforts as we regularly discuss teaching and learning concerns observed from our practice. The APR process requiring direct measures of learning, however, felt antithetical to much of the qualitative data we find most informative. Moving forward, we seek to find ways to systematize the collection of qualitative data for program evaluation that can propel us beyond the reporting of outcome achievement to more descriptive analyses of student development. Another limitation we faced was the extent to which we could draw upon the SLO data to inform our understanding of our students’ learning processes across the program curriculum. Due to the implementation timeline of embedding SLO assessments in course- based assignments, the milestone and course-level data represented different student cohorts and failed to account for course progressions. Accordingly, we were unable to probe the data to examine how cohorts were developmentally progressing through coursework and milestones in order to anticipate and inform approaches to further scaffold students’ curricular experiences in preparation for more rigorous demands of advanced standing. We further realized that because the course-level and milestone assessments were unique evaluation points, they captured student achievement of the SLOs in isolation from the actual learning process that took place and were further disconnected from the integrated nature of doctoral students’ experiences. For example, although the high pass rate on our dissertation proposal hearing demonstrates students’ achievement of the SLOs, the real faculty concern is the time-to-hearing rate of the doctoral candidates that is not captured in the current evaluation and is a priority issue in terms of the observed extended time and advising needed by candidates to achieve this milestone. This reductive practice of reviewing SLO data in isolation from larger programmatic experiences and outcomes was insufficient in helping us address a more complex issue of determining the efficacy of our mentoring practices in supporting student persistence. As a result, our efforts with the APR fell short in supporting “a more developmental approach to review” that prioritizes student learning and is emphasized by leaders in doctoral education and further valued by our department (Hakkola & King, 2016, p. 140). Finally, we would be remiss if we did not touch upon the importance of cultivating departmental ownership and stewardship of SLOs for sustained program and student learning improvement. While our process is emergent, our approach is grounded in a genuine desire to bridge individual efforts to improve teaching and learning with institutional goals for continuous improvement (Openo, 2017). This work necessitates a departmental commitment to move SLOs from the periphery to formally integrate them into the daily discourse and practices surrounding program improvement. Because student performance evaluation is embedded in the everyday work of teaching (Kuh & Ewell, 2010), we need to be able to leverage assessment data to ask practical questions about the nature and efficacy of our teaching and learning practices for advancing student learning. Although our initial APR work supported our ability to offer assurance of program rigor and quality as they related to past student learning outcome achievement, we felt it was rudimentary in addressing questions about the vitality of our teaching and learning practices for future quality enhancement (Openo, 2017). Specifically, these experiences lead us to wonder about the efficacy of our methods to authentically speak to the hallmarks of quality we articulated in our student learning outcomes: What is adequate evidence for SLO assessments? How do assessments authentically capture the holistic experiences of doctoral students, including beyond the class mentoring through research, practicum, and scholarly writing endeavors? How do assessments generate new understanding about what students are learning and how such efforts can be improved? How can data and SLOs be decoupled and disaggregated to reveal trends across different students and different experiential trajectories? How much rigor is acceptable for doctoral programs? And what balance must be achieved between such standards of rigor and the enhancement of program and student outcomes? While these issues remain a persistent tension in our efforts, we raise these questions in recognition of the developmental nature of program review work and for the benefit of discussions about program quality before assessment begins. Brooks and Heiland (2007) remind us that “the work of assessment must begin by thinking about the relationship of assessment to overall programme goals” (p. 357). We see the need for synergy as critical in our work to rightsize the reporting of quality assurance with the achievement of innovation in doctoral teaching and learning (Kuh & Ewell, 2010). FUTURE DIRECTIONS Defining, assessing, and assuring program quality in doctoral education are critical, if not integrated, processes that inform programmatic change and continuous improvement efforts. Yet, as discussed, scholars have long noted the tensions inherent in the work of quality assurance, including the challenges associated with evidencing student learning (Ewell, 2010), variation in local, program-based and larger, system-level conceptualizations of quality (Bogue, 1998), and a balancing of internal and external forces that impinge on accountability concerns with an authentic desire to engage continuous improvement in order to support rigorous practitioner preparation (Brooks & Heiland, 2007). In our program assessment work, we aim to implement a scalable, student learning- driven approach that is directly grounded in the CPED framework and guiding principles for program design (The Carnegie Project on the Education Doctorate, 2019). We believe that a synthesis of our Finch et al. Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 6 No. 4 (2021) DOI 10.5195/ie.2021.171 46 program assessment work with these guiding principles supports a more informed evaluation of program rigor, applicability, and, when contextualized to the education doctorate, consistency. Our reflection on our program assessment system illuminates the benefits of a multi-faceted approach while simultaneously identifying areas in need of further development and refinement. We contend that these matters are relevant to and have merit for other graduate and doctoral program assessment practices. For instance, our practice of tracking student progress throughout the program relies heavily on quantitative data reporting student outcomes. Although we do collect some qualitative information predominantly in the form of our exit survey, developing a method for capturing descriptive analyses of student development over the course of the program would better position us to move beyond accounting the state of the program and toward refinement and improvement (Koslowski, 2006). Furthermore, in its present form, our assessment practices rely heavily on faculty voice both in their development and evaluation, the exception being, again, our student exit survey. Identifying ways students can more fully participate in program assessment would provide a robust picture of their perspectives of and experiences with the program, as well as prepare them take up assessment practices in pursuit of improving teaching and learning in their own professional contexts (Brooks & Heiland, 2007). As we look to the next phase of our program review cycle, we have a renewed sense of understanding and purpose for the agency we have in building a culture of assessment that bridges the divide between assurance and improvement (Openo et al., 2017). Assessment is part and parcel of classroom teaching and it is a practice faculty embrace, formally and informally, as part of our daily work; however, the APR process seemed to decouple the generative nature of this kind of inquiry from the requisite documentation of outcome achievement (Mårtensson et al., 2014). It is this sense of practicality that we believe is essential to reclaiming a utilitarian process of program review that reflects and responds to authentic issues of teaching and learning and, in turn, better serves students and faculty. Although we are early in our reformative work, we are energized by the ingenuity through which researchers in the Scholarship of Teaching and Learning (SoTL) have approached closing the gap between institutional compliance and programmatic enhancement. While SoTL has traditionally been driven by practitioners to understand and resolve problems of practice within the classroom, many scholars have noted the synergy between this facet of individual academic inquiry and the collective work of program assessment (e.g. Hakkola & King, 2016; McKenny & Anderson, 2019; Openo et al., 2017). As McKenny and Anderson reminded us “The work of SoTL is, after all, the work of providing evidence about learning.” (p. 29). Yet, it also prompts us to problematize such demonstrations to elicit deeper understanding about our approaches, our beliefs and values, and our purpose (McKenny & Anderson, 2019). SoTL’s emphasis on asking complex questions about student learning in relation to the broader context of the educational experience has the potential to address some of the limitations we saw in our initial APR, e.g. prioritizing questions about how students are learning within, across and beyond coursework, while not precluding us from satisfying regulatory requirements that necessitate the reporting of what they have learned. With this shift in reframing the questions we prioritize, we also broaden what counts for evidence and whose perspective is included in our efforts to integrate student voices and descriptive accounts of learning into the inquiry process. Furthermore, just as we benefited from looking to CPED as an external standard of quality for establishing our SLOs, we see an opportunity to embrace SoTL as a leading heuristic through which we can ensure our local practices reflect and align with standards of excellence for assessment that advance changes for student learning. Felten’s (2013) discussion of five pillars for quality and rigor in SoTL offers one pathway forward as does Hakkola and King’s (2016) account of an innovative approach to graduate program review at the University of Minnesota. We conclude our discussion about program assessment and quality with a reflection of Bogue’s (1998) call for a community of caring in quality assurance. It is not enough, he argued, to concern ourselves with only the technical and systemic aspects of program quality; we must also consider the moral and personal because quality cannot be sustained without integrity. Our efforts to develop and sustain a worthy doctoral program are motivated by a genuine commitment to serve our students, and by extension, their literacy communities of practice; a purpose that cannot be fulfilled without an investment “in the minds and hearts, in the values and courage, of the faculty and administrators who hold our climates of learning in trust” (Bogue, 1998, p. 16). REFERENCES Bogue, E. G. (1998). Quality assurance in higher education: The evolution of systems and design ideals. New Directions for Institutional Research, 99(99), 7-18. https://doi.org/10.1002/ir.9901 Bonvillian, G., & Dennis, T. L. (1995). Total quality management in higher education: Opportunities and obstacles. In S. J. Simms & R. R. Simms (Eds.), Total quality management in higher education: Is it working? (pp. 37-50). Praeger Publishers. Bowker, L. (2017). Aligning accreditation and academic program reviews: a Canadian case study. Quality Assurance in Education, 25(3), 287-302. https://doi.org/10.1108/QAE-11-2016-0061 Brooks, R. L., & Heiland, D. (2007). Accountability, assessment and doctoral education: Recommendations for moving forward. European Journal of Education, 42(3), 351-362. https://doi.org/10.1111/j.1465-3435.2007.00311.x Carnegie Project on the Education Doctorate. (2019). Guiding principles for program design. https://www.cpedinitiative.org/the-framework Denecke, D., Kent, J., & McCarthy, M. T. (2017). Articulating learning outcomes in doctoral education. Council of Graduate Schools. Ewell, P. (2010). Twenty years of quality assurance in higher education: What’s happened and what’s different? Quality in Higher Education, 16(2), 173-175. https://doi.org/10.1080/13538322.2010.485728 Felten, P. (2013). Principles of good practice in SoTL. Teaching and Learning Inquiry, 1(1), 121-125. Hakkola, L., & King, J. A. (2016). A developmental approach to graduate education review: A new take on a traditional process. Innovative Higher Education. 41, 137-152. https://doi.org/https://doi.org/10.1007/s10755-015-9338-3 Harvey, L., & Knight, P. T. (1996). Transforming higher education. Society for Research in Higher Education & Open University Press. https://eric.ed.gov/?id=ED418640 Koslowski, F. A. (2006). Quality and assessment in context: A brief review. Quality Assurance in Education, 14(3), 277-288. https://doi.org/10.1108/09684880610678586 Kuh, G. D., & Ewell, P. T. (2010). The state of learning outcomes assessment in the United States. Higher Education Policy and Management, 22(1). https://doi.org/https://doi.org/10.1787/hemp-22-5ks5dlhqbfr1 Mårtensson, K., Roxå, T., & Stensaker, B. (20014). From quality assurance to quality practices: An investigation of strong microcultures in teaching and learning. Studies in Higher Education, 39(4), 534-545. https://doi.org/10.1080/03075079.2012.709493 McKenny, P., & Anderson, C. (2019). Quality with integrity: Working in partnership to conduct a program review. International Journal for Establishing Rigor and Quality in Doctoral Programs Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 6 No. 4 (2021) DOI 10.5195/ie.2021.171 47 Students as Partners, 3(2), 27-43. https://doi.org/https://doi.org/10.15173/ijsap.v3i2.3757 National Commission on the Future of Higher Education. (2006). A test of leadership: Charting the Future of U.S. Higher Education. U.S. Department of Education. Openo, J. A., Laverty, C., Kolomitro, K., Borin, P., Goff, L., Stranach, M., & Gomaa, N. (2017). Bridging the divide: Leveraging the Scholarship of Teaching and Learning for quality enhancement. Canadian Journal for the Scholarship of Teaching and Learning, 8(2), 1-18. https://doi.org/10.5206/cjsotl-rcacea.2017.2.6 Shulman, L. S., Golde, C. M., Bueschel, A. C., Kristen, J., Shulman, L. S., Golde, C. M., Bueschel, A. C., & Garabedian, K. J. (2016). Reclaiming education’s doctorates: A critique and a proposal. Educational Researcher, 35(3), 25-32. https://doi.org/https://doi.org/10.3102%2F0013189X035003025 Tagg, J. (2010). The learning‐paradigm campus: From single‐to double‐loop learning. New Directions for Teaching and Learning, 123(119), 51-61. https://doi.org/10.1002/tl Maida Finch Salisbury University Jake D. Follmer University of West Virginia Heather Porter Salisbury University (1) Program Context Components of Program Assessment Practices Our First Academic Program Review What we learned about our assessment system from first APR Future Directions References