Impacting Education: Journal on Transforming Professional Practice New articles in this journal are licensed under a Creative Commons Attribution 4.0 United States License. This journal is published by the University Library System of the University of Pittsburgh as part of its D-Scribe Digital Publishing Program and is cosponsored by the University of Pittsburgh Press. 16 This journal is supported by the Carnegie Project on the Education Doctorate: A Knowledge Forum on the EdD (CPED) cpedinitiative.org impactinged.pitt.edu ISSN 2472-5889 (online) Vol. 7 No. 1 (2022) DOI 10.5195/ie.2022.226 Professional and Practical Considerations for the Program Evaluation Dissertation Mary Alice Varga University of West Georgia maryalice@westga.edu Katherine B. Green University of West Georgia kbgreen@westga.edu Justin Grant Lindsey University of West Georgia jlindsey8010@gmail.com ABSTRACT This essay discusses the emergence of the program evaluation dissertation in our doctoral program as the preferred dissertation in practice (DiP) option. We also outline important considerations that must be reviewed with students when considering this approach. Our students are professional educational leaders in the settings where they conduct their dissertation research, emphasizing the importance of our doctoral faculty in addressing the potential implications of blending dissertation research with professional practice. Using the utilization-focused evaluation approach as a framework, we address professional and practical considerations to ensure effective evaluation designs to examine a specific problem of practice. These considerations include the student's relationship to the program being evaluated, the impact on their professional position, support for the evaluation, access to data collection, potential consent concerns, and the utilization of findings. We conclude with additional ethical considerations to be considered when supporting program evaluation dissertation work. KEYWORDS program evaluation, dissertation, problem of practice, dissertation in practice For the past 20 years, our school improvement doctoral program at a state-funded comprehensive university has trained many educators to initiate and sustain significant change in their educational settings. Since educational leaders are tasked with generating and using data to inform their practice in various ways (Bauer & Brazer, 2012), we have experimented with various dissertation formats. The traditional five-chapter dissertation has been the most prevalent format that students use, primarily due to faculty preference. This approach is utilized in both Ph.D. and Ed.D. programs and mirrors the scientific method process, but is also used for qualitative research. This format includes chapters pertaining to introduction/rationale, review of literature, methods, results, and interpretations/recommendations (Calabrese, 2006). Despite the constant demand for educational data, it is argued that educational environments rarely use research-based information to inform their practice (Dagenais et al., 2012). In response to doctoral student feedback, our program piloted a program evaluation dissertation format. This format is becoming increasingly popular within educational research since program evaluation is crucial to educational initiatives and even recommended by the United States Department of Education (U.S. Department of Education, 2014). As Stewart (2016) noted, this trend will continue as schools become increasingly reliant on federal funding that requires evaluative components. The program evaluation dissertation format is also of interest to the doctoral students in our school improvement Ed.D. program involved in designing, implementing, or evaluating various problems of practice in their real-world capacities. Utilization-Focused Evaluation Process We strive to have students use a process similar to the utilization-focused evaluation (U-FE) process (Patton, 2012) when initially outlining their program evaluation dissertation proposals. UFE is based on the principle that evaluations should be useful both during the evaluation process and after findings are generated. U-FE fits well within the Ed.D. philosophy and the dissertation in practice (DiP) approach because it is personal and situational. The evaluation context is significant, and findings are expected to be used in practical ways that make sense for the program. There are 17 steps to the U-FE process that begins with assessing and building capacity for the evaluation. The U-FE process ends with a focus on the use of http://www.library.pitt.edu/ http://www.pitt.edu/ http://www.library.pitt.edu/articles/digpubtype/index.html http://www.library.pitt.edu/articles/digpubtype/index.html http://upress.pitt.edu/ http://creativecommons.org/licenses/by-nc-nd/3.0/us/ http://cpedinitiative.org/ https://orcid.org/0000-0003-2416-3253 https://orcid.org/0000-0001-8280-5061 Professional and Practical Considerations Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 7 No. 1 (2022) DOI 10.5195/ie.2022.226 17 findings (Patton, 2012). As we explain the professional and practical considerations we address with students, we also outline their connection to specific steps in the U-FE process and the CPED DiP framework. Connection to the Dissertation in Practice (DiP) Since program evaluation is inherently collaborative, this DiP approach fits well with the Carnegie Project on the Education Doctorate (CPED) scholarly practitioner focus in two ways. First, students are more engaged with reflective problem-solving by their interactions with various program stakeholders. Secondly, this process guides students through various systematic complexities specific to the program context. The DiP approach allows students to more deeply and holistically examine their problems of practice in a way that may not be available in traditional research methods. While the program evaluation approach works well with the DiP philosophy, there are professional and practical considerations that faculty must address with their doctoral students before implementation. This manuscript outlines these considerations from both the faculty and student perspective and discusses how these conversations facilitate student development into their role as scholarly practitioners in an educational evaluator capacity. PROFESSIONAL CONSIDERATIONS The professional considerations we address with students speak to their dissertation work in an educational leadership capacity. As with most Ed.D. programs, our students are full-time practitioners, often in school, district, or state-level leadership positions. We specifically discuss their role with the programs they seek to examine. As a highly regarded best practice, our program encourages students to focus their dissertation research on a topic in which they are passionate (Butin, 2010). For our program, the topic should center on identifying a problem of practice related to school improvement. Students who choose the program evaluation dissertation often seek to evaluate programs housed in their school environments. Their evaluations are not limited to DiPs and can involve a range of programs they would like to investigate with the intention of school improvement. However, professional and ethical concerns arise when our students are in positions of authority or obligation directly related to the program or its stakeholders. Therefore, before the student begins their proposal, we ask them the following questions: 1. What is your relationship with this program? 2. Will the findings of your evaluation impact your professional position? 3. Is your school/district/state supportive of evaluating this program? These questions align with the first three U-FE tasks centered on the program, organizational, and evaluator readiness for a utilization- focused evaluation and engaging the appropriate individuals in the process (Patton, 2012). Answers to these questions can immediately alter the program evaluation dissertation's trajectory in the design and implementation stages, as described below. Student Relationship to the Program In our initial dissertation conversations, the first consideration we outline is the student's relationship to the program they seek to evaluate. For example, STUDENT AUTHOR initially wanted to collect data on the Autism Support Group he developed and led ten years ago. While the support group is now sustained by volunteers, STUDENT AUTHOR still attends the meetings periodically and has a presence within the support group (Author and Author). During conversations with his committee (AUTHOR and AUTHOR), it was agreed that his direct relationship with the program could cause a conflict of interest when interviewing participants and conducting focus groups. As a result, STUDENT AUTHOR enlisted a research assistant to collect data for this program evaluation dissertation. STUDENT AUTHOR also performed member checks on his data analysis (Merriam & Tisdell, 2015) to increase the trustworthiness and dependability of his analysis. This additional layer of analysis allowed participants to ensure STUDENT AUTHOR accurately interpreted their experiences in the program. Impact on Professional Positions In addition to their relationship with the program, we also discuss the potential impact the evaluation could have on the students’ professional position. If our student's professional performance is tied to a program's outcomes, we consider the implications and various ways to approach these concerns. For example, instead of examining an entire program that a student oversees in a professional capacity, we may recommend that the student evaluate one portion of the program that another individual administers. This approach does not always work well, depending on the evaluation needs, but it can be an option for students who need a safeguard. While our goal is to help our students develop and grow into strong scholarly practitioners who initiate and sustain school improvement, we also strive to respect and protect them professionally. Program Evaluation Support Professionalism is also tied to the support needed for the program evaluation dissertation. Early in dissertation discussions, we ask students about the anticipated support they will receive from their school, district, state, etc. We also ask about the potential findings. PRACTICAL CONSIDERATIONS Once the professional considerations for the program evaluation dissertation are discussed, we outline potential practical considerations. Since our doctoral students are practitioners, these considerations are important to discuss early in the dissertation process. As experienced educators and evaluators, faculty must guide students through thoughtful concerns for their dissertation research requirements, as well as their school/educational environments. Given the political nature of some program evaluations, the questions need to be considered from a variety of angles. The student is encouraged to investigate background information on the program, such as when the program began, who started the program, who or which departments support and monitor the program, and if there are any potential issues with evaluating the program in advance, as student success should be at the forefront of Varga, et al. Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 7 No. 1 (2022) DOI 10.5195/ie.2022.226 18 the dissertation. Students may need guidance and support in order to navigate challenging situations. Example questions we ask students include: 1. Will you be able to collect the data you need to effectively evaluate the program? 2. Are there any anticipated issues with gaining consent, assent, parental permission, etc.? 3. How will findings from this evaluation be used? These questions align with U-FE tasks 5-17 that begin with outlining the evaluation's intended uses and priorities and end with the use of findings for improvement purposes (Patton, 2012). Answers to these questions may significantly affect the data collection process and the use of findings. Effective Program Evaluation Data Collection Research in educational settings, particularly public schools, can be cumbersome and difficult to navigate. Schools, districts, and states have varying policies and procedures for conducting research. The recent emergence of COVID-19 has further complicated these processes. Due to these barriers, we ask students to outline multiple data collection plans. For example, Dr. STUDENT AUTHOR initially planned on face-to-face focus groups to examine his Autism Support Group. He planned a backup option to utilize an online platform for the focus groups, as well as to evaluate another similar support group. Due to COVID-19, he was forced to use the virtual option of data collection. Regardless of the dissertation approach, we encourage all doctoral students to consider creating contingency options and plans and incorporate them into their initial IRB application materials to ensure continuity in data collection. Although Dr. STUDENT AUTHOR was fortunate to have additional data collection options available during his dissertation process, additional limitations may arise specific to program evaluation dissertations. For example, programs can be limited in the number of participants available for data collection. Therefore, Dr. STUDENT AUTHOR had another data collection contingency plan specific to participants. Since the emergence of COVID-19 left many things unpredictable, his second option was to conduct focus groups with a similar program in a neighboring area. While this option would still provide beneficial information for his evaluation, it would not have been specific to the support group he developed and led. Permissions and Consents Since an educational setting's culture can significantly impact the evaluation and research efforts, considerations about consent must be discussed. This is an important practice where students combine their schools' practical knowledge into their emerging knowledge as scholars. For example, students who know their target population is largely non-English speaking will outline thoughtful approaches to articulating evaluation efforts, translating consents, and modifying data collection tools as needed. Utilization of Evaluation Findings The final practical consideration we discuss with students is the utilization of the evaluation. As we strive to have students use the utilization-focused evaluation approach (Patton, 2012), this requires that stakeholders commit to using the evaluation findings to improve the practice in some capacity. The utilization of findings also connects to the appeal of the program evaluation approach to our doctoral students. The practical nature of this work is meaningful for our students who are passionate about improving schools and appears to strengthen their identities as scholarly practitioners. Since our dissertations are publicly accessed, conversations about confidentiality and permissions are critical (Patton, 2012). Masking educational settings or participants' identities is always practiced in our program and often sufficient; however, we require students to have conversations with stakeholders about potential negative, unwelcome, or unanticipated evaluation findings. As STUDENT AUTHOR shared in his post-dissertation experience, he highlights that "students should remember that program evaluations are effective at answering questions within organizations and that the answers to their questions may reveal problems or truths that are not easily accepted." We have had instances of students not being granted access to externally evaluate programs in schools due to stakeholder's concerns with data and the results of the program evaluation. As in traditional program evaluations, stakeholders connected to the program often request updates or debriefs as programs are being examined. We discuss the context of their situation and the optimal feedback process. For example, sharing preliminary findings with important stakeholders can prepare them for the results in final reports (Patton, 2012). If the program a student is evaluating is highly sensitive, visible, political, or controversial, we encourage them to have regular conversations with the appropriate individuals. CONTINUED CONSIDERATIONS As we continue to implement the program evaluation dissertation in practical settings using the U-FE approach (Patton, 2012), we have outlined six specific professional and practical considerations to address with our students; however, additional questions and considerations remain. As our students work as full- time practitioners and emerging scholarly practitioners examining important problems of practice, continued discussions around ethics in program evaluation are necessary. Faculty in our program have posed the following questions: Is it ethical to conduct a program evaluation in one's own educational environment as an internal evaluator? How can we ensure "no harm" and avoid coercion when EdD students are evaluating students, teachers, and staff they oversee? Should EdD students only be allowed to conduct external program evaluations, as opposed to internal program evaluations? These questions will continue to be discussed as we work with students through their program evaluation dissertations while helping them balance the reality of their practitioner work. Similar to the ever- changing landscape of education, our work with doctoral students must adapt as well. We may never have a perfect program evaluation dissertation model that fits with all students and their circumstances. Challenges and barriers will continue to present themselves in various ways. Having individual conversations with students to guide them through professional and practical considerations in their dissertation will strengthen their work and their scholarly practitioner identities. Professional and Practical Considerations Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 7 No. 1 (2022) DOI 10.5195/ie.2022.226 19 REFERENCES Bauer, S. C., & Brazer, S. D. (2012). Using research to lead school improvement: Turning evidence into action. Sage. Butin, D. W. (2010). The education dissertation. Corwin. Calabrese, R. L. (2006). The elements of an effective dissertation and thesis: A step-by-step guide to getting it right the first time. R&L Education. Dagenais, C., Lysenko, L., Abrami, P. C., Bernard, R. M., Ramde, J., & Janosz, M. (2012). Use of research-based information by school practitioners and determinants of use: a review of empirical research. Evidence & Policy: A Journal of Research, Debate and Practice, 8(3). Policy Press. https://doi.org/10.1332/174426412X654031 Merriam, S. B. & Tisdell, E. J. (2015). Qualitative research: A guide to design and implementation (4th ed.). Jossey-Bass. Patton, M. Q. (2012). Essentials of utilization-focused evaluation. Sage. Stewart, Tricia. (2016). Educational Leaders and the Program Evaluation Dissertation with Logic Model. 10.4018/978-1-5225-0445-0.ch014. https://www.igi-global.com/chapter/educational-leaders-and-the-program- evaluation-dissertation-with-logic-model/169003 U.S. Department of Education. (2014). Evaluation Matters: Getting the information you need from your evaluation. https://www2.ed.gov/about/offices/list/oese/sst/evaluationmatters.pdf https://doi.org/10.1332/174426412X654031 https://www.igi-global.com/chapter/educational-leaders-and-the-program-evaluation-dissertation-with-logic-model/169003 https://www.igi-global.com/chapter/educational-leaders-and-the-program-evaluation-dissertation-with-logic-model/169003 https://www2.ed.gov/about/offices/list/oese/sst/evaluationmatters.pdf Mary Alice Varga University of West Georgia Katherine B. Green University of West Georgia (1) Justin Grant Lindsey University of West Georgia (2) Utilization-Focused Evaluation Process Connection to the Dissertation in Practice (DiP) Professional Considerations Student Relationship to the Program Impact on Professional Positions Program Evaluation Support Practical Considerations Effective Program Evaluation Data Collection Permissions and Consents Utilization of Evaluation Findings Continued Considerations References