Impacting Education: Journal on Transforming Professional Practice New articles in this journal are licensed under a Creative Commons Attribution 4.0 United States License. This journal is published by Pitt Open Library Publishing. 47 This journal is supported by the Carnegie Project on the Education Doctorate: A Knowledge Forum on the EdD (CPED) cpedinitiative.org impactinged.pitt.edu ISSN 2472-5889 (online) Vol. 8, No. 2 (2023) DOI 10.5195/ie.2023.326 Rigor and Relevance in Research Methods: Reflections from a Professor and Alumni of an Online EdD Program Carey Borkoski Loyola University Maryland cborkoski@jhu.edu Jeannie Chipps Montana State University jchipps3@jhu.edu Brianne Roos Loyola University Maryland bhiggins@loyola.edu ABSTRACT EdD students represent diverse individuals with established professional identities who enroll in doctoral programs seeking relevant, useable content. Instructors and program directors must find ways to incorporate rigor and relevance into the readings, assessments, and training for EdD students. This essay explores the evolution of research methods courses in an online education doctoral program. Hochbein and Perry (2013) noted that “research training need not be diluted, but rather tailored to the specific needs of scholarly practitioners” (p.182). Our narrative integrates the unique perspectives of a former EdD research methods faculty member and two recent alumni to describe one program’s efforts to maintain the rigor of doctoral research methods courses and better align student experiences to their needs and professional context. Relying on research literature and experiential evidence, we offer a rich recounting of a rationale for change and how these adjustments contributed to scholarly practitioner training and research journeys. The essay identifies requisite knowledge of scholarly practitioners and describes the integration of learning opportunities across the courses. KEYWORDS research methods, application-based methods, practitioner-scholar EdD students represent diverse individuals with established professional identities who enroll in doctoral programs seeking relevant, useable content. Instructors and program directors must find ways to incorporate rigor and relevance into the readings, assessments, and training for EdD students. This essay describes the evolution of a sequence of research methods courses in an online education doctoral program. It includes a brief review of the program and discussion of the original design and changes that occurred over several iterations of the methods sequence. Our narrative integrates the unique perspectives of three alumni of the program, including a former research methods faculty member in this EdD program. Relying on research literature and experiential evidence, this essay offers a rich recounting of rationale for change, descriptions, and examples of some of the changes, and how these iterations contributed to scholarly practitioners’ training and research journeys. This essay also identifies requisite knowledge and skills of scholarly practitioners and how changes to the research methods course support deeper learning and more effective integration of opportunities to learn and practice these competencies across the courses. Moreover, we will share how meaningful pedagogical practices reflect theories of adult (Knowles, 1978) and experiential (Kolb, 1984) learning-informed instruction strategies to better facilitate practitioner development and growth as researchers. The discussion that follows includes some background about the program and motivation for the changes, specific examples of the changes and students’ experiences in and with those changes, and implications for the implementation of the course changes as well as learning that occurred because of the change process. BACKGROUND This program is a fully online, three-year doctoral program, with three required research methods courses including Research Methods I (RMI), Research Methods II (RMII), and Program Evaluation (PE). RMI is an introductory course in research methods that explores quantitative, qualitative, and mixed methods designs through readings, discussion posts, and assignments. RMII https://library.pitt.edu/e-journals http://creativecommons.org/licenses/by-nc-nd/3.0/us/ http://cpedinitiative.org/ https://orcid.org/0000-0003-2375-7439 https://orcid.org/0000-0002-9029-2304 https://orcid.org/0000-0002-7212-6740 Borkoski et al. Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8, No. 2 (2023) DOI 10.5195/ie.2023.326 48 represents the data analysis course where students take a deep dive into descriptive and inferential analysis techniques and qualitative coding through reading literature, discussion, and practical application. PE offers training in knowledge and techniques related to intervention research and evaluation. While this paper will touch on aspects of all courses in the sequence, much of the discussion will reference RMI and RMII and the changes in faculty approaches to teaching of research design and facilitating students’ engagement and practice with data analysis. The students in this online program, like the co-authors, are as diverse as the problems and research questions they examine, including differences in professional contexts, research interests, and the methods they use for inquiry. Students represent professionals from myriad contexts, including faculty, administrators, and district leaders in K-12 and higher education, as well as leaders in areas adjacent to and supportive of education systems. The EdD students are also diverse in ethnicity, country of origin, age, and reasons for enrolling in doctoral studies. Furthermore, students also arrive to the program with a range of prior coursework and practical experience with using research methods. For example, the first author was trained in quantitative methods and entered the EdD program with a Ph.D. She joined the methods faculty in this EdD program while completing her EdD. The second author also came to the EdD program with extensive quantitative training (but no qualitative training), while the third author was entirely new to the formal study of research methods upon entering the EdD. The variety of EdD students’ prior experience with research methods, coupled with the contextual nature of the research and the multiple research roles of practitioners may require different methodological approaches, knowledge, and skills. Translated, this means that EdD programs must include rigorous and evidence- based methods training that is relevant and applicable to varied contexts (Lochmiller & Lester, 2017). In fact, Hochbein and Perry (2013) suggest that the diverse needs of practitioners may call for deeper training to examine the complexities of localized research. Using a faculty member’s and two alumni experiences and reflections, our essay explores how one EdD program strengthened the research methods sequence while maintaining a high level of rigor and meeting the unique needs of practitioners. The program changes came about because of methods faculty and program leaders soliciting and listening to students’ feedback. Aware of her positionality as a faculty member with extensive methods experience and the diversity of her students’ experience and comfort with methods, the first author regularly checked in with her students (including the second and third authors) to ask what they needed and how faculty could better support their learning. She was intentional about doing this in a way that set the conditions for honest feedback, and the third author and her classmates asked for a simple dataset to better understand the concepts in RMI and RMII. Responding to students’ feedback and recognizing the need to integrate practice, learning, and context to make connections and deepen understanding, research methods faculty developed a simple dataset about puppies and offered multiple opportunities to engage with the puppy data. Furthermore, students also questioned the volume and type of required work, and this was elevated during the Spring 2020 semester as the pandemic began to be felt. Demands on students’ time and cognitive bandwidth were extensive and consuming, and the faculty and program leaders reevaluated methods requirements to focus on what was most important to promoting students’ learning and success in the program and as emerging researchers. The methods courses, RMI, RMII, and PE, evolved into a place of pragmatism for the students. The revised methods courses discussed reflect a true acknowledgement of adult learning theory (Knowles, 1978): addressing problems, relevance, and immediate application to their work. This approach to research integrates learning and experiences to improve a specific, real practice, rooted in rigor but not limited by disciplinary parameters that might be irrelevant to the goal (Lochmiller & Lester, 2017). The knowledge, skills, and dispositions to conduct application-based contextual research with a variety of stakeholders (i.e., scholarly practitioner research methods competencies) must be grounded in evidence- based science and flexible enough to address student needs, while providing time and space for students to practice and collaborate. There must also be opportunities for students to leverage their and their peers’ professional expertise and that of other relevant stakeholders to address a research question or problem of practice. Guided by the Carnegie Project on the Education Doctorate (CPED) definition of scholarly practitioner as “[an individual who] blends practical wisdom with professional skills and knowledge to name, frame, and solve problems of practice…they have an obligation to resolve problems of practice by collaborating with key stakeholders…” (n.d., Design Concepts Section, para. 1). To that end, EdD research methods courses must include content, activities, and experiential opportunities to deepen students’ methods and data analysis skills beyond their dissertation research interests so that they might address other and different research questions after graduating from the doctoral program and engage meaningfully with a multitude of diverse stakeholders. Originally, the dissertation dictated the methods sequence objectives, content, and activities. This often meant faculty taught students addressing 15-17 different research projects in a course. Consequently, these courses tried to be all things for all students. This resulted in courses with extensive required readings, several textbooks, and many individual student meetings to support their learning and dissertation progress. Moreover, the time spent in optional synchronous sessions required disseminating a high volume of information about research methods and answering diverse and student-specific questions about assignments and their dissertation progress. From the perspective of this former faculty member, the sessions included too much required reading (even for a doctoral program) and too little time for thinking, discussing, sharing, and peer-to-peer and instructor-peer learning (Mezirow, 2003). This continued through the course sequence, with data analysis activities as a function of students’ research, rather than aligned with the overall research methods competencies for EdD students. To be clear, students experienced rigorous and relevant content and gained required skills. However, their skills and competencies were often narrowly focused on their area of inquiry. Paradoxically, the initial intent of customizing to meet the individual student needs resulted in information overload for both faculty and students, making it difficult for students to develop and practice skills most relevant to scholar practitioners. The changes implemented shifted the course foci away from centering the dissertation, created space and time for students both to gain scholarly practitioner competencies, and empowered students to engage in and reflect on course skills most relevant to their research. In the end, faculty and EdD leadership understood that even with the rich diversity of student backgrounds, professional contexts, and Rigor and Relevance in Research Methods Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8, No. 2 (2023) DOI 10.5195/ie.2023.326 49 problems of practice, there were skills and competencies that everyone needed. The faculty and leaders of this online EdD program recognized that scholarly practitioners must gain knowledge and skills in the tenets of research design, learn the language of methods, and gain the skills to examine quantitative and qualitative data. Student feedback and faculty experiences revealed that learning these competencies required space, time, practice, feedback, and independent and collaborative efforts. The following provides a description of some of the ways the research methods courses evolved to meet the needs of doctoral students in an online EdD program. PROGRAM CHANGES Changes in the methods sequence range from course objectives and assessments to learning opportunities and types of effort and engagement. As previously discussed, the sequence shifted away from trying to be all things to all students with expansive required readings, multiple textbooks, and many individual meetings and discussion forums. The revised sequence, in the spirit of Hochbein and Perry (2015), moved not to a diluted set of topics, but a series of courses that included intentional focus on scholarly practitioner competencies. This included time and attention to deliberate research methods and data analysis practices relevant to the EdD students and their varied backgrounds and contexts. Table 1 offers a brief overview of the changes discussed in the rest of this article. Subsequent sections will use examples to describe the adjustments in some detail. Objectives, Assessments, and Content The revision of the methods courses included realignment of course objectives, content, assignments and assessments, and a corresponding diverse array of learning opportunities. As previously noted, the primary objective of the courses shifted from teaching methods via a dissertation-focused model to an approach focused on teaching and learning the requisite skills for application-based, contextual education research to be conducted during and beyond students’ time in the program. Consequently, assignments and assessments expanded beyond specific dissertation milestones and integrated the reality that graduates, over the course of their careers, will face multiple problems of practice. This necessitates skills to collect and analyze data related to the identified problem. Finally, the content of the RMI and RMII courses was also redesigned to complement the revised approach. With the goal of affording students more time to think, apply, and understand the research methods content, reading lists were reduced and redesigned, textbooks were selected more judiciously. By embracing a definition of rigor that includes intellectual challenge, engagement, and opportunities for empowerment (NAIS, 2021), reducing the volume of required work did not change the high expectations of instructors of the level of academic rigor. Diversity of Learning Opportunities Perhaps the most noteworthy changes to the methods courses are found in the diversity of learning opportunities in the reimagined versions of RMI and RMII. Changes were made to asynchronous deliverables, synchronous sessions, and students’ opportunities for data analysis and practice. Upon review of and reflection on the research methods courses and student feedback, methods faculty determined that students needed more time to practice the skills and competencies of research methods. They also needed opportunities outside of their dissertation research to learn, practice, and collaborate on data analysis techniques and skills. To create time and space for this shift in focus, the methods faculty reduced the number of discussion forums in the methods courses, integrated practice and analysis activities into the remaining posts, and focused the discussion questions and responses on application of the doing of methods and analysis, rather than only reading about the methods. Whereas the asynchronous work in previous courses included numerous required discussion posts focused on remembering and recounting important principles and learning from assigned readings, the new courses expanded to include activities that range from examining readings to creating data tables and evaluating other researchers’ discussions of findings in a research article. The change supports students in building methods efficacy and developing a researcher identity. Moreover, graduates will be consumers and producers of research and must have skills to critically examine and use evidence. Synchronous learning also changed, as students in each course are provided with optional opportunities to meet online to connect with faculty, teaching assistants (TAs), and classmates. In the old model, these meetings were used to answer questions, share course logistics (e.g., schedules, due dates), and to review expectations for course deliverables. The approach was adjusted to a model in which faculty and TAs host workshops on a variety of relevant topics, including writing research questions, defining, and measuring constructs, and coding qualitative data. The rationale for the change comes from the benefits of repeated practice (Karpicke & Roediger, 2007) and community building in a synchronous learning Table 1. Changes to the Research Methods Sequence Type of Change Original Design Change to Design Rationale Course Goals/ Objectives Focused on completing dissertation milestones (chapters and/or projects related to the chapters) Focus on research methods competencies for scholarly practitioners, with dissertation and research support EdD students need to build knowledge and skills to address a diversity of topics that will emerge over time. Students must gain understanding and practice in technical skills and the contextualized nature of many of the research questions that emerge. Assignments/ Assessments Dictated by progress on dissertation Guided by scholarly practitioner competencies EdD graduates live and work in diverse and ever- changing professional contexts. Need to prepare students to address issues that may extend beyond their current dissertation research. Diversity of Learning Opportunities Learning exclusively embedded in readings and centered in students’ problems of practice Learning embedded across resources, research experiences, group discussions, workshops, and real-world data Interprofessional approach to methods training. Opportunities for students to engage in communities of practice and leverage student talent and experiences. Borkoski et al. Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8, No. 2 (2023) DOI 10.5195/ie.2023.326 50 environment (Borup et al., 2020). Workshops offer more space, time, and scaffolding to examine and navigate the complexities of methods and data analysis. We also found that the restructured workshops resulted in strong attendance even when offered as optional. Student presence and engagement may be attributed to their enjoyment and seeing the value of the time together. In informal feedback, students reported having a lot of fun working together and appreciated knowing and seeing that others also experience the productive struggle that is research methods. Synchronous learning also changed, as students in each course are provided with optional opportunities to meet online to connect with faculty, teaching assistants (TAs), and classmates. In the old model, these meetings were used to answer questions, share course logistics (e.g., schedules, due dates), and to review expectations for course deliverables. The approach was adjusted to a model in which faculty and TAs host workshops on a variety of relevant topics, including writing research questions, defining, and measuring constructs, and coding qualitative data. The rationale for the change comes from the benefits of repeated practice (Karpicke & Roediger, 2007) and community building in a synchronous learning environment (Borup et al., 2020). Workshops offer more space, time, and scaffolding to examine and navigate the complexities of methods and data analysis. We also found that the restructured workshops resulted in strong attendance even when offered as optional. Student presence and engagement may be attributed to their enjoyment and seeing the value of the time together. In informal feedback, students reported having a lot of fun working together and appreciated knowing and seeing that others also experience the productive struggle that is research methods. During these workshops, faculty and TAs provide students with quantitative datasets and qualitative transcripts, and the opportunities to analyze quantitative and qualitative data live with a supportive community of learners. The time together is intentionally designed to allow for doing (i.e., using SPSS, engaging in coding and theme development), reflecting, practicing, and asking live questions. Faculty, TAs, and students model these practices for and with one another, thus building community (Shea et al., 2022), increasing efficacy through repeated and successful practice (Bandura, 1977), and learning. These workshops provided invaluable feedback and the opportunity for practice through what Garrison (2011) refers to as “collaborative constructivism” (p. 4). It represents a space for online learners to connect synchronously and talk about their areas of challenge, celebrate achievements, and program milestones, and continue to develop relationships and connections that are vital sources of collegial support. The final category of change in the research methods course sequence was the intentional addition of multiple opportunities for data analysis training and practice. Previously, students learned traditional quantitative and qualitative analysis techniques and relied on their problems of practice as both their project and example of research. Now, in addition to their problems of practice, students collaborate to clean, review, analyze, present, and write about multiple real-world (and messy) data sets. This approach affords more opportunities to practice with data, and students support and learn from one another as they work with the same dataset. Using the same dataset also provides opportunities for students to see a research process from start to finish. The real-world data sets that students now use in the research methods courses provide context and relevance to their learning of methodologies and are accessible within the program evaluation course to further extend their understanding. The first data set made available to students was labeled puppy data and the emphasis within this data set was collecting descriptive information that could be used when first learning the statistical software, descriptive statistics, and research method terminology. Briefly, the puppy data, created by the methods faculty, included easy-to-follow and understand data about training puppies that represented a way for students to learn and practice unfamiliar and often confusing data analysis practices and techniques. The students first work within teams to identify constructs and variables within the data set, then consider these research terms within their own doctoral work. Students also generate research questions that can be addressed through descriptive statistics, suggest techniques to answer these questions, and generate initial findings tables. By design, the puppy data provides scaffolding for learning research design steps, including research questions and summary statistics. Revisions to RMII integrated an additional data set to scaffold inferential statistical analysis. This data set contained information gathered from students within the cohort during their entry to the program in year one and included demographic information (including binary gender options, which enabled discussion of the importance of designing inclusive surveys), information on the specialization, and online learning self-efficacy (Zimmerman & Kulikowich, 2016) As with the first data set, students work collaboratively to produce initial attempts at designing research questions and presenting results and conclusions. This data set offers opportunities for students to conduct a second analysis and advance their knowledge of data analysis. Like RMI, students work in groups to both apply and discuss approaches to the analyses. Throughout this process, students engage in feedback, reflection, and production of data analysis and research findings. The second data set extends student learning and practice with a broad range of data analysis skills and leverages the opportunity to collaborate and discuss with peers their approaches and reflections on the analysis processes. The use of these two data sets is intentional. The puppy data is somewhat clean and students can easily conduct descriptive statistics and learn the software program without concerns of missing data or the need to transform data. The second data set, however, includes missing data and a combination of categorical and numerical data, requiring students to decide on the appropriate inferential statistics and how to address missing data. The second data set also serves as more of a bridge to the students’ research and dissertation methods needs. Finally, in addition to the survey data, focus group transcripts were included so that students can perform both qualitative analysis and mixed methods research. The combination of data sets provides students with the opportunity to engage in discussions as to the benefits or drawbacks of using different methodologies that would likely be considered for their dissertation work and within the field of education more broadly (Libman, 2010). CONCLUSION When designing research methods courses for a three-year online EdD program, it is easy to focus on the dissertation as the outcome. In this format, students concurrently work on courses and their research, so it is natural and expected that their doctoral research should connect with the courses they take. What we have Rigor and Relevance in Research Methods Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8, No. 2 (2023) DOI 10.5195/ie.2023.326 51 learned, however, through our collective experience, empirical research, and student and alumni feedback, is that while dissertation work is paramount to success in the EdD, scholarly practitioners must also learn a depth and breadth of research methodology, practice with a diversity of real-world data sources and types, and collaboratively engage with faculty and their peers to understand the benefits and challenges of contextualized research. Courses and other program experiences should integrate research experiences and professional expertise of EdD students while also exposing students to contexts and data that may not be aligned with their current research interests. Scholarly practitioners must engage in practice-based research that challenges them to think, discuss, and practice the traditional skills learned in research methods and how they can and when they should be used in localized research. Unlike a traditional PhD program, students and faculty should leverage and integrate the wealth of expertise and practical experience EdD students bring into the program. In order for this to happen, course structures must include time and space for synchronous workshops for practice and collaboration, group discussions and work with data, and many opportunities to give and receive feedback on a variety of activities and assessments. Like doctoral students, faculty in EdD programs continually reflect, learn, and iterate and, as a result, the revised methods courses meet the diverse research interests, backgrounds, and needs of online EdD students. With intentionally designed space to think, discuss, and practice, EdD students consume and conduct research as well-prepared scholarly practitioners. REFERENCES Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84(2), 191–215. https://doi.org/10.1037/0033-295X.84.2.191 Borup, J., Graham, C. R., West, R. E., Archambault, L., & Spring, K. J. (2020). Academic communities of engagement: An expansive lens for examining support structures in blended and online learning. Educational Technology Research and Development, 68(2), 807–832. https://doi.org/10.1007/s11423-020-09744-x Carnegie Project on the Education Doctorate. (n.d.). The CPED framework https://www.cpedinitiative.org/the-framework Garrison, D. R. (2011). E-learning in the 21st century: A framework for research and practice. Taylor & Francis. https://doi.org/10.4324/9780203838761 Hochbein, C., & Perry, J. A. (2013). The role of research in the professional doctorate. Planning and Changing, 44(3/4), 181–195. Karpicke, J. D., & Roediger III, H. L. (2007). Repeated retrieval during learning is the key to long-term retention. Journal of Memory and Language, 57(2), 151–162. https://doi.org/10.1016/j.jml.2006.09.004 Knowles, M. S. (1978). Andragogy: Adult learning theory in perspective. Community College Review, 5(3), 9–20. Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. Prentice Hall. Lochmiller, C. R., & Lester, J. N. (2017). Conceptualizing practitioner- scholarship for educational leadership research and practice. Journal of Research on Leadership Education, 12(1), 3–25. https://doi.org/10.1177/1942775116668525 Libman, Z. (2010). Integrating real-life data analysis in teaching descriptive statistics: A constructivist approach. Journal of Statistics Education, 18(1), 1–22. https://doi.org/10.1080/10691898.2010.11889477 Mezirow, J. (2003). Transformative learning as discourse. Journal of Transformative Education, 1(1), 58–63. https://doi.org/10.1177/1541344603252172 Shea, P., Richardson, J., & Swan, K. (2022). Building bridges to advance the community of inquiry framework for online learning. Educational Psychologist, 57(3), 148–161. https://doi.org/10.1080/00461520.2022.2089989 Zimmerman, W. A., & Kulikowich, J. M. (2016). Online learning self-efficacy in students with and without online learning experience. American Journal of Distance Education, 30(3), 180–191. https://doi.org/10.1080/08923647.2016.1193801 https://psycnet.apa.org/doi/10.1037/0033-295X.84.2.191 https://doi.org/10.1007/s11423-020-09744-x https://doi.org/10.4324/9780203838761 https://psycnet.apa.org/doi/10.1016/j.jml.2006.09.004 https://doi.org/10.1177/1942775116668525 https://doi.org/10.1080/10691898.2010.11889477 https://doi.org/10.1177/1541344603252172 https://doi.org/10.1080/00461520.2022.2089989 https://doi.org/10.1080/08923647.2016.1193801