Impacting Education: Journal on Transforming Professional Practice New articles in this journal are licensed under a Creative Commons Attribution 4.0 United States License. This journal is published by Pitt Open Library Publishing. 25 This journal is supported by the Carnegie Project on the Education Doctorate: A Knowledge Forum on the EdD (CPED) cpedinitiative.org impactinged.pitt.edu ISSN 2472-5889 (online) Vol. 8, No. 2 (2023) DOI 10.5195/ie.2023.342 Research Methods Courses Redesigned for an EdD in Instructional and Performance Technology Holley Handley University of West Florida hhandley@uwf.edu Nancy B. Hastings University of West Florida nhastings@uwf.edu ABSTRACT This essay describes the design, and subsequent redesign, of the research methods courses included in an Instructional and Performance Technology (IPT) EdD program at a regional comprehensive university in the southeast United States. The program under examination was developed based on the principles of the Carnegie Project for the Education Doctorate (CPED) and research and best practices aligned with the practice of performance improvement. The curriculum includes three research methods courses. The first introduces the students to the principles of action research as applied to the analysis of performance problems in organizational settings. The second addresses instrumentation and data collection processes used in quantitative, qualitative, and mixed methods research, and the third examines analyzing and reporting quantitative, qualitative, and mixed methods research. Collectively these courses provide students with the knowledge, skills, and abilities necessary to serve as scholarly practitioners, examining any type of problem of practice in any organizational setting. KEYWORDS research curriculum, performance improvement, scholarly practitioner The EdD in Instructional and Performance Technology (IPT) program discussed in this essay grew out of a long-standing specialization in Instructional Design and Technology (IDT) in an EdD in Curriculum and Instruction (C&I) program offered by a regional comprehensive university in the southeastern United States. The C&I program, like many EdD programs, was designed based on the standard PhD format, including a heavy focus on scholarly research and requiring students to conduct empirical research and report their work in a traditional five chapter dissertation. Over time, it became clear that this structure was not meeting the needs of our students. The first challenge was that many of the students pursuing the IDT specialization were employed in positions outside of traditional educational settings, where a degree in C&I is not highly valued. The second challenge was that the coursework portion of the program was very heavily focused on the C&I component of the degree, allotting only 18 credit hours for academic preparation related to the area of specialization. This is equal to about half of a Master’s degree; not at all sufficient for the level of education we felt was needed for doctoral students. Finally, because our students were planning to continue to work in the field post-graduation, we felt it was important to move away from the traditional requirements of comprehensive exams and five chapter, empirical research-based dissertations, providing students with opportunities to engage in the type of research they would be expected to perform in their organizations post-graduation. To address the above-noted challenges, the university sought and obtained membership in the Carnegie Project for the Education Doctorate (CPED) in 2018 and developed a standalone EdD in Instructional and Performance Technology, which was approved by the Board of Governors in June 2019 and enrolled its first cohort of students in Fall 2020. This new program was designed based on the CPED principles and framework, which are focused on structuring the EdD as a professional practice doctoral degree and preparing students to serve as leaders, solving organizational performance problems, and facilitating change in their areas of expertise (CPED, 2022). Many of the CPED member institutions offer EdD degrees focused on Educational Leadership, Higher Education Administration, and/or Curriculum and Instruction (CPED, 2022). The program discussed in this essay is unique because it leads to a degree in Instructional and Performance Technology (IPT), which is a very broad field, relevant for professionals in any organization that has an interest in improving performance, integrating technology, and/or designing and delivering well-designed instruction for training and development purposes (University of West Florida, 2022). Due to the unique nature of our field, our program redesign presented some challenges that stretched beyond those faced by most CPED https://library.pitt.edu/e-journals http://creativecommons.org/licenses/by-nc-nd/3.0/us/ http://cpedinitiative.org/ https://orcid.org/0000-0002-6008-1979 https://orcid.org/0000-0001-9048-9855 Handley & Hastings Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8, No. 2 (2023) DOI 10.5195/ie.2023.342 26 institutions. They included identifying an appropriate guiding framework that would allow students to envision the entire performance improvement process, aligning the program deliverables with that framework, and finally, reimagining the research methods portion of the curriculum to align with the data collection and analysis activities associated with the performance improvement framework. Performance improvement (PI) is widely accepted as being concerned with identifying, designing, developing, and implementing instructional and non-instructional solutions to close gaps between desired and actual performance. It relies heavily on comprehensive front-end analyses from a solution-neutral perspective to identify root causes of performance issues before making recommendations as to how to address current or potential future performance issues (Van Tiem et al., 2012). It is the overarching field that encompasses the various aspects of IPT research and practice, and therefore, was seen as a suitable framework on which to build the program. PI is both systemic and systematic, providing students with a logical, step- by-step process to follow when investigating organizational performance problems, while also reinforcing the importance of systems thinking. The PI framework also provided the structure necessary to resolve the second challenge of aligning the program deliverables with the framework. The first deliverable is a proposed analysis plan, identifying the problem of practice and the student’s strategies for collecting and analyzing data to complete the performance, gap, and cause analysis processes. The second deliverable reports the findings and proposes appropriate solutions based on the data and stakeholder input. The final deliverable, the dissertation in practice, includes the information from the previous two steps as well as information about the design, development, implementation, and initial evaluation of their identified solutions (University of West Florida, 2022). Faculty worked collaboratively to address the third challenge noted: reimagining the research methods portion of the curriculum to align with the data collection and analysis activities associated with the performance improvement framework. We started from the traditional perspective of individual classes on quantitative, qualitative, and mixed-methods research designs, but quickly realized that sub-dividing PI research in this manner did not provide students with a realistic view of the analysis activities they would be engaging in post-graduation. We knew we needed something different, and we knew that it needed to fit into no more than three dedicated research methods courses. The design of the research courses is rooted in adult learning theory (Knowles 2014), action research methodology (Stringer, 2020), and principles of performance improvement (Van Tiem et al., 2012). The decision to split the research content into three courses aligns with these theories and research methods. The action research course is taught first to give the students foundation in action research. The other two courses, data collection and then data analysis, are designed and offered in this order to align with the processes involved in conducting a performance improvement action research project. The content of these two courses is situated in adult learning theory, qualitative and quantitative research methods, and performance improvement processes. According to the tenants of adult learning theory (Knowles, 2014), adults learn by the information being relevant at the time of instruction and the content being “oriented to learning as problem-centered and contextual (p. 6)”. The research courses are aligned with the other courses in such a way that the data collection course is taught when students need to know how to design a data collection plan and create data collection instruments, allowing them to apply these concepts immediately. Similarly, the data analysis research course is taught when the students are preparing to analyze their data, allowing them to apply the concepts immediately. The courses and assignments are aligned so the students can apply them to the current problem in the workplace context that serves as the basis for the dissertation in practice. The structure of the research courses and the order in which they are presented supports three additional principles of adult learning theory: learning from experience, readiness to learn, and self-concept (Knowles, 2014). The students are learning by creating data collection plans and instruments and gaining immediate experience by using these instruments in real-world practice. The students are ready to learn as they are motivated to address problems of practice in their workplaces and have a vested interest in doing so. The balance of this essay focuses on the solution, how it was implemented, and our initial assessment of its effectiveness. COURSE OFFERINGS The EdD in IDT integrates the principles of action research and the processes associated with the International Society for Performance Improvement (ISPI)/Human Performance Technology (HPT) model to provide students with the knowledge, skills, and abilities necessary to conduct practitioner-based research. To achieve this goal without exceeding the intended nine credit hour research sequence, we opted to teach one course on practitioner- based/action research and then integrate quantitative, qualitative, and mixed methods research activities into two comprehensive courses: one covering data collection processes for all three designs and the other covering data analysis processes for all three. Fundamentals of Practitioner-based Research The first of the three research courses, Fundamentals of Practitioner-based Research, introduces students to the principles of action research and their applications in the field of instructional and performance technology. This course explores action research, defining a problem of practice, and contextualizing identified problems in the relevant literature. Processes and strategies for conducting research within an individual's organization are discussed extensively. This course provides the foundation necessary to conduct scholarly research using the performance improvement process. Students are taught how to critically analyze research literature. The course highlights trends for investigating problems and issues. Methods to ensure quality research are detailed throughout the course. In this course, the students are introduced to the six principles of CPED research. This course teaches the CPED signature pedagogy of preparing scholarly practitioners “to think, to perform, and to act with integrity” (Shulman, 2005, p. 52). The students are introduced to the concepts of action research and “Inquiry as Practice” (CPED, 2022, para. 8). They are guided through the process of identifying their problems of practice and contextualizing their problems of practice in the literature. The course requires the students to analyze the literature to facilitate the process of viewing their research at the intersection of theory, inquiry, and practice. Research Methods Course Redesign Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8, No. 2 (2023) DOI 10.5195/ie.2023.342 27 Course outcomes include: • examining the components of improvement science and performance improvement, • critiquing existing research regarding problems in professional educational practice, • generating a formal problem of practice document, • evaluating applied research design to address a problem of practice, and • assessing ethical standards of research. Specific topics that are examined during the course include the dissertation in practice, improvement science framework, performance improvement, diversity, equity, inclusion, social justice, scholarly literature review, problem-of-practice, scope and risks of a problem of practice, applied research, and research ethics. The first deliverable in the course is an investigation into various action research methods and how they relate to the dissertation and problem of practice. The students examine action research and the various data types, data collection methods, data analysis, and data presentations. As students are guided through the assignments, they learn about the dissertation in practice and the process of becoming a scholarly practitioner and the alignment of action research and improving performance within an organization. The assignment also provides students the opportunity to compare and contrast the different types of data and data collection processes involved in action research. The second deliverable of the action research course is the student’s in-depth problem of practice statement and description. They create documents outlining how their proposed problems of practice are urgent, actionable, feasible, and strategic. The students prepare their problem of practice documents using the CPED definition of a problem of practice, “a persistent, contextualized, and specific issue embedded in the work of a professional practitioner, the addressing of which has the potential to result in improved understanding, experience, and outcomes” (CPED, 2022, para. 11). Students provide a contextualized examination of the identified problem and how the resolution of this problem results in outcomes that improve both individual and organizational performance. The third deliverable is a review of the research as it relates to the identified problem of practice. The review of the relevant research includes an introduction and summary of the problem of practice, justification of elements included/excluded from the review, the broader situational context, historical context, definitions, resolution of ambiguities, methodologies, and research techniques that have been used in the field, the practical and scholarly significance of the research problem, and the impact of the problem’s resolution. Data Collection in Performance Improvement The second course in the student’s research curriculum sequence is Data Collection in Performance Improvement. Students examine data types, collection methods, sources, and instrumentation. Students learn how, when, where, and why to collect quantitative and qualitative data for performance improvement projects. Students critically investigate researcher bias, reflexivity, and positionality when conducting data collection for performance improvement research. In the data collection course, students learn to develop data collection instruments. This course coincides with the front-end analysis portion of the performance improvement process. The first step in using the performance improvement model is to gather information relative to an organization's current and desired performance. Students learn how to develop instruments to gather relevant quantitative and qualitative data to determine the actual and desired performance. In this course, students also learn how to prepare the documents necessary to apply for university Institutional Review Board (IRB) approval. The IRB is a university committee charged with protecting the rights and welfare of human participants involved in university research. The IRB process requires the students to include the data collection instruments. These documents are created during the semester and become part of the IRB application. Student learning outcomes include: • evaluating quantitative and qualitative data collection instruments, • evaluating strategies, methods, and timing for collecting quantitative and qualitative data for performance improvement projects, • justifying the alignment between identified problems of practice and chosen data collection elements, • creating data collection instruments for quantitative and qualitative performance improvement research, • implementing performance improvement data collection instruments aligned with a specific problem of practice, and • critically analyzing researcher bias and positionality in relation to performance improvement projects. Students produce three specific deliverables during this second research course. The deliverables for the course are 1) the completed quantitative instrument protocol, including planning for the assessment of validity and reliability, 2) the completed qualitative instrument protocol, including planning for the assessment of face and content validity, and 3) a completed IRB application. In addition to the completed assignments, the students are led through three comprehensive, structured discussions. These discussions provide support for the students and the opportunity for them to collaborate and critically examine key elements for creating data collection instruments for both quantitative and qualitative data collection. The three structured discussions are: 1) proposed problem of practice and research-focused elevator speech providing the essence of the planned study, 2) data collection instruments, participants, and required procedures before, during, and after data collection, and 3) qualitative instrument content and face validity. In the first structured discussion, the students provide a concise two to three sentence overviews of their problems of practice, a one- sentence purpose statement for their dissertations in practice, one overarching research question for their studies aligned with the purpose of their studies addressing the problems of practice, and potential research sub-questions that guide the dissertations in practice. For each research sub-question, the students include the a) type of data/data collection method and b) type of analysis that will provide formal responses to the research sub-questions. In the second structured discussion, the students discuss data collection instruments, participants, and the steps required relative to one of their proposed data collection methods. In this discussion, students provide brief descriptions of the instruments they will use for data collection and identify the participants that will be the sources of Handley & Hastings Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8, No. 2 (2023) DOI 10.5195/ie.2023.342 28 the data. The students list the steps required before, during, and after data collection. These steps include requirements such as obtaining their organization’s approval and the institution’s IRB approval, creating consent forms, confirming protocol validation and reliability, implementing the instrument, gathering data, recording interviews, downloading and cleaning data, transcribing qualitative interviews, and preparing data for analysis. In the third structured discussion, the students examine procedures to ensure content and face validity for qualitative instruments. The students also provide concise statements summarizing their data collection instruments, the participants, and the steps they will follow to ensure face and content validity for the instruments. The students collaborate and share ideas on the best practices for planning, communicating, and completing the process for instrument validation. Data Analysis Processes The third and final course in the student’s research curriculum sequence is Data Analysis in Performance Improvement. Students examine the various types of data analysis consistent with action research and the performance improvement framework. Appropriate qualitative and quantitative data analysis procedures and alignment with research questions and study purposes are also addressed in this course. Students learn how to analyze the data they collect using the instruments created in the previously completed data collection course. It is important to note, that while the students create instruments and analyze data relative to their specific problems of practice, they are taught all methods of data collection and analysis. The goal is to prepare students to collect and analyze data in future projects and contexts; therefore, data collection and data analysis techniques are not limited to the current problem of practice. Learning outcomes for this course include: • comparing and contrasting strategies for analyzing quantitative and qualitative data in performance improvement, • analyzing quantitative and qualitative data using appropriate methods in performance improvement, • mapping data analysis results to performance improvement research questions, and • presenting results in the context of the problem of practice. Specific topics that are examined during the semester include quantitative data, descriptive and inferential statistics, parametric and nonparametric tests, qualitative data, qualitative data analysis and coding, research question alignment, and writing and presentation of results. Students produce six specific deliverables during this third research course. The deliverables for the course are 1) study population, sample, and demographics, 2) quantitative data analysis, 3) qualitative data analysis, 4) performance analysis findings, 5) gap analysis findings, and 6) root cause analysis findings. At the conclusion of the course, the students are prepared to create comprehensive reports of their problems of practice, data collection and analysis activities, and findings and recommendations for presentation to their stakeholders. The purpose of this report is to obtain approval to move forward with the design, development, implementation, and initial evaluation of the recommended solutions. COURSE ROTATION Fundamentals of practitioner-based research is offered during the student’s first semester in the program. This has remained consistent across all three cohorts. The timing of the data collection and data analysis courses has changed based on student feedback and performance. We admit one cohort each fall, with our third cohort of students having started the program in Fall 2022. As the students have completed the research courses and are progressing in the program, we have made a few adjustments in the timing for the delivery of the courses in relation to their program of study. These changes are shown in Table 1. Table 1. Research Course Timing Research Course Cohort One Cohort Two Cohort Three Fundamentals 1st semester 1st semester 1st semester Data Collection 3rd semester 3rd semester 2nd semester Data Analysis 5th semester 4th semester 4th semester Note. Change in research course teaching from cohort one to cohort three. As the students have progressed in the program, we have found it beneficial to move the data collection and analysis courses earlier in the rotation. Initially, students took the foundations course in their first semester, the data collection course the following summer, and data analysis in the fifth of their six semesters of coursework. We quickly realized this timing was back-end loaded, making it difficult for students to progress at the expected rate. The first change we made from Cohort 1 to Cohort 2 was to move the data analysis course earlier in the program. This modification was intended to facilitate the students being prepared to present their findings and recommendations to their stakeholders earlier. This modification allowed students to engage stakeholders in the decision-making process sooner and provided students with more time to design, develop, implement, and initially evaluate identified solutions, all of which is necessary for degree completion. The second change in the timing of research course delivery was moving the data collection course one semester sooner in the rotation. The expectation is for students to collect data in the beginning of their second year. We found that IRB and student site approvals were taking longer than initially anticipated, delaying planned data collection. To address this challenge, we moved the data collection class one semester earlier starting with Cohort 3. The revised spacing of the research courses facilitates earlier data collection. The semester between data collection and data analysis allows the students adequate time for data collection. LESSONS LEARNED We gather feedback from the students in course reviews, personal communications, and written reflections for each course. The feedback from the students has been constructive, reflective, and overall positive. The students relay how they are provided the resources and research tools when needed, making the courses and program immediately relevant. Students have expressed how they appreciate the layering of the courses and how each semester, the research methods taught build on each other. Students also express how exciting it is to be able to apply how to conduct action research Research Methods Course Redesign Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8, No. 2 (2023) DOI 10.5195/ie.2023.342 29 directly in their organizations. They feel they are empowered to make positive impacts in their workplaces. Having the research presented in the sequence of collection and then analysis, the students have expressed the advantage of being able to compare and contrast qualitative and quantitative methods in terms of the type of data needed in the performance improvement context. Students comment on what they perceive to be the advantages of having the research content arranged as data collection and then data analysis compared to the more traditional, separate qualitative and quantitative courses and how it supports the scholarly practitioner focus of the degree program. Based on the student feedback and student deliverables, we added a semester between the data collection methods course and the data analysis course. The additional time has allowed most students to obtain IRB approval and collect data that can be used during the analysis course. This is a lesson learned as we continually monitor student progress through the program and the various phases of their research. One of the challenges we face is the need to present some of the analysis processes when introducing data collection to inform the students as to how they will use the data. We have added a big picture overview of data analysis, describing the entire process during the data collection course. This general introduction to analysis aids the student in designing data collection instruments. The students appreciate seeing how the data will be used and presented when designing data collection tools and instruments. Another challenge we are facing is relative to the quantitative research methods content. We have found that most performance improvement dissertation in practice research projects do not include high-level inferential statistics. The course introducing data collection for quantitative research has been modified to focus primarily on descriptive statistics as most of the projects do not go into in-depth inferential statistical analysis. As we further modify the curriculum and expand the program, we are planning to offer an optional, in- depth statistics course for those individuals that need or want a more comprehensive examination of inferential statistics. CONCLUSION We continue to assess the effectiveness of the research courses and our curriculum as students progress through the program. Based on student deliverables and student and faculty reflections, the timing and the content of the courses appear to be better aligned with the structure of the dissertation in practice and best practices in performance improvement. Providing the fundamentals in the first semester, the data collection course earlier in the program, and the data analysis course with only one semester separating it from the data collection course give students the structure necessary to move through the program at the desired pace, applying skills as learned. The revised structure provides opportunities for students to control their learning (Knowles, 2014) while ensuring students complete the major components of the dissertation in practice when expected, preparing them to finish the program in the intended three years. We regularly apply the principles of action research and performance improvement in the course and program design. This continuous improvement mindset, which is also supported by CPED, encourages us to regularly assess student performance to identify opportunities to enhance outcomes through modifications in curriculum, course design, and course rotation. It also provides us with an opportunity to model the best practices we are developing in our students. REFERENCES CPED. (2022). The CPED framework. https://cped.memberclicks.net/the- framework Knowles, M. S., Holton, E. F., & Swanson, R. A. (2014). The adult learner (8th ed.). Taylor & Francis. Shulman, L. S. (2005). Signature pedagogies in the professions. Daedalus, 134(3), 52–59. https://doi.org/10.1162/0011526054622015 Stringer, E. T., & Alfredo, O. A. (2020). Action research (5th ed.). SAGE Publications, Inc. University of West Florida. (2022). University of West Florida Online: Ed.D. instructional and performance technology. https://onlinedegrees.uwf.edu/online-degrees/edd-instructional- performance-technology/ Van Tiem, D. M., Moseley, J. L., & Dessinger, J. C. (2012). Fundamentals of performance improvement: Optimizing results through people, process, and organizations (3rd ed.). Pfeiffer. https://cped.memberclicks.net/the-framework https://cped.memberclicks.net/the-framework https://doi.org/10.1162/0011526054622015 https://onlinedegrees.uwf.edu/online-degrees/edd-instructional-performance-technology/ https://onlinedegrees.uwf.edu/online-degrees/edd-instructional-performance-technology/