Impacting Education: Journal on Transforming Professional Practice New articles in this journal are licensed under a Creative Commons Attribution 4.0 United States License. This journal is published by Pitt Open Library Publishing. 34 This journal is supported by the Carnegie Project on the Education Doctorate: A Knowledge Forum on the EdD (CPED) cpedinitiative.org impactinged.pitt.edu ISSN 2472-5889 (online) Vol. 8 No. 4 (2023) DOI 10.5195/ie.2023.296 Assessing Online Doctoral Student Research Competencies John C. Gillham University of Findlay gillham@findlay.edu Nicole V. Schilling University of Findlay nicole.schilling@findlay.edu ABSTRACT The purpose of this study was to investigate online education doctoral students’ perceptions of their research competencies. The researchers utilized the Scholar-Practitioner Research Development Scale (Rockinson- Szapkiw, 2018) which consisted of 24 Likert-based items to assess research competency in five areas: attitude toward or value of research, critical evaluation and application of research, research knowledge, research skills, and research dissemination. For each of the five research competencies students reported their competency at 4.0 or above (on a scale of 1- 5) each year, with students most strongly agreeing with statements related to the value of research, evaluation and application skills, and research knowledge. This study illuminates the development of research competencies in online doctoral students. Recommendations include using the Scholar-Practitioner Research Scale to assess program effectiveness, track program improvements, and identify gaps in the curriculum. KEYWORDS EdD student development, Scholar-Practitioner Research Development Scale (SPRDS), research competencies, Carnegie Project on the Education Doctorate (CPED) In 2006, Schulman et al. noted that “The problems of the education doctorates [Ed.D.] are chronic and crippling. The purposes of preparing scholars and practitioners are confused; as a result, neither is done well” (p. 25). Schulman et al. (2006) were part of a chorus of scholars (Andrews & Grogan, 2005; Archbald, 2008; Deering, 1998; Grogan & Andrews, 2002; Levine, 2005; Malen & Prestine, 2005; Murphy & Vriesenga, 2005; Osguthorpe & Wong, 1993; Toma, 2002; Townsend, 2002) who criticized Academia for failing to distinguish the Ed.D. degree from the Ph.D. degree and called for reform or redesign of the Ed.D. as a professional degree to prepare educators for practice in the field. It was for these purposes the Carnegie Foundation for the Advancement of Teaching initiated the Carnegie Project on the Education Doctorate (CPED) in 2007 (Perry et al., 2015). Over time, the traditional Ph.D. model was modified to meet the unique needs of the Ed.D. student, leading to the concept of the scholar–practitioner (also described in the literature as the scholarly practitioner or the practitioner-scholar). LITERATURE REVIEW Scholarship conducted since Shulman and colleagues’ (2006) challenge provides a more robust conception of the role and development of scholar-practitioners. Jenlink (2014) envisioned scholar-practitioners as educational leaders who engage in research as practice, view knowledge and practice as one, are situated in critically oriented inquiry and noted “scholar–practitioner leadership is grounded in a postmodern—post-positivist inquiry of leadership, which seeks to blur boundaries in the knowledge-practice and inquiry-practice relationship” (p. 8). Hochbein and Perry (2013) stated that scholar-practitioners will utilize three habits of inquiry— decipher, debate, and design—to solve their problems of practice. A central focus of these habits is to utilize the body of research in the field and apply it to a real-world context. Murakami-Ramalho et al. (2013) propose scholarly practitioners need to understand and use research to set goals, implement research design, and measure growth. Developing research competencies begins with critically consuming the literature (Slayton & Samkian, 2017), progressing to learning research methods, and culminates in the defense and publication of the dissertation (Baker & Pifer, 2014). Golde (2013) conceptualizes the development of scholar-practitioners as occurring in three stages: Entry and Integration, Integrating New Knowledge, and Completion and Exit. Focusing on impact, one scholar notes that “scholar–practitioners conjoin the strategies and knowledge gained through meticulous academic endeavors with experiences and knowledge...to form the basis of effective, change-centered practices” (Bouck, 2011, p. 203). Rockinson-Szapkiw (2018) posited scholar- practitioners should: 1. Develop a value of research, including an understanding and appreciation of research as a means to solve problems of practice and to advocate for social https://library.pitt.edu/e-journals http://creativecommons.org/licenses/by-nc-nd/3.0/us/ http://cpedinitiative.org/ Assessing Student Research Competencies Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8 No. 4 (2023) DOI 10.5195/ie.2023.296 35 justice and equity; 2. Obtain skills and knowledge to critically consume (e.g., information literacy), apply, and conduct research to inform practice; and 3. Gain the ability to disseminate research to professionals in the field to transform practice. (p. 2) Therefore, the primary objectives of a scholar-practitioner are to value, critically evaluate, and apply the necessary knowledge and skills to disseminate research to solve problems in the field. At its heart, “the scholar-practitioner’s goal is to bridge research, theory, and practice” (Suss, 2015, p. 50). As Ed.D. students begin to develop as scholar-practitioners, they become aware of problems of practice and approach these problems through the application of rigorous research methods. They generate, analyze, and disseminate data among stakeholders to produce change. Lasater et al. (2016) conducted a case study of the perceptions of students and alumni from the CPED affiliated Ed.D. program at University of Arkansas. Specifically, the researchers sought to answer three questions: 1. How do current students and alumni describe their experiences in the Educational Leadership online doctoral program at University of Arkansas? 2. How do current students and alumni perceive the online experience as impacting their professional practice? 3. How do current students and alumni describe the value of the Educational Leadership program? (Methods, para. 1). Utilizing a qualitative approach, the researchers distributed surveys and conducted semi-structured interviews with students who were enrolled in the program between 2010 and 2014. First among the three findings, students perceived “the doctoral program led to a change in their thinking” (Lasater et al., 2016, Findings, para. 1). One participant reported their experiences changed the way they looked at problems and how they approached problem-solving. They noted: I think about “what does the literature say?” Those are questions that are popping up in my head that didn’t pop up in my head before. I definitely can read those stats parts of the findings, when you read an article, now I am like, “oh, I think I know what that means” (Doctoral Program Led to a Change in Student Thinking, para. 2). Another participant shared they have become more critical of data distributed in the workplace and need proof that data is valid and meaningful. These findings reflect graduates who value, critically consume, and apply research to their professional context, characteristics of scholar-practitioners. Other findings included students valued the program design and the relationships they built with faculty and their peers in the program (Lasater et al., 2016). Perry and Zambo (2019) examined data gather from CPED- influenced programs to investigate whether the CPED framework was being used to develop the research capabilities of students. The researchers specifically examined Inquiry as Practice, a CPED design concept for scholar practitioner research development. The examination was part of a broader study that sought to assess the impact of the CPED framework upon member institutions’ doctoral programs. The study investigated teaching inquiry to improve practice in CPED-influenced programs, how students are socialized in this method, the methodologies used by these programs, and the skills and abilities that resulted from being taught inquiry. For this qualitative study, online, open-ended surveys were sent to individuals who were the primary contacts at CPED-influenced programs. Fifty-three (60%) member institutions responded. Participants were asked to provide student and faculty demographics, to describe how they enacted the CPED framework, and to provide syllabi to illustrate how the framework was enacted at their institutions. A two-level analysis consisting of generating categories and subcategories of elements of the CPED framework that were enacted was used to analyze open-ended responses. Syllabi were analyzed using content analysis until reaching saturation and providing evidence for the findings. Data gathered to answer the research questions allowed Perry and Zambo (2019) to better understand how participating institutions “understand inquiry as a form of practice for professional practitioners, how inquiry is taught, and what graduates are expected to be able to know and do as a result of having completed one of these programs” (p. 10). Findings indicated participating CPED-influenced programs put problems in the professional context at the center of doctoral and research training. The practice of inquiry and reliance on the research were understood to be essential skills for educational leaders. Programs also placed an emphasis on the socialization of students into their roles as scholar-practitioners. A focus on inquiry was seen as an integral part of being a scholar-practitioner and was present across the curriculum, in field experiences and in the dissertation process. Participating programs utilized various research frameworks and methodologies, with learning objectives centered on the dissertation. Research methodologies tended to permeate the curricula and increase in sophistication as students progressed through their programs. Participating CPED-influenced programs also utilized inquiry to develop scholar-practitioners who can do the following: think systematically, creatively, and reflectively; to blend their own practical knowledge with the theories and ideas learned in their program; to use literature to deeply understand problems of practice; to analyze and critique policies, theory, and research and to wisely consume literature and apply it to develop solutions. (Perry & Zambo, 2019, p. 18) Finally, participating CPED-influenced programs emphasized the “creation and transformation of professional knowledge and practice” (Perry & Zambo, 2019, p. 18), one of CPED’s Six Guiding Principles. CPED-influenced programs foster students’ identities as scholar- practitioners and students are inculcated with the belief that ongoing professional learning must take place to renew professional practice and programs. Because many education doctoral students lack experience conducting research, they may not feel competent in research or developed an identity as a researcher (Caskey et al., 2020), and scholars note developing an identity as a researcher for doctoral students is a complicated problem (Labaree, 2003; Murakami- Ramalho et al., 2013; Shulman et al., 2006). In their review of the literature, Choi et al. (2019) investigated how doctoral students (both Ph.D. and Ed.D.) develop their identities as scholars. The study was built upon Gee’s (2000) theory of identity and Engeström’s (1987, 2001) cultural-historical activity theory. The researchers defined a scholar’s identity as “as an individual’s felt or recognized abilities allowing association with communities doing scholarship pertaining to an academic discipline” (Choi et al., 2019, pp. 7-8). Choi et al. (2019) sought to understand the conceptualizations of researcher identity relevant to doctoral students, determine how doctoral Gillham & Schilling Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8 No. 4 (2023) DOI 10.5195/ie.2023.296 36 students’ identities as scholars evolve, and determine the practical implications for stakeholders based on the literature of how to cultivate doctoral students’ identities as scholars. After unstructured and structured phases of literature identification, 62 potential research studies were identified. An additional round of reviewing the exclusion/inclusion criteria resulted in 36 empirical studies suitable for review. Independent coding by the authors and subsequent discussions resulted in cogent codes and findings. The literature investigated in the Choi et al. (2019) study revealed how doctoral students begin to see themselves as scholars and become recognized as scholars by others. Multiple methods of attaining scholarly identity were noted, including engaging with other doctoral scholars, seeking and receiving feedback from the faculty, engaging with undergraduate students and earning their respect, garnering respect for their work from non-academics, participating in affinity groups, and attending academic conferences. Student identities as scholars increased as faculty shared their vulnerabilities and students reflected on their research journey, noting the challenges associated with producing research. Choi et al. (2019) also noted how doctoral students developed a sense of competence as scholar-practitioners. Choi et al. (2019) identified 27 articles from the literature that indicated competence and confidence often go hand-in-hand. As students develop research competence, they also develop confidence and become more invested in their identities as scholars. One method to promote competence was participation in student-created writing groups for faculty and peers. An additional layer of challenge in developing scholar practitioner research competencies is the increasing reliance on online education as a medium for teaching the value and application of research, as with the program in this study. For most students, the dissertation represents the pinnacle of the research experience during their preparation as scholar-practitioners and requires multiple research competencies. Scholars have investigated what dissertation chairs can do to support doctoral (Ed.D.) students as they craft their dissertations in an online learning environment. In their qualitative study, Burrington et al. (2020) conducted semi- structured interviews with dissertation chairs they contacted through social media. The six female and five male participants had a range of 1.5 to 16 years’ experience working with dissertation students in an online environment, with a mean of 5.5 years. Feedback was a frequent theme in the findings, including the frequency, modes, timelines, and effectiveness and relevance of feedback. Burrington et al. (2020) noted “the importance of providing frequent feedback through various modes of communication, emphasizing a tailored approach to the students’ needs. Timely, thorough feedback was supported, stressing effectiveness and relevancy, which was most commonly achieved through one-on-one communication” (p. 1). The researchers also noted the importance of the dissertation chair’s expectations for doctoral scholarship; scholar-practitioners must learn to represent themselves “in a scholarly voice, learning to conduct original research, and learning to tell the story of that research in a way that is consistent with a field of scholarship and practice, as well as with the research methodology chosen” (Burrington et al., 2020, pp. 7-8). The researchers found other factors in effectively supporting online dissertation students, including building a caring, trusting relationship, providing individualized guidance, and balancing the needs of the student with university’s requirements. To assess the student’s development, the research competency of the scholar-practitioner must be clearly examined using diverse criteria (Perry, 2015; Rockinson-Szapkiw, 2018; Rolfe & Davies, 2009; Servage, 2009; Shulman et. al., 2006). The program in this study sought to understand the development of its students’ research competencies as a means of assessing its ability to produce scholar- practitioners. To prepare students for their role as scholar- practitioners, CPED-influenced and other doctoral programs need a comprehensive, valid assessment of student growth to assess program effectiveness and track program improvements. Such an instrument might also serve as a guidepost for program faculty, revealing areas of strength as well as areas of need in the curriculum. PURPOSE AND CONTEXT OF THE STUDY This study sought to assess the impact of a Doctor of Education (Ed.D.) program on students’ perceptions of their ability to effectively use research in the field. The purpose of this study was to explore how online doctoral students, who are predominantly educational leaders, perceive the way they value, understand, and can use research. The program in this study prepares students to become scholar-practitioners who solve problems of practice using research- based strategies. Students apply the skills gained in their research courses to their dissertations and professional context over a period of three or more years. Across the curriculum, students are required to use the literature to solve problems in their professional practice. Graduates of the program take leadership roles in public education, assume responsibility for the school districts in which they apply their research skills, and serve as first responders in the gap between scholarship and systemic change. The setting for this study is a private residential liberal arts institution located in the mid-western United States. Founded in 1882, the institution offers 86 Bachelor’s degrees, 10 Master’s degrees, five Doctoral degrees, and a variety of non-degree license and certificate programs. Approximately one-third of the institution’s 4,829 students are enrolled in graduate programs, both online and face-to-face. Doctor of Education degree requirements at this institution include 60 course credits composed of core courses (24 credits), research and dissertation courses (21 credits), and electives (15 credits). Students may choose from two elective strands, Teaching and Learning or Administration. The Administrative strand includes courses required by the state for a superintendent’s license. Students may complete the program in as little as three years, with no penalty for taking fewer courses each semester and extending their timelines. To satisfy the residency requirement, students attend three face-to-face Summer Institutes, each lasting three days. The Summer Institute is an opportunity for students to present their research, give and receive feedback to and from their peers, and attend research training workshops. A signature pedagogy of the program is the embedded dissertation. Students begin working on the dissertation their first semester in the program and continue to craft it as they progress through the program. Several of the research courses are associated with specific chapters of the dissertation so students apply what they learn as they write the chapter in that course. As a member of CPED, the program supports and promotes the CPED Framework and the Guiding Principles for Program Design. Assessing Student Research Competencies Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8 No. 4 (2023) DOI 10.5195/ie.2023.296 37 The expectation is for students to develop as scholar-practitioners who will apply research to problems of practice (PoP) in their professional contexts. CPED (2021) posits that: Scholarly Practitioners blend practical wisdom with professional skills and knowledge to name, frame, and solve problems of practice. They use practical research and applied theories as tools for change because they understand the importance of equity and social justice. They disseminate their work in multiple ways, and they have an obligation to resolve problems of practice by collaborating with key stakeholders, including the university, the educational institution, the community, and individuals (Design-Concepts Upon Which to Build Programs, para. 2). Each core course in the program contains a key assessment that, taken together, compose the comprehensive exams. For each key assessment, students are required to analyze a PoP in their professional context relative to the subject of the course (e.g., policy, leadership, diversity, etc.), investigate the literature on their PoP, then create a plan to solve the PoP using the research they have identified. Students in this program are primarily individuals working full time in P12 education (teachers, principals, and superintendents), with some students from other fields such as higher education, health professions, and business. Students have an average age of approximately 39.7 years, with an age range of 22 to 70 years. Participants across the four years of the study ranged from one to five years of involvement in the Doctor of Education program. Offered entirely online (aside from the three-day Summer Institutes), this program is representative of a larger trend in education. Online graduate programs have witnessed consistent growth since the year 2000. At that time, 2.2 million students were enrolled in online graduate programs, a figure which increased to 3.1 million students by 2019 (NCES, 2021). Programs such as these require doctoral students to develop research competencies in a virtual context, thus presenting an addition layer of challenge to both faculty and students. The informal assessments faculty might make of research competencies in regularly scheduled face-to-face settings are not possible in online, asynchronous courses such as the one utilized in this program. In the same way, students’ self- assessment of research competencies are done in isolation without face-to-face context with peers and faculty, making the need for a specific, objective assessment of research competencies even more pressing. Similar challenges may be prevalent among CPED member institutions’ Doctor of Education programs, as 25% are offered entirely online (CPED, 2020). While online enrollment in undergraduate programs decreased during the COVID-19 pandemic, graduate programs grew at higher-than-expected rates (Quality Matters & Eduventures Research, 2021). Nearly 30% of graduate students enrolled in private, not-for-profit institutions like the one in this study are enrolled in exclusively online programs (NCES, 2020). One study of Chief Online Officers at private universities found that 88% expect online enrollment to continue to increase (Quality Matters & Eduventures Research, 2021). The study was guided by the following research question: How do scholar-practitioners perceive their research competency (value of research, evaluation and application skills, research knowledge, research skills, and research dissemination)? RESEARCH METHODS The researchers, who are online Doctor of Education program faculty, in this study sought to explore students’ perceptions of their research competency as scholar-practitioners through dissemination of the Scholar-Practitioner Research Development Scale (Rockinson-Szapkiw, 2018). The survey consisted of 24 Likert-based items (strongly agree - 5, agree - 4, neutral - 3, disagree - 2, and strongly disagree - 1) developed to access research competency in five areas: value of research (six items), evaluation and application skills (three items), research knowledge (six items), research skills (six items), and research dissemination (three items). For example, in respect to attitude or value of research, participants were asked to indicate their level of agreement with each statement: 1. Acquiring research knowledge and skills during my program is important. 2. Research can improve the lives of those served in my professional practice. 3. Research can improve my professional practice. 4. Research is useful to solve complex problems I face in my professional practice. 5. Research is important to promote equity and social justice in my professional practice. 6. Disseminating my research to various audiences is important to improve professional practice. Rockinson-Szapkiw (2018) reported in her development of the Scholar-Practitioner Research Development Scale that “evidence from both an exploratory factor analysis and internal consistency analysis demonstrate that the self-report scale has both validity and reliability” (p. 20). The Cronbach’s alpha was .93 and each subscale demonstrated good internal consistency (value of research = .88, evaluation and application skills = .82, research knowledge = .85, research skills = .90, and research dissemination = .93). A link to the online survey was distributed electronically to all Ed.D. students during the summer for four consecutive years (2018 - 2021). The primary use of the collected data was for program evaluation and program improvement. In addition to the questions on the Scholar Practitioner scale, four demographic questions were asked: (1) years in the Ed.D. program, (2) gender, (3) professional role, and (4) the number of years in their professional role. No identifying data was collected, which protected student confidentiality and encouraged truthful responses. However, due to the lack of identifiable data, it was not feasible to track individual students or cohorts longitudinally. For each iteration of the survey, the data was downloaded into an Excel spreadsheet. Means and standard deviations were calculated for each of the 24 items and for the five constructs for each of the four years. RESULTS A total of 129 students participated in the study over a four-year period. A large majority of participants reported they worked in P12 education as teachers or administrators. Other participants worked in fields such as higher education, health care, and business. Other participant demographics can be found in Table 1. Gillham & Schilling Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8 No. 4 (2023) DOI 10.5195/ie.2023.296 38 Table 1. Participant Demographics Study Year Participants (n) Year in the Program (%) Gender Average Years in Professional Role 2018 33 1 (28) 2 (34) 3 (22) 4 (16) Female: 64% Male: 36% 9.7 2019 33 1 (13) 2 (41) 3 (31) 4 (13) 5 (3) Female: 61% Male: 39% 10.3 2020 28 1 (18) 2 (36) 3 (32) 4 (14) Female: 61% Male: 39% 11.2 2021 35 1 (26) 2 (23) 3 (37) 4 (11) 5 (3) Female: 54% Male: 46% 10.4 Note. Percentages are rounded and may not equal 100. To answer the research question, an initial descriptive analysis of the data was conducted. Due to the lack of identifiable data, it was not feasible to track individual students or cohorts longitudinally, regardless patterns in the data emerged. Across all four years of the survey, participants perceived their competency highest in value of research, evaluation and application skills, and research knowledge, respectively. Value of Research Participants perceived their highest competency in the value of research construct in each of the four years of the study (M = 4.59, 4.65, 4.60, and 4.57 respectively). Within the value of research construct, in the first year of the study, participants perceived their competency to be equally highest (M = 4.71) for the items “Acquiring research knowledge and skills during my program is important” and “Research can improve the lives of those served in my professional practice.” The item which participants reported their lowest perceived competency (M = 4.49) was “Disseminating my research to various audiences is important to improve professional practice.” In the second year of the study, participants also perceived their competency to be highest (M = 4.79) for the item “Acquiring research knowledge and skills during my program is important.” The item which participants reported their lowest perceived competency (M = 4.52) was “Research is important to promote equity and social justice in my professional practice.” In the third year of the study, participants perceived their competency to be highest (M = 4.79) for the item “Acquiring research knowledge and skills during my program is important.” The item which participants reported their lowest perceived competency (M = 4.36) was “Disseminating my research to various audiences is important to improve professional practice.” In the fourth year of the study, participants perceived their competency to be highest (M = 4.89) for the item “Acquiring research knowledge and skills during my program is important.” The item which participants reported their lowest perceived competency (M = 4.34) was “Research is important to promote equity and social justice in my professional practice” (See Table 2). Table 2. Participant Perceived Competency in Value of Research Survey Item 2018 2019 2020 2021 M SD M SD M SD M SD Acquiring research knowledge and skills during my program is important. 4.71 0.46 4.79 0.48 4.79 0.42 4.89 0.32 Research can improve the lives of those served in my professional practice. 4.71 0.57 4.70 0.53 4.71 0.46 4.54 0.70 Research can improve my professional practice. 4.63 0.55 4.76 0.44 4.75 0.52 4.69 0.58 Research is useful to solve complex problems I face in my professional practice. 4.57 0.50 4.58 0.56 4.61 0.63 4.54 0.74 Research is important to promote equity and social justice in my professional practice. 4.53 0.61 4.52 0.62 4.39 0.69 4.34 0.84 Disseminating my research to various audiences is important to improve professional practice. 4.49 0.70 4.55 0.62 4.36 0.73 4.43 0.65 Evaluation and Application Skills Within the evaluation and application skills construct, participants reported their highest perceived competency for the item “I can identify scholarly resources to solve problems I encounter in my professional practice” across all four years of the study (M = 4.65, 4.79, 4.61, and 4.71 respectively). In the first three years of the study, participants reported their lowest perceived competency for the item “I can apply theory to solve problems I encounter in my professional practice” (M = 4.15, 4.36, and 4.29 respectively). In the fourth year of the study, participants reported their lowest perceived competency (M = 4.31) for the item “I can apply empirical research to solve problems I encounter in my professional practice” (See Table 3). Research Knowledge Within the research knowledge construct, participants reported their highest perceived competency for the item “I understand ethical guidelines for research in my profession (e.g., obtain IRB approval, do not harm participants)” across all four years of the study (M = 4.74, 4.81, 4.75, and 4.60 respectively). In the first two years of the study, participants reported their lowest perceived competency for the item “I understand analytic procedures to analyze data collected in my professional practice” (M = 3.94 and 4.14 respectively). In the third year of the study, participants reported their lowest perceived competency (M = 4.04) for the item “I understand how theories and paradigms are used to develop investigations to solve problems in my professional practice.” In the fourth year of the study, participants reported their lowest perceived competency (M = 4.20) for the item “I understand how to engage in the research process, from conceptualization to dissemination (e.g., communication to key stakeholders), to address problems in my professional practice” (See Table 4). Assessing Student Research Competencies Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8 No. 4 (2023) DOI 10.5195/ie.2023.296 39 Table 3. Participant Perceived Competency in Evaluation and Application Skills Survey Item 2018 2019 2020 2021 M SD M SD M SD M SD I can apply empirical research to solve problems I encounter in my professional practice. 4.32 0.64 4.55 0.56 4.36 0.68 4.31 0.58 I can apply theory to solve problems I encounter in my professional practice. 4.15 0.74 4.36 0.74 4.29 0.66 4.37 0.60 I can identify scholarly resources to solve problems I encounter in my professional practice. 4.65 0.60 4.79 0.42 4.61 0.69 4.71 0.46 Table 4. Participant Perceived Competency in Research Knowledge Survey Item 2018 2019 2020 2021 M SD M SD M SD M SD I understand ethical guidelines for research in my profession (e.g., obtain IRB approval, do not harm participants). 4.74 0.51 4.81 0.40 4.75 0.44 4.60 0.74 I understand how to formulate questions to investigate problems in my professional practice. 4.32 0.68 4.56 0.67 4.29 0.76 4.54 0.51 I understand research methods (e.g., quantitative, qualitative, and mixed) I can use to investigate problems in my professional practice. 4.41 0.66 4.63 0.49 4.43 0.69 4.43 0.74 I understand analytic procedures to analyze data collected in my professional practice. 3.94 0.89 4.14 0.87 4.18 0.86 4.23 0.73 I understand how theories and paradigms are used to develop investigations to solve problems in my professional practice. 4.00 0.92 4.39 0.72 4.04 0.84 4.31 0.72 I understand how to engage in the research process, from conceptualization to dissemination (e.g., communication to key stakeholders), to address problems in my professional practice. 4.29 0.80 4.37 0.62 4.18 0.90 4.20 0.93 Table 5. Participant Perceived Competency in Research Skills Survey Item 2018 2019 2020 2021 M SD M SD M SD M SD I can design meaningful research investigations to address problems in my professional practice. 4.06 0.83 4.38 0.79 4.04 0.88 4.29 0.67 I can choose the appropriate method of inquiry (e.g., quantitative, qualitative, and mixed) to address problems in my professional practice. 4.15 0.83 4.31 0.69 4.25 0.80 4.37 0.73 I can conduct rigorous research investigations to address problems in my professional practice. 4.12 0.89 4.22 0.83 4.07 0.86 4.03 0.92 I can interpret results from the data I analyze. 4.15 0.67 4.19 0.83 4.21 0.74 4.09 0.90 I can analyze data (e.g., quantitative, qualitative, and mixed) that I collect to address problems in my professional practice. 4.06 0.86 4.22 0.79 4.18 0.77 3.94 0.91 I can develop investigation questions to examine problems in my professional practice. 4.33 0.60 4.39 0.76 4.32 0.67 4.40 0.65 Table 6. Participant Perceived Competency in Research Dissemination Survey Item 2018 2019 2020 2021 M SD M SD M SD M SD I can communicate (e.g., present, write) the results of research investigations I conduct to key stakeholders. 4.18 0.85 4.44 0.72 4.25 0.70 4.17 0.86 I can discuss the results of research investigations in light of empirical and theoretical literature, drawing connections between the practice and the knowledge of the profession. 4.12 0.89 4.31 0.74 4.00 0.90 4.17 0.82 I can communicate implications to improve practice based on the results of research investigations I conduct. 4.27 0.72 4.38 0.79 4.14 0.85 4.29 0.83 Gillham & Schilling Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8 No. 4 (2023) DOI 10.5195/ie.2023.296 40 Research Skills Within the research skills construct, in the first year of the study participants perceived their competency to be equally highest (M = 4.15) for the items “I can choose the appropriate method of inquiry (e.g., quantitative, qualitative, and mixed) to address problems in my professional practice” and “I can interpret results from the data I analyze.” In that same year, participants perceived their competency to be equally lowest (M = 4.06) for the items “I can design meaningful research investigations to address problems in my professional practice” and “I can analyze data (e.g., quantitative, qualitative, and mixed) that I collect to address problems in my professional practice.” In the second, third, and fourth year of the study, participants perceived their competency to be highest for the item “I can develop investigation questions to examine problems in my professional practice” (M = 4.39, 4.32, and 4.40 respectively). In the second year of the study, the item which participants reported as their lowest perceived competency (M = 4.19) was “I can interpret results from the data I analyze.” In the third year of the study, participants perceived their competency to be highest (M = 4.04) for the item “I can design meaningful research investigations to address problems in my professional practice.” In the fourth year of the study, participants perceived their competency to be highest (M = 3.94) for the item “I can analyze data (e.g., quantitative, qualitative, and mixed) that I collect to address problems in my professional practice” (See Table 5). Research Dissemination Within the research dissemination construct, participants reported their lowest perceived competency for the item “I can discuss the results of research investigations in light of empirical and theoretical literature, drawing connections between the practice and the knowledge of the profession” across all four years of the study (M = 4.12, 4.31, 4.00, and 4.17 respectively). In the first year of the study, participants perceived their competency to be highest (M = 4.27) for the item “I can communicate implications to improve practice based on the results of research investigations I conduct.” In the second year of the study, participants perceived their competency to be highest (M = 4.44) for the item “I can communicate (e.g., present, write) the results of research investigations I conduct to key stakeholders.” In the third year of the study, participants perceived their competency to be highest (M = 4.25) for the item “I can communicate (e.g., present, write) the results of research investigations I conduct to key stakeholders.” In the fourth year of the study, participants perceived their competency to be highest (M = 4.29) for the item “I can communicate implications to improve practice based on the results of research investigations I conduct” (See Table 6). CONCLUSIONS AND DISCUSSION This study illuminates the development of research competencies in online doctoral students. Participants in this study are developing scholar-practitioner research competencies through an online medium. Participants rated their competencies with a weighted average of 4.0 or higher in all five constructs across all four years. Even the weakest responses to an individual survey item (M = 3.94) barely fell below the 4.0 threshold. On average, students strongly agreed or agreed with statements related to their research competency in all five constructs. This may be attributed to the emphasis placed in the program on using research to solve problems of practice, as is evidenced in other CPED member programs (Perry & Zambo, 2019). Another possible explanation for the high degree of perceived competency is the expectation that students will emerge from the program as a scholar-practitioner equipped to impact their professional context, a factor found to impact doctoral student development (Burrington et al., 2020). Rockinson-Szapkiw (2018) posits scholar-practitioners have three purposes, the first of which is “Develop a value of research, including an understanding and appreciation of research as a means to solve problems of practice” (p. 2). Students self-assessed their competency to be highest in three categories: value of research, evaluation and application skills, and research knowledge, aligning with Rockinson-Szapkiw’s (2018) first purpose. This also supports the findings of Perry and Zambo (2019) who found among CPED member programs that application of research skills is an essential element of preparing scholar-practitioners. These programs put inquiry at the center of their curriculum, as does the program in this study. Rockinson-Szapkiw’s (2018) second and third purposes of scholar-practitioners are to “Obtain skills and knowledge to critically consume (e.g., information literacy), apply, and conduct research to inform practice” and “Gain the ability to disseminate research to professionals in the field to transform practice (p. 2).” These purposes align with the research skill and research dissemination competencies. In the last two years of the survey, participants most strongly agreed with the statement “acquiring research knowledge and skills during my program is important,” which suggests the importance of research skills had been emphasized in the program and, more importantly, internalized by the online doctoral students in this study. Similarly, in Lasater et al.’s (2016) study, program graduates approached problems and problem-solving differently and became critical consumers of data. Although the means were lower for research skills and research dissemination than the other competencies, this may be explained by the experience of participants in the study. Research value and knowledge is emphasized from the first semester of the program; however, the application of research skill and dissemination culminate in a dissertation defense in the program’s third and final year. Scholars have noted that acquiring research competencies in doctorate programs begins with evaluating and consuming literature (Slayton & Samkian, 2017), progresses to research methods, and culminates in the dissertation defense and publication (Baker & Pifer, 2014). Between 49% and 61% of participants surveyed each year were in their second year of the program, which may explain higher perceived competency in the value of research and lower perceived competency in research skill and research dissemination. Additionally, as students progress through the program, they attend more Summer Institutes and become more active in attending and presenting at education research conferences. Choi et al. (2016) found that participating in affinity groups and attending academic conferences were both associated with development of a scholar identity. These factors suggest why the skill to conduct and disseminate research develops more slowly even as the value of research, evaluation and application skills, and research knowledge are developing throughout the program. In conclusion, the findings indicate that participants in this study, in accordance with Rockinson-Szapkiw (2018), are developing as scholar-practitioners in their abilities to value research as a means to Assessing Student Research Competencies Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8 No. 4 (2023) DOI 10.5195/ie.2023.296 41 solve problems of practice, obtain skills and knowledge to critically consume, apply, and conduct research to inform practice, and gain the ability to disseminate research to professionals in the field to transform practice. RECOMMENDATIONS AND CALLS FOR FUTURE RESEARCH First, the researchers recommend using the Scholar- Practitioner Research Scale to assess program effectiveness and track program improvements. The scale represents a valid means for doctoral faculty to assess whether students are emerging as scholar- practitioners who can “bridge research, theory, and practice” (Suss, 2015, p. 50). Our data indicated students perceived their competencies to be highest in value of research, suggesting the program has been successful in fostering a belief in the important role research plays in solving problems of practice. On the other hand, students perceived research dissemination to be one of their lowest competencies. However, growth in student perceptions of dissemination of research over the last few iterations of the survey reflect an increased focus on research dissemination in the program as a result of the findings from earlier iterations of the Scholar- Practitioner Research Scale. In response to data collected in the earlier iterations of the survey, the program put a greater emphasis on research dissemination earlier in the program, alumni shared how they disseminated their research, and students were encouraged to present at regional and national conferences, even during the research-in-progress stage. Subsequently, student perceptions of dissemination of research increased. Thus, the Scholar-Practitioner Research Scale helped identify a need and empowered the program to target and improve that research competency. Golde (2013) conceptualizes the development of scholar- practitioners as occurring in three stages: Entry and Integration, Integrating New Knowledge, and Completion and Exit. It is recommended that CPED-influenced Ed.D. program develop a strategy for developing research competencies across all three stages. Similarly, it is recommended that CPED-influenced institutions and other institutions that prepare scholar-practitioners, review individual items on the Scholar-Practitioner Research Scale to identify possible gaps in the curriculum. For example, a Doctor of Education program may emphasize using research to solve PoPs, yet the item “I can identify scholarly resources to solve problems I encounter in my professional practice” might lead faculty to question the ways in which students are taught to access research, especially after they graduate from the program. The literature highlights the skills required of scholar- practitioners, yet additional research is needed to determine the impact of specific pedagogical practices and faculty behaviors upon the development of individual research competencies. This is particularly important for online doctoral education, a growing segment of doctoral study. Further research is required to determine the manner in which scholar-practitioner research competencies are attained by students and whether these competencies are attained sequentially, concurrently, or in splintered fashion. Though Golde (2013) conceptualized the development of scholar-practitioners as occurring in three stages, the scholarship on doctoral students has yet to determine if the growth and development of research competencies are developed in a sequential, linear fashion or if development is staggered and splintered. In other words, research competencies may not be developed in a specific order or at a specific pace. Perhaps students experience periods of rapid research competency development and growth, interspersed with periods of relatively slower growth. Likewise, some competencies may be developed simultaneously. More research is needed to make these determinations. As Choi et al.’s (2019) literature review reveals, multiple studies found a link between research competence and researcher confidence. Further research is needed to determine if competencies in particular skills is associated with researcher confidence and if scholar-practitioner attitudes (e.g., value of research) are associated with competence and confidence. Choi et al. (2019) also note that specific activities such as participation in student-created writing groups promote research competence, however additional research is needed to determine how these might function in an online, asynchronous environment, such as the one in this program. While this study investigated the development of scholar- practitioner competencies in an online Doctor of Education program, it could not separate the impact of the program from the online medium itself. Therefore, further investigation is needed to determine the impact of an online learning environment on the development of scholar-practitioner competencies. Replication of this study using identifiable individuals and groups is recommended to better understand the growth of research competencies over time, including sequence and pace. A qualitative study of student experiences while developing research competencies would help illuminate this phenomenon and therefore is also recommended. REFERENCES Andrews, R., & Grogan, M. (2005). Form should follow function: Removing the Ed.D. dissertation from the Ph.D. straight jacket. UCEA Review 46(2), 10–12. Archbald, D. (2008). Research versus problem solving for the education leadership doctoral thesis: Implications for form and function. Educational Administration Quarterly, 44(5), 704–739. Baker, V. L., & Pifer, M. J. (2014). Preparing for practice: Parallel processes of identity development in stage 3 of doctoral education. International Journal of Doctoral Studies, 9, 137–154. https://doi.org/10.28945/2041 Bouck, G. M. (2011). Scholar–practitioner identity, A liminal perspective. Scholar-Practitioner Quarterly, 5(2), 201–210. Burrington, D., Madison, R. D., & Schmitt, A. (2020). Dissertation committee chairs’ current practices to support doctoral students in an online doctoral program. Online Journal of Distance Learning Administration, 23(3), 1–13. Carnegie Project on the Education Doctorate. (2020). 2020 CPED Member Report. https://cped.memberclicks.net/assets/2020%20Member%20Report.pdf Carnegie Project on the Education Doctorate. (2021). The CPED Framework. CPEDinitiative.org. https://www.cpedinitiative.org/the-framework Caskey, M., Stevens, D., & Yeo, M. (2020). Examining doctoral student development of a researcher identity: Using the draw a researcher test. Impacting Education: Journal on Transforming Professional Practice, 5(1). https://doi.org/10.5195/ie.2020.92 Choi, Y. H., Bouwma-Gearhart, J., & Ermis, G. L. (2019, April 8). Identity development in doctoral education: Literature reviews and implications [Paper presentation]. Annual Meeting of the American Educational Research Association, Toronto, CA. Golde, C. M. (2013). Afterward: Mapping the transformation of the Ed.D student. In J. A. Perry & D. L. Carlson’s (Eds.), In their own words, A journey to the stewardship of the practice in education (pp. 139-148). Information Age. https://doi.org/10.28945/2041 https://cped.memberclicks.net/assets/2020%20Member%20Report.pdf https://www.cpedinitiative.org/the-framework https://doi.org/10.5195/ie.2020.92 Gillham & Schilling Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8 No. 4 (2023) DOI 10.5195/ie.2023.296 42 Hochbein, C., & Perry, J. A. (2013). The role of research in the professional doctorate. Planning and Changing 44(3/4), 181–195. Jenlink, P. M. (2014). The cultural ecology of scholar-practitioner leaders: An ethnographic study of leadership. Education Leadership Review of Doctoral Research, 1(1), 1–24. Labaree, D. F. (2003). The peculiar problems of preparing educational researchers. Educational Researcher, 32(4), 13–22. https://doi.org/10.3102/0013189X032004013 Lasater, K., Bengston, E., & Murphy-Lee, M. (2016). An online CPED educational leadership program, Student perspectives on its value and influence on professional practice. Impacting Education, Journal on Transforming Professional Practice, 1(1). https://doi.org/10.5195/ie.2016.8 Levine, A. (2005). Educating school leaders. Columbia University Teachers’ College. Malen, B., & Prestine, N. (2005). The case for revitalizing the dissertation. Review 46(2), 7–9. Murakami-Ramalho, E., Militello, M., & Piert, J. (2013). A view from within: How doctoral students in educational administration develop research knowledge and identity. Studies in Higher Education, 38(2), 256–271. https://doi.org/10.1080/03075079.2011.578738 Murphy, J., & Vriesenga, M. (2005). Developing professionally anchored dissertations. School Leadership Review, 1(1), 33–57. National Center for Education Statistics (2020). Number and percentage of students enrolled in degree-granting postsecondary institutions, by distance education participation, location of student, level of enrollment, and control and level of institution: Fall 2018 and fall 2019. https://nces.ed.gov/programs/digest/d20/tables/dt20_311.15.asp National Center for Education Statistics (2021). Postbaccalaureate enrollment. https://nces.ed.gov/programs/coe/pdf/2021/chb_508c.pdf Perry, J. A. (2015). The Carnegie Project on the Education Doctorate. Change: The Magazine of Higher Learning, 47(3), 56–61. Perry, J. A., Zambo, D., & Wunder, S. (2015). Understanding how schools of education have redesigned the doctorate of education. Journal of School Public Relations 36(1), 58–85. Perry, J. A., & Zambo, D. (2019, April 8). Developing educational leaders with the capacity to connect research and practice [Paper presentation]. Annual Meeting of the American Educational Research Association, Toronto, CA. Quality Matters & Eduventures Research (2020). The changing landscape of online education (CHLOE) 4: Navigating the mainstream. https://www.qualitymatters.org/sites/default/files/research-docs- pdfs/CHLOE-4-Report-2020-Navigating-the-Mainstream.pdf Rockinson-Szapkiw, A. (2018). The development and validation of the scholar– practitioner research development scale for students enrolled in professional doctoral programs. Journal of Applied Research in Higher Education, 10(4). 478–492. https://doi.org/10.1108/JARHE-01-2018-0011 Rolfe, G., & Davies, R. (2009). Second generation professional doctorates in nursing. International Journal of Nursing Studies, 46(9), 1265–1273. Servage, L. (2009). Alternative and professional doctoral programs: What is driving the demand? Studies in Higher Education, 34(7), 765–779. Shulman, L. S., Golde, C. M., Bueschel, A. C., & Garabedian, K. J. (2006). Reclaiming education’s doctorates: A critique and a proposal. Educational Researcher, 35(3), 25–32. Slayton, J., & Samkian, A. (2017). Scaffolding learning for practitioner-scholars: The philosophy and design of a qualitative research methods course. Journal of Research on Leadership Education, 12(1), 51–71. Suss, D. D. (2015). T4 MAP™: A scholar-practitioner model for performance improvement. Performance Improvement Quarterly, 27(4), 49–75. https://doi.org/10.1002/piq.21179 Toma, J. D. (2002, November 22). Legitimacy, differentiation, and the promise of the EdD in higher education [Paper presentation]. Annual Meeting of the Association for the Study of Higher Education, Sacramento, CA. Townsend, B. (2002, November 22). Rethinking the EdD or what’s in a name [Paper presentation]. Annual Meeting of the Association for the Study of Higher Education, Sacramento, CA. https://nces.ed.gov/programs/digest/d20/tables/dt20_311.15.asp https://nces.ed.gov/programs/digest/d20/tables/dt20_311.15.asp https://nces.ed.gov/programs/digest/d20/tables/dt20_311.15.asp https://nces.ed.gov/programs/coe/pdf/2021/chb_508c.pdf https://www.qualitymatters.org/sites/default/files/research-docs-pdfs/CHLOE-4-Report-2020-Navigating-the-Mainstream.pdf https://www.qualitymatters.org/sites/default/files/research-docs-pdfs/CHLOE-4-Report-2020-Navigating-the-Mainstream.pdf https://doi.org/10.1002/piq.21179