Impacting Education: Journal on Transforming Professional Practice New articles in this journal are licensed under a Creative Commons Attribution 4.0 United States License. This journal is published by Pitt Open Library Publishing. 20 This journal is supported by the Carnegie Project on the Education Doctorate: A Knowledge Forum on the EdD (CPED) cpedinitiative.org impactinged.pitt.edu ISSN 2472-5889 (online) Vol. 8 No. 4 (2023) DOI 10.5195/ie.2023.333 Pilot Testing as a Strategy to Develop Interview and Questionnaire Skills for Scholar Practitioners: A Selection of Education Doctorate Students’ Reflective Vignettes Ruth Tate University of Toronto ruth.tate@utoronto.ca Fatima Beauregard University of Toronto fatima.beauregard@utoronto.ca Cristina Peter University of Toronto cristina.peter@utoronto.ca Laura Marotta University of Toronto laura.marotta@utoronto.ca ABSTRACT This essay presents the reflections of four Education Doctorate (EdD) students on the pilot testing strategies used during an online research methods course. Rigorous questionnaire and interview development skills are challenging to acquire. Pilot testing is an under-researched stage of instrument design, yet it is crucial to ensure validity and reliability, reduce bias, and psychologically prepare researchers for data collection. A structured, multi-step pilot testing process led to the collective development of stronger scholar-practitioner identities, the use of innovative synchronous/asynchronous methods during COVID-19 and increased academic rigor. These reflections demonstrate how several types of pilot testing can support the development of rigorous data collection instruments and prepare post-graduate students for the psychological and technical challenges they may encounter in future research. KEYWORDS research methods, pilot testing, critical self-reflections, Education Doctorate, course design, learning during COVID-19 INTRODUCTION The Education Doctorate (EdD) has historically been the awkward middle child of faculties of education - hard to differentiate from the PhD yet often considered intellectually inferior and lacking the status of the “higher professions” (Labaree, 2003, p. 13). Moreover, critics of the EdD argue that it has focused on traditional research methods that inadequately prepare educational leaders to conduct sound, rigorous, research that is relevant to their practice (Buss et al., 2014). In recent years, there has been a focus on how the EdD can be intentionally structured so that graduate students can transition from learners with extensive professional experience to scholarly, influential practitioners with the “combined practical wisdom, professional knowledge, and research skills to identify, frame, and solve problems of practice in their workplace settings” (Buss et al., 2014, p. 139). Rigorous data collection that produces meaningful contributions to the field can be a challenging skill for novice and experienced scholars alike. Naturally, a researcher’s skills determine the reliability and validity of instruments and the quality of both qualitative and quantitative research. When conducting interviews, the interviewer is “the primary instrument” (Majid et al., 2017, p.1) in data collection, and when using questionnaires, the wording of questions is “crucial to their success” (Cohen et al., 2018, p. 496). As EdD students cultivating our research skills, we have found limited empirical research examining how scholars acquire the methodological acumen that is central to producing quality educational research. There is little formal training available, few verified resources, and a gap in the literature examining how to develop excellent research skills in scholar practitioners, especially with regards to developing interview protocols and questionnaires (Roulston et al., 2003). This essay presents our critical self-reflections as four EdD students on the instrument development processes and metacognitive exercises used during a research methods course titled “Introduction to Research Methods for the EdD” with synchronous and asynchronous components. The Carnegie Project https://library.pitt.edu/e-journals http://creativecommons.org/licenses/by-nc-nd/3.0/us/ http://cpedinitiative.org/ https://orcid.org/0000-0001-7033-0898 https://orcid.org/0009-0000-5992-2569 https://orcid.org/0000-0002-7916-6913 https://orcid.org/0000-0002-0361-8178 Pilot Testing as a Strategy to Develop Interview Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8 No. 4 (2023) DOI 10.5195/ie.2023.333 21 on the Education Doctorate (CPED)’s (2021) “Design-Concepts upon which to build EdD programs”, recommends that scholarly practitioners should develop “a professional knowledge base that integrates both practical and research knowledge, that links theory with systemic and systematic inquiry” (para. 6) that they can use to “name, frame, and solve problems of practice” (para. 8). The first half of this course aimed to provide students with an understanding of research ethics, the main paradigms and methodologies in educational research, how to formulate research questions, and different research designs. In the second half of the course, there was a focus on providing active learning opportunities that would allow students to “acquire essential research skills” (Acton, 2022, p. 3). During the course, students developed a research question linked to a problem of practice and designed and conducted interviews and survey questionnaires. The aim was for the skills, readings, and considerations that students used to explore their research questions in this course to inform their doctoral dissertation research later in the course. In this essay, we share our experiences of following a multi-step, iterative, pre-testing, and revision pilot testing method of our interview protocol and questionnaires. We then reflect on the process and impact of gathering feedback from peers, recording think-aloud feedback, and conducting recorded cognitive interviews, and conclude by sharing recommendations for promising practices in future graduate research methods courses. PURPOSE This reflective essay uses Castillo-Montoya's (2016) Interview Protocol Refinement (IPR) framework to showcase how the stages of pilot testing, including peer feedback and cognitive interviewing, are effective pedagogical practices in developing both quantitative and qualitative research skills. Its purpose is to provide insights from EdD students that can support scholars’ and instructors' integration of peer-to-peer pilot testing as a pedagogical tactic to support rigorous, quality data collection instrument development and prepare students for the potential psychological and technical challenges of the research process. LITERATURE REVIEW While surveys and interviews are widely used by scholars, there is limited research and guidance on how doctoral students develop these skills through the course of their programs (Cohen et al., 2018; Collins, 2003; Merriam & Tisdell, 2016). As emerging researchers and scholars, graduate students aim to learn how to produce “results that are valid, reliable, sensitive, unbiased and complete” (Collins, 2003, p. 229) both for methodological rigor and to uphold participants’ rights to engage in quality research (Cohen et al., 2018). The quality of the data collected relies heavily on the data collection instrument yet creating an interview protocol or survey instrument is not an intuitive or simple task. In quantitative research, Cohen et al. (2018) argue that every stage of the questionnaire development process should be critically reflected upon, including the questions, their impact on participants, the data analysis, and the reporting. In qualitative research, scholars agree that interview protocols must be carefully designed to ensure validity and reliability and reduce bias (Castillo-Montoya, 2016; Collins, 2003; Merriam & Tisdell, 2016). Intentionally embedding pilot testing within a design- based research methods course can be a meaningful and effective pedagogical practice to support EdD students in developing their skills as research practitioners. Pilot testing prepares students for the potential challenges encountered during data collection and increases developing scholars’ competence and confidence in their research processes (Castillo-Montoya, 2016; Collins, 2003; Merriam & Tisdell, 2016). With the goal of creating surveys and questionnaires that are “clear, unambiguous and useful” (De Vaus, 2014, p. 97), participating in pilot testing allows students to check clarity of questions, gain feedback on validity of questionnaire items, test whether questions are leading, eliminate ambiguities, and check readability (Cohen et al., 2018; De Vaus, 2014; Privitera & Ahlgrim-Delzell, 2018). In addition to the creation of a reliable and valid instrument, students can also use piloting activities to monitor the participant experience, such as checking how long the questionnaire takes to complete and how difficult the questions are to answer. Finally, piloting can support students’ understanding of data analysis as they can use the pilot phase to generate categories from open-ended response items, identify redundant or superfluous questions, and try out potential coding/classification systems for data analysis (Cohen et al., 2018). In the design of an interview protocol, pilot testing supports students in generating interview protocols that are clear, consistent, and participant-centered (Truong, 2021). Piloting, more importantly, allows students to practice conducting interviews - which can be surprisingly challenging! Many researchers state they have never received formal interview training; therefore, piloting interviews within doctoral programs can provide future researchers with an informal and low stakes way to hone and practice their identity and skills as interviewers (Majid et al., 2017; Roulston et al., 2003). USING THE IPR FRAMEWORK AS A GUIDE FOR PILOTING INTERVIEWS AND SURVEYS While there is scant literature on the teaching of research design to doctoral students, Castillo-Montoya's (2016) four phase IPR framework offered an elegant process that guided us as doctoral students when piloting interviews and surveys during this research methods course. The IPR framework includes: (1) ensuring that research questions inform the questions asked of participants, (2) creating questions that are inquiry-focused, (3) inviting and incorporating feedback on protocols, and (4) piloting the protocol. In addition to using Castillo-Montoya's (2016) IPR framework to structure each step of the instrument development process, our course instructor introduced the use of supplemental cognitive interviewing methods to pilot student surveys and questionnaires. Willis (1999) describes two components of cognitive interviewing: the think aloud and verbal probing. The think aloud pilot empowers the participant to share their thought process while responding to each survey question. The researcher then notes whether questions are eliciting thought processes that produce reliable and valid responses. The verbal probing procedure allows the researcher to gather further information regarding participants’ understanding of each question and identify how and if a question meets its measurement purpose (Collins, 2003). These components of cognitive interviewing ensure that individual questions and the survey, as a whole, receive a rigorous and critical review before being presented to participants (De Vaus, 2014). Despite the limited literature on pilot testing and its role in supporting emerging researchers in gaining interview and survey protocol creation skills, Castillo-Montoya's (2016) framework and the cognitive interview processes appear to be rigorous methods for students to use to improve the reliability and validity of their Tate et al. Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8 No. 4 (2023) DOI 10.5195/ie.2023.333 22 research protocols and served as the basis for piloting activities within this course. METHOD For the purposes of this course, students were asked to explore a small-scale research question of interest related to their potential dissertation topic that could be further developed at a later stage. Peer-to-peer pilot testing was used as a pedagogical practice for an online research methods course for EdD students. Assignments in the course included the hands-on experience of creating, testing, and conducting a survey questionnaire and interview protocol using asynchronous and synchronous virtual activities. Castillo-Montoya's (2016) four-phase IPR framework and additional cognitive interviewing strategies were used by students to systematically develop and refine their research protocols. Phase 1: Aligning Interview and Survey Questions with Research Questions As part of the pre-testing process, students created a matrix to support the alignment of their research questions to the interview and questionnaire question protocols. These matrices were shared with peers who engaged in a close reading of questions. lose reading of an interview protocol “entails a colleague, research team member, or research assistant examining the protocol for structure, length, writing style, and comprehension” (Castillo-Montoya, 206, p. 826). Asynchronous feedback was provided via discussion boards, and students then included the proposed changes within their matrices. Students engaged in personal reflection and used the feedback they received to inform revisions to their instruments. Phase 2: Constructing an Inquiry-Based Conversation After pre-testing the interview questions, students created a script - “a written text that guides the interviewer during the interview” (Castillo-Montoya, 2016, p. 824) - to support smoother transitions between questions and achieve a more natural, conversation-based interview protocol. The consideration of prompts and follow-up questions also occurred at this stage of the process. Students created a chart to track changes made to the interview protocols and the reasoning behind those changes. This step was not relevant for the questionnaire, as the questionnaires were to be completed asynchronously and anonymously by participants without the presence of a researcher. Phase 3: Receiving Feedback on Interview Protocols and Questionnaires Castillo-Montoya (2016) identifies that a close reading of the interview protocol can help provide important feedback to enhance reliability of the protocol. Willis (1999) identifies a comparable version of close reading for questionnaires and survey questions through think aloud activities and verbal probing. Students participated in both activities with peers from our cohort, using slightly different processes for the interview and questionnaire protocols. When evaluating the questionnaires, students used the rules for writing survey items (Privitera & Ahlgrim-Delzell, 2018) to guide them as they provided feedback on the overall flow, design, and length of their peers’ surveys. For the questionnaire, the think-aloud activities were conducted asynchronously with this group of students, recording their close reading thoughts in an audio file as they completed the survey questions. The questionnaire was revised a second time using this feedback before the second cognitive interview activity. During the second cognitive interview activity, the EdD student (as interviewer) used a verbal probing approach to ask the respondent questions about their thoughts on the survey questions as they completed the questionnaire. Changes based on this verbal probing activity were then made to the survey to create the finished protocol. For the interview protocols, feedback was given asynchronously in a first close reading, then synchronously in a think-aloud cognitive interview. After each of these exercises, students were encouraged to refine their questions and track their changes using a chart to illustrate the evolution of their protocols. Phase 4: Piloting the Interview Protocol and Survey Questionnaire For the final step of this process, a pilot test of the survey questionnaire was conducted via the discussion board where students could take one another's survey and offer final feedback. The final stage of interview pilot testing also included a practice interview with an individual, who either had familiarity with the topic or was a member of the target population, within or outside of our program. This practice interview was conducted in person or using a video meeting platform. Simulating the context of the interview provided the interviewer with a realistic idea of how long the interview would take and allowed the interviewer to make necessary improvements or changes to the protocol (Castillo-Montoya, 2016). This final stage of testing interviews and surveys in full with members of the target population helped to confirm the validity of the questions being used, helped students practice their roles as interviewers, and ultimately resulted in methodologically rigorous data collection instruments that students were confident in using for research (Kelley et al., 2003). A FINISHED PRODUCT The final project of the course was a scholarly paper itemizing the development of survey tools and interview protocols and reflecting on the evolution of these products throughout the course based on readings, lectures, peer-to-peer feedback, and pilot testing. After working through this course together, we continued to meet virtually to compare our experiences. This time spent together after the course solidified our learning, allowing us to gain different perspectives and a richer understanding of the different facets of research design as experienced by our peers. The personal experiences are reflected in the following vignettes, as each student discusses their primary takeaways. Vignettes Vignette 1: Collective Learning and the Formation of our Scholar-Practitioner Identities Fatima: Many students entering graduate education programs, such as our EdD program, are working professionals, and depending on their previous education and work experience, some students have not had any prior undergraduate experience conducting research (Coronel Llamas & Boza, 2010). I started this program with Pilot Testing as a Strategy to Develop Interview Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8 No. 4 (2023) DOI 10.5195/ie.2023.333 23 years of professional working experience, confident in identifying myself as a practitioner but doubting my skills and abilities as a researcher. Although I had experience conducting a research study for my master's degree, I completed my research two decades ago. It was quantitative research, and it was not in the field of education. As such, I began the Research Methods course with some theory but no hands-on experience with qualitative research methods. The organization and content of the lectures and the extent of peer-to- peer, active learning exercises provided the necessary structure and support to guide my learning as an adult learner. As a member of a cohort in our EdD program, the tasks of giving and receiving feedback, role-playing, and conducting cognitive interviews provided multiple opportunities to collaborate, build relationships, and share knowledge with a diverse group of peers. Interviewing can include not only technical challenges but psychological difficulties, especially for novice researchers. Feedback can provide the researcher with information about how well participants understand the interview questions and whether their understanding is close to what the researcher intends or expects (Patton, 2015). Having multiple perspectives from my peer group on my interview questions provided valuable insights that I would not have had on my own. The pilot testing process that we engaged in, due to its ongoing nature and the intentional collective involvement of fellow students, created a dynamic learning opportunity to support our development as scholarly practitioners within a safe space. I appreciated the time and planning that went into designing and guiding the course to provide for the “open communication, critical friendships, and peer-to-peer support with reciprocal interactions and responsibilities” that I feel helped us to form a true “community of learners” throughout the pilot testing process (CPED, 2021, para. 14). The scaffolded learning that we received during this course was purposeful, and the assigned readings in conjunction with the peer-peer activities and feedback increased my skills, knowledge, and confidence with the interview process. By engaging in experiential learning as a group, we developed competence with developing and conducting interviews by sharing knowledge with each other, reflecting, and providing meaningful feedback. Active learning opportunities such as these helped to scaffold learning within an authentic context, provided us opportunities to engage in meaningful practices with peers and faculty, exposed us to the complexities and messiness of the research process, and empowered us to be more confident in the research process (Saeed & Al Qunayeer, 2021). An integral part of the CPED (2021) framework is to build an EdD program using a set of program principles, including “Laboratories of Practice” where “theory and practice inform and enrich each other” (para. 11) The pilot interview process contributed to my growth as a novice qualitative researcher, and I hope that I was able to contribute to the growth of others. The impact of the peer-to-peer support during this learning cannot be underestimated – these intentional, collective, collaborative experiences helped to strengthen my expertise, social, professional, and academic relationships and the formation of my identity as a scholarly practitioner. Vignette 2: Increased Academic Rigor Ruth: The feedback received was instrumental in allowing me to reflect on the survey’s clarity, structure, validity, and reliability and in improving it ahead of sending participants the final iteration. Going through the process of crafting, revising, and testing was an opportunity to bridge theory and practice and improve my interview protocol. When considering feedback, I was able to rephrase questions, clarify the wording where necessary, and prepare additional prompts (Roulston et al., 2003). When my classmates indicated a question was difficult to understand, redrafting, and having the opportunity to discuss further increased my confidence that respondents would not satisfice, as understanding questions is key to respondents giving authentic, considered responses rather than perfunctory answers that fulfil the minimum requirements (Cohen, et al., 2018; Collins, 2003). The recorded audio file was particularly useful when piloting the survey. If both the pilot respondents understood a question in the same way, it indicated clarity, as they had recorded separately and were not influenced by each other’s responses. When discrepant feedback was given, I saw how specific terms and questions could be interpreted differently and adapted the wording before our in-person meeting. Some of the feedback also identified biased language I corrected. One question was initially phrased “Our school critically examines the curriculum to determine whether it reinforces negative cultural stereotypes.” One classmate reflected that any cultural stereotype can be harmful, which allowed me to rephrase the question in the next iteration. After multiple steps of pilot testing, I felt more confident in the research process, and that although eliminating all bias is unrealistic, the interview protocol and questionnaire would yield valid, reliable data that authentically represented participants’ views (Cohen et al., 2018). Vignette 3: Learning During COVID-19: Hybrid Pedagogies and Innovation Cristina: As we were participating in this course in winter of 2022 during the peak of the Omicron wave in Ontario, all class activities were required to be virtual. While virtual classes can limit peer-to-peer activities, I greatly appreciated our instructor’s creativity and intentionality in using virtual tools to support peer-to-peer piloting, and as a result, I believe there were benefits and improvements on how peer-to-peer piloting can be conducted in the future. For our first piloting activity, for both the interview and survey, we were asked to review multiple classmates' protocols asynchronously via a discussion board. I gave and received more insightful feedback on protocols, as these reflections were not conducted in a noisy think- pair-share type classroom activity. I had the time and quiet I needed to deeply reflect on my peers’ work, and I reviewed far more peers’ protocols asynchronously than I could during a single class period. By seeing more of my peers’ work, I saw a diversity of unique approaches to using class readings and lectures in the creation of protocols (De Vaus, 2014). While in person think-pair-share reviews of protocols are a more traditional approach, moving this initial step of piloting asynchronously and online was a great benefit for my learning. As a scholar-practitioner, I find it challenging to practice research skills, especially since I cannot participate in the research assistant roles, as my full-time peers in PhD programs can. The asynchronous activities allowed for flexibility in the timing of activities that we adapted as needed to accommodate our schedules. Flexible and intentional activities, like piloting via discussion boards, limit the amount of rushed class assignments and focus on developing tangible skills through deep and reflective learning to ensure students have these research skills prior to embarking on the dissertation process. Tate et al. Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8 No. 4 (2023) DOI 10.5195/ie.2023.333 24 My other favorite pivot was conducting the think aloud pilot asynchronously by recording ourselves reading aloud and talking through one another's surveys. Recording and sending our read aloud to each other electronically rather than meeting to conduct them was a quick change our group had to make due to scheduling conflicts that week. Reflecting on this process, participating in asynchronous think alouds eliminated the possibility of us interjecting or reacting to our peers’ commentary on our survey questions and potentially stifling honest responses (Willis, 1999). Hearing the recording of my peers’ unfiltered thought processes as they worked through my survey instrument was helpful in generating verbal probing questions to dig deeply into some of the more challenging concepts I was trying to measure. Lockdown or none, I would highly recommend instructors and students incorporate asynchronous components of pilot testing via discussion board reviews and a recorded think aloud, as I felt these were more beneficial than synchronous piloting activities! Asynchronous, spacious learning activities to explore and practice research skills are both efficient and effective in training scholar practitioners in the nuances of research. Vignette 4: The Challenges Experienced by a Would- be Quantitative Research/Practitioner Laura: As a student in the EdD program, I participate in compulsory courses with my classmates along a prescribed timeline. The course where my fellow authors and I practiced our research methods, and in particular pilot testing, focused almost entirely on a qualitative research approach. Although I was fairly certain, even at this early stage in the program, that I would not engage in interviews in my research, I learned significantly from using this pilot testing protocol. Now that I am further along in my studies and have determined that I will pursue a quantitative methodology, I provide a reflection on a slight gap I now recognize in the course structure of the EdD related to pilot testing for quantitative research or rather a mixed methods methodology. Creswell and Guetterman (2019) recommend locating, modifying, or developing instruments as the fourth step in the process of quantitative data collection and “the easiest procedure is to use an existing instrument or modify one, rather than develop your own” (p. 170). I benefited as a novice researcher from participating in pilot testing through the think-aloud activities because I tested an existing instrument and considered potential modifications. The cognitive think-aloud, like the practice we engaged to test our interview questions, provided significant insight on examining my bias and decisively explaining the choices I made regarding the specific language used in the final instrument I chose to send to participants. It was essential to ensure my questions had the same meaning for respondents as they did for me, the researcher (Collins, 2003; DeVaus, 2014). The cohort EdD structure, however, specifies that certain compulsory courses will be taken in a particular order with the cohort members (Ontario Institute for Studies in Education, n.d.). Research Methods, the course upon which this paper is focused, was scheduled as the fourth compulsory course, and it is expected that all members of our cohort be enrolled during the stated term without opportunity for an elective until after this term. Wanting to pursue quantitative research, I would have benefited from testing the questions related to my hypothesis or how variables might or might not interact with one another (Creswell & Guetterman, 2019) in order to prepare for the specific psychological and technical challenges associated with this process as well. I had not yet even learned about these and therefore did not take full advantage of pilot testing to develop and test them. I advocate, therefore, for a pause in the EdD course's timeline to allow for an introductory statistics elective related to a student’s interest in pursuing quantitative research. This would allow the researcher to “consider how the research questions or hypotheses will be answered when deciding on what data type(s) to use, identify the variables, operationally define them, and select measures (e.g., performance and attitudes, observations of behavior, and factual and personal data) that fit the operational definitions” (Creswell & Guetterman, 2019, p. 170) and then engage with classmates in pilot testing. I might recommend testing not only the language of the questions but also the type of statistical analysis that would be employed to examine relationships, if any related to the variables in question. A novice researcher would need, at minimum, a cursory understanding of statistical practices and measures which I did not possess. While it is evident I did not select the most appropriate research methods course, without the opportunity of a statistics elective, I certainly was not adequately prepared in any research methods course to pilot test a questionnaire for statistical analysis. Furthermore, it was challenging in the subsequent compulsory Data Analysis course I enrolled in to engage in meaningful measures, as much of my previous work had been focused on an interview protocol I would not use. RECOMMENDATIONS These reflections point to recommendations for supporting students in participating in peer-to-peer pilot testing to improve their qualitative and quantitative instruments and learn valuable research skills. Based on promising practices used by our course instructor, we highlight the following recommendations for instructors who wish to incorporate new and innovative practices into research methods courses: 1. Include multi-step piloting integrated as an explicit, structured component of research methods courses to scaffold deep understanding of each step of the research method. 2. EdD students, as scholar-practitioners, have limited resources and availability to participate in and conduct research prior to their dissertations. Offer synchronous and asynchronous opportunities for both face-to-face and online course options to accommodate EdD students with different learning preferences and time commitments to reflect on and critique one another's work. Asynchronous activities also encourage diverse pairings of students. 3. Consider integrating piloting models that balance the focus on both qualitative and quantitative methods within educational research. For additional depth, consider using piloting frameworks beyond their intended purpose for new approaches, like using Castillo-Montoya's (2016) IPR framework to support quantitative as well as qualitative testing. 4. Continue to create and share empirical research to fill the gap in the literature on effective pilot testing and feedback strategies. 5. Although focus groups and other methodologies were not included in this research course, instructors may consider using these pilot testing strategies in the teaching of other methodologies to EdD students. Pilot Testing as a Strategy to Develop Interview Impacting Education: Journal on Transforming Professional Practice impactinged.pitt.edu Vol. 8 No. 4 (2023) DOI 10.5195/ie.2023.333 25 CONCLUSION This essay highlights the reflections from four EdD students on conducting systematic and rigorous multi-phase pilot testing of survey questions and interview protocols during a Research Methods course. Coronel Llamas and Boza (2011) describe the journey that doctoral students take as they engage in this type of active learning of research methods by stating that, students range from being passive recipients of knowledge about research methods and prior research to fully engaging in the process of research, and thus becoming active agents in creating their own research agenda and further developing their own professional and personal understandings of the world around them (p. 78). As a professional doctorate, the EdD presents specific considerations when teaching the applied theories and research methods necessary for scholarly practitioners to be able to pose and address “significant questions that focus on complex problems of practice” (CPED, 2021, para. 10). As a result of the intentional course scaffolding of our instructor and the accidental necessities of the pandemic, we experienced a research methods course that fostered collaborative and reflective learning to begin to hone our work as active agents who are continuing to make sense of the world around us. Inviting and incorporating peer feedback on protocols using a variety of methods, including asynchronous review on the university’s online portal, recorded think aloud feedback, and cognitive interviews with probing questions, led to improvements in the clarity and flow of the questions used to pilot the interview and questionnaire question protocols (Castillo-Montoya, 2016; Collins, 2003; Majid et al., 2017; Willis, 1999). Pilot testing the interview protocols provided students with the opportunity to collectively role- play, reflect, revise, learn, and grow as emerging researchers. The flexible nature of the asynchronous strategies provided were particularly effective in developing applied research skills for scholarly practitioners with full-time professional commitments. Most PhD students dedicate themselves full-time to their studies and can develop research skills through teaching and/or research assistant positions in which graduate students work collaboratively with professors. In contrast, the EdD requires the integration of flexible opportunities for collaborative critical reflection as students develop their problems of practice and research skills. All four of us found that pilot testing helped us reciprocally develop scholar-practitioner identities and skills through an iterative and collaborative cycle of sharing knowledge and reflecting. Starting the EdD during COVID-19 presented undeniable challenges, but also opportunities for innovation and flexibility. The relevance of pilot testing for doctoral candidates from such different backgrounds and with such varying interests indicate these reflections are relevant to a wide range of scholars seeking to develop their research skills. After receiving feedback, redrafting, and retesting our interview protocols and questionnaires several times in authentic contexts, we were better prepared for psychological and technical challenges and confident in our research instruments’ reliability and validity. We hope these potential recommendations can help future cohorts of students. ACKNOWLEDGEMENTS We are grateful to Dr. Karen Acton for her guidance and support in the class and conversations that inspired this article. Dr. Acton’s expertise as both an educational scholar and practitioner afforded our cohort a rigorous foundation by which to explore survey and interview protocols. As students who truly benefited from this process, we hope that other faculty are inspired to use Dr Acton’s methods for teaching survey and interview skills. REFERENCES Acton, K. (2022). LHA3005: Introduction to Research Methods for the EdD. [Syllabus]. Toronto: Ontario Institute for Studies in Education. Buss, R., Zambo, R., Zambo, D., & Williams, T. (2014). Developing research professionals in an EdD program: From learners and leaders to scholarly and influential practitioners. Higher Education, Skills and Work-Based Learning, 4(2), 137–160. https://doi.org/10.1108/HESWBL-11-2013-0022 Carnegie Project on the Education Doctorate. (2021). The CPED framework. https://www.cpedinitiative.org/the-framework Castillo-Montoya, M. (2016). Preparing for interview research: The interview protocol refinement framework. The Qualitative Report, 21(5), 811–831. https://doi.org/10.46743/2160-3715/2016.2337 Cohen, L., Manion, L., & Morrison, K. (2018). Research methods in education (8th ed.). Routledge. Collins, D. (2003). Pretesting survey instruments: An overview of cognitive methods. Quality of Life Research, 12(3), 229–238. https://www.jstor.org/stable/4038871?seq=1&cid=pdfreference#referenc es_tab_contents Coronel Llamas, J. M., & Boza, Á. (2011). Teaching research methods for doctoral students in education: learning to enquire in the university. International Journal of Social Research Methodology, 14(1), 77–90. https://doi.org/10.1080/13645579.2010.492136 Creswell, J.W. & Guetterman, T. C. (2019). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (6th ed.). Pearson. De Vaus, D. (2014). Surveys in social research (6th ed.). Routledge. Kelley, K., Clark, B., Brown, V., & Sitzia, J. (2003). Good practice in the conduct and reporting of survey research. International Journal for Quality in Health Care, 15(3), 261–266. https://doi.org/10.1093/intqhc/mzg031 Labaree, D. F. (2003). The peculiar problems of preparing educational researchers. Educational Researcher, 32(4), 13–22. https://doi.org/10.3102/0013189X032004013 Majid, M. A. A., Othman, M., Mohamad, S. F., Lim, S. A. H., & Yusof, A. (2017). Piloting for interviews in qualitative research: Operationalization and lessons learnt. International Journal of Academic Research in Business and Social Sciences, 7(4). https://doi.org/10.6007/IJARBSS/v7-i4/2916 Merriam, S. B., & Tisdell, E. J. (2016). Qualitative research: A guide to design and implementation. John Wiley & Sons. Ontario Institute for Studies in Education. (n.d.). Doctor of Education in Educational Leadership & Policy. Retrieved October 31, 2022, from https://www.oise.utoronto.ca/lhae/doctoral-degrees/doctor-education- educational-leadership-policy Patton, M. Q. (2015). Qualitative research & evaluation methods: Integrating theory and practice (4th ed.). SAGE Publications, Inc. Privitera, G. J., & Ahlgrim-Delzell, L. (2018). Research methods for education. Sage Publications. Roulston, K., deMarrais, K., & Lewis, J. B. (2003). Learning to interview in the social sciences. Qualitative Inquiry, 9(4), 643–668. https://doi.org/10.1177/1077800403252736 Saeed, M. A., & Al Qunayeer, H. S. (2021). Can we engage postgraduates in active research methodology learning? Challenges, strategies and evaluation of learning. International Journal of Research & Method in Education, 44(1), 3-19. https://doi.org/10.1080/1743727X.2020.1728526 Truong, T., (2021). Critical reflections on using interviews - A research method. Academia Letters. (Article 2335), 1–5. https://doi.org/10.20935/al2335 Willis, G. B., (1999). Cognitive interviewing: A “how to” guide. [Short course] Research Triangle Park, NC: Research Triangle Institute. Meeting of the American Statistical Association. https://doi.org/10.1108/HESWBL-11-2013-0022 https://doi.org/10.46743/2160-3715/2016.2337 https://www.jstor.org/stable/4038871?seq=1&cid=pdfreference#references_tab_contents https://www.jstor.org/stable/4038871?seq=1&cid=pdfreference#references_tab_contents https://doi.org/10.1080/13645579.2010.492136 https://doi.org/10.1093/intqhc/mzg031 https://doi.org/10.3102/0013189X032004013 https://doi.org/10.6007/IJARBSS/v7-i4/2916 https://www.oise.utoronto.ca/lhae/doctoral-degrees/doctor-education-educational-leadership-policy https://www.oise.utoronto.ca/lhae/doctoral-degrees/doctor-education-educational-leadership-policy https://doi.org/10.1177/1077800403252736 https://doi.org/10.1080/1743727X.2020.1728526 https://doi.org/10.20935/al2335